DoorDash's internal data agent, Vera, answers business questions across fragmented data and knowledge sources, according to Data Engineering Weekly's summary. The newsletter describes an agent that retrieves only vetted tables and documents, uses a model of how those sources connect, and improves through evaluations reviewed by the people who own each business domain.
The useful detail is that domain owners review the evaluations. That gives the business a role in judging whether the agent's answers are useful. The summary does not provide benchmark results or enough detail to establish which part of Vera's design contributes most to its performance.
For a data leader preparing an agent for use, this suggests a practical starting point: agree how its answers will be judged. Choosing approved sources and deciding whether an answer meets the business need are related decisions. Both deserve explicit attention.
Make the questions testable
An answer key is a useful way to think about this, provided it includes the reasoning needed to judge an answer. For a question with a reproducible calculation, record the expected result and how it was obtained. For a question that calls for interpretation, write down what a satisfactory answer must address and where uncertainty should remain visible.
Those are proposed evaluation practices, not details established by the newsletter's account of Vera. The distinction matters because the summary supports a role for domain review without explaining exactly how DoorDash carries it out.
Pick one domain where the agent is already being used or is about to be. Collect questions that people in that area have actually asked, along with the answers they accepted and the supporting material. Ask the domain expert to review whether those answers are still valid and sufficiently clear to test against.
Keep disputed answers visible. If reviewers disagree, resolve the disagreement or narrow the question before treating the answer as a standard. Otherwise, the evaluation would ask the agent to satisfy a requirement the reviewers have not settled themselves.
Use the evaluations to guide changes
Run the agent against the reviewed questions and inspect the failures. Check whether it selected an appropriate source, used the relevant context and answered the question that was asked. Use those findings to choose what to change, then repeat the evaluation.
Keep conclusions within the scope of the questions tested. Include questions the agent should decline to answer when the approved material is insufficient. Review the expected answers when the underlying business information changes.
The immediate deliverable is a set of reviewed questions that the team can use to assess its next change. Before expanding the agent's access or scope, use that set to identify which answers still need work.
Sources
- Inside Vera, DoorDash’s Data Agent, DoorDash
- Data Engineering Weekly #289, Data Engineering Weekly