Few companies face higher stakes when deploying artificial intelligence than Waymo, the autonomous vehicle company under Alphabet that spun out of Google. Its models do not merely generate text or automate back-office tasks: they help vehicles navigate unpredictable streets, respond to human drivers, and make split-second decisions in the physical world. The methods Waymo uses to manage those risks — continuous evaluation, carefully curated data, human oversight, and clearly defined business outcomes — offer a broader playbook for enterprises deploying AI agents in nearly any industry. At the core of this approach lies a principle that has reshaped Waymo’s entire engineering culture: the success of an AI launch hinges on evals, not model performance.
Waymo’s Eval-Forced Development: Why Evaluation Maturity Defines AI Readiness
Manasi Joshi, Waymo’s director of engineering for systems intelligence and machine learning, explained at VB Transform 2026 how the autonomous vehicle company trains, tests, and deploys AI at scale. To date, Waymo has driven more than 220 million fully autonomous, or “rider-only,” miles, with 17 times fewer serious crash injuries than human drivers over the same distance. Achieving these impressive results required a shift in mindset. Joshi described what she calls “eval-forced development” or “eval-centric development,” making evaluation a core part of engineering rather than a final check performed before deployment.
“The stage at which our projects are maturing can be easily kind of transpired based on the eval maturity that they showcase,” Joshi said. In practice, Waymo assesses a project’s readiness partly by examining the maturity of the tests surrounding it. That approach has clear implications for enterprises building customer service agents, coding assistants, financial systems, or other AI applications: If a company cannot reliably measure a system’s performance, it may not be ready to place that system into production.
What is eval-forced development? Eval-forced development is a methodology where evaluation is not an afterthought but drives the entire engineering lifecycle. Waymo embeds evaluation into model training, post-training analysis, and both open-loop and closed-loop simulations. The maturity of the evaluation suite becomes a proxy for project maturity, forcing teams to build robust measurement frameworks before claiming readiness.
For enterprise leaders, this represents a fundamental departure from the common practice of benchmarking a model on public datasets and then declaring it production-ready. Waymo’s experience shows that the quality of any AI system is only as trustworthy as the evaluation data behind it. Joshi emphasized that Waymo pairs its performance claims with detailed information about the properties of the datasets used to test its systems, providing transparency that builds trust.
Continuous Evaluation from Training Through Deployment
Joshi said much of Waymo’s quality work has shifted toward evaluations, including tests conducted during model training, after training, and inside open-loop and closed-loop simulations. “Eval is not a one-time task to launch a model,” she said. Waymo treats evaluation as a continuous process spanning driving, simulation, and validation. Its methodology combines datasets, performance metrics, and infrastructure capable of operating efficiently at scale.
For enterprises, that means testing an agent before launch is insufficient. Teams must continue evaluating it as underlying models, business processes, user behavior, and incoming data change. Those evaluations should also connect to actual business outcomes rather than relying solely on broad industry benchmarks. A model that scores high on a generic question-answering dataset may fail disastrously in a domain-specific context with nuanced regulations or safety-critical constraints.
Waymo’s continuous evaluation cycle includes real-time monitoring of vehicle telemetry, periodic simulation runs that test edge cases, and rigorous analysis of every disengagement or near-miss event. This feedback loop feeds directly into model retraining, creating a virtuous cycle of improvement. The company also evaluates its evaluation infrastructure itself — ensuring that the tools used to measure performance are accurate, scalable, and free from bias.
Testing the Rare and Dangerous Cases: Human Oversight Remains Essential
Waymo’s evaluation hierarchy remains grounded in one overriding objective: safety. The company draws on first-party driving logs, some third-party data, and realistic simulations that expose its systems to scenarios spanning billions of synthetic miles. Task owners choose specialized data and metrics for situations involving vulnerable road users, railroad crossings, construction zones, and other complex environments.
The same principle applies outside autonomous driving. Enterprises need to test not only the routine requests their agents handle successfully, but also uncommon situations where errors could create financial, legal, security, or reputational damage. A customer service agent for a healthcare provider must handle not just appointment scheduling but also unusual medical billing inquiries or privacy-sensitive complaints. A coding assistant must be tested on edge cases like deprecated library functions or security vulnerabilities.
Joshi emphasized that Waymo does not leave release decisions entirely to automated systems. Its production-readiness reviews include extensive human oversight, while internal safety leaders approve software releases and service-area expansions. “This is not AI-driven and completely automated and zero human oversight,” she said. “Human lives are at stake.”
For enterprises, this is a crucial lesson: evaluation data and automated metrics are powerful, but they cannot replace the judgment of domain experts and responsible leaders. The most sophisticated evaluation suite will still miss subtle contextual risks that only human review can catch. Waymo’s structure — with named safety leaders accountable for each release — provides a model for enterprise AI governance.
Efficiency Cannot Come at the Expense of Reliability
Waymo faces another problem familiar to enterprise AI teams: demand for compute, storage, memory, and network capacity is growing faster than the resources available. The company pursues efficiency across data extraction and storage, distributed model training, model distillation, simulation, and evaluation. It also emphasizes “data efficiency,” selecting the most useful training examples instead of treating greater volume as inherently better.
Waymo began using transformers in 2017 and subsequently expanded into large language models, vision-language models, and vision-language-action models. Joshi said the company now uses generative multimodal models as part of its foundation-model strategy. These models are computationally expensive, making efficiency a critical enabler of scale. But Joshi is clear: efficiency must never come at the expense of reliability. Cutting corners on evaluation to save compute costs would be a catastrophic mistake for any safety-critical application.
Waymo divides its technology between onboard systems inside each vehicle and off-board infrastructure used for model development, data processing, and simulation. That combination forces the company to optimize both real-time inference and the larger systems supporting it. For enterprise teams, this means investing in evaluation infrastructure that can keep pace with model complexity. A lightweight evaluation script that runs on a laptop may not suffice when deploying a multimodal agent serving millions of users.
Agents Need Their Own Evals: Internal Productivity Tools Also Get Tested
Waymo also uses AI agents internally as productivity tools for engineers. Joshi said agents help analyze data distributions, assess data efficiency, and triage problems found in vehicle telemetry, training runs, and failed evaluation jobs. The goal is to accelerate investigative work so engineers can devote more time to judgment and difficult technical problems. But Waymo also evaluates those agents to ensure they produce trustworthy, accurate results rather than sending employees down unproductive paths.
This is a point that many enterprises overlook. As organizations deploy AI agents for internal use — to summarize meetings, generate code, draft reports, or support decision-making — they must apply the same rigorous evaluation standards. An internal coding agent that introduces subtle bugs into production software can be as damaging as a customer-facing agent that gives wrong information. Waymo’s approach shows that even agents intended to augment human work require their own dedicated evaluation pipelines, with metrics tied to downstream impact on engineering velocity and quality.
For enterprise leaders, Waymo’s larger lesson is that agentic AI requires more than choosing a powerful model. Organizations need a clearly defined objective, representative evaluation data, continuous testing, infrastructure that can operate efficiently, and named human decision-makers who remain accountable for deployment. “Earning trust is supremely important,” Joshi said.
Trust is earned not by boasting about model performance but by demonstrating a disciplined, transparent, and safety-first evaluation culture. As AI agents move from chatbots to tools that act autonomously in the physical and digital worlds, the Waymo playbook will become increasingly relevant. The companies that invest in eval-forced development today will be the ones best positioned to deploy AI agents at scale tomorrow — without compromising reliability or safety.
