Few corporations face increased stakes when deploying AI than Waymo, the self-driving automobile firm below Alphabet that spun out of Google. Its fashions don’t merely generate textual content or automate back-office duties: They assist autos navigate unpredictable streets, reply to human drivers and make split-second choices within the bodily world.
However the strategies Waymo makes use of to handle these dangers — steady analysis, rigorously curated information, human oversight and clearly outlined enterprise outcomes — supply a broader playbook for enterprises deploying AI brokers in practically any trade.
Manasi Joshi, Waymo’s director of engineering for techniques intelligence and machine studying, defined at VB Transform 2026 how the autonomous automobile firm trains, checks and deploys AI at scale. Thus far, Waymo has pushed greater than 220 million totally autonomous, or "rider-only," miles, with 17 instances fewer severe crash accidents than human drivers over the identical distance, in response to the corporate.
To realize these spectacular outcomes, Joshi mentioned Waymo has adopted what she known as “eval-forced growth” or “eval-centric growth,” making analysis a core a part of engineering reasonably than a closing examine carried out earlier than deployment.
“The stage at which our tasks are maturing might be simply sort of transpired based mostly on the eval maturity that they showcase,” Joshi mentioned.
In follow, Waymo assesses a venture’s readiness partly by inspecting the maturity of the checks surrounding it. That method has clear implications for enterprises constructing customer support brokers, coding assistants, monetary techniques or different AI purposes: If an organization can’t reliably measure a system’s efficiency, it might not be prepared to put that system into manufacturing.
Evals should proceed after launch
Joshi mentioned a lot of Waymo’s high quality work has shifted towards evaluations, together with checks performed throughout mannequin coaching, after coaching and inside open-loop and closed-loop simulations.
“Eval isn’t a one-time process to launch a mannequin,” she mentioned.
Waymo as a substitute treats analysis as a steady course of spanning driving, simulation and validation. Its methodology combines datasets, efficiency metrics and infrastructure able to working effectively at scale.
For enterprises, meaning testing an agent earlier than launch is inadequate. Groups should proceed evaluating it as underlying fashions, enterprise processes, consumer habits and incoming information change. These evaluations also needs to hook up with precise enterprise outcomes reasonably than relying solely on broad trade benchmarks.
Joshi cautioned that model-quality measurements are solely as reliable because the analysis information behind them. Waymo due to this fact pairs its efficiency claims with details about the properties of the datasets used to check its techniques.
Testing the uncommon and harmful instances
Waymo’s analysis hierarchy stays grounded in a single overriding goal: security.
The corporate attracts on first-party driving logs, some third-party information and lifelike simulations that expose its techniques to eventualities spanning billions of artificial miles. Activity homeowners select specialised information and metrics for conditions involving susceptible street customers, railroad crossings, building zones and different advanced environments.
The identical precept applies exterior autonomous driving. Enterprises want to check not solely the routine requests their brokers deal with efficiently, but additionally unusual conditions the place errors may create monetary, authorized, safety or reputational harm.
Joshi emphasised that Waymo doesn’t go away launch choices totally to automated techniques. Its production-readiness opinions embrace intensive human oversight, whereas inside security leaders approve software program releases and service-area expansions.
“This isn’t AI-driven and utterly automated and nil human oversight,” she mentioned. “Human lives are at stake.”
Effectivity can’t come on the expense of reliability
Waymo faces one other downside acquainted to enterprise AI groups: Demand for compute, storage, reminiscence and community capability is rising quicker than the assets out there.
The corporate pursues effectivity throughout information extraction and storage, distributed mannequin coaching, mannequin distillation, simulation and analysis. It additionally emphasizes “information effectivity,” choosing probably the most helpful coaching examples as a substitute of treating larger quantity as inherently higher.
Waymo started utilizing transformers in 2017 and subsequently expanded into massive language fashions, vision-language fashions and vision-language-action fashions. Joshi mentioned the corporate now makes use of generative multimodal fashions as a part of its foundation-model technique.
Waymo divides its expertise between onboard techniques inside every automobile and off-board infrastructure used for mannequin growth, information processing and simulation. That mixture forces the corporate to optimize each real-time inference and the bigger techniques supporting it.
Brokers want their very own evals
Waymo additionally makes use of AI brokers internally as productiveness instruments for engineers. Joshi mentioned brokers assist analyze information distributions, assess information effectivity and triage issues present in automobile telemetry, coaching runs and failed analysis jobs.
The objective is to speed up investigative work so engineers can dedicate extra time to judgment and troublesome technical issues. However Waymo additionally evaluates these brokers to make sure they produce reliable, correct outcomes reasonably than sending workers down unproductive paths.
For enterprise leaders, Waymo’s bigger lesson is that agentic AI requires greater than selecting a strong mannequin. Organizations want a clearly outlined goal, consultant analysis information, steady testing, infrastructure that may function effectively and named human decision-makers who stay accountable for deployment.
"Incomes belief is supremely essential," Joshi mentioned.
