Expedia AI chief says evals are becoming product requirements

Expedia Group’s first chief AI and data officer, Xavi Amatriain, told VB Transform 2026 that evaluation suites are taking over a role once filled by product requirements documents in AI development. The point for operators is concrete: as code generation and agent autonomy increase, the test harness becomes the place where product intent, security assumptions and release gates are defined.

Amatriain, who joined Expedia in December 2025 after serving as VP of AI and Compute Enablement at Google on systems behind Gemini and Search, said teams should put more of their planning into evals before software is built. He told the Menlo Park audience that evals can include red teaming and security requirements, meaning product design is increasingly expressed through what the system must pass rather than through prose specifications.

The industry context is uncomfortable. VentureBeat’s VB Pulse research found that 66% of 157 surveyed enterprises already allow some production deployment without human review or are preparing to do so within 12 months. Only 5% said they fully trust the automated evaluations needed to support that choice. Half reported shipping an agent that passed internal evals and later failed with a real customer.

Expedia wants risk-based release gates

Amatriain argued against piling on rigid guardrails, saying too many business rules can distort user behavior and damage the feedback loop that improves AI systems. He described guardrails as a “necessary evil” whose impact should shrink over time. Other speakers at Transform took a stricter view, saying high-risk actions still need firm constraints.

Expedia’s approach, as described by Amatriain, has three layers: principles, processes and tools, then automation. Principles are meant to guide distributed decision-making inside a large company, while processes and tooling give those principles enforcement. Automation then applies those rules through the development and release process.

The company calls its release checkpoints “agent release toll gates.” Amatriain said those gates are tied to the risk level of a given agent, with evaluation rounds, red teaming and security review becoming more mandatory as the potential impact rises. He did not disclose quantitative results showing how those gates have reduced incidents or improved launch quality.

Smaller agents, composed systems

Amatriain also rejected the idea of a single general-purpose model as the right design target for Expedia’s agentic systems. He said the company prefers specialized agents that perform narrow jobs and are composed into broader workflows.

In Expedia’s architecture, tools are grouped into skills, skills are assembled into sub-agents, and sub-agents are orchestrated into larger systems. Amatriain said common design rules still have to govern tone, user interaction, context and memory across those components. He framed the problem as system design rather than model selection.

That structure also gives Expedia a security rationale. Narrower agents can be evaluated and restricted individually before they are combined into a larger workflow. The tradeoff is operational complexity: teams have to maintain the evaluation logic across many components, not just one chatbot surface.

Travel keeps the user in the final step

Amatriain said travel creates specific constraints for agents because prices, availability and supplier claims change quickly. Expedia uses a mix of retrieval-augmented generation and direct API calls, choosing between cached answers and live checks based on the user’s request and latency expectations.

For example, a broad question about typical hotel prices in Chicago can use cached information, while a more specific request involving location, rating, pets and amenities may justify a longer reasoning window. Amatriain said Expedia can also compare supplier descriptions against traveler reviews, such as cases where guests report that a pool was unavailable or closed earlier than advertised.

Expedia does not want its agent to complete the booking on behalf of the traveler, according to Amatriain. He said the system may recommend, suggest and discuss options, but the customer must make the final click to book a hotel or buy a plane ticket. He described that as both a product principle and a security choice.

Agent security is becoming agent-on-agent security

Amatriain said security should be designed earlier in the process, and that needing a late guardrail often means the team failed to address the issue during design. He also described a feedback loop where production monitoring signals are fed back into the eval suite so failures can be reported and fixed faster.

VentureBeat’s June Pulse survey on agent security, based on 107 enterprises, shows why that loop is under pressure. Fifty-four percent of respondents said they had experienced an agent security incident or near-miss. Fifty-nine percent planned to adopt, add or replace agent security tools within 12 months, including 29% this quarter. Among companies with more than 1,000 employees, incident or near-miss rates reached 63%, compared with 49% for companies with 101 to 1,000 employees.

Amatriain warned that future attacks will increasingly come from other AI systems rather than only human adversaries. His view puts evals in a broader role than QA: for AI agents, they are becoming part specification, part security control and part production feedback system.

This story draws on original reporting from VentureBeat.