Zillow says AI ROI depends on measuring before deployment

Zillow engineering chief Toby Roberts said the company’s reported AI productivity gains are credible because it had measurement in place before the rollout, not because it added metrics after the fact. Speaking at VB Transform 2026 with Glean co-founder and CEO Arvind Jain, Roberts described an enterprise AI architecture built around persistent customer and employee context rather than a single chatbot or a single model API.

The point is practical for large software buyers and builders: Zillow’s customer interactions can span a phone screen, a loan officer, a real estate agent and long transaction timelines. Roberts said that made a one-conversation assistant inadequate for a real estate workflow where users expect prior context to move with them.

Zillow, which says its products touch about 80% of U.S. real estate transactions each year, has used machine learning since well before the generative AI boom. Roberts pointed to the company’s long-running Zestimate work as part of that history. The current AI effort, as described by Roberts, relies on a company-controlled harness and smaller fine-tuned models for specific tasks, rather than sending every workflow through a general-purpose model.

Context became the hard problem

Roberts said Zillow began with the data foundation, including a data mesh approach, lineage, governance, permissions and identity tied to data. He said that work was necessary, but it did not solve the hardest part of the system.

The more difficult requirement was maintaining a context layer that could support a customer at different points in a real estate journey, regardless of which product surface or professional they encountered next. Roberts said Zillow chose to control that layer itself after looking at the shape of a full transaction, rather than optimizing for a single chat session.

That framing is a useful distinction for enterprise AI programs. Many vendors sell access to models or chat interfaces, but Zillow’s account suggests the integration and memory layer can be the more strategic asset, particularly where a workflow crosses teams, tools and months of elapsed time.

Glean’s role inside Zillow

Roberts said Zillow runs its internal AI harness alongside Glean and now has thousands of Glean agents in production, with tens of thousands of executions across the company. He described those agents as handling repetitive work, though the companies did not disclose a dollar value for the labor savings or a total AI budget.

Jain said Glean’s argument is that enterprises should centralize integrations through the company’s MCP gateway instead of letting departments such as finance, legal and marketing each rebuild connections to the same systems. That is both an operating model claim and a cost claim.

Jain pointed to model routing and precomputed context as two ways to reduce AI spend. Model routing sends routine work to smaller, less expensive models rather than defaulting to frontier models. Precomputed context reduces the amount of token-heavy assembly an agent has to perform before answering or acting. Jain said routing through Glean can reduce token consumption by as much as half.

Measurement came before the AI claim

Roberts said Zillow can attribute a 40% increase in shipped code to AI adoption because it had already established a DORA metrics baseline years earlier. That caveat matters: without a pre-existing baseline, productivity claims around coding assistants are difficult to separate from team mix, project timing or changes in engineering process.

The session also showed limits to relying on permission inheritance alone. Roberts said Zillow added hard rules and an ongoing compliance check for its most sensitive data categories, even with a permissions-aware context platform in place.

Jain said models alone are insufficient for enterprise automation because they need access to company context. For buyers, the less promotional version is that model choice may be a secondary decision. The larger implementation burden sits in context, permissions, measurement and cost control, none of which is solved by adopting a chatbot interface.

This story draws on original reporting from VentureBeat.