Enterprise AI failures put data observability back on the roadmap
Stale and incomplete data, rather than model choice alone, is emerging as a common cause of wrong answers from production AI agents.
By Wei-Lin Zhao · AI Correspondent
· 3 min read
Enterprise AI teams are finding that production agents can return confident, incorrect answers when the data underneath them falls out of date or loses required fields. The issue matters for AI operators because many retrieval systems still measure whether content is relevant or available, while leaving correctness, freshness and lineage weakly tested.
Financial terms are not part of this story: the relevant activity is product and infrastructure investment, not a disclosed funding round. VentureBeat has reported that AWS has entered the context-layer market with a knowledge graph that learns from agent behavior, while Snowflake has introduced Horizon Context and Cortex Sense for governance around the business logic used by agents.
Those products address a visible symptom: AI systems producing plausible answers from stale or incomplete context. The underlying problem, according to the analysis, is often earlier in the pipeline. A vector store, document index or API call can all return material that appears authoritative to the model, even when the document is old, a price has changed or a field has disappeared upstream.
Why green pipelines can still serve bad answers
Standard data monitoring often confirms that a job ran, a file arrived or a query completed. That does not prove the data is still accurate for the use case consuming it. A stale policy document can rank highly in retrieval. A malformed record can move through a pipeline if the system checks availability rather than the data’s actual contents.
The pattern predates generative AI. The analysis describes a fintech pipeline where an upstream field changed without warning downstream users. The pipeline continued to run and pushed bad values into dashboards. The problem surfaced only after a customer noticed the inconsistency.
For AI systems, that same failure mode is harder to see because the model can turn defective inputs into fluent output. Dashboards may show healthy infrastructure while users receive answers based on outdated business rules or incomplete records.
Data observability gets renewed attention
The proposed fix is stronger data observability across four areas: correctness, freshness, consistency and lineage. Correctness covers schema, field types, nulls and value ranges. Freshness measures whether data has been updated within the right source-specific SLA. Consistency checks whether the same fact agrees across downstream systems. Lineage records where data came from and which transformations touched it.
Uber built its Unified Data Quality platform before retrieval-augmented generation became common. According to Uber, the system supports more than 2,000 critical datasets and catches about 90% of data quality incidents before they reach downstream users.
Netflix has described its own company-wide lineage system, built to show where datasets originate and how they move through infrastructure. Netflix said the system maps dependencies across Kafka topics, machine learning models and experimentation systems, not only warehouse tables.
Tools cited for row- and column-level validation include Great Expectations and Soda. The analysis also points to write-audit-publish patterns, where data lands in staging, is checked and only then moves to downstream consumers.
What operators need to verify
For teams running retrieval-based AI, the practical test is whether they can identify the oldest content being served with high confidence, detect disagreement between chunks from the same source and trace a bad answer back to its origin.
Socure is cited as an example where client data arrived in inconsistent shapes and sometimes with silent errors. The described implementation used Great Expectations for ingestion checks, per-source freshness SLAs, cross-system consistency tests and file-level lineage before data moved downstream.
The broader signal for enterprise AI buyers is that context-layer products do not remove the need for data engineering controls. Knowledge graphs and retrieval tools still depend on the systems feeding them, and agents will expose gaps that reporting and machine learning pipelines may have tolerated for years.
This story draws on original reporting from VentureBeat.
