EY proposes graph-based multimodal RAG for enterprise AI retrieval

EY has outlined a multimodal retrieval-augmented generation framework that uses knowledge graphs to connect enterprise text with charts, tables, diagrams, equations and images before content is sent to a large language model. The company did not disclose product revenue, customer counts or benchmark results, but says the approach is aimed at a common enterprise AI problem: text-only RAG missing evidence embedded in non-text documents.

The work comes from EY, the business name of Ernst & Young LLP, and was described in new research and a related white paper on building generative AI applications at scale. Dipanjan Sengupta, EY Global Delivery Services Consulting distinguished technologist and AI engineering leader, said the effort grew out of limits the firm saw in client implementations.

Conventional RAG systems are often set up to search text passages and pass the retrieved material into a model prompt. EY’s argument is that this leaves out a large share of enterprise knowledge in sectors where key information sits in engineering drawings, scientific graphs or other visual artifacts. Sengupta said RAG performs well on written material, but many industries keep important facts in illustrative content.

How EY’s framework changes retrieval

EY’s approach does not modify the underlying LLM. It changes how company content is ingested, indexed, connected and retrieved at inference time.

The framework separates text and visual material into distinct processing pipelines. Text is broken into segments and enriched through keyword extraction and named-entity resolution, which EY uses to connect variants of names or records that refer to the same real-world entity. Illustrations receive metadata from captions, nearby text, bounding-box analysis, optical character recognition and descriptions generated by a language model.

Those assets are then stored in separate vector indexes, allowing searches to target text, images or both. Each text segment and illustration also becomes a node in a knowledge graph. EY says weighted links between those nodes let the system retrieve related evidence across modalities rather than relying on vector similarity alone.

The company’s paper lists three ways to create relationships in the graph: deterministic keyword matching, semantic similarity based on embeddings and machine-learning inference for less obvious associations. A secondary process looks for missing connections, ambiguous entities and related information across documents.

At query time, the system first searches the relevant modality-specific index. It then uses the returned identifiers to traverse nearby graph nodes. Depending on the application, retrieval can stay close to the original result, expand through graph communities or combine both approaches. A multimodal reranker decides which passages and illustrations are inserted into the prompt.

Configurable RAG, with unquantified claims

EY says the system should be tuned by use case. A compliance workflow may require narrow retrieval and deterministic matching, while a research workflow may need broader semantic search. The firm argues that chunking, embeddings, graph-building rules, reranking and retrieval scope should remain configurable rather than locked into a single RAG pattern.

Sengupta also tied the work to AI agents, saying agents need current and domain-specific information to choose actions. EY’s position is that multimodal retrieval can improve the evidence available to those systems and reduce errors that might otherwise carry through an automated workflow.

The qualification is material. EY says it has seen a “manyfold” increase in accuracy and better answer narratives in client work, but the public materials do not provide comparative benchmarks or quantify the gains. That leaves buyers with an architecture claim rather than a published performance case. The logic is plausible for document-heavy enterprises, especially those with technical or regulated content, but evaluation will depend on the quality of each company’s metadata, graph relationships and governance controls.

Sengupta also argued that larger model context windows will not remove the need for RAG. In his view, bigger windows may reduce the pressure to split documents into small chunks, but they do not solve evidence selection. For enterprise teams building generative AI systems, EY’s proposal is a reminder that retrieval quality may matter more than model selection once a deployment moves beyond clean text.

This story draws on original reporting from SiliconANGLE.