DataFlow-Harness AI pipelines reach 93.3% pass rate in benchmark

DataFlow-Harness AI pipelines reached a 93.3% observed end-to-end pass rate in a 12-task data-engineering benchmark, according to researchers at Peking University, Zhongguancun Academy and Shanghai’s Institute for Advanced Algorithms Research. The open-source framework is aimed at a common enterprise AI problem: coding agents can produce scripts quickly, but struggle when asked to build persistent, governed data workflows for retrieval-augmented generation, synthetic data and training pipelines.

The researchers introduced DataFlow-Harness to move agents away from disposable Python output and toward structured workflow graphs. In their experiments, standard Claude Code with codebase context reached a 94.2% success rate when writing free-form scripts. When an agent had to use platform-native workflow building blocks instead, the rate fell to 83.3%, a 10.9 percentage-point gap the researchers call the “NL2Pipeline gap.”

What is DataFlow-Harness?

DataFlow-Harness is an Apache 2.0-licensed framework that guides an LLM agent to assemble data-processing pipelines as editable directed acyclic graphs, rather than asking it to generate raw code. The point is to leave behind artifacts that engineers can inspect, revise and run through existing production controls.

Runming He, first author of the paper, told VentureBeat that the hard part is less about writing Python and more about grounding the work in a live platform: installed operators, real dataset schemas, registered models and stage dependencies. General coding agents often assume dependencies or platform capabilities that do not exist, according to the researchers.

How does DataFlow-Harness work?

The system has four main parts: a Data Pipeline Backend, DataFlow-WebUI, an MCP Tools Layer and DataFlow-Skills. The backend is the source of truth for the pipeline graph, including data sources, configured processing modules and execution order. Agents change that graph through typed actions, such as adding an operator or connecting stages.

DataFlow-Skills are markdown files that add domain procedures and compatibility rules to the model context. The MCP Tools Layer exposes the operator registry and current pipeline state, then validates proposed changes against metadata. DataFlow-WebUI gives developers a chat interface and a visual graph editor, so the AI’s proposed pipeline can be inspected and changed by a human.

The researchers said DataFlow-Harness beat vanilla Claude Code’s 91.7% pass rate and came within 0.9 percentage points of the context-aware Claude Code baseline. They also reported API cost of $0.261 per task, down 72.5% versus vanilla Claude Code and 42.8% versus the context-aware baseline. Workflow generation latency was 49.9% lower than vanilla Claude Code and 17.6% lower than the context-aware version.

The benchmark covered 12 tasks across six industrial data-processing scenarios, including QA generation, review governance and schema normalization, using Claude Opus 4.7 as the backbone model. In a textbook-to-VQA extraction task, the framework combined PDF parsing, layout recovery, OCR, figure extraction, multimodal understanding and long-range question matching. The researchers reported 97.2% precision and 87.3% coverage, ahead of the baselines.

They also tested synthetic instruction-data generation, where the agent built a multi-stage pipeline to generate candidate instruction-response pairs, critique and rewrite them, score them with an LLM judge and filter weak outputs before training. In a math data cleaning and synthesis task, data from the DataFlow-Harness pipeline trained a model with higher average accuracy on AIME24 and AIME25 than data produced by vanilla Claude Code, according to the paper.

The framework is not a drop-in replacement for common orchestration systems. He told VentureBeat that the current implementation is native to the DataFlow platform and is not a turnkey Airflow, Prefect or Spark plug-in. Teams would need adapters for their registries, metadata and execution interfaces, plus maintained operator registries, schemas and encoded domain procedures. The project’s code is available on GitHub.

This story draws on original reporting from VentureBeat.