AI coding agents research software report finds speed gains, validation gaps

An OpenAI field report with academic partners says AI coding agents research software projects can produce large speedups and modernize aging tools, but the same systems cannot be trusted to decide whether the science is correct. The report covers eight case studies, mostly in biology, using coding agents including Codex and Claude Code for maintenance, optimization, migrations and rewrites.

The work targets a known weak spot in academic computing: software that began as code for a paper, then became infrastructure for a field without the testing, maintenance budget or engineering attention typical of commercial products. The report’s main claim is operational rather than scientific. Agents helped write or rewrite code, while human researchers set the goals, tests and acceptance criteria.

Can AI coding agents make research software reliable?

The report’s answer is limited. Coding agents can make research software faster or easier to maintain when the task is well specified, but reliability still depends on domain experts building independent validation tests and checking scientific behavior.

One small case involved cyvcf2, a Python library used to read genetic data. GPT-5.5 replaced an outdated build and installation setup with a more current process. A more complex migration moved MHCflurry, an immunology model for predicting immune-cell targets, from TensorFlow to PyTorch across about 10,000 lines of code, with Claude Code and Codex alternating between implementation and review roles.

Other projects went further. Rustar-aligner rebuilt STAR, a sequencing-read aligner with more than 20,000 lines of C and C++, in Rust. In a test using 10,000 short yeast sequencing reads, rustar-aligner matched STAR on 99.815% of single-end reads and 99.883% of paired-end reads. The comparison included mapped genome position and other output fields, and neither program mapped reads the other failed to map.

The largest reported runtime gain came from RustQC, which consolidated 15 quality-control tools into one program. On a large dataset, runtime dropped from 15 hours and 34 minutes to 14 minutes and 54 seconds, more than 60 times faster. HelixForge, a GPU-native replacement for a synthetic genomic data tool, completed a pipeline 59.6 times faster than BamSurgeon in a test using one donor and a ten-million-base-pair genome section. Its main compute step ran 98.6 times faster.

Where the agents failed

The report is clear that fast code did not mean correct code. Participants said agents were useful on defined engineering tasks but could present flawed work confidently. Philip Ewels, who led RustQC, said he did not let models judge their own accuracy and instead used an independent test harness.

The bayesm rewrite shows the risk. The Rust version ran between two and twenty times faster than the original, but early versions of two advanced methods contained errors that were not obvious from plausible outputs. In one case, an agent reversed a key control parameter by using reciprocal values. Researchers found a separate calculation bug only after calibration against thousands of synthetic datasets with known answers.

For hifiasm, a genome assembly tool, the researcher built separate training and validation datasets before asking GPT-5.5 to optimize the code. The resulting changes reduced runtime on real human genome data by nearly 15%. For HI.SIM, a genetic data simulation library, GPT-5.2 and a later model found optimizations that cut runtime by about 31% without changing output.

Maintenance may become the next bottleneck

The report estimates that if agents solved one-quarter to one-half of installation problems across 100 research packages, the value of saved research time would range from $600,000 to nearly $5 million. For NumPy alone, it estimates about 650 hours of annual maintenance work could be saved.

Those savings come with a governance problem. Cheaper rewrites could split user communities and add to the workload of scarce maintainers. Some changes were merged into existing projects. Because STAR is no longer actively maintained, rustar-aligner moved to the scverse research consortium. The author of FastQC declined to replace the Java tool with a Rust rewrite, so the team applied the improvements to the original Java version and achieved the same threefold speedup.

The authors describe the work as exploratory and not a representative study. Their conclusion is still useful for operators building AI-assisted developer tools for science: the coding bottleneck is shrinking, while validation, review and long-term ownership remain unresolved.

This story draws on original reporting from The Decoder.