FineBooks OCR benchmark puts a price on cleaning historical books for AI

The FineBooks OCR benchmark, a Hugging Face and EleutherAI collaboration, has evaluated 14 open-weight text-recognition models on 2,165 pages of historical books and published the results as a leaderboard. The project says improved OCR could make public-domain book collections more useful for language-model training, with its highest-ranked system reporting 97.6% character accuracy at an estimated $1.94 per 1,000 pages.

The claim addresses a data-quality problem in open model development. According to The Decoder, the Talkie project found that a language model trained on OCR-derived text learned at 30% of the efficiency of one trained on human transcriptions of the same books. FineBooks’ authors argue that rerunning legacy scans is a high-leverage way to improve open training corpora, where training-data quality can matter as much as volume.

What did the FineBooks OCR benchmark find?

FineBooks used ground-truth transcriptions from six Biodiversity Heritage Library volumes, produced by experts in 2011 and 2012. The material covers English, French, German and Latin; The Decoder reports that the reference transcriptions had roughly one error for every 2,000 characters.

The benchmark measures Character Error Rate, the proportion of characters read incorrectly. It provides a “diplomatic” score, which treats updates to historical forms such as the long s as errors, and a “reading” score that permits them.

• dots.mocr, with 3 billion parameters, was listed at 97.6% accuracy and $1.94 per 1,000 pages.
• OvisOCR2, with 0.9 billion parameters, was listed at 96.9% accuracy and $0.46 per 1,000 pages.
• PaddleOCR-VL-1.6, with 1 billion parameters, was listed at 96.1% accuracy and $0.34 per 1,000 pages.

Those are benchmarked estimates, not universal operating prices. Still, the results cut against the assumption that a larger model will produce better historical-document OCR. FineBooks ranked dots.mocr above Qwen3.5-9B, which has 9.7 billion parameters and was listed at 94.9% accuracy. All 14 tested models can run locally without an API key, according to The Decoder.

A scaling plan with clear boundaries

The Biodiversity Heritage Library holds more than 300,000 digitized natural-history documents and more than 64 million pages, according to The Decoder. FineBooks plans to reprocess about 200,000 public-domain BHL documents with a leading model and publish the resulting text as an open dataset. The size of that planned run turns a model leaderboard into a potential upstream change for open language-model datasets.

Its evidence does not establish that the models are ready for archives or scholarship. FineBooks says the top outputs are adequate for language-model training but still too error-prone for scientific and scholarly transcription. The models also return Markdown or plain text, rather than the word-level coordinates required by library workflows built around ALTO XML.

The test set is narrow: single-column pages in Antiqua typefaces across four languages. It excludes Fraktur, non-Latin scripts, handwriting and more complex page layouts. That scope matters: a separate 16-system study in Information Processing & Management found no OCR approach was best across every document type; multimodal language models did better on unstructured and handwritten inputs in that study, while specialized OCR systems were more reliable on structured and printed documents.

This story draws on original reporting from The Decoder.