The new leaderboard, ExtractBench, is designed to accelerate progress building and evaluating document extraction systems that can pull structured data from the messy, high-stakes files enterprises run on.
Check out the new leaderboard on Kaggle: https://www.kaggle.com/benchmarks/llamaindex-org/extractbench-leaderboard/leaderboard
Enterprise agents are increasingly making decisions on structured data pulled from unstructured documents. A claims agent extracts coverage limits from a 90-page filing and routes the claim. A compliance workflow pulls values from scanned contracts and flags exceptions. The extraction system is the critical dependency, and until now there hasn't been a rigorous way to evaluate how these systems perform on the documents enterprises actually process.
Today, LlamaIndex is partnering with Kaggle to launch ExtractBench, an open benchmark that evaluates extraction systems on 370 enterprise documents (4,869 pages) across eight business domains. The benchmark scores 14 systems on accuracy, perception quality, table structure, document length, and cost. The leaderboard is live on Kaggle now.
The challenge of enterprise document extraction
Enterprise documents don't just get read anymore; they get extracted. An agent pulls structured fields from a 90-page insurance filing and routes the claim automatically. A compliance workflow extracts values from scanned contracts and flags exceptions without a human in the loop. When the extraction is wrong, the agent acts on bad data. If a coverage limit is misread, the claim is adjudicated incorrectly. If a date field is skipped on page 60, a deadline passes silently. If a handwritten annotation is missed, the signed terms don't match the system of record. The bar for extraction has shifted from "good enough for a human to QA" to "reliable enough for an agent to act on."
This has led to a wave of extraction systems, from frontier VLMs to specialized APIs to coding agents, all promising production-grade accuracy. The challenge is telling them apart. Existing extraction benchmarks test short, born-digital documents with clean formatting and fixed schemas. They don't cover scanned pages, handwritten fields, or rotated inputs. They don't measure what happens past page ten. They don't score whether an extracted value can be traced back to its source. And they don't report cost. Even the most thorough benchmarks cover just a slice of the workload that enterprise teams actually run.
Introducing ExtractBench
LlamaIndex is on a mission to unlock document understanding for AI agents. Earlier this year, we released ParseBench, a benchmark that brings scientific rigor to document parsing and OCR evaluation. ExtractBench is the next step: where ParseBench measures how well a system can parse a page, ExtractBench measures how well a system can extract structured data from an entire document and get every field right.
ExtractBench evaluates 14 systems across 370 enterprise documents (4,869 pages) spanning eight business domains and 67 document types, with over [X] evaluation rules across five axes that actually separate systems in production: task challenge, perception quality, table structure, document length, and domain coverage. In practice, that means tasks like pulling every line item from a multi-page invoice with merged cells, extracting nested coverage limits from a scanned insurance filing, reading handwritten annotations on a signed contract, or recovering structured records from a 100-page regulatory submission where the tables start on page 40.
The benchmark tests frontier VLMs (GPT-5.5, Claude Opus 4.8, Gemini 3 Flash), coding agents (Codex, Claude Code), and specialized extraction APIs (Reducto, LlamaIndex's Extract in both Cost Effective and Agentic Plus tiers). Every system ran on the same frozen schemas so results are directly comparable. The evaluation is deterministic and rule-based; there is no LLM-as-judge.
Agentic extraction systems take this further. Rather than passing an entire document through a single model call, they break the document into sections, route each section through the right model, cross-check extracted values against the source page, and retry on low-confidence fields. ExtractBench is designed to show where that extra work pays off, particularly on long documents and messy inputs where simpler approaches fall apart.
Where systems diverge
The Kaggle leaderboard covers frontier models. The full paper evaluates 14 systems, including coding agents and specialized extraction APIs, and that's where the sharper differences show up.
While the overall leaderboard shows which systems score highest, the per-challenge breakdowns reveal which document challenges separate them.
Long documents are where the systems really start to separate. Gemini 3.5 Flash drops from 87.9% on short documents to 27.9% on long ones. It often gets individual values right, but stops before it reaches the end of a long record list. We saw the same pattern across commercial VLMs: accuracy held up through the first ten pages, then fell as documents got longer.
LlamaIndex's Extract Agentic Plus holds at 94.4% on the same long documents. Its agentic pipeline splits each document into sections, then checks the extracted values against the source pages.
Coding agents come close to the same accuracy, but they cost much more. Codex GPT-5.5 reaches 93.5% overall at 27.8¢ per page, compared with 95.6% at 8.1¢ for LlamaIndex's Extract Agentic Plus. At a million pages, that’s $278,000 versus $81,000.
Accuracy and cost aren’t the only differences. Commercial VLMs and coding agents return extracted values without word-level bounding boxes, so reviewers cannot see exactly where each value came from. LlamaIndex's Extract was the only system we tested that returned a bounding box for every extracted value. Reviewers get an exact source location for each field without having to search the document manually.
What goes wrong also depends on the document. Long record lists get cut off or come back with duplicate rows. On dense forms, systems may invent values for blank fields or assign a nearby date, identifier, or amount to the wrong field. Complex tables create their own problems: merged headers can send values to the wrong column, while page breaks can make the system lose track of how a table continues.
Because ExtractBench tags every document by challenge type, you can see which failure modes are hidden behind a system’s overall score, from truncated record lists and over-extracted fields to complex tables, scans, and handwriting.
Why Kaggle
Kaggle gives us transparent, reproducible evaluation on neutral ground. Anyone can submit a system and see exactly how it was scored. It also removes the operational complexity of managing submissions and versioning results. We'd rather spend the time making the benchmark harder than maintaining leaderboard infrastructure. And Kaggle is where practitioners and researchers already go to compare methods. We want ExtractBench to be the place where extraction systems get evaluated on real enterprise workloads, not demos.
What's next
We plan to expand ExtractBench with new document types and harder edge cases, deepen the failure-mode analysis for each evaluation axis, and introduce end-to-end agentic evaluation where extraction is scored as part of a complete agent workflow, not just as an isolated step.
Everything is open:
Run your own system against it. If it does well, submit to the leaderboard. If it doesn't, tell us what we're missing.
