A 180-page credit agreement goes into an AI document extraction pipeline with a schema that asks for interest_rate . The value comes back as 6.25%. It's on the page and was read correctly.
It's also wrong, because the first amendment on page 174 replaced that rate with SOFR plus 350 basis points, and there's a default rate of 11% that applies after a covenant breach. Three real numbers, all legible, all sitting in the same document. Run the file through three different extraction systems and you can get three different answers without a reading error.
Nothing in the schema said which rate, as of when, under what condition. It asked a question the document answers three ways, then took the first answer as the truth.
AI document extraction is the use of machine learning and vision models to pull specified values out of documents and return them as structured data, usually JSON, against a schema a developer defines. The reading half of that job has gotten very good. The specifying half is where production pipelines lose accuracy, and this half almost never gets audited.
Parsing Reads the Page. Extraction Answers a Question.
Two different jobs get called extraction, and conflating them means every failure gets diagnosed as a model problem. Parsing turns a PDF or a scan into a machine-readable structure: reading order, table boundaries, headers, figures, everything a human eye resolves without thinking. Extraction takes that structure plus a schema and returns the values you asked for. We've written about the distinction at length, and it matters here for one reason: the two layers fail differently, and the failures look identical in the output.
A parse failure produces a wrong representation. Columns merge into a single stream, table rows shift by one, a footnote lands in the middle of a sentence. A spec failure produces a perfectly well-formed answer to the wrong question. Both are returned as valid JSON, but only one of them gets fixed by a better model.
Teams keep buying the wrong fix because the parse layer is visible. It has artifacts you can see, public benchmarks you can read, and a vendor market you can shop, which is where the evaluation work goes and why document parsing API comparisons get read. The spec layer has no benchmark and no vendor. It's a JSON file somebody wrote in an afternoon eighteen months ago and nobody has opened since. Optical character recognition, meanwhile, is the oldest layer in the stack and the one that has improved most, which makes it the least likely explanation for a wrong field in your database today.
Your Schema Is the Program and No One Reviews It
A schema field is a claim. It asserts that this document contains exactly one answer to a question, that the answer has a particular type, and that the question is meaningful for every document in the population.
JSON Schema hands you required , nullable , enum , arrays, and nested objects, and every one of those is load-bearing. Schema-based extraction inherits all of it, and the schema is where the assertions live. Get one wrong and you get a silent contract violation that validates cleanly and ships. The trap underneath is that a model asked for a required field will produce one, because refusing isn't default behavior and absence gets rendered as a plausible value.
Teams code-review the pipeline, the prompts, the retry logic, and the model choice. The schema decides what "correct" means, and it usually goes unreviewed.
Fields the Document Doesn't Have
The schema marks po_number required. This vendor doesn't print one. The model has to return something, so it finds the nearest number-shaped string on the page, often a quote number or an account reference, and hands it over with a straight face. The correct output was null, and the schema forbade null.
That's the most common defect in an unaudited invoice extraction pipeline. required belongs to fields that are genuinely universal across the population, and that set is much smaller than a first draft assumes. Marking a field required where it isn't universal is a standing instruction to invent one.
Fields That Mean Something Different per Issuer
"Total" is the canonical case, and it has nothing to do with reading. Pre-tax on one issuer's layout, post-tax on another's, net of credits on a third. One schema field, three definitions, and the extraction is correct by its own lights every time.
The same pattern runs through contract metadata, where effective date, execution date, and commencement date appear under overlapping labels, and through lab reports, where one analyte carries different names by instrument vendor. The fix is semantic: the field description has to carry the disambiguation, because the layout can't.
One-to-Many Collapsed Into One
Documents are full of repeating structures: line items, payment schedules, rent escalations, test results, remittance adjustment codes. Declaring one of them scalar doesn't throw an error. It returns the first match, and the pipeline runs clean for months.
Tables are where this bites hardest, since a table can parse perfectly and still get flattened by a schema that asked for a single value. LlamaExtract carries an extraction_target of per_table_row for exactly this case, and using it is a schema decision rather than a model decision.
Here's what those three fixes look like on one AP schema:
html
// Naive: three assertions the document population doesn't support
{
"po_number": { "type": "string" },
"total": { "type": "number" },
"line_item": { "type": "object" },
"required": ["po_number", "total", "line_item"]
}
// Audited: nullable, disambiguated, and plural where the document is
{
"po_number": {
"type": ["string", "null"],
"description": "PO number as printed. Null if this issuer prints none. Never substitute a quote or account number."
},
"invoice_total_due": {
"type": "number",
"description": "Amount payable after tax and credits. Never the subtotal or pre-tax total."
},
"line_items": {
"type": "array",
"items": { "type": "object" },
"description": "Every billed line, including continuation pages."
},
"required": ["invoice_total_due", "line_items"]Value Accuracy and Record Completeness Are Two Different Numbers
Most reported field-level accuracy in AI document extraction is measured per value, on documents short enough that everything fits in one pass. That measurement is blind to the failure that dominates at scale.
ExtractBench, which we publish, is worth reading on this. It evaluates schema-guided enterprise extraction across 4,869 pages, 370 documents, 8 business domains, and 67 document types, and it deliberately splits three things that usually collapse into one headline number: order-insensitive value F1, record completeness at scale, and word- and page-level grounding F1.
The finding worth remembering is that commercial VLMs do well on short documents and truncate record lists on long ones. A 300-row payment schedule comes back with 260 rows, schema-valid and materially incomplete, and value accuracy on the 260 survivors can look excellent. Coding agents hold their accuracy on long documents at much higher cost. LlamaExtract Agentic Plus ranks first on all three metrics, with accuracy comparable to those agents at a fraction of the price.
| Value accuracy | Record completeness | Grounding | |
|---|---|---|---|
| Question it answers | Is this field's value right? | Did we get all of them? | Where did this come from? |
| Typical failure | wrong scope, wrong issuer semantics | truncated list on a long document | value asserted with no source |
| Shows up in the output | no, the value is well-formed | no, the payload is schema-valid | no, absence is the default |
| What catches it | a second reader with the source | reconciling against a declared count | citations per field |
The parsing side landed on the same idea. ParseBench, which we also publish, scores visual grounding as one of five independent dimensions instead of folding it into an accuracy figure. Weigh our own leaderboard however you like; the design choice is the part worth noticing, because ExtractBench made the same one on the other half of the pipeline. Two teams, two halves, both concluding that "was this right" and "show me where it came from" have to be scored apart. The dimension breakdowns are public.
So measure completeness on its own and reconcile against numbers the document already carries: stated invoice totals, "page X of Y" footers, declared record counts. Straight-through processing rates follow completeness much more closely than they follow per-value accuracy, since one missing line item costs you the whole document either way.
Something Has to Negotiate Between a Rigid Schema and a Document That Ignores It
Every failure above has the same shape: a fixed spec meeting a variable document. One model call can't resolve that mismatch. It gets a single pass, no way to check itself, and an instruction to produce a value for every field you declared.
Schema-guided agentic extraction treats a schema as a set of questions to be answered and defended rather than slots to be filled, and the difference shows up in four places.
The first is order of operations. Layout-aware computer vision splits the page into components before anything transcribes it, so a payment schedule reaches the extractor as a set of rows rather than a stretch of prose a decoder can wander out of. Collapsing a repeating structure into its first entry gets much harder once the structure was identified as repeating.
The second is routing. Instead of sending a whole page to one model, agentic orchestration hands each component to whatever reads it best, a dedicated OCR model for a clean typed block, a vision language model for a stamped or hand-filled field, with a cost optimizer keeping frontier pricing off pages that don't need it. The mechanics of that orchestration go deeper than there's room for here.
The third is counting. Validation passes re-check extracted records against the document's own arithmetic and against the cardinality the schema declared. A model that stopped early has no way to know it stopped early. A loop that compares 260 rows against a stated 300 does.
The fourth is evidence. With cite_sources and confidence_scores enabled, each value returns with the page region it was read from and a score you can route on. The credit agreement's rate stops being arguable: open the citation, see the number came from the defined-terms section rather than the amendment on page 174, and you know which of the three answers you're holding. Only genuinely uncertain fields need a person, which is what gets financial data extraction off a blanket review policy.
LlamaExtract is where the schema lives and LlamaParse does the reading underneath it. Worth being explicit about the architecture: this is the OCR stage, not a repair layer behind one. Point an agentic extractor at a raw Tesseract text dump and it has a flattened document to reason over, because no amount of reasoning rebuilds a table that stopped being a table two steps earlier. The same ceiling caps RAG pipelines, where retrieval can't exceed what extraction preserved.
The Documents Where Writing the Schema Is Harder Than Reading the Page
These three share a property that makes them useful test cases. The parse is easy, the typography is clean, and the schema will still be wrong.
Credit Agreements and Lease Abstraction
Defined-terms sections, amendments and riders that supersede the base document, conditional rates, rent escalation schedules, renewal options with notice windows. The applicable value depends on effective dates and conditions living somewhere other than the value itself. A schema field here has to carry scope rather than just a name: not interest_rate but the rate in effect on a stated date under a stated condition. Contract clause extraction and legal document workflows hit this on every deal.
Certificates of Analysis in Regulated Manufacturing
Every CoA prints a measured value next to a specification range. They sit adjacent and mean opposite things, and a schema asking for "result" without binding it to a test method and a unit will cheerfully return the spec limit. Analytic names and units vary by lab, so comparing suppliers means normalizing that inside the schema, which is why manufacturing document workflows stall on data quality rather than OCR.
Clinical and Laboratory Reports
Reference ranges shift by lab, by instrument, and by patient demographic, so a result without its range isn't interpretable. Panels repeat across pages. A scalar field for a repeating analyte is the one-to-many failure with a clinical consequence attached, and it's much of why clinical data extraction is harder than the page images suggest.
The Next Gain in AI Document Extraction Comes From the Spec
Model quality is converging, and the interesting question has moved to what surrounds the model. An automation rate that won't climb is usually a spec problem hiding inside an accuracy conversation: fields that ask documents questions those documents can't answer, reported through a single number covering three things that fail independently.
A design problem has no upgrade path, which is the uncomfortable part. It gets fixed by writing better questions and demanding evidence for each answer.
So audit the schema you're already running in production. Go field by field. How many are answerable from the document alone, and how many require business logic the document doesn't carry? How many are marked required against a population where they aren't universal? How many repeating structures did somebody declare scalar? Most teams find at least one of each, and the fields they find tend to be the ones already generating exceptions.
Then run the audited version somewhere that returns grounding per field, so every answer arrives with the evidence that justifies it. LlamaExtract does that with confidence scores and citations on each value, which is what separates an automated document extraction system from a JSON payload you have to take on faith. If you'd rather survey the field first, we keep a running breakdown of document extraction platforms, including the ones we compete against. Otherwise, feed it the schema you just audited and count how many fields come back with nothing to point at. LlamaParse is free to try with 10,000 credits upon signup.
