OCR Document Processing: Why a Better Extractor Doesn't Shrink Your Review Queue

A team swaps out the extraction model in their document pipeline. Field accuracy on the internal test set climbs from 94% to 97%. That's a real gain, and it took most of a quarter to get. Then they look at the exception queue, and it's the same size it was before.

The usual explanations come out fast. The test set wasn't representative. The new model regressed on some document type nobody checked. Occasionally that's true. Much more often, the model really did get better, and it got better at something the pipeline doesn't route on.

OCR document processing is the pipeline that carries a document from intake, through optical character recognition that turns pixels into machine-readable text, then field extraction, validation, and finally into a system of record. The OCR engine is one stage inside that pipeline. The accuracy of that stage turns out to be a surprisingly weak predictor of how much work actually gets automated, and the reason has nothing to do with how good the model is at reading.

What decides the outcome is whether the system knows which fields it got wrong. That's a different property from getting them right, it's measurable, and almost nobody measures it.

A 97% Accurate Extractor Can Still Leave Half Your Documents in the Queue

There's a unit mismatch buried in every document automation project, and it survives because the two numbers involved get quoted by different people.

Vendors quote accuracy per field. Operations measures straight-through processing per document: the share of documents that reach the system of record without a human touching them. Those two numbers have different denominators, and the gap between them compounds.

Take an AP invoice with 20 extracted fields, which is a modest schema. At 97% per-field accuracy, the chance that all twenty fields are correct is 0.97²⁰, or about 54%. Nearly half of the documents contain at least one bad field. One bad field is enough to send the whole document to a person, because you can't post an invoice with a wrong remit-to account and call it 95% posted.

Push per-field accuracy to 99% and clean documents rise to about 82%. Drop to 94% and they collapse to 29%.

That's an idealized model, and worth being honest about: field errors aren't independent in practice. A bad scan tends to blow up many fields at once, which clusters the damage into fewer documents and makes the real numbers somewhat kinder than the arithmetic suggests. The direction holds regardless. Per-document cleanliness degrades much faster than per-field accuracy, which is why a 3-point accuracy win lands on the ops team as a rounding error.

The 94% to 97% jump, in this model, moves clean documents from 29% to 54%. That's a genuine improvement and a large one. It still leaves 46% of documents with something wrong in them, and here's the part that actually determines the queue size: the pipeline has no idea which 46%.

Field-level accuracy is necessary and nowhere near sufficient. Chasing extraction accuracy on its own gets you a better model and the same queue.

Your Automation Rate Is Decided at the Routing Gate

Every document in a production pipeline hits the same fork. It gets auto-posted, or it gets sent to a human. Something has to make that call, and what it makes the call on is a confidence signal.

Which means straight-through processing is a property of the confidence scores, not of the accuracy sitting behind them. Two pipelines, same documents, same 99.9% quality bar. Read these as document-level numbers, the share of documents that come out fully correct, since that's what the arithmetic above actually leaves you holding:

Pipeline A gets 97% of documents fully right and returns bare strings. No per-field signal. The 3% that are wrong are invisible, with nothing to tell them apart from the other 97%. To hold the quality bar, a human reviews everything, so straight-through processing is zero. The alternative is to auto-post the lot and let a 3% document error rate into the ledger, which no controller signs off on. That 97% bought nothing.

Pipeline B gets 94% of documents fully right, three points worse, but it returns per-field confidence, and those scores separate the good values from the bad ones. Its uncertainty concentrates: about 8% of documents carry at least one low-confidence field, and nearly all of the 6% that are genuinely wrong sit inside that flagged 8%. Review the 8%, auto-post the other 92%, and the quality bar holds.

Pipeline B is three points less accurate, and it automates 92% of the work that Pipeline A hands to a human. That inversion is the whole game, and it's why buying accuracy in isolation is such a reliable way to spend a quarter and move nothing.

The published automation ranges are usually quoted as though accuracy produced them. LlamaIndex's own agentic OCR breakdown puts traditional OCR at 60-80% straight-through processing and agentic approaches at 90-95%+, and the broader document AI overview tells a similar story. Those numbers are real. The mechanism behind them is routing quality, not raw reading skill.

What a Confidence Signal Has to Be Before You Can Route On It

A number between 0 and 1 attached to an extraction is not automatically useful. To gate on it, it has to be two things.

• Separating. Wrong values have to score lower than right ones, reliably enough that some threshold sits between them. That's a weaker property than calibration, and it's the one a gate actually runs on. Calibration, where a 0.90 is right 90% of the time, is the luxury version: it lets you pick the threshold on paper instead of by experiment. Almost nothing on the market is calibrated in that sense, so you sweep the threshold against your own corrections log and keep the point where the flagged set covers the errors. The signal that kills a gate is one that doesn't separate at all, where wrong values sit at 0.9 as often as right ones do.
• Per-field, not per-page. A page-level score can't tell a reviewer which value to check, so it has no choice but to push the entire document into review. It converts uncertainty about one number into manual handling of forty.

Most of the signals teams actually have fail one or both tests:

The Textract row is the one most teams think they've already solved. Textract does return a confidence per key-value pair, and that number is measuring something narrower than it looks. In AWS's own documented example, the pair scores 51.6 while the individual words inside it score 99.5 and up. Those two numbers aren't in conflict. The engine read Ana Carolina almost perfectly and was near a coin flip on whether Ana Carolina is what belongs next to Name:. That's real information about the layout, and it still isn't a claim about whether the value is correct.

That last row is the one that changes the economics, and it's rarer in production than the marketing suggests.

Catch Rate: The Number That Predicts Your Automation Rate

Here's the metric worth adding to your dashboard: of the documents your pipeline got wrong, what fraction did it flag?

Call it catch rate, or error recall. It's cheap to compute, since it only needs your corrections log, and it's a far better predictor of your automation ceiling than accuracy is. An engine with no confidence output has a catch rate of zero at any accuracy whatsoever, which is the formal reason accuracy alone can't buy automation. Pipeline A above is a 97% accurate system with a catch rate of zero. Pipeline B is a 94% accurate system with a catch rate near 1.

Catch rate and your review threshold together determine straight-through processing at a given quality bar. Accuracy is an input to that, and only one of several.

The Documents That Never Reach the Extractor

Benchmarks run on clean corpora. Document ingestion in production does not, and the pipeline leaks well before the OCR engine is called. None of it shows up in an accuracy number, because a document that was never parsed was never scored.

The intake failures are boring and relentless: password-protected and permission-locked PDFs, HEIC photos taken on a phone and emailed in, screenshots of screens, truncated and 0-byte uploads, Office files renamed to .pdf . Then there are faxes arriving at 200 DPI when the resolution floor for reliable recognition sits at 300, and 900-page bundles holding twelve document types that have to be split and classified before a single field can be pulled from any of them.

A document processing platform that silently drops 4% of intake has capped its straight-through processing at 96% before the model does anything at all. Ceiling effects compound downward through the stages, and they're invisible to every metric pointed at the extractor.

The other thing most pipelines skip is the second signal. Validation against something outside the model: line items that must sum to the stated total, MRZ and routing-number checksums, service dates that must fall inside a policy period, vendor IDs that must exist in a reference table. Cross-field validation raises catch rate specifically because it isn't correlated with the model's own confidence. It catches the errors the model felt good about, which are the ones that hurt.

Building the Signal Instead of Buying More Accuracy

If routing decides the outcome, the question to ask an intelligent document processing vendor stops being how accurate they are and starts being what comes back alongside each value. Accuracy is table stakes across most of the current OCR field anyway. The routable output is what's scarce.

LlamaParse handles the parse layer with layout-aware computer vision that segments the page before anything tries to read it, so a table is processed as a table and a 900-page bundle is split into its constituent documents first. Agentic orchestration routes each element to the model that reads it best, with a cost optimizer that avoids paying frontier-VLM rates on a page of clean body text. Every page carries provenance back to its source.

The routing signal itself lives in LlamaExtract, which returns per-field confidence against a schema you define. That distinction matters in practice, and the two products do different jobs: parsing gives you faithful structure out of complex layouts and extraction gives you the gated, schema-conforming structured data a downstream system can actually post.

import os, time
from llama_cloud import LlamaCloud

client = LlamaCloud(api_key=os.environ["LLAMA_CLOUD_API_KEY"])
file_obj = client.files.create(file="vendor-invoice.pdf", purpose="extract")

job = client.extract.create(
    file_input=file_obj.id,
    configuration={
        "data_schema": InvoiceSchema.model_json_schema(),
        "extraction_target": "per_doc",
        "tier": "agentic",
        "confidence_scores": True,
        "cite_sources": True,  # bounding-box citations back to the source pixels
    },
)

while job.status not in ("COMPLETED", "FAILED", "CANCELLED"):
    time.sleep(2)
    job = client.extract.get(job.id)  # poll until the job reaches a terminal state

if job.status != "COMPLETED":
    raise RuntimeError(f"extraction {job.status} for {file_obj.id}")

# expand is what pulls back the per-field confidence and citations
job = client.extract.get(job.id, expand=["extract_metadata"])
field_meta = job.extract_metadata.field_metadata.document_metadata

# The gate. A document goes to a human because a field that matters is
# uncertain, not because a page average dipped. The scores are uncalibrated by
# design, so 0.95 here is not "95% likely right." Treat it as a rank and tune the
# cutoff against your own corrections log.
CRITICAL = {"invoice_total", "vendor_tax_id", "remit_to_account"}
needs_review = [f for f in CRITICAL if field_meta[f]["confidence"] < 0.95]

if needs_review:
    route_to_exception_queue(job, flagged_fields=needs_review)
else:
    post_to_erp(job.extract_result)

That is what a routable output looks like, and it's the difference between an automated document extraction system and a text dump you still have to babysit. The confidence score names the suspect field. The citation lets a reviewer confirm it against the pixels it came from in seconds rather than re-reading the page, which is what turns human-in-the-loop review from a rubber stamp into an actual control. Validation loops catch known hallucination patterns before the value ever reaches the gate, which raises catch rate rather than just nudging accuracy.

This replaces the extraction stage rather than sitting behind Tesseract or Textract as a cleanup pass, and teams who bolt it on that way inherit the ceiling of whatever ran first. The same architecture is what moves the needle in loan document automation, invoice data extraction, and financial data extraction, where the review queue is the entire cost center.

The Ceiling on Automation Is What Your Pipeline Knows About Itself

Models cleared the "can it read this page" bar a while ago. The remaining differentiator in OCR document processing is whether the system can tell you where it's unsure and prove where each value came from. The queue, the headcount, the audit trail, and the ROI are all downstream of that one property.

Teams stuck at 60% automation usually have a perfectly decent extractor sitting behind a gate that can't tell them which documents to look at. More accuracy won't move them, because accuracy was never the thing the gate was reading.

So change the evaluation question. The next time an OCR vendor leads with an accuracy number, ask what ships alongside each extracted field. Confidence per value, or a page average? A citation back to the source pixels, or a bare string? If the honest answer is a bare string, then that accuracy number is the only thing you will ever get, and you'll be reviewing everything to protect it.

LlamaParse answers it the other way, with layout-aware parsing underneath and per-field confidence and citations on top, which is the machinery a routing gate actually runs on. You can point it at your own documents and check the flagged set against your corrections log. The comparison worth running is not which engine reads your documents best, but which one tells you when it didn't.