A practical guide to extraction confidence scores

For this evaluation, we calculated AUGRC as:

Here, N is the number of evaluated rows: returned fields, unmatched predictions, and omitted blanks credited as correct. After sorting by confidence, Eₖ counts the errors among the first k rows.

At each acceptance level, we divide the accepted errors by all evaluated rows, then average over those levels. Multiplying by 1,000 expresses the result as average accepted errors per thousand evaluated rows.

Random baseline. Keeping the same evaluated results fixed and randomizing their acceptance order gives an expected AUGRC × 1,000 of 26.5 for Agentic Plus, compared with 15.0 using its actual ranking. Overall extraction accuracy stays the same.

Unscored rows get a score of 0, so they tie with rows scored 0. Tied rows are averaged exactly over every ordering within the tie, with no random sampling. Missing fields other than credited blanks are outside its denominator. See the AUGRC paper for the underlying definition.

Score-path correction. We corrected a benchmark field-path mapping that had dropped confidence scores for fields named properties. Every returned value in the Extract runs has a score. All affected metrics were recomputed from the archived grades and restored scores; extraction values and correctness grades are unchanged. Reducto and Extend results are unchanged by this correction.

Dataset and runs. Every system processed all 370 ExtractBench documents between September 5 and 9, 2026. The dataset contains 4,869 pages and 841,895 ground truth fields. The confidence model was trained and calibrated on documents outside ExtractBench.

Grading. We used archived field grades, with blank and missing-record treatment based on the ExtractBench paper, Table 9. All ground truth fields count, including blanks. A null or omitted key for a blank field is correct; an extracted value in that field is wrong. Every field in a dropped record counts as a miss. Every returned scalar field is graded, whether or not it has a confidence score. Empty record arrays contribute no fields.

Precision and recall. Precision before filtering is correct evaluated rows divided by all evaluated rows. These rows include extra predictions with no ground truth match and omitted blanks credited as correct. Recall before filtering is correct fields divided by all 841,895 ground truth fields. Recall in the threshold chart is correctly accepted fields divided by that same ground truth total.

These metrics pool fields across all 370 documents. Long documents therefore carry more weight than they do on the ExtractBench leaderboard, which scores each document and then averages.

Comparison with ExtractBench. This analysis counts scalar list elements individually. Precision before filtering uses evaluated rows and excludes other missing fields; the paper also counts missing scalar keys inside represented records against precision. Those missing fields count against recall here. These confidence results therefore should not be read as a reproduction of the leaderboard.

Thresholds and missing scores. For each precision target, we used the lowest actual score boundary that still met it on the pooled benchmark results. These are retrospectively selected benchmark operating points. For deployment, choose a cutoff on one representative labeled sample and verify it on another. This guidance is not a second evaluation reported here. Values without scores could not be accepted automatically when confidence filtering was enabled. For Reducto, only numeric extraction confidence counted as a score.

Deployment checks. Evaluate fields separately when they need different accuracy targets. Route values below the cutoff for review, handle unscored values explicitly, and check for missing expected fields. To prioritize review by confidence, verify that the lowest scores contain a higher share of errors. Keep checking accepted-value accuracy as your documents change.

Extend's Review Agent returned ratings from 1 to 5. Its documentation describes 5 as no issues detected and 1 as critical issues; its workflow example routes scores of 3 or below to review. We mapped the ratings to 0, 0.25, 0.5, 0.75, and 1.0 for plotting, preserving their order and ties. This mapping does not treat the ratings as probabilities.

Cost. The cost chart uses the billed credits for each run at published credit prices, divided by 4,869 pages. Both Extract tiers shown used the Agentic parse tier in this evaluation, as shown in the configurations below. These are measured run costs, so the chart should be read in the context of that setup.

Review counts and source data. The baseline is every returned value from the same run, excluding omitted blanks credited as correct. Needs review equals returned values minus accepted correct and accepted wrong. Each system has a different returned-value total, and missing expected fields are outside this denominator. Counts at the original operating points were recovered from archived full-precision threshold percentages and checked against integer totals. The cutoff explorer adds measurements at 0.001 intervals from the same archived field grades, with no interpolation. The figure downloads use the same values as the figures. The source archive contains the corrected threshold and system-summary CSVs, the previous snapshot, and a record of restored scores.

These excerpts show the job submission calls used in the benchmark. DOCUMENT and SCHEMA stand for the document and its ExtractBench schema. Credentials must be supplied for each service.

# Extract
from llama_cloud import LlamaCloud

client = LlamaCloud()
file = client.files.create(file=DOCUMENT, purpose="extract")
parse_config = client.configurations.create(
    name="benchmark-parse",
    parameters={
        "product_type": "parse_v2",
        "tier": "agentic",
        "version": "latest",
        "page_ranges": {"max_pages": 200},
        "output_options": {"granular_bboxes": ["word", "cell"]},
    },
)
job = client.extract.create(
    file_input=file.id,
    configuration={
        "tier": "agentic_plus",  # "agentic" for the other tier shown; the parse config stays the same
        "extraction_target": "per_doc",
        "confidence_scores": True,
        "cite_sources": True,
        "parse_config_id": parse_config.id,
        "data_schema": SCHEMA,
    },
)

# Reducto Deep Extract
from reducto import Reducto

client = Reducto()
upload = client.upload(file=DOCUMENT)
job = client.extract.run_job(
    input=upload,
    instructions={"schema": SCHEMA},
    settings={
        "citations": {"enabled": True},
        "deep_extract": True,
    },
)

# Extend with Review Agent
from extend_ai import Extend

client = Extend(token=EXTEND_API_KEY)
file = client.files.upload(file=DOCUMENT)
processor = client.processor.create(
    name="benchmark-extract",
    type="EXTRACT",
    config={
        "type": "EXTRACT",
        "baseProcessor": "extraction_performance",
        "baseVersion": "4.8.1",
        "schema": SCHEMA,
        "advancedOptions": {
            "citationsEnabled": True,
            "citationMode": "line",
            "arrayCitationStrategy": "property",
            "arrayStrategy": {"type": "large_array_max_context"},
            "reviewAgent": {"enabled": True},
        },
    },
)
job = client.processor_run.create(
    processor_id=processor.processor.id,
    file={"fileId": file.id},
    sync=False,
)