ExtractBench: A New Benchmark for Schema-Guided Document Extraction

ExtractBench, a new benchmark from LlamaIndex, aims to measure how well AI systems perform schema-guided extraction from enterprise documents. The benchmark, which is now available on GitHub and HuggingFace, tests systems on their ability to take a document and a user-defined JSON Schema and return valid, schema-compliant JSON with correct values.

The benchmark includes 370 documents (4,869 pages) across 8 business domains and 67 document types, each with its own schema. Documents are tagged along independent axes—task challenge, perception challenge, table structure, length, and business domain—allowing for detailed analysis of where systems fail.

The headline metric is "Unified value F1," an unweighted mean over documents. A second metric, "Grounding F1," requires that each extracted value point to the correct source evidence, ensuring traceability.

Unlike many benchmarks, ExtractBench avoids LLM-as-a-judge by using deterministic value scoring. Each document counts equally, and failed or missing documents score zero, preventing systems from gaming the average by skipping hard cases. The dataset is compiled from public records, including SEC filings, government procurement forms, and court exhibits, with ground truth established through adjudicated agreement across independent extraction systems, human verification for forms, and pre-defined values for synthetic documents.

Running a full evaluation costs between $10 and $1,677 depending on the system, so the benchmark includes a --test mode that runs six documents for cents to verify setup. The tooling supports custom pipelines and can integrate new extraction systems via Claude Code.

For a developer building document-processing agents, ExtractBench highlights a critical challenge: extraction systems must handle schemas and documents they've never seen, and every error can propagate downstream. If an agent acts on a wrong payment amount or a truncated schedule, the consequences are real. This benchmark provides a way to measure and compare how well systems handle the long tail of enterprise documents with their messy layouts and varied formats.

The project's emphasis on grounding and completeness is a step toward more reliable automation, especially as AI agents increasingly take actions on extracted data before human review. While ExtractBench is just one benchmark, it fills a gap for a common but overlooked task, and its rigorous methodology is a model for future evaluations.