Introducing OfficeQA Pro V2: A New Benchmark for Enterprise Grounded-Reasoning
Databricks released OfficeQA Pro V2, a benchmark testing AI agents' ability to generalize grounded reasoning across unfamiliar enterprise documents, following the original OfficeQA benchmark introduced seven months prior.
OfficeQA Pro V2 evaluates whether AI systems can generalize grounded reasoning beyond a single document collection, addressing a gap identified in enterprise settings where agents rarely operate on stable corpora. The benchmark was first used in the Databricks Grounded Reasoning Cup, a competition involving 11 academic teams supported by OpenAI, Anthropic, and Google DeepMind, evaluated on a previously unseen corpus of approximately 120,000 pages from the U.S. Treasury’s Accounts of Receipts and Expenditures. The dataset, released by the U.S. Treasury for the first time to mark the 250th anniversary of the United States, spans accounting records from 1793 to 2024, including dense tables, evolving conventions, and institutional complexities typical of enterprise documents.
Out-of-the-box frontier agents achieved an average accuracy of 37.5% on OfficeQA Pro V2, while agents developed specifically for the competition averaged 41.1%, with the winning team reaching 63.3%. Databricks’ Genie agent, using the same underlying models, improved accuracy by an average of 24.0 percentage points, with its strongest configuration reaching 60%. Performance varied significantly by model and cost, with higher-priced models not consistently outperforming cheaper ones, highlighting the role of agent harness design in improving results.
OfficeQA Pro V2 introduces more rigorous challenges than its predecessor, requiring evidence from an average of 6.7 source documents per question—nearly triple the 2 documents required in OfficeQA Pro. The benchmark includes 90 questions, with 7% requiring visual interpretation of charts or graphs and 10% necessitating supplemental web search for external data such as inflation indices. These demands reflect the diverse, real-world analytical tasks encountered in enterprise workflows, where agents must retrieve, parse, and reason across multiple, evolving documents.
The benchmark’s difficulty stems from the historical evolution of the underlying documents and reporting conventions, which have changed significantly over centuries. Despite advances in agent harnesses like Genie, which leverages Databricks’ ai_parse for efficient document parsing, OfficeQA Pro V2 remains a formidable test. Systems continue to struggle with parsing fidelity, temporal reconciliation of revised figures, and misinterpretation of entity scope, underscoring that grounded reasoning remains an unsolved challenge in AI.