Evaluating AI Agents Live at the Grounded Reasoning Cup
Databricks hosted the Grounded Reasoning Cup, a live AI competition testing 11 academic teams on enterprise document reasoning. Stanford’s agent achieved 63.3% accuracy, outperforming others by up to 35 points, while highlighting persistent challenges in grounded reasoning.
The Grounded Reasoning Cup, hosted by Databricks, challenged 11 top academic teams from the U.S. and Canada to develop AI agents capable of reasoning over complex enterprise document collections. Teams worked for two months using the OfficeQA benchmark to refine their systems, then competed in real time on a newly released OfficeQA Pro V2 benchmark to test generalization. Stanford’s winning agent achieved 63.3% accuracy, surpassing the average team score by 22 points and the average frontier agent baseline by 35 points, demonstrating measurable progress in grounded reasoning.
Stanford’s victory stemmed from a system that transformed common failure modes into reusable skills for its Claude Opus 4.8 agent, including document parsing strategies and answer verification procedures. The team prepared over 100 skills but adapted during the competition, removing a verification step to reduce latency and later re-enabling it to correct a critical answer in the final round. UMass Amherst prioritized speed, using parallel agents and a metadata catalog to achieve the fastest average submission time for correct answers, earning 36 speed bonuses despite finishing second.
Yale’s approach focused on robustness through a multi-arm verification system, running four independent agents with distinct strategies and a meta-verifier to select the final answer. The team’s emphasis on parallel verification and fallback mechanisms earned third place with 49 correct answers out of 90. Across all top teams, success depended on holistic system design, including document preprocessing, targeted retrieval, structured tool use, and verification steps, rather than relying solely on model choice.
The competition underscored both progress and remaining challenges in enterprise grounded reasoning, with 18.8% of questions unsolved by every team. Databricks acknowledged support from industry sponsors OpenAI, Anthropic, and Google DeepMind, as well as partners USAFacts and the U.S. Treasury for their roles in developing the OfficeQA Pro V2 benchmark and dataset.