Enhancing Agent Retrieval with Structured Chart Extraction
Databricks introduces structured chart extraction in ai_parse_document to improve retrieval and answer accuracy for chart-based questions in enterprise documents, outperforming larger multimodal models while using lighter text embeddings.
Databricks reports that many enterprise agents struggle to interpret charts in proprietary documents, leading to incorrect answers on questions requiring chart data. In tests, a frontier agent miscounted local maxima on a chart (answering 17 instead of 18) after 50 seconds of reasoning, while Databricks Genie correctly identified the count using structured chart extraction. The company highlights gaps in existing benchmarks like OfficeQA Pro, where models perform worse on chart-based questions compared to text-only queries, particularly in financial services where precise data retrieval is critical.
The new approach uses ai_parse_document to extract charts as structured JSON, enabling precise value retrieval and improving both retrieval accuracy and answer correctness. Evaluations on ViDoRe V3 and a synthetic Chart-RAG dataset show that structured chart JSON outperforms large multimodal embedding models, including ColQwen2.5-3B and Qwen3-VL-Embedding-2B, while using a lightweight 300-million-parameter text embedding model. Retrieval gains extend beyond chart-specific questions, as extracted values enhance page retrieval even when answers aren’t directly on the chart.
Adding visual context—by providing images of the top three retrieved text chunks alongside structured JSON—further boosts answer accuracy by 4 percentage points on Chart-RAG and 2.6 points on ViDoRe V3. The structured chart JSON enrichment for ai_parse_document will soon be available, automatically including chart values in parsed documents without changing the function’s interface. For retrieval, Databricks recommends pairing it with ai_prep_search to optimize performance.
The capability will also enhance Genie One’s ability to answer chart-related questions over PDFs for Databricks customers. Structured chart preprocessing addresses a key limitation in traditional RAG systems by providing precise, retrievable data from charts, improving both retrieval quality and agent accuracy. Future research may explore how different structured extraction formats impact performance.