The useful question is what changes for users, developers or buyers, and whether the announcement stays industry context or becomes something people can actually use.

Databricks engineers built an internal benchmark to test coding agents on real-world tasks within their multi-million line codebase, covering languages like Python, Go, Typescript, and Scala. The benchmark evaluated tools based on actual edits performed by engineers, ensuring accuracy through careful review of tasks and solutions. Results highlighted performance variations among models and harnesses, with clear distinctions emerging across three capability tiers rather than minor score differences. The analysis aimed to identify patterns that could inform model selection for different task complexities, from simple operational changes to complex design explorations.

The benchmark revealed that while high-intelligence models excel at solving diverse problems, they come at a premium cost. Medium and lower intelligence models proved highly effective for common tasks while being significantly cheaper, prompting Databricks to shift more workloads to models like Haiku and GPT 5.4 Mini. GLM 5.2 emerged as a strong daily driver, matching Opus 4.8 in quality at a lower cost of $1.28 per task versus $1.94. Developers piloting GLM reported consistent performance, reinforcing its suitability for everyday coding tasks.

Token costs often misrepresent actual task expenses due to variations in reasoning efficiency among models. For example, Sonnet 5 was 1.7x cheaper per token than Opus 4.8 but cost $2.09 per task compared to Opus’s $1.94, while scoring six points lower on completion. Harness configurations also impacted costs, with some frameworks consuming over twice the tokens per task despite identical quality. The findings underscored the importance of task-level benchmarking, as model choice alone does not determine efficiency. Databricks invested in Omnigent to enable flexible model-and-harness swaps, addressing these nuances.

Public benchmarks like SWE-Bench and TerminalBench were insufficient for Databricks’ needs, as they could not account for the company’s specific codebase and task complexity. To build a reliable benchmark, Databricks used Unity AI Gateway to log coding interactions, revealing that about 60% of tasks were medium complexity and 25% low complexity. Engineers merge thousands of code changes daily, providing a rich dataset for analysis. The benchmark required manual quality checks, including rewriting tests to avoid anchoring on exact string matches and sealing Git history to prevent agents from exploiting prior solutions, ensuring accurate and fair evaluations.