Cutting our inference bill by 68% by being obsessive about optimization

In April, Cloudflare’s AI code reviewer averaged $1.19 per review, but costs rose to $2.46 by August due to increased model usage and more complex code changes. The company identified token inefficiencies and model dependencies as primary cost drivers, prompting a mid-September optimization push. By aggressively trimming token usage and refining model selection, Cloudflare slashed average review costs to $0.77, a 68% reduction from the peak $2.43 in the two most expensive weeks prior to optimization.

The optimization efforts also improved review speed, reducing the median review time from 4 minutes 9 seconds to 1 minute 32 seconds. About 40% of savings came from code changes to the reviewer itself, while the remainder stemmed from experimenting with model configurations and effort levels. The cache hit rate rose from 85.7% in April to 97% by September, though the number of tokens per review tripled from 0.91 million to 2.99 million during the same period.

Cloudflare transitioned its reviewer to OpenCode v2, a ground-up rewrite that achieved 98% agreement with v1 in shadow testing before full deployment in early October. Full-tier reviews on v2 completed about 13% faster than on v1 when comparing identical model setups. The company also experimented with running the reviewer on Durable Objects using pi-durable, though further details were not provided.

Before optimization, output tokens accounted for 23% of costs, cache writes 43%, and cache reads 34%. Cloudflare traced the inefficiency to repeated context loading, where each session’s first request rarely reused cached prefixes due to poor prompt construction in OpenCode v1. A fix on September 16 split the system prompt to isolate shared content, enabling cache reuse and reducing the internal standards reviewer’s LLM calls per run from 15.9 to 9.6.