Model Routing Is Simple. Until It Isn’t.

Model routing is often framed as a simple classification task—send easy requests to cheaper models and reserve expensive ones for harder tasks. However, Hugging Face found that actual costs depend on interactions between models, workloads, and infrastructure, such as caching behavior. In tests using the AppWorld Test Challenge with a CodeAct agent, Sonnet 4.6 was cheaper than GPT-4.1 despite higher sticker pricing due to lower cache-read costs, demonstrating that pricing sheets alone do not determine efficiency.

Estimating task difficulty at routing time is unreliable because many tasks appear simple but require multiple steps, while specialized models may handle complex prompts efficiently. Routing must also account for enterprise constraints like compliance, data residency, and approved model lists, which can override ideal model choices. These factors mean routing is not just about difficulty but a multi-dimensional optimization problem balancing cost, quality, latency, and governance.

Latency is not solely determined by model size; serving conditions such as hardware, cache warmth, and endpoint load often dominate response times. Routing granularity further complicates latency—deciding once per task minimizes overhead, while step-by-step routing increases latency and operational complexity. A router must account for these system-level factors rather than treating latency as a model attribute alone.

Hugging Face’s router shifts from a classification approach to an optimization framework that balances cost, quality, and latency while remaining lightweight. In tests, their latency-optimized configuration reduced costs by 21% and latency by 9% compared to running Opus alone, with only a 4% accuracy drop. The router explores tradeoffs across configurations, outperforming difficulty-based routers by considering the full system context, including caching and infrastructure state.