Global AI routing with <1% overhead on multi-cluster GKE Inference Gateway

Google Cloud’s new multi-cluster GKE Inference Gateway addresses the challenge of fragmented AI infrastructure by treating geographically dispersed accelerator clusters as a unified resource. The system uses real-time telemetry, including KV-cache utilization metrics from inference engines, to dynamically route requests based on actual memory and compute pressure rather than traditional round-robin methods. This approach ensures high utilization of expensive hardware, reducing idle capacity and improving cost efficiency across regions.

In a benchmark involving 17,000 compute nodes across three GKE clusters in the US and Europe, the Gateway demonstrated its scalability by delivering a near-linear throughput increase when expanding from one to three clusters. The deployment, serving a leading Mixture of Experts model via SGLang, maintained a 99.9% success rate even under heavy multi-client concurrency, proving its reliability for production-scale AI workloads.

The Gateway’s architecture integrates with existing distributed inference engines, such as those using tensor parallelism, by routing traffic to leader pods while respecting regional service topologies. This design ensures that cross-region load balancing aligns with underlying multi-node configurations, avoiding inefficiencies that typically arise in distributed AI deployments.

Routing overhead remained below 1%, with the Gateway delivering 99.5% of the throughput of a direct, local cluster call. The system’s ability to detect and respond to memory pressure—such as spilling traffic when KV-cache utilization exceeds 40%—eliminates the need for manual intervention, simplifying operations while maximizing accelerator fleet performance.