Announcing llm-d 1.0: Open distributed inference for agents on Google Cloud TPUs and GPUs

llm-d 1.0, an open-source Kubernetes-native stack for distributed LLM inference, was announced at the PyTorch Conference by Google and collaborators including Red Hat and IBM. Developed under the CNCF Sandbox with over 500 contributors across 25 organizations, the project bridges PyTorch serving engines like vLLM and SGLang with Kubernetes-based distributed systems. The release targets the shift from simple chat inference to complex multi-step agent workflows that require coordinated handling across accelerators.

The software turns a cluster of model servers into a unified serving pool, using cache-aware routing and tiered KV cache to keep agent context on the same replicas. During a PyTorch Conference demo, 1,000 sandboxed agents on Agent Substrate accessed a single llm-d endpoint across vLLM, SGLang, and mixed Google Cloud TPUs and GPUs, demonstrating scalable coordination without recomputing prompts or duplicating state.

llm-d 1.0 introduces components such as prefix- and load-aware routing, KV cache offloading, prefill/decode disaggregation, and wide expert parallelism across Google Cloud TPUs and GPUs. These features are designed to be composable, allowing teams to adopt only the serving components needed while retaining their existing engines and clusters. The stack integrates upstream Kubernetes primitives with PyTorch serving engines and accelerator-aware scheduling.

TorchTPU enables native PyTorch model execution on Google Cloud TPUs within vLLM and SGLang without model rewrites, while GKE Inference Gateway runs the llm-d router as a managed service on GKE. The router tracks cached prefixes and balances load and latency, and KV cache offloading to host memory or SSD prevents prefix-eviction storms under heavy multi-turn loads across mixed fleets.