llm-d 1.0 Advances Open, Cost-Efficient AI Inference for Production Scale
The llm-d 1.0 open source project launches to simplify and reduce costs of deploying large language model inference at production scale, addressing challenges in agentic workloads and infrastructure efficiency.
The llm-d project, governed by the Cloud Native Computing Foundation and founded by Red Hat, IBM, Google Cloud, CoreWeave, and NVIDIA, released version 1.0 to help organizations deploy and scale large language model inference efficiently across modern infrastructure. The project addresses the growing challenge of running AI at scale, where operational costs and performance demands often rival those of model development itself. llm-d 1.0 introduces capabilities such as prefix-cache and load-aware routing, KV cache offloading, and disaggregated prefill/decode to improve latency, throughput, and memory efficiency in production environments.
Agentic AI workloads, which require continuous context access and parallel agent support, place unique stress on infrastructure systems, often consuming more resources managing context than generating output. llm-d 1.0 is engineered to handle these demands by supporting multi-tenant serving, zero-downtime operations, and complex workloads without sacrificing responsiveness or efficiency. The project already integrates with multiple model servers, including vLLM and SGLang, and is used by over 500 contributors from more than 25 organizations across the AI infrastructure ecosystem.
IBM Research demonstrated llm-d’s cost efficiency by achieving a 98% prefix-cache hit rate and a blended serving cost of approximately $0.11 per million tokens, reducing inference costs by 11.5 times compared to an all-proprietary configuration. In another test, llm-d served a 753-billion-parameter open-weight model on NVIDIA GPUs, delivering over 6.6 million output tokens per minute at peak while supporting up to 3,000 concurrent coding agents with zero preemptions.
The project’s CNCF governance model and broad industry participation aim to standardize distributed inference, providing an open alternative for managing complex AI infrastructure. Contributors like Google Cloud are integrating llm-d 1.0 into Google Kubernetes Engine and enabling it across TPU and NVIDIA GPU portfolios, while CoreWeave and NVIDIA are supporting deployments on their platforms to optimize performance and reduce costs for enterprises.