The open AI-native stack on Kubernetes: From open models to distributed inference and agents

Kubernetes has become the standard for orchestrating containerized applications, but AI workloads now require a new architectural approach to handle large-scale, multi-tenant production fleets. Google Cloud outlines an open-source stack designed to transition AI from isolated research prototypes to scalable, domain-specialized systems. The stack includes components like OpenRL for reinforcement learning post-training and llm-d 1.0 for distributed inference, which aim to improve accelerator utilization and reduce serving costs. Organizations can test these tools on their own Kubernetes clusters or explore them at KubeCon + CloudNativeCon North America 2026 through live demonstrations.

Scaling reinforcement learning (RL) post-training and distributed inference on traditional Kubernetes infrastructure presents significant challenges, including underutilized accelerators during training and inefficient load balancing for multi-turn agent calls. Google Cloud addresses these issues with OpenRL, a Kubernetes-native post-training service that maximizes GPU and TPU utilization by interleaving training steps with rollout inferences. The service also reduces network saturation by transmitting only parameter deltas to rollout workers, eliminating cold starts and improving efficiency.

Serving interactive, bursty agent traffic requires a distributed inference control plane that can manage stateless HTTP requests while preserving key-value (KV) cache locality. Google Cloud, in collaboration with Red Hat, IBM Research, CoreWeave, and NVIDIA, co-founded llm-d 1.0, a CNCF Sandbox project that transforms standalone inference engines like vLLM and SGLang into a coordinated, cluster-wide serving fabric. The system provides prefix-cache-aware routing and disaggregated serving to handle multi-tier KV cache management effectively.

Executing untrusted, model-generated code in multi-tenant clusters introduces security risks such as sandbox escapes and privilege escalation. To mitigate these challenges, Google Cloud introduces Agent Substrate and gVisor, both part of the CNCF Sandbox. Agent Substrate provides lifecycle management for multi-turn AI agent workflows, while gVisor delivers kernel-level container sandboxing to isolate untrusted agent code. These tools aim to enhance security and reliability in shared Kubernetes environments.