Networking for AI inference model serving

Google Cloud introduces two reference architectures for AI inference model serving, one tailored for Google Kubernetes Engine (GKE) and another for other backend types, aiming to simplify model invocation while centralizing governance. Both designs share common components such as secure entry points, policy enforcement, and optional API management integrations. The architectures leverage Private Service Connect endpoints to keep inference traffic within private networks, ensuring data remains internal. Security measures include TLS termination, API management for client verification, and Model Armor for screening prompts and outputs against policy violations.

The GKE-specific architecture introduces a GKE Inference Gateway deployed as an internal Application Load Balancer, which parses request payloads, evaluates routing rules, and directs queries to appropriate model-serving targets. Inference pools group model replicas that can autoscale dynamically, while model replica sets consist of uniform instances across GPU or TPU node pools. The flow begins with a client call routed via Private Service Connect to the GKE Inference Gateway, where payload inspection, control plane validation, and backend selection occur before inference execution.

For architectures spanning mixed environments, including Cloud Run, Agent Platform, on-premises data centers, or external clouds, a regional internal Application Load Balancer serves as the central Layer 7 routing proxy. This design uses a Cloud Run callout to inspect JSON payloads and inject headers for routing, enabling body-based routing without requiring GKE-specific components. Network Endpoint Groups (NEGs) provide flexible routing to heterogeneous backends based on the injected model header, ensuring consistent handling across diverse environments.

The architectures emphasize secure, scalable, and policy-enforced inference routing, with both designs supporting Private Service Connect endpoints for private network traffic. Model Armor acts as an inline safety checkpoint for screening prompts and outputs, while API management components like Apigee can enforce client identity verification, rate limits, and quota enforcement. The designs aim to streamline model serving by centralizing routing logic, security policies, and governance across varied backend environments.