ZenteiQ scales batch inference 4x with GKE Inference Gateway powered by llm-d
ZenteiQ increased batch inference throughput 4.07x and daily token output from 1.47 billion to 5.97 billion using GKE Inference Gateway and llm-d, accelerating scientific foundation model training.
ZenteiQ, an AI startup spun out of the Indian Institute of Science under the IndiaAI Mission, is building sovereign scientific foundation models trained on governing equations and numerical solvers. The company developed a large-scale batch synthetic data-generation pipeline to pre-train models that interpret physical laws across domains like thermodynamics and electromagnetics. The pipeline generates, evaluates, and filters candidate derivations before feeding verified datasets into MaxText pre-training jobs.
The pipeline operates on a fixed compute allocation, where delays in data generation idle multi-host accelerator clusters reserved for pre-training. To maximize speed, ZenteiQ separates prompt preparation on worker pods from model serving on vLLM across multi-node accelerator pools. Each batch moves through four stages: expanding scientific seed concepts, generating reasoning traces, scoring responses, and running guardrail checks before packaging verified datasets.
Initially, ZenteiQ used Kubernetes with standard ClusterIP round-robin load balancing, which caused uneven queues and redundant prefill work. These bottlenecks slowed dataset creation, delaying pre-training runs. The team needed routing that understood live vLLM queue depths and prompt caches without diverting researchers to build a custom scheduler.
Google Cloud recommended deploying GKE Inference Gateway, powered by the open-source llm-d framework, to address routing bottlenecks. The gateway manages traffic across all model families and serving nodes, routing generation, evaluation, and guardrail requests using live serving signals from vLLM instances instead of static rotation.