Not All LLM Workloads Are Equal: Benchmarking TPU Performance on Classification vs. Generation
Google Cloud benchmarking shows TPU v6e performance varies sharply between classification and generation workloads, with model size and concurrency shaping throughput and latency ceilings.
Enterprise deployments of large language models reveal that infrastructure limits and unit economics are shaped by workload type, not just model size. Google Cloud’s benchmarking of Gemma 3 12B and 27B on TPU v6e demonstrates how decode-heavy generation tasks hit performance ceilings far sooner than prefill-heavy classification tasks, challenging assumptions that larger models always scale better under load.
For generation workloads, the Gemma 3 27B model’s throughput plateaus at 4.12x normalized throughput at 128 concurrent users, while the 12B model scales to 8.19x, indicating memory or compute constraints in the larger model under high concurrency. This suggests organizations should either downsize to the 12B model for generation or enforce strict concurrency limits to avoid saturation.
In classification tasks, both models scale similarly, achieving peak normalized throughputs of approximately 6.04x to 6.37x at 128 users, with latency remaining stable until extreme concurrency levels. This indicates that parameter size matters less for prefill-heavy workloads, allowing deployment of larger models without significant throughput penalties.
Hardware saturation manifests as latency spikes and silent request dropouts, requiring scaling based on end-to-end latency metrics rather than CPU or memory triggers. Google’s evaluation used a GKE Autopilot cluster with a TPU v6e node pool and vLLM serving framework, with standardized configurations to isolate workload-specific performance differences across 16 to 128 concurrent users.