Making Knowledge Distillation Cheap Enough to Run at Scale
Hugging Face introduces a cost-efficient method for knowledge distillation in large language models, reducing VRAM requirements by up to 15.6× and enabling training on a single GPU instead of hundreds.
Knowledge distillation compresses large language models into smaller versions by training a student model to replicate a teacher’s outputs, but the process is computationally expensive. Recent open-source models like Kimi-K3 require 3TB of VRAM just to load, making distillation impractical without significant hardware. Companies such as Nvidia and Multiverse Computing have released compressed models, but the distillation step itself remains a major bottleneck due to high memory and compute demands. The new method addresses this by caching the teacher’s top-K logits and using a memory-efficient loss function to avoid storing large intermediate tensors during training.
The default online distillation approach requires both teacher and student models to be loaded simultaneously, producing a full vocabulary distribution for every token. For a model like gpt-oss-120b with a 201,088-token vocabulary and a 32K sequence length, this results in tensors exceeding 50GB per token position, pushing VRAM requirements beyond even high-end GPUs like the H200 or B200. The proposed offline method caches only the top-100 logits per token, eliminating the need to keep the teacher model in memory during training. This reduces peak VRAM usage from 250GB to around 128GB, making long-context distillation feasible on a single GPU.
The fused chunked KL loss further reduces memory by processing data in chunks rather than building a full vocabulary-by-sequence matrix. At 32K tokens, this cuts peak memory from 85.2 GiB to 5.45 GiB—a 15.6× reduction—while at 256K tokens, it uses 11.6 GiB compared to 134.2 GiB for the next-best variant. The method also improves speed, with step time dropping from 57.0 to 12.23 seconds when distilling a GPT-OSS 20B model at 32,768 tokens, and throughput rising from 74.2 to 345.7 TFLOP/s per GPU. These gains enable large-scale distillation campaigns that were previously unaffordable.
The resulting student model, distilled from Llama 3.1 8B Instruct to 3.2B parameters, retains most of the teacher’s accuracy on benchmarks like BoolQ and HellaSwag, with a slight drop of about nine points on MMLU. The research, part of Multiverse Computing’s ongoing work, aims to make distillation practical for iterative experimentation rather than one-off projects. Additional ablations explore how loss functions and sequence packing affect recovery quality, with full technical details available in the published paper.