Distillation in 2026 (so far): which frontier models use it and how sergiopaniego • 5 days ago • 14
Distillation in 2026 is reshaping frontier models by compressing large models into smaller ones, merging specialized RL experts, and enabling self-improvement without additional compute overhead.
The useful question is what changes for users, developers or buyers, and whether the announcement stays industry context or becomes something people can actually use.
Distillation remains a core technique in 2026’s frontier model post-training, with labs adopting three distinct approaches. The traditional method involves training a smaller student model to replicate a larger teacher’s outputs, as seen in Google’s Gemma 3 and Gemma 4 tech reports, where instruction-tuned teachers guide student models through improved distillation techniques. DeepSeek-R1-Distill exemplifies this by using reasoning traces from the R1 model to fine-tune compact Qwen and Llama students, demonstrating both soft-label and hard-label distillation methods to transfer knowledge efficiently.
A newer trend involves training separate RL experts for specialized domains—such as math, coding, or agentic tasks—and then distilling their combined expertise into a single student model. This on-policy distillation allows the student to generate outputs while teachers evaluate each token, enabling faster convergence on desired behaviors. Thinking Machines’ approach highlights this method, matching RL baselines with significantly reduced compute costs by leveraging specialized, domain-specific teachers rather than larger models.
Self-distillation introduces a paradigm where models teach themselves by using a privileged version of themselves as the teacher. Cursor’s Composer 2.5 employs this technique by embedding hints into the context, with the hint-conditioned model acting as the teacher for its unhinted counterpart. This method eliminates the need for hints during inference, as the model internalizes the desired behavior through per-token KL divergence, effectively learning from a better version of itself.
Thinking Machines further explores self-distillation by using a pre-fine-tune checkpoint as the teacher to restore lost behaviors after domain-specific fine-tuning, addressing the challenge of continual learning. This approach mirrors solutions like GLM-5’s cross-stage distillation, ensuring models retain prior knowledge while adapting to new tasks. Across these methods, distillation in 2026 hinges on the teacher-student dynamic, with open-source tools like TRL enabling reproducible implementations for broader adoption.