Quantization-Aware Healing: a compressed, 4-bit model that outperforms its full-precision original

Large language models are typically compressed through structural changes and 4-bit quantization to reduce size and compute costs, but these steps often degrade reasoning, math, and code generation. A recovery step, or healing, is usually added to restore performance before deployment. Recent models like gpt-oss and NVIDIA’s Nemotron family use a compress-then-heal approach, but the effectiveness of healing after both compression and quantization remained unclear until now.

Quantization-Aware Healing (QAH) addresses this gap by distilling directly from the original full-precision model rather than the recovered checkpoint. Applied to a GPT-OSS 120B model compressed to 60B parameters and quantized to MXFP4, QAH produced a 4-bit model that outperformed its own bfloat16 version on 7 of 9 benchmarks, including gains in long-context reasoning and math. The method avoids the instability and high costs of traditional quantization-aware training (QAT), which requires re-running expensive fine-tuning processes.

A comparison between QAH and QAT on a GPT-OSS 9B model showed QAH reached peak performance in about 100 steps—7 times faster than QAT—and maintained stability, while QAT degraded sharply after its peak. The stability advantage stems from QAH’s use of a fixed teacher distribution via KL divergence, preventing drift that can erode model capabilities. This reduces deployment risks, as QAH checkpoints remain reliable without careful early stopping.

The efficiency gains are substantial: the QAH model uses roughly 4 times less weight memory than its bfloat16 counterpart and halves compute per token due to its reduced parameter count. For models shipped in bfloat16, the combined reduction could approach 8 times less compute per token. QAH reframes quantization as an opportunity to teach the model further, enabling smaller, cheaper, and more accurate deployments without sacrificing performance.