Bringing Nunchaku 4-bit Diffusion Inference to Diffusers
Hugging Face integrates Nunchaku 4-bit diffusion inference into Diffusers, enabling faster and more memory-efficient image generation on supported NVIDIA GPUs without requiring custom pipelines or local compilation.
Hugging Face has integrated Nunchaku's 4-bit weight and activation (W4A4) quantization method into Diffusers, allowing users to load and run quantized diffusion models with reduced VRAM usage and improved inference speed. The Nunchaku Lite checkpoints, which pair a 4-bit transformer with a bitsandbytes NF4 text encoder, can generate 1024x1024 images in about 1.7 seconds on an RTX 5090 with peak memory usage of 12 GB, compared to 24 GB for the BF16 pipeline. The integration eliminates the need for separate inference engines or local CUDA compilation, as kernels are downloaded from the Hub via the Nunchaku Lite kernels page during first use.
Nunchaku Lite supports NVIDIA Blackwell GPUs (RTX 50 series, RTX PRO 6000, B200) for NVFP4 checkpoints, while earlier generations can use INT4 variants. The method addresses challenges in quantizing diffusion transformers by moving activation outliers into weights and using a small 16-bit low-rank branch for the hardest parts of each weight matrix, with the remainder quantized to 4 bits. Runtime SVDQ/AWQ linear layers patch stock Diffusers models, delivering around 30% speedup and 50% VRAM reduction compared to BF16 baselines, though without architecture-specific fused kernels, it does not match the original Nunchaku engine's speed.
The diffuse-compressor toolkit enables users to quantize new architectures and publish them as standard Diffusers repositories, supporting end-to-end workflows from calibration to publishing. Quantized models retain the original Diffusers structure, allowing compatibility with schedulers, LoRA loading hooks, offloading, and torch.compile. The toolkit includes a generic scanner to identify target modules for quantization, such as SVDQ W4A4 for transformer blocks and AWQ W4A16 for modulation linears, while leaving other components dense.
Benchmark results on an NVIDIA RTX PRO 6000 (Blackwell) show Nunchaku Lite reducing peak VRAM by up to 50% and improving latency by roughly 30%, with torch.compile further reducing the full pipeline latency to 1.68 seconds—1.8x faster than the BF16 baseline. The integration and toolkit aim to make advanced diffusion models more accessible on consumer GPUs by lowering memory and computational requirements without sacrificing image quality or requiring custom pipelines.