Transformers now runs llama.cpp quants

Hugging Face has added native support for running GGUF-formatted models within the transformers library, allowing users to load quantized checkpoints directly from the Hugging Face Hub and generate text locally using standard transformers APIs. This integration simplifies local inference by eliminating the need for separate tools, enabling developers to work with GGUF models in environments where they already use transformers. The move aligns with the growing trend of running AI models on personal devices, where tools like llama.cpp have made local inference practical for everyday tasks.

GGUF, developed by the llama.cpp team, is a widely adopted format for local inference that supports various quantization levels to balance precision and memory usage. Quantized checkpoints are available from publishers such as Unsloth, LM Studio Community, and bartowski, offering options like Q4_K_M, Q5_K_M, and Q6_K_M. These formats allow users to select models that fit their hardware constraints, with more aggressive quantization enabling larger models to run on limited memory. The Hugging Face Hub provides documentation on available quantization types and loading options.

To load a GGUF model, users specify the Hub repository and filename using the from_pretrained method, with transformers automatically selecting compatible ggml/Metal kernels for Apple Silicon when available. If no compatible kernel is found, the model falls back to standard attention mechanisms, though this may increase memory usage. Benchmarks on a MacBook Pro M2 Max show that transformers achieves token-generation rates close to llama.cpp across multiple GGUF checkpoints, including dense and mixture-of-experts models, demonstrating competitive local inference performance.

The integration bridges transformers and llama.cpp, combining transformers' model definition capabilities with llama.cpp's efficient local inference engine. While llama.cpp remains the recommended choice for pure inference speed, this update enables developers to use GGUF models within transformers for broader compatibility and easier integration into existing workflows. The approach also paves the way for accelerating other architectures and modalities, such as computer vision and audio models, by reusing compatible kernels without requiring full llama.cpp implementations.