Accelerating vision-language models with LFM2.5-VL-DSpark

Hugging Face has released an experimental DSpark draft model for its vision-language model LFM2.5-VL-3B, designed to accelerate inference by adding a speculative decoding path. The draft model uses the same architecture as text-based DSpark drafters, capturing hidden states at fixed layers to generate candidate tokens. Image patches and text tokens are projected into a shared representation, allowing the drafter to operate on identical dimensionality hidden-state vectors regardless of input type. The inference algorithm remains unchanged from text models, ensuring consistency in behavior.

The vision drafter model is trained using a mixture of vision-language supervised fine-tuning data, weighted toward expected workloads. It features a simplified attention-only architecture with 4 layers and a block size of 9, trained for 10 epochs. Ablation studies across 3, 4, and 5 layers informed the final design. The resulting drafter model has approximately 280 million parameters and increases the deployed model’s parameter count by 8.9%.

The DSpark draft model for LFM2.5-VL-3B supports day-one integration with llama.cpp, MLX-VLM, and SGLang. Performance benchmarks on six vision-based tasks, including general VQA and image captioning, show decoding speedups ranging from 1.57x to 3.13x on edge devices and 2.66x to 20.4x on H100 GPUs. End-to-end latency improvements vary from 1.30x to 2.62x depending on hardware and task.

Speculative decoding accelerates only the decode phase, not vision encoding or prefill, which can limit overall speed gains on edge devices. The draft model is available in Safetensors and GGUF formats on Hugging Face. Users must use compatible builds of llama.cpp, MLX-VLM, or SGLang to enable DSpark support for LFM2.5 targets.