Your Experts Have Partners. There's HOPE for Keeping Them Together

Mixture-of-Experts (MoE) models, such as Qwen3.5-122B, use hundreds of experts per layer but only activate a small fraction per token, leading to significant memory waste. For instance, 95% of the model's 250 GB memory footprint is consumed by idle expert MLPs, despite only 8 experts being used per layer per token. Expert pruning permanently removes the least useful experts to reduce memory usage without requiring retraining or introducing latency from dynamic loading.

Existing pruning methods score experts independently, ignoring their cooperative behavior, which can degrade performance, especially at high pruning rates. Research shows experts form stable cooperative groups that are consistently co-activated, meaning their interactions are critical to model performance. First-order methods, which rank experts based solely on individual scores, fail to account for these dependencies, leading to suboptimal pruning decisions.

HOPE (Higher-Order Pruning of Experts) addresses this by considering pairwise expert interactions, providing a second-order approach to pruning. At a 50% pruning rate, HOPE achieves the best average rank among pruning methods and delivers up to a 6.1% improvement on agentic coding tasks compared to the next-best method. The method minimizes pruning error by modeling expert interactions through a quadratic program, ensuring more accurate and efficient expert removal.

The paper details HOPE's methodology, compares its performance against existing pruning techniques, and provides guidance on implementing it for custom MoE models. The approach is designed to be practical, composable with quantization, and scalable, making it a viable solution for reducing the memory footprint of large MoE models without sacrificing performance.