Pruning LLMs Like a Physicist: Block Removal as an Ising Optimization Problem

Transformer block removal can speed up large language models by shortening their architecture, but selecting which blocks to delete is complex because the impact of each block depends on others removed simultaneously. This interdependence makes the problem combinatorial, akin to spin systems in physics, where interactions between elements determine overall behavior. The new approach models block selection as a constrained binary optimization problem, mapping it directly to an Ising glass—a disordered spin system with all-to-all interactions and a fixed number of removals.

The method leverages a second-order Taylor expansion of the model’s loss to derive a Hessian matrix, where diagonal entries represent individual block importance and off-diagonal entries capture pairwise interactions. This reformulation allows the energy of the spin system to serve as a strong proxy for downstream benchmark performance, enabling efficient ranking of candidate configurations without full model evaluation. The energy minimization problem is then solved using classical, quantum, or quantum-inspired solvers, depending on problem size.

In practical terms, the Hessian is computed once from forward and backward passes on a small calibration dataset, making subsequent energy evaluations computationally inexpensive. This allows brute-force enumeration of billions of configurations on a single GPU for smaller problems, while larger cases are handled by optimized solvers like tabu search or quantum annealing. The approach scales effectively, even for models where exhaustive search is infeasible, by prioritizing low-energy states over absolute ground states.

The framework also provides a spectrum of high-quality pruning candidates by exploring low-energy excited states, not just the optimal solution. This flexibility aligns with practitioner needs, as the energy proxy is strong but not perfect, and different configurations may yield better performance in practice. The method demonstrates significant gains, such as nearly 23 percentage points improvement on MMLU at 50% compression for Llama-3.3-70B-Instruct compared to existing block-removal techniques.