Hopscotch: Discovering and Skipping Redundancies in Language Models (2025)

Abstract: Modern causal language models stack many attention blocks to improve performance, but not all blocks are necessary for every task. We propose Hopscotch, a simple yet effective method that identifies and skips attention blocks with least contributions to a task and adapts to preserve output quality. Hopscotch jointly optimizes which blocks to skip and how to scale the outputs of the remaining layers. By introducing lightweight, trainable scaling parameters to attention and MLP blocks, it mitigates distribution shifts in hidden states caused by removing attention blocks. Hopscotch does not modify model weights or require access to pretraining or instruction-tuning data, and is compatible with existing model compression techniques. When applied to \texttt{Llama-3.1-8B} and \texttt{Qwen-2.5-7B} , Hopscotch achieves less than a 2% drop in performance even after skipping four attention blocks.

## introduction large language models (llms) continue to grow in size, driven by "scaling laws" that suggest larger models tend to yield better performance @xcite @xcite . adding more attention blocks increases model capacity, but self-attention is also the most expensive operation in llms. unlike mlp blocks, attention incurs a quadratic computational cost with respect to sequence length, making it a dominant factor in inference-time efficiency. however, not all attention blocks are equally important for every task, and some may carry redundant information. in this paper, we explore whether entire attention blocks can be skipped without significant performance degradation. we introduce hopscotch foot_0 , a method that jointly learns which attention blocks to skip and how to rescale the outputs of the remaining attention and mlp blocks. hopscotch iteratively identifies attention blocks with minimal contribution to the target task and introduces lightweight, trainable scaling parameters to mitigate distribution shifts in hidden states caused by block removal. hopscotch requires no changes to model weights and no access to pretraining or instruction-tuning data. additionally, it is compatible with fine-grained compression techniques, such as model sparsification @xcite or kv cache quantization @xcite @xcite , and can be combined with them to further reduce llm inference costs. we evaluate hopscotch on instruction-tuned models, including llama-3.1-8b-instruct and qwen2.5-7b-instruct, and find that it can successfully skip up to 7 attention blocks with less than a 3% average performance drop across diverse tasks while yielding up to 15% inference speedup (see section 4.6). with 4 blocks removed, we retain over 98.6% of baseline accuracy. these results highlight structural redundancy in attention blocks. we measure distributional shift in hidden representations and find that hopscotch significantly reduces deviation from the original model, compared to unscaled attention block skipping. finally, we show that hopscotch is compatible with leading quantization techniques, such as gptq @xcite and awq @xcite . a r c _ c h a ll e n g e g p q a s o c ia l_ iq a g s m 8 k _ fe g s m 8 k _ s m p a w s x 30 40 50 60 70 80 90 score ## conclusion we introduce hopscotch, a simple yet effective method for skipping attention blocks in llms. to preserve model outputs, hopscotch learns scaling factors for attention and mlp blocks in remaining layers, compensating for distributional changes in hidden states. we present promising results, achieving near-lossless performance on standard benchmarks. hopscotch is also compatible with existing model compression techniques. we hope this work paves the way for future research on identifying and reducing redundant computations in attention mechanisms to build more efficient foundation models.