D iff S kip: Differential Layer Skipping in Large Language Models (2025)

Abstract: Existing Large Language Models (LLMs) enforce uniform computation across all tokens. We analyze the correlation between the input-output difference of self-attention block and Feed-Forward Network (FFN) within the same transformer layer, and find that these two differential vectors are highly correlated. Thus, we propose to dynamically skip the FFN blocks based on the self-attention difference and introduce Diffential Layer Skipping (DiffSkip) to show that LLMs are inherently dynamic-depth models, capable of adjusting computational depth when generating different tokens. DiffSkip employs a lightweight router module to dynamically skip a set of FFN blocks in LLMs and only requires efficient fine-tuning while keeping the whole LLM frozen. Experimental results demonstrate that DiffSkip effectively enables dynamic FFN skipping in decoder-only language models, even in continuous token generation tasks where many layer-skipping methods struggle.

## introduction large language models (llms) @xcite @xcite have demonstrated remarkable capabilities across diverse tasks @xcite @xcite . these models operate through a next-token-prediction mechanism, enabling them to tackle complex problems via stepby-step reasoning. however, regardless of the prediction complexity, tokens are processed through the same number of transformer layers. to enable dynamic computation, prior works @xcite introduce a router that makes a binary decision at each layer, determining whether to skip the layer or not. in their implementation, the router is jointly trained with the transformer from scratch. although this approach allows the transformer to learn representations that provide valuable signals for routing, it comes with significant training overhead. this motivates us to explore whether effective routing signals can be directly obtained from a pre-trained model. in this paper, we demonstrate that pre-trained large language models already possess the routing signals for dynamic computation. our key insight is that the self-attention input-output difference, ∆ @xmath0, is correlated with the feed-forward network (ffn) input-output difference, ∆ f f @xmath1. here, x and h represent the inputs to the self-attention block and the ffn block, respectively. as shown in figure 1 , ∆ attn and ∆ f f n exhibit high cosine similarity across diverse tasks, including single-token generation tasks like arc @xcite and hellaswag @xcite , as well as multitoken generation tasks like gsm8k @xcite and bbh @xcite . this correlation enables us to use ∆ attn as a proxy for ∆ f f n . by feeding ∆ attn to a learnable router, we enable the router to predict the potential impact of the ffn transformation without executing it. this allows the router to make informed decisions about whether to skip or execute the subsequent ffn block based on its anticipated contribution. based on this insight, we propose differential layer skipping (diffskip), a method that uses the self-attention input-output difference ∆ attn as a routing signal for dynamic ffn skipping. specifically, diffskip employs two key components in each layer: (1) a router that takes ∆ attn as input to determine whether individual tokens skip or pass through the ffn, and (2) an adaptor that aligns the latent spaces of tokens that skip the ffn with those that undergo ffn computation. importantly, the router and the adaptor are the only components that are tunable, while the rest of the transformer parameters remain frozen. by stacking multiple such layers, diffskip enables tokens to dynamically skip ffn blocks during inference, effectively creating paths of varying depths through the network for each token. to optimize diffskip in an end-to-end manner, we introduce a skipping loss that works in conjunction with the original next-token prediction loss. this loss function encourages feed-forward network (ffn) skipping by penalizing the number of ffn blocks used. it is defined as the l2 loss on the expected number of ffn blocks used to generate each token. additionally, to tailor the skipping strategy to the complexity of the generation task, we incorporate token-wise weighting into the loss. high-perplexity tokens, which indicate more challenging predictions, incur smaller penal-ties to preserve ffn usage, while low-perplexity tokens, associated with simpler predictions, are incentivized to skip more. although we use nexttoken perplexity as a straightforward weighting measure in this work, future research could explore more sophisticated schemes for adaptive allocation. experimental results demonstrate that diffskip effectively enables dynamic ffn skipping in decoder-only language models, even in continuous token generation tasks where many pruning methods struggle. for example, when applying diffskip to llama-3-8b with 8 ffn blocks skipped, it maintains 91.3% of the original performance. to better understand how diffskip dynamically allocates computational depth to different tokens, we visualize the layer usage of each token in figure 2 . diffskip processes answer tokens that require computation through more blocks, such as "180" in "= 180 students" and the first "36" in "= 36 teachers". interestingly, although the final answer "36" is numerically identical to the first "36" in "= 36 teachers", it is assigned fewer blocks since it is simply a copy of the previous result and does not require computation. diffskip also demonstrates that the ffn blocks within the network exhibit different usage patterns across layers. the ffns in the first and last few layers are utilized more frequently than those in the middle layers, with the ffn utilization pattern approximately following a long-tail distribution. this selective activation pattern reveals that pre-trained llms naturally exhibit varying computational requirements across different tokens and ffns. by leveraging these inherent variations, diffskip transforms static llms into dynamic models that allocate computation based on actual needs rather than a fixed architecture. ## methods diffskip is a plug-in method that applies to a pretrained decoder-only language model. it adds a router and an adaptor to enable dynamic ffn skipping. the router takes the self-attention inputoutput difference ∆ attn as input and produces the gating scores @xmath2, g 2 , ..., g t ], where t is the number of tokens. based on the gating scores g and a pre-defined threshold τ , tokens are partitioned into two groups: those routed to the ffn (figure 3 left) and those routed to skip the ffn (figure 3 right). in the following subsections, we describe the implementation details. ## conclusion in this paper, we show that large language models inherently possess a capacity for dynamicdepth processing without modifying their original parameters. we introduced differential layer skipping (diffskip). this method leverages the self-attention input-output difference as a routing signal, enabling a lightweight router mechanism to decide whether each token's hidden state should undergo or skip ffn transformations. consequently, computational depth is allocated dynamically based on token difficulty and show interesting phenomenon on various tasks. we hope these findings encourage further research into exploiting the inherent signals and dynamic-depth properties of large language models to create more adaptive and efficient ai systems.