S hort GPT : Layers in Large Language Models are More Redundant Than You Expect (2025)

Abstract: As Large Language Models (LLMs) continue to advance, their computational overhead has increased significantly. In this study, we identify notable redundancy across the layers of LLMs, where some layers contribute minimally to the overall network functionality. To quantify this, we introduce a metric called Block Influence (BI), which measures the importance of each layer based on the similarity between its input and output. Based on the observation of layer redundancy, we propose straightforward pruning methods for different tasks: ShortGPT for multiple-choice tasks and ShortGPT-gen for generative tasks. They prune redundant layers based on their BI scores. Our methods demonstrate superior performance over previous pruning methods. The ability to achieve better results through simple layer pruning, as opposed to more complex pruning techniques, suggests a high degree of redundancy across layers. We hope this work will contribute to future research for improving LLM efficiency.

## introduction the field of large language models (llms) has witnessed rapid development recently, with llms achieving impressive performance across various domains. guided by the scaling laws @xcite , modern llms require significant computational resources, creating substantial barriers to their practical use. to mitigate the computational demands of large models, techniques for improving model efficiency have become a critical area. these techniques are generally divided into quantization @xcite @xcite and pruning @xcite @xcite . quantization reduces the precision of model parameters, but its effectiveness often requires specific hardware support. in contrast, pruning reduces the participating networks to decrease the model's computation, offering a more flexible approach. despite its advantages, many existing pruning methods are complex. for example, some require gradient information @xcite , which limits their practicality. in this paper, we focus on layer redundancy in llms and propose methods to improve llm efficiency. we introduce block influence (bi), a metric that quantifies how much hidden states change after passing through each layer, providing a direct measure of layers' importance. leveraging bi, we propose simple yet effective pruning methods for multiple-choice and generative tasks. for multiplechoice tasks, we propose shortgpt, which identifies and removes redundant layers with lower bi scores, significantly reducing model size without sacrificing much performance. for generative tasks, we propose a dynamic pruning method, shortgptgen, where the input tokens skip the same redundant layers as in shortgpt while the generated token pass through all layers to resolve the accumulated errors of shortgpt during generation. to evaluate our methods, we conduct evaluation across comprehensive benchmarks. our experiments reveal that our methods exhibit a smaller performance decrement than previous methods. for instance, shortgpt removes 10 layers (25% of the total 40 layers) from the llama2-13b model, resulting in only a slight drop in performance on the mmlu benchmark @xcite , from 55.0 to 52.2. shortgpt-gen skips 10 layers of llama2-13b, leading to only a minimal decrease in performance on the gsm8k @xcite benchmark without requiring any training, from 28.89 to 26.91. our findings highlight substantial redundancy in current llms and suggest potential avenues for improving the efficiency of models by reducing inherent redundancy in the future. moreover, our methods are orthogonal to quantization methods, meaning it can be combined with quantization techniques to further reduce the computational overhead of llms. our main contributions are as follows: based on bi, we propose layer pruning methods: shortgpt for multiple-choice tasks and shortgpt-gen for generative tasks. ## methodology in this section, we begin by introducing block influence (bi), a novel metric designed to assess the hidden states transformation of each layer. leveraging bi, we then detail our methods for different tasks, i.e. shortgpt for multiple-choice tasks and shortgpt-gen for generative tasks. ## conclusion in this work, we uncovered the significant layerwise redundancy of llms, our research demonstrates that certain layers contribute minimally to 20199 overall network functionality and can be removed without substantially compromising model performance. based on our observation, we introduce block influence to quantify the importance of each layer and propose simple and straightforward pruning methods for multiple-choice tasks and generative tasks. our experiments demonstrates that it is possible to maintain approximately 90% of a llm's performance while reducing the computational requirements in approximately 25% of the llm's layers. our work suggests potential avenues for improving the efficiency of models by reducing their inherent redundancy.