Change Is the Only Constant: Dynamic LLM Slicing based on Layer Redundancy (2024)

Abstract: This paper introduces a novel model compression approach through dynamic layer-specific pruning in Large Language Models (LLMs), enhancing the traditional methodology established by SliceGPT. By transitioning from constant to dynamic slicing, our method leverages the newly proposed Layer Redundancy (LR) score, which assesses how much change each layer changes its input by measuring the cosine similarity of the input to the output of the layer. We use this score to prune parts of individual layers based on redundancy in such a way that the average pruned percentage for all layers is a fixed value. We conducted extensive experiments using models like Llama3-8B and Mistral-7B on multiple datasets, evaluating different slicing bases and percentages to determine optimal configurations that balance efficiency and performance. Our findings show that our dynamic slicing approach not only maintains but, in many cases, enhances model performance compared to the baseline established by constant slicing methods. For instance, in several settings, we see performance improvements of up to 5% over the SliceGPT baseline.Additionally, a perplexity decrease by as much as 7% was observed across multiple benchmarks, validating the effectiveness of our method. The code, model weights, and datasets are open-sourced at - https://github.com/RazvanDu/DynamicSlicing

## introduction large language models (llms), characterized by their massive scale, often consist of billions to trillions of parameters, enabling them to perform a wide range of complex tasks with remarkable proficiency @xcite @xcite . however, the deployment of these models poses significant challenges, primarily due to the extensive computational resources requirements. as the scale of these models grows, so does the urgency to develop more efficient methods for their deployment. this has led to increased interest in model compression techniques that aim to reduce the computational burden without substantially sacrificing performance. techniques such as knowledge distillation, quantization, or pruning variants have emerged as viable solutions @xcite , each offering a different approach to streamlining model architecture and operations @xcite . in this paper, we improve the work on model pruning introduced by @xcite , a pruning technique via a constant slicing percentage of each layer. while this approach reduces computational demands and maintains a level of performance, it does not account for the varying significance of different layers within the network. we propose a more nuanced, dynamic pruning method that adapts the degree of pruning based on the individual characteristics and contributions of each layer. our method aims to optimize both the efficiency and the efficacy of the pruning process by preserving more functionality in critical areas of the model, leading to better performance and less degradation in tasks. more specifically, we develop a new metric, namely layer redundancy (lr) score, to quantify the impact of each layer on the model's overall performance. this evaluation is essential, as it guides the order in which layers are pruned, ensuring that the most influential layers are preserved while less critical layers are removed. our approach involves generating slicing functions tailored to the importance of each layer, allowing for a dynamic and informed pruning strategy. our results from the extensive empirical studies across various datasets and base models show a substantial improvement in model accuracy across all datasets tested, accompanied by a notable reduction in perplexity. in order to thoroughly evaluate the effectiveness of our proposed dynamic slicing pattern, we also analyzed the median accuracy and perplexity across a range of models. the results consistently show the superiority of our method over conventional constant slicing techniques. the main contributions of our paper are: ## related work recent advancements in model compression techniques @xcite have markedly improved the efficiency of deploying llms while striving to retain their performance. the field has seen a variety of approaches including knowledge distillation @xcite , quantization @xcite , pruning @xcite , low-rank adaptation @xcite or hybrid variants @xcite , each designed to address the growing computational and memory requirements of these models. innovative approaches such as llm-pruner @xcite and laco (layer collapse) @xcite offer novel perspectives on model pruning. llm-pruner focuses on structured pruning by identifying and removing dependency groups within the model, aiming to minimize dependency on the original training corpus while preserving linguistic capabilities. similarly, laco presents a layer-wise pruning strategy where subsequent layers collapse into preceding ones, achieving notable size reduction while maintaining good performance. a third approach @xcite explores the potential of simple layer-pruning strategies combined with parameter-efficient finetuning (peft), demonstrating minimal performance loss even when half of the model's layers are removed. among the innovative strategies in llm optimization, @xcite emerges as a significant breakthrough in model compression. developed to address the intensive computational and memory demands of deploying llms, slicegpt employs a unique post-training sparsification technique. although effective in practice, previous research has illustrated that the order in which layers are removed plays a critical role in model performance @xcite . this insight led us to explore variable slicing percentages across different layers, challenging the constant slice for all layers. initial attempts by the creators of slicegpt to implement this through spectral analysis of layers did not yield a reliable method for determining the optimal percentage to be removed from each layer, as spectral analysis only tells part of the story and doesn't correlate with how much can be sliced out of a layer. ## experiments the first step of the process is to evaluate how redundant each one of the layers is using the procedure described above. we have done this for llama3-8b and mistral-7b @xcite using the full validation split of the pg-19 data-set @xcite . this will give us a lr score for each layer that we then need to process into a slicing pattern. figure 1 shows an example of a slicing pattern evaluation process for llama3-8b having a target s p of 30% and a s b of 10%. furthermore, we evaluate llama3-8b and mistral-7b using different slice base (s b ) values in increments of 2%. intuitively as we decrease the base percentage of the layers we have more extreme slicing patterns. in all our experiments we compare with the constant slice as a baseline. we experiment with slice percentages (s p ) of 30%, 35%, 40%, these representing the percentage of the llm that we prune. for evaluations, we use: piqa (bisk et al., 2019), hellaswag (zellers et al., 2019), winogrande (sakaguchi et al., 2019), arc easy (clark et al., 2018), and wikitextv2 (merity et al., 2016) within the library lm-evaluation-harness (gao et al., 2023). for more experimental details, please refer to section a.1 in the appendix. ## results we will first explore how the accuracy is affected by our dynamic slicing. figure 2 ( presented above), and figures 3, 4, 5, 6, 7 (presented in appendix) show the behaviors of llama3-8b and mistral-7b on 30%, 35%, 40% pruned percentage with respect to the accuracy on four data sets and perplexity on wikitextv2. as observed in figure 2, for the llama3-8b model, the accuracy is improved across all of the 5 evaluated datasets, and the perplexity decreases by as much as 1.4 while pruning the exact same amount from the model. we also see huge accuracy improvements from 52% to 57% on winogrande which has a baseline accuracy of 50%. an important finding in all cases is that the perplexity decreases and task accuracy (mostly) increases when we started to decrease the s b (until a certain point) showcasing llms benefit more from dynamic slicing as proposed in our work. to estimate a good s b in our proposed dynamic slicing method, we evaluated the accuracy at the s b point that achieves the minimum perplexity on wikitextv2, thus using it as a calibration data set. the results shown in ## conclusions in conclusion, we propose a novel dynamic slicing strategy that shows considerable improvements in accuracy when compared against a fixed slicing method that prunes the same amount of parameters on average. we also show that layer redundancy is a powerful metric when removing percentages of layers, and also that there is room for improvement, leading to possible future work in the field.