Pruning via Merging: Compressing LLM s via Manifold Alignment Based Layer Merging (2024)

Abstract: While large language models (LLMs) excel in many domains, their complexity and scale challenge deployment in resource-limited environments. Current compression techniques, such as parameter pruning, often fail to effectively utilize the knowledge from pruned parameters. To address these challenges, we propose Manifold-Based Knowledge Alignment and Layer Merging Compression (MKA), a novel approach that uses manifold learning and the Information Bottleneck (IB) measure to merge similar layers, reducing model size while preserving essential performance. We evaluate MKA on multiple benchmark datasets and various LLMs. Our findings show that MKA not only preserves model performance but also achieves substantial compression ratios, outperforming traditional pruning methods. Moreover, when coupled with quantization, MKA delivers even greater compression. Specifically, on the MMLU dataset using the Llama3-8B model, MKA achieves a compression ratio of 43.75% with a minimal performance decrease of only 2.82%. The proposed MKA method offers a resource-efficient and performance-preserving model compression technique for LLMs. We make our code available at https://github.com/SempraETY/Pruning-via-Merging

## introduction large language models (llms), such as @xcite , llama-3 2 , llama-2 @xcite and @xcite , have demonstrated remarkable proficiency in language understanding and generation. these models, with billions of parameters trained on trillions of tokens, can handle complex tasks and exhibit emergent abilities @xcite . while these models have achieved unprecedented success, their growing complexity and scale have brought to the fore significant challenges in terms of computational resources, memory requirements, and energy consumption @xcite , raising concerns about their sustainability. to mitigate these challenges, researchers have developed various model compression techniques in llm to reduce its parameter size while preserving performance @xcite @xcite @xcite . these techniques can be roughly categorized into two main mainstreams @xcite : quantization @xcite @xcite @xcite and pruning @xcite @xcite . quantization based methods aid in the reduction of the memory consumption of weights, activations, and kv caches by using the low-precision values with fewer bits instead of the high-precision values. however, the acceleration benefits of quantization are seriously dependent on hardware support @xcite and sometimes require additional fine-tuning to maintain performance @xcite . compared to quantization, pruning, especially structural pruning @xcite , eliminates redundant llm's parameters to decrease the overall parameter count, and can be applied directly to a trained llm without retraining and is generally more hardware-friendly than quantization approaches. while effective, pruning usually risks losing valuable model structures and determining how to prune the llm with minimal disruption to the origin remains an unsolved problem @xcite . to tackle this issue head-on, we delve into the realm of model merging @xcite , a powerful technique that seamlessly weaves together the strengths and knowledge of multiple models, creating a robust and efficient aggregation. this technique, through averaging the weights of multiple models with the same architecture, can retain essential features without significant additional resources @xcite . furthermore, by offsetting the biases and errors of individual models, model merging often leads to greatly improved performance @xcite . additional, the number of models in the merging process can be gradually and naturally reduced. however, such a useful technology are limited to merging between models currently, and few studies pay attention on merging the same internal structures within a model. this raises the question of whether model compression could be achieved by reducing the total number of layers through the progressive aggregation of knowledge between layers. to answer this question, we introduce manifold-based knowledge alignment and layer merging compression (mka) in this paper. mka combines manifold learning and layer merging to preserve essential information while significantly reducing llm parameter size. as illustrated in figure 1 , our method mainly comprises two primary components: manifold learning for llm knowledge: we employ manifold learning techniques to align knowledge across layers by extracting layer activations from a llm and applying the diffusion kernel algorithm @xcite to learn low-dimensional manifold representations. this approach captures the nonlinear structure in the activation and achieves dimensionality reduction while preserving important activation features, enabling more effective comparison of knowledge patterns across different layers. similarity alignment layer merging: following manifold learning, we use the normalized pairwise information bottleneck (npib) measure @xcite to construct a similarity matrix that quantifies the similarity between layers by maximizing their mutual information while considering the entropy of each layer. based on this similarity matrix, we select the most similar layer pairs for merging. to rigorously validate the effectiveness of mka, we conduct extensive empirical evaluations on a diverse array of benchmark datasets, like mmlu and piqa, and a wide range of state-of-the-art large language models, including llama-3 series with 8b and 70b parameters, llama-2 series with 7b and 13b parameters, and mixtral-7b. our experimental results indicate that mka can maintain good performance while achieving a significant compression ratio, outperforming existing pruning methods and achieving even greater compression when combined with quantization. for example, on the mmlu dataset with llama3-8b, mka can achieve a compression ratio of 43.75% with only a 2.82% performance drop. in summary, the main contributions of this paper are as follows: ## manifold-based knowledge alignment and layer merging our mka method relies on the redundancy present in the latter layers of post-training llms @xcite . by merging layers with high inputoutput similarity from back to front, we maintain the model's performance while reducing its size. in this section, we first describe the extraction and dimensionality reduction processes for the intermediate states, as high-dimensional intermediate states are challenging to analyze. then, we propose our layer merging method based on similarity alignment, which aims to maintain performance by aligning intermediate states through merging techniques. ## experiments we conduct a comprehensive set of experiments to evaluate the effectiveness and generalizability of our mka method across various domains. moreover, we aim to compare our approach with pruning techniques to assess whether it offers improvements and to investigate if it can be combined with quantization methods to achieve even higher compression ratios.