1+1>2: A Synergistic Sparse and Low-Rank Compression Method for Large Language Models (2025)

Abstract: Large Language Models (LLMs) have demonstrated remarkable proficiency in language comprehension and generation; however, their widespread adoption is constrained by substantial bandwidth and computational demands. While pruning and low-rank approximation have each demonstrated promising performance individually, their synergy for LLMs remains underexplored. We introduce Synergistic Sparse and Low-Rank Compression (SSLC) methods for LLMs, which leverages the strengths of both techniques: low-rank approximation compresses the model by retaining its essential structure with minimal information loss, whereas sparse optimization eliminates non-essential weights, preserving those crucial for generalization. Based on theoretical analysis, we first formulate the joint low-rank approximation and sparse optimization as a unified problem and solve it by iterative optimization algorithm. Experiments on LLaMA and Qwen2.5 models (7B-70B) show that SSLC, without any additional training steps, consistently surpasses standalone methods, achieving state-of-the-arts results. Notably, SSLC compresses Qwen2.5 by 50% with no performance drop and achieves at least 1.63 \times speedup, offering a practical solution for efficient LLM deployment.

## introduction in the research field of natural language processing (nlp), large language models (llms) @xcite @xcite , as an emerging technology, have achieved remarkable success in handling complex linguistic tasks and have significantly influenced the evolutionary direction of nlp @xcite @xcite . however, their vast parameters require extensive computational resources and substantial memory bandwidth, thereby constraining their deployment in practical applications. to address the memory consumption issues of llms, various post-training compression (ptc) techniques that do not require retraining have been explored. these include model quantization @xcite @xcite , pruning @xcite @xcite and low-rank approximation @xcite @xcite . pruning simplifies the network by removing non-critical weights or structures, while low-rank approximation methods reduces the model's complexity by decomposing the weight matrix into two orthogonal low-dimensional matrices. recent studies @xcite @xcite @xcite have formulated llm pruning as a layer-wise reconstruction problem and pruned redundant neurons using a metric derived from the second taylor approximation of reconstruction error @xcite . this metric, referred to as weight salience @xcite and detailed in the preliminaries section, evaluates the quadratic error associated with changes in matrix elements, which directly correlates with model performance: higher salience indicate a greater impact on performance. as illustrated in figure 1 (a), the original weight salience, approximated from the calibration dataset that is conventionally employed by prevailing methodologies @xcite , exhibits a discrete distribution of outliers against a consistent pattern of moderate values. unfortunately, existing pruning approaches retain neurons with high salience from a discrete perspective, failing to maximize the extraction of the coherent part in salience space. in contrast, low-rank approximation (lra) methods, such as singular value decomposition (svd) @xcite @xcite , are particularly suitable for compressing the coherent components within the salience and extracting a set of orthogonal bases that form a subspace, maximizing the preservation of the energy of the original space. however, these methods for llms still lead to severe performance degradation at a high compression ratio @xcite . this degradation arises because low-rank approximation effectively preserves the weight-sharing common basis, but fails to retain the full-rank, noncoherent parts that are crucial for maintaining the model's knowledge and performance. given these insights, there is an urgent need to combine sparsification and low-rank approximation techniques. this integration can enhance compression efficiency while ensuring that critical information is preserved. figure 1 demonstrates that the outliers in salience space are effectively extracted after low-rank approximation, and this phenomenon is quantitatively analyzed in section 5.1. consequently, with the same compression rate, the synergistic method, by truncating at a smaller salience threshold and increasing the proportion of neurons with less salience, leads to fewer reconstruction errors and thus less performance degradation. inspired by these experimental observations, we propose the synergistic sparse and low-rank compression (sslc) method. sslc decouples the coherent and non-coherent parts of the neuron, allowing the model to benefit from both sparse and low-rank approximation. the low-rank approximation uses orthogonal bases to maximize the extraction of energy from the salience space, while the sparse part preserves key incoherent neurons to maintain the network's essential expressive power. by synergizing these two techniques, sslc ensures a dense, expressive layer with the low-rank part, mitigating the loss of expressive capacity caused by pure pruning/sparsification. furthermore, we model the joint compression problem as a unified data-aware mathematical optimization objective, considering the effect of low-rank and sparse components on reconstruction loss. then, a synergistic optimization algorithm has been proposed to solve the problem. consequently, our method possesses the orthogonality property of low-rank approximation and the full-rank property of sparsification mathematically, ensuring effective preservation of the model's expressive capacity while reducing redundant information. another advantage, based on the assumption that weight changes during model adaptation exhibit a low "intrinsic rank" @xcite , the low-rank component can effectively adapts to downstream tasks. through comprehensive experiments on the llama @xcite and qwen2.5 @xcite models with 7b to 70b parameters, the results demonstrate that sslc achieves state-of-the-art performance. the main contributions are summarized as follows: in addition, sslc provides an optimized initialization for subsequent low-rank part finetuning. specifically, sslc yields a 1.63× speedup on qwen2.5-7b (within about 3 gpu hours of pruning and fine-tuning) without performance drop across various zero-shot tasks. ## preliminaries current post-training compression methods focus on compressing pre-trained weights without retraining, ensuring model performance by minimizing the output discrepancy between the compressed and original models. due to the computational infeasibility of global minimization, this task is typically framed as a layer-wise reconstruction problem for llms. let @xmath0,n) and w ′ ∈ r (m,n) denote the original and compressed weights of a given layer, where m and n represent the number of output and input channels, respectively. the input activation is represented as @xmath1,n ×l) , where n is the number of calibration samples and l is the sequence length respectively. this problem can be expressed as follows: ## evaluation a comprehensive evaluation of the llama and qwen2.5 model family has been conducted to as-sess the effectiveness of sslc. detailed experimental setups, pre-trained models, datasets, and baselines are provided in appendix b. here, we present the performance analysis of the compressed models, focusing on perplexity and zero-shot capability. additionally, we performed ablation studies to illustrate the impact of key hyperparameters such as rank, iteration count and weight preservation strategy. finally, we evaluated the acceleration potential of our method using the simulated vit-cod @xcite accelerator, as detailed in appendix c. ## conclusion in this paper, we systematically analyze the strengths and weaknesses of two previously independent compression techniques for llms: pruning and low-rank approximation. based on the theoretical analysis, sslc (synergistic sparse and low-rank compression) is introduced for efficient llm deployment, which maximizes the energy in the low-rank component using orthogonal bases, while simultaneously achieving discrete full-rank information in the sparse part. by modeling the joint compression for llms as a unified optimization problem, we apply an iterative optimization algorithm that offers a novel theoretical perspective and achieves significant performance improvements in practice. experiments on language modeling and zero-shot tasks show that our method significantly outperforms previous compression approaches. furthermore, comprehensive fine-tuning experiments demonstrate sslc's effectiveness in restoring model accuracy, validating its practicality for real-world deployment.