SVD - LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression (2025)

Abstract: Despite significant advancements, the practical deployment of Large Language Models (LLMs) is often hampered by their immense sizes, highlighting the need for effective compression techniques. Singular Value Decomposition (SVD) emerges as a promising method for compressing LLMs. However, existing SVD-based compression approaches suffer from substantial truncation losses, leading to severe performance degradation in compressed models. In this work, we introduce , a novel SVD-based LLM compression method that optimizes singular value truncation in SVD compression with two key strategies. First, employs dynamic compression ratio allocation to effectively balance the extremely large truncation loss across different layers. Second, it implements loss-optimized weight truncation to ensure that the truncated singular values result in a lower and more stable truncation loss in practice. We evaluate on ten datasets and five models on various scales and demonstrated that outperforms current state-of-the-art methods. The source code is available at https://github.com/AIoT-MLSys-Lab/SVD-LLM .

## introduction despite the outstanding performance large language models (llms) exhibit in various tasks @xcite , the significant resources consumed limit their widespread accessibility @xcite @xcite . model compression @xcite is one effective approach to reduce resource consumption. to avoid resource-intensive retraining, llm compression is often conducted in a post-training manner. techniques such as llm quantization @xcite , unstructured pruning @xcite , and structured svd-llm ## svd-llm v2 figure 2 provides an overview of svd-llm v2. specifically, svd-llm v2 groups the weight matrices across all the layers in the original llm by type, such as query (q) and key (k) in attention blocks and gate (g) and up (u ) in mlp blocks. it then computes the theoretical truncation loss of the weight matrices and assigns a unique compression ratio to each weight matrix within each group based on the computed truncation loss. lastly, svd-llm v2 performs loss-optimized weight truncation to obtain the compressed llm. below, we describe the details of the two main components of svd-llm v2: (1) heterogeneous compression ratio allocation and (2) loss-optimized weight truncation. ## experiments and analysis baselines. we compare svd-llm v2 against two groups of methods. (1) three state-of-the-art svdbased llm compression methods: fwsvd @xcite , asvd @xcite , and svd-llm @xcite (section 4.1). (2) other types of llm compression methods. these include three state-of-the-art pruning-based llm compression methods: llm-pruner @xcite , @xcite , and block-pruner @xcite , and two state-of-theart quantization-based llm compression methods: pb-llm @xcite , and billm @xcite (section 4.4). ## limitations while svd-llm v2 outperforms existing svdbased llm compression methods, there is still space for further improvement. for example, under 90% compression ratio, there is a small performance gap compared with state-of-the-art quantization methods. we aim to fill this gap in future. ## acknowledgement this study is supported in part by nsf award nets2312674.