Adaptive Feature-based Low-Rank Compression of Large Language Models via B ayesian Optimization (2024)

Abstract: In recent years, large language models (LLMs) have driven advances in natural language processing. Still, their growing scale has increased the computational burden, necessitating a balance between efficiency and performance. Low-rank compression, a promising technique, reduces non-essential parameters by decomposing weight matrices into products of two low-rank matrices. Yet, its application in LLMs has not been extensively studied. The key to low-rank compression lies in low-rank factorization and low-rank dimensions allocation. To address the challenges of low-rank compression in LLMs, we conduct empirical research on the low-rank characteristics of large models. We propose a low-rank compression method suitable for LLMs. This approach involves precise estimation of feature distributions through pooled covariance matrices and a Bayesian optimization strategy for allocating low-rank dimensions. Experiments on the LLaMA-2 models demonstrate that our method outperforms existing strong structured pruning and low-rank compression techniques in maintaining model performance at the same compression ratio.

## introduction in recent years, the emergence and application of large language models (llms) have served as a powerful stimulant for natural language processing and artificial intelligence @xcite @xcite . adhering to the scaling law @xcite , researchers are continually seeking llms with more parameters and training data, aiming to achieve general models closer to human capabilities. however, larger language models imply a larger overhead of computing resources. therefore, when deploying llms, it is necessary to strike a balance between efficiency and performance @xcite . to achieve efficient llms, many compression techniques for llms are proposed, such as pruning @xcite @xcite , quantization @xcite @xcite and knowledge distillation @xcite . among these methods, unstructured pruning and quantization can reduce the number of parameters or memory requirements by half or even more without significant performance degradation, but they require specialized gpu kernels to fully realize their acceleration potential. in contrast, structured pruning can produce lightweight models that do not rely on specialized hardware. despite extensive research, the performance of structured pruning still lags significantly behind that of the original model. low-rank compression (lrc) @xcite is another promising compression technique. it decomposes the weight matrix into the product of two dense low-rank matrices, discarding unimportant parameter information during the decomposition process. however, lrc remains under-explored in llms. the keys to lrc are low-rank decomposition methods and low-rank dimension allocation. existing decomposition methods can generally be categorized into two types: weight-based and featurebased decomposition. the former minimizes the reconstruction error of weight matrices by applying truncated svd or weighted svd @xcite @xcite . however, recent research @xcite has discovered that the weights of most transformer-based language models are typical of high rank or even close to full rank; thus, direct decomposition might result in significant error. in contrast, the model's features usually exhibit low-rank characteristics. thus, more work focuses on the feature-based decomposition @xcite @xcite , which aims to minimize the reconstruction error of features. on the other hand, allocating suitable low-rank dimensions to different weight matrices according to the target compression ratio can also reduce the downside on the model's overall performance since they exhibit varying sensitivities to low-rank compression. when lrc is applied to llms, it encounters more new challenges. first, it is challenging for llms to maintain their generality while achieving feature-based low-rank compression. this is because the feature space of llms is extremely high dimensional, making the feature distribution more complex, and the presence of outlier features may interfere with the accurate distribution estimation. thus, we utilize the pooled covariance matrix instead of the sample covariance matrix, which enables a more accurate estimation of feature distributions @xcite . then, for low-rank dimension allocation, manual design struggles to achieve optimal results, and due to its vast search space, grid search requires a considerable amount of time. we conduct empirical studies on the lowrank sensitivity of different types of parameters and observe significant variations among them. based on these findings, we categorize the parameters into groups, allowing each group to share the same lowrank dimensions. this approach effectively narrows down the search space, and furthermore, we utilize sample-efficient bayesian optimization to determine the optimal low-rank allocation. to evaluate the effectiveness of our proposed lrc method, we conduct experiments on two commonly used llama-2 models @xcite . experimental results demonstrate our proposed method can perform better than existing strong structured pruning and lrc methods in llms. when combined with efficient post-training, our method obtains the latest state-of-art for the same settings, maintaining 98% of the model's performance at the 20% compression rate. overall, our main contributions include: ## preliminary in this section, we summarily introduce the foundation of low-rank factorization in model compression, and then empirically show that different layers of the transformers-based generative large language model have different low-rank sensitivities. ## related work a common technique for low-rank factorization is svd, which retains only the top-r largest singular values and their corresponding singular vectors to obtain two rank-r matrices. goldberg (2020) first combine svd with knowledge distillation, applying it to compress bert. directly applying svd decomposition implies an assumption that each parameter in the weight matrix equally affects the model performance. this contradicts many previous research, therefore, fwsvd @xcite and tfwsvd @xcite consider weighting the weight matrix using fisher information. @xcite observe that plms' weights are not inherently low-rank matrices. therefore, directly applying svd will result in significant reconstruction loss. however, they find that the product of data representation and weights is low-rank. hence, they perform a global low-rank decomposition on it. following this observation, @xcite propose the atomic feature mimicking (afm) method to decompose the output features. @xcite also observe the high rank phenomenon of plm weights. they utilize iterative first-order unstructured pruning to reduce the rank of the weight matrix, and then apply fisher information-weighted svd decomposition for low-rank compression. for llms, low-rank compression has not yet received the attention it deserves. lord @xcite applies afm to code llms, demonstrating the potential of low-rank decomposition in compressing llm. recently, @xcite conduct an in-depth study on the weight decomposition of llms and discover that the low-rank components of the weights encapsulate low-frequency information. by meticulously selecting low-rank components, it is possible to eliminate interfering signals and further improve llms' performance. however, their research does not propose a practical low-rank compression algorithm. ## conclusion in this paper, we attempt to unearth the potential of low-rank compression for lightweight universal llms. we thoroughly investigate the challenges of low-rank compression in llms and the low-rank characteristics of features within llms. we propose a bayesian optimization-based feature low-rank compression to address these challenges, incorporating pooled covariance estimation and bayesian optimization for more precise feature distribution estimation and low-rank dimension allocation, respectively. experimental results on the llama 2 model demonstrate that our method significantly outperforms existing structured pruning and other low-rank compression techniques.