Dynamic Low-rank Estimation for Transformer-based Language Models (2023)

Abstract: Matrix decomposition methods, such as Singular Value Decomposition (SVD) and its importance-weighted variants, have been widely used for compressing Transformer-based language models. While importance-weighted decomposition methods alleviate the strong assumption of equal importance for each parameter in SVD, they still rely on two fundamental assumptions: 1) unchanged importance distribution during further fine-tuning, 2) equal importance across weight matrices in different layers. Furthermore, these methods necessitate a well-trained task-specific model as the starting point and require additional fine-tuning after compression. In this work, we proposed RankDyna, a matrix decomposition method that enables dynamic rank resource allocation among matrices across different layers during the training process. Starting from a general pre-trained model, RankDyna accomplishes the dual goals of compression and adaptation to the downstream task, all within a single round of fine-tuning. The extensive evaluations demonstrate that RankDyna can outperform current SOTA methods under various parameter budget levels, and the advantage of RankDyna is further enhanced with higher compression rates.

## introduction transformer-based language models @xcite have obtained significant success in a variety of natural language processing tasks. however, these models are usually pre-trained by the huge corpus with millions or even billions of parameters, and hard to be deployed to resource-constrained devices. therefore, the compression of the transformer-based language model have attracted extensive attentions. the transformer blocks are primarily built with linear layers, which will convert the input features into output features through a weight matrix. by applying low-rank estimation methods @xcite , the large weight matrix in a layer can be decomposed into two smaller matrices, resulting in the creation of two reduced-size linear layers. also, it is important to note that matrix factorization technologies are orthogonal to other compression approaches such as knowledge distillation @xcite @xcite or quantization @xcite . the standard svd aims to minimize the reconstruction error, which results in the removal of the portion associated with small singular values. however, previous work have revealed that this objective does not always align with the objective of preserving task performance @xcite . this is primarily due to the fact that parameters have various influences on the overall performance, whereas the standard svd treats all parameters as equally significant. to tackle this issue, previous work fwsvd @xcite and tfwsvd @xcite attempt to conduct low-rank estimation on the matrix weighted by fisher information. these approaches need an already fine-tuned model as their starting point to calculate importance scores based on fisher information, and often require an additional round of fine-tuning to attain better performance. besides, they are based on the assumption that the fisher information of parameters, computed from a fine-tuned task-specific model, can accurately represent the importance distributions of those parameters. additionally, they also assumed that these importance distributions remain unchanged throughout the additional round of fine-tuning. in this paper, we propose a novel method called rankdyna, which conducts low-rank estimation throughout the fine-tuning process. starting with a full-rank matrix, rankdyna gradually reduces the parameter budget level using a scheduler. at each step, the least important singular groups are removed based on their importance scores, which are determined by the amount of impact they contribute to the loss. rankdyna collectively evaluate the importances of all singular groups within the model, and therefore capable of assigning different ranks to each matrix. the advantages of the proposed rankdyna can be summarized as follows: dynamic tracking of importance for singular groups. previous importance-aware decomposition approaches aimed to optimize an objective weighted by static importance assigned to each parameter. however, assuming that the importance distribution of parameters remains unchanged during fine-tuning is unrealistic. unlike these methods, rankdyna can capture the dynamic changes in importance associated with singular groups, thus improving performance compared to the methods based on the static importance assumption. ## methodology in this section, we introduce rankdyna, a solution that enables dynamic resource allocation by monitoring changes in parameter importance during fine-tuning. this approach effectively addresses the above mentioned issues in previous work. as an overview, we outline our proposed rankdyna in algorithm 1. ## conclusion rankdyna provides a comprehensive solution for compressing transformer-based models. through the integration of importance tracking and dynamic allocation of ranks across different layers, rank-dyna addresses the unsolved issues left by the previous importance-aware decomposition methods. extensive results demonstrate that rankdyna outperforms current state-of-the-art (sota) models by a significant margin in terms of performance.