Numerical Optimizations for Weighted Low-rank Estimation on Language Models (2022)

Abstract: Singular value decomposition (SVD) is one of the most popular compression methods that approximate a target matrix with smaller matrices. However, standard SVD treats the parameters within the matrix with equal importance, which is a simple but unrealistic assumption. The parameters of a trained neural network model may affect the task performance unevenly, which suggests non-equal importance among the parameters. Compared to SVD, the decomposition method aware of parameter importance is the more practical choice in real cases. Unlike standard SVD, weighed value decomposition is a non-convex optimization problem that lacks a closed-form solution. We systematically investigated multiple optimization strategies to tackle the problem and examined our method by compressing Transformer-based language models.Further, we designed a metric to predict when the SVD may introduce a significant performance drop, for which our method can be a rescue strategy.The extensive evaluations demonstrate that our method can perform better than current SOTA methods in compressing Transformer-based language models.

## introduction transformer-based language models such as bert @xcite have obtained significant success in a variety of natural language processing tasks, such as language modeling @xcite ), text classification @xcite , question answering @xcite , and summarization @xcite . despite their success, these models usually contain millions or even billions of parameters, pre-trained by the large corpus. however, the downstream tasks may only focus on a specific scenario, such that only a small amount of parameters in the big transformer model will contribute to the performance of the target task. also, the massive size of transformer models prohibits * these authors contributed equally to this work. low-rank factorization @xcite aims to approximate each parameter matrix in the trained model by two smaller matrices. this line of compression strategy will naturally inherit the knowledge of the big trained model without expensive generic retraining, and is the orthogonal direction to other compression approaches such as knowledge distillation @xcite @xcite or quantization @xcite . however, applying standard svd to approximate the learned weights often results in a significant task performance drop. previous work shows that this phenomenon may be caused by a strong assumption held by the standard svd, that the parameters in the matrix are equally crucial to the performance @xcite . also, it has been observed that different parameters in transformer models have different impacts on the overall task performance @xcite . following fwsvd @xcite , we utilize fisher information @xcite to weigh the importance of parameters, so that the objective of matrix factorization will jointly consider matrix reconstruction error and the target task performance. in the standard svd, all the local minima are saddle points, ensuring a closed-form global optimal solution @xcite . this property no longer holds true to our new objective weighted by fisher information. without the closed-form solution, we revert to the numerical optimization methods to minimize the weighted objective. as our method can provide a more accurate solution than fwsvd @xcite , we name our proposed method as tfwsvd (true fisher weighted svd). our results reveal the hybrid optimizer we called adam_sgd can best fit our problem, with its switching point estimated by the row-based analytic solution. we also investigated the scenarios where svd fails, under the guidance of the metric we introduced to measure the variance of parameter importance, with the example of analyzing the matrices within the transformer blocks. in summary, this work makes the following contributions: (1) we provide several optimization methods to search for the best numerical solution for low-rank estimation weighted by the fisher information; (2) we perform extensive evaluations on various language tasks, showing our tfwsvd achieves better performance than the sota compression methods, and can further compress already compact models; (3) through the analysis of factorizing sub-structures inside the transformer blocks, we provide the guide about when svd may fail but tfwsvd can retain the performance. ## conclusion unlike svd, there is no closed-form solution for the weighted low-rank estimation problem, which therefore has to be approximated via numerical optimization methods. we managed to obtain the practical solutions through our hybrid adam_sgd optimizer with the specially designed switching point. our tfwsvd consistently works better than other low-rank factorization methods (fwsvd, tvd, and svd). compared to sota methods that requiring expensive generic re-training, our tfwsvd shows more stable performance on various tasks. also, tfwsvd can efficiently further compress and optimize the already compact models. we also investigate the properties of the targeted matrix, where that svd may fail, and tfwsvd can be the rescuer. we believe our tfwsvd could be the best alternative to svd for language model compression.