Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators (2021)

Abstract: This paper presents a novel pre-trained language models (PLM) compression approach based on the matrix product operator (short as MPO) from quantum many-body physics. It can decompose an original matrix into central tensors (containing the core information) and auxiliary tensors (with only a small proportion of parameters). With the decomposed MPO structure, we propose a novel fine-tuning strategy by only updating the parameters from the auxiliary tensors, and design an optimization algorithm for MPO-based approximation over stacked network architectures. Our approach can be applied to the original or the compressed PLMs in a general way, which derives a lighter network and significantly reduces the parameters to be fine-tuned. Extensive experiments have demonstrated the effectiveness of the proposed approach in model compression, especially the reduction in fine-tuning parameters (91% reduction on average). The code to reproduce the results of this paper can be found at https://github.com/RUCAIBox/MPOP .

## introduction recently, pre-trained language models (plms) @xcite @xcite have made significant progress in various natural language processing tasks. instead of training a model from scratch, one can fine-tune a plm to solve some specific task through the paradigm of "pre-training and fine-tuning". typically, plms are constructed with stacked transformer layers @xcite , involving a huge number of parameters to be learned. though effective, the large model size makes it impractical for resource-limited devices. therefore, there is an increasing number of studies focused on the parameter reduction or memory reduction of plms (noach and goldberg, 2020), including parameter sharing @xcite , knowledge distillation @xcite , low-rank approximation @xcite and data quantization @xcite . however, these studies mainly apply parameter reduction techniques to plm compression, which may not be intrinsically appropriate for the learning paradigm and architecture of plms. the compressed parameters are highly coupled so that it is difficult to directly manipulate different parts with specific strategies. for example, most plm compression methods need to fine-tune the whole network architecture, although only a small proportion of parameters will significantly change during fine-tuning @xcite . in this paper, we introduce a novel matrix product operator (mpo) technique from quantum manybody physics for compressing plms @xcite . the mpo is an algorithm that factorizes a matrix into a sequential product of local tensors (i.e., a multi way array). here, we call the tensor right in the middle as central tensor and the rest as auxiliary tensors. an important merit of the mpo decomposition is structural in terms of information distribution: the central tensor with most of the parameters encode the core information of the original matrix, while the auxiliary tensors with only a small proportion of parameters play the role of complementing the central tensor. such a property motivates us to investigate whether such an mpo can be applied to derive a better plm compression approach: can we compress the central tensor for parameter reduction and update the auxiliary tensors for lightweight fine-tuning? if this could be achieved, we can derive a lighter network meanwhile reduce the parameters to be fine-tuned. to this end, we propose an mpo-based compression approach for plms, called mpop. it is developed based on the mpo decomposition tech-nique @xcite . we have made two critical technical contributions for compressing plms with mpo. first, we introduce a new fine-tuning strategy that only focuses on the parameters of auxiliary tensors, so the number of fine-tuning parameters can be largely reduced. we present both theoretical analysis and experimental verification for the effectiveness of the proposed fine-tuning strategy. second, we propose a new optimization algorithm, called dimension squeezing, tailored for stacked neural layers. since mainstream plms usually consist of multiple transformer layers, this will produce accumulated reconstruction error by directly applying low-rank approximation with mpo at each layer. the dimension squeezing algorithm is able to gradually perform the dimension truncation in a more stable way so that it can dramatically alleviate the accumulation error in the stacked architecture. to our knowledge, it is the first time that mpo is applied to the plm compression, which is well suited for both the learning paradigm and the architecture of plms. we construct experiments to evaluate the effectiveness of the proposed compression approach for albert, bert, distillbert and mobilebert, respectively, on glue benchmark. extensive experiments have demonstrated the effectiveness of the proposed approach in model compression, especially dramatically reducing the finetuning parameters (91% reduction on average). ## related work we review the related works in three aspects. ## preliminary in this paper, scalars are denoted by lowercase letters (e.g., a), vectors are denoted by boldface lowercase letters (e.g., v), matrices are denoted by boldface capital letters (e.g., m), and high-order (order three or higher) tensors are denoted by boldface euler script letters (e.g., t ). an n-order tensor t i 1 ,i 2 ,...in can be considered as a multidimensional array with n indices {i 1 , i 2 , ..., i n }. ## approach so far, most of pre-trained language models (plm) are developed based on stacked transformer layers @xcite . based on such an architecture, it has become a paradigm to first pre-train plms and then fine-tunes them on task-specific data. the involved parameters of plms can be generally represented in the matrix format. hence, it would be natural to apply mpo-based approximation for compressing the parameter matrices in plms by truncating tensor dimensions. in particular, we propose two major improvements for mpo-based plm compression, which can largely reduce the fine-tuning parameters and effectively improve the optimization of stacked architecture, respectively. ## experiments in this section, we first set up the experiments, and then report the results and analysis. ## conclusion we proposed an mpo-based plm compression method. with mpo decomposition, we were able to reorganize and aggregate information in central tensors effectively. inspired by this, we designed a novel fine-tuning strategy that only needs to finetune the parameters in auxiliary tensors. we also developed a dimension squeezing training algorithm for optimizing low-rank approximation over stacked network architectures. extensive experiments had demonstrated the effectiveness of our approach, especially on the reduction of fine-tuning parameters. we also empirically found that such a fine-tuning way was more robust to generalize on small training datasets. to our knowledge, it is the first time that mpo decomposition had been applied to compress plms. in future work, we will consider exploring more decomposition structures for mpo.