Small Pre-trained Language Models Can be Fine-tuned as Large Models via Over-Parameterization (2023)

Abstract: By scaling the model size, large pre-trained language models (PLMs) have shown remarkable performance in various natural language processing tasks, mostly outperforming small PLMs by a large margin. However, due to the high computational cost, the huge number of parameters also restricts the applicability of large PLMs in real-world systems. In this paper, we focus on scaling up the parameters of PLMs only during fine-tuning, to benefit from the over-parameterization, while without increasing the inference latency . Given a relatively small PLM, we over-parameterize it by employing a matrix product operator, an efficient and almost lossless decomposition method to factorize its contained parameter matrices into a set of higher-dimensional tensors.Considering the efficiency, we further propose both static and dynamic strategies to select the most important parameter matrices for over-parameterization.Extensive experiments have demonstrated that our approach can significantly boost the fine-tuning performance of small PLMs and even help small PLMs outperform 3\times parameterized larger ones.Our code is publicly available at https://github.com/zfgao66/OPF .

## introduction due to the remarkable performance, finetuning large-scale pre-trained language models (plms) @xcite @xcite has become the de facto method in the natural language process (nlp) field. with the help of large-scale pre-trained data and parameters, these large-scale plms are able to process an extensive range of world knowledge @xcite and generalize well on a variety of tasks @xcite . following this direction, more data and more parameters have been widely explored in improving the model capacity of plms in recent years @xcite @xcite , leading to the expansion of plm parameters from millions to billions. despite the effectiveness, the huge model size also limits the use of large plms in both research and industry communities. specifically, the computational cost of pre-training and the efficiency of utilizing large plms are often unaffordable for researchers as well as resource-limited applications. therefore, as a compromise, a number of studies @xcite @xcite focus on pre-training relatively smaller language models (e.g., bert-baseuncased) on the domain-specific or task-specific corpus. however, small plms are not highly over-parameterized as larger models. as a result, their generalization capabilities are relatively weaker @xcite , leading to a suboptimal fine-tuning performance on downstream tasks. in order to narrow the performance gap between small and large plms, this work seeks to overparameterize small plms as large models during fine-tuning, in pursuit of improving their generalization capability. in the essential form, plms are based on the transformer architecture @xcite , and most of their parameters are stored as matrices. according to the matrix @xcite @xcite (e.g., singular value decomposition), each matrix can be factorized as the multiplication of a set of matrices. in this way, the total number of parameters would be enlarged during fine-tuning. while, after convergence, the factorized matrices can be merged to re-organize the parameter matrix of the small plms. such a paradigm effectively harnesses the advantages of over-parameterization, without incurring any additional inference latency for small plms during the fine-tuning process. although it is promising to incorporate the matrix decomposition to over-parameterize small plms, there are two major concerns remaining to be investigated. first, the potential information loss caused by the matrix decomposition strategy should be greatly reduced, since small computation errors might be exponentially accumulated and propagated in the stacked multiple transformer layer of plms. second, small plms are comprised of multiple parameter matrices, although they may not always fulfill pivotal roles in fine-tuning various downstream tasks @xcite . thus, it is costly and unnecessary to over-parameterize all of them during fine-tuning. therefore, there is a need to select the proper matrix decomposition method and leverage it to overparameterize these selected (more important) parameter matrices. to address the above issues, we introduce the technique of matrix product operator (mpo) @xcite as the matrix decomposition strategy. mpo has been widely used in the quantum manybody physics area, as it can efficiently factorize any matrix with arbitrary dimensions into a set of higher-dimensional tensors with arbitrary scales, and the factorized tensors can reconstruct the original matrix in almost lossless condition @xcite . such merits make mpo a potentially good method for over-parameterizing small plms during fine-tuning. based on mpo, we also devise static and dynamic strategies to adaptively select important parameter matrices for over-parameterization. specially, the static strategy estimates the importance of each parameter matrix based on the variation of the loss values after removing it from a fine-tuned model @xcite and then over-parameterizes the top-n important ones. the dynamic strategy computes the variation of gradients within several fine-tuning steps, which is the approximation of the above loss variation @xcite , which can dynamically guide the matrix over-parameterization process during fine-tuning. to this end, in this paper, we propose a general over-parameterization framework, namely opf to improve the fine-tuning performance of small plms. given the pre-trained parameter matrices of a small plm, we first utilize the static or dynamic strategies to select the most important ones and then over-parameterize them by the mpo decomposition. such a framework does not affect the fine-tuning process, hence it is general to various small plms and nlp tasks. we conduct extensive experiments on the glue benchmark @xcite , a widely-used natural language understanding benchmark. experimental results show that our opf can boost the performance of small plms on glue significantly, e.g., improving bert-base by +2.64 in average, improving t5-base by +2.41 in average. besides, our approach also helps small plms outperform 3× parameterized ones, e.g., bert-base+ours (83.68) v.s. bert-large (83.60) in average metrics on glue. ## preliminary tensor. a tensor t i 1 ,i 2 ,...,im can be viewed as an array with m indices, where {i 1 , i 2 , . . . , i m } denotes the dimensions of the m indices, respectively. in this way, a vector (i.e., v) and a matrix (i.e., w) can be regarded as a 1-order tensor and 2-order tensor, respectively. tensor product. suppose {ψ 1 , . . . , ψ p } and {ϕ 1 , . . . , ϕ q } are the orthonormal basis of tensors t (1) and t (2) , respectively. the tensor product can be derived by contraction of t (1) and t (2) , denoted as ⊗. formally, the tensor contraction of tensor decomposition tensor decomposition can be seen as the inverse operation of the tensor product. a widely-used way is the singular value decomposition (svd) algorithm. given a tensor @xmath0, the m times svd operation can decompose this tensor into m local tensors transformer layer 1 transformer layer n … feed forward network 2 ∈ ℝ 3072×768 =0.95 =0.1 unselected structure … mpo decomposition svd () () () () … multi-head attension embedding layer plm (e.g. bert) 1 ∈ ℝ 768×3072 reshape () () () reshape ## approach in this part, we describe our proposed overparameterization framework for improving the finetuning performance of small plms. we first give an overview of our approach, and then introduce the details of matrix decomposition and over-parameterized matrices selection strategies. ## experiments in this section, we first set up the experiments, then report the results and give a detailed analysis. ## conclusion in this paper, we proposed opf, a novel overparameterization framework to scale up the number of parameters for plms just during fine-tuning, for benefiting from more parameters. in our opf, we incorporated the matrix product operator method, which decomposes the parameter matrices in plms into high-order tensors for increasing the parameter number, and also devised the static and dynamic strategies to select the most important parameter matrices for over-parameterization. extensive experiments have demonstrated that our opf approach can boost the performance of small plms significantly, and even help small plms outperform big ones. in future work, we will investigate more efficient and effective tensor decomposition methods for plm over-parameterization. in addition, we will also apply opf to other important backbone models in computer vision and multimodal domains.