Enhancing Scalability of Pre-trained Language Models via Efficient Parameter Sharing (2023)

Abstract: In this paper, we propose a highly parameter-efficient approach to scaling pre-trained language models (PLMs) to a deeper model depth. Unlike prior work that shares all parameters or uses extra blocks, we design a more capable parameter-sharing architecture based on matrix product operator (MPO), an efficient tensor decomposition method to factorize the parameter matrix into a set of local tensors. Based on such a decomposition, we share the important local tensor across all layers for reducing the model size and meanwhile keep layer-specific tensors (also using Adapters) for enhancing the adaptation flexibility. To improve the model training, we further propose a stable initialization algorithm tailored for the MPO-based architecture. Extensive experiments have demonstrated the effectiveness of our proposed model in enhancing scalability and achieving higher performance (i.e., with fewer parameters than BERT-base, we successfully scale the model depth by a factor of 4x and even achieve 0.1 points higher than BERT-large for GLUE score). The code to reproduce the results of this paper can be found at https://github.com/RUCAIBox/MPOBERT-code.

## introduction recently, pre-trained language models (plms) have achieved huge success in a variety of nlp tasks by exploring ever larger model architecture @xcite . it has been shown that there potentially exists a scaling law between the model size and model capacity for plms @xcite , attracting many efforts to enhance the performance by scaling model size @xcite . as a straightforward approach, we can directly increase the layer number of networks for improving the model capacity @xcite ; huang et al., 2020). while, a very deep architecture typically corresponds to a significantly large model size, leading to high costs in both computation and storage @xcite . and, it is difficult to deploy deep networks in resource-limited settings, though it usually has a stronger model capacity. therefore, there is an urgent need for developing a parameter-efficient way for scaling the model depth. to reduce the parameters in deep networks, weight sharing has proven to be very useful to design lightweight architectures @xcite . as a representative one by across-layer parameter sharing, albert @xcite keeps only ten percent of the whole parameters of bert while maintaining comparable performance. although the idea of parameter sharing is simple yet (to some extent) effective, it has been found that identical weights across different layers are the main cause of performance degradation @xcite . to address this issue, extra blocks are designed to elevate parameter diversity in each layer @xcite . while they still use the rigid architecture of shared layer weights, having a limited model capacity. besides, it is difficult to optimize very deep models, especially when shared components are involved. although recent studies @xcite propose improved initialization methods, they do not consider the case with parameter sharing, thus likely leading to a suboptimal performance on a parameter-sharing architecture. to address these challenges, in this paper, we propose a highly parameter-efficient approach to scaling plms to a deeper model architecture. as the core contribution, we propose a matrix product operator (mpo) based parameter-sharing architecture for deep transformer networks. via mpo decomposition, a parameter matrix can be decomposed into central tensors (containing the major information) and auxiliary tensors (containing the supplementary information). our approach shares the central tensors of the parameter matrices across all layers for reducing the model size, and meanwhile keeps layer-specific auxiliary tensors for enhancing the adaptation flexibility. in order to train such a deep architecture, we propose an mpobased initialization method by utilizing the mpo decomposition results of albert. further, for the auxiliary tensors of higher layers (more than 24 layers in albert), we propose to set the parameters with scaling coefficients derived from theoretical analysis. we theoretically show it can address the training instability regardless of the model depth. our work provides a novel parameter-sharing way for scaling model depth, which can be generally applied to various transformer-based models @xcite . we conduct extensive experiments to evaluate the performance of the proposed model on the glue benchmark in comparison to plms with varied model sizes (tiny, small and large). experiments results have demonstrated the effectiveness of the proposed model in reducing the model size and achieving competitive performance. with fewer parameters than bert base , we scale the model depth by a factor of 4x and achieve 0.1 points higher than bert large for glue score. parameter-efficient plms. existing efforts to reduce the parameters of plms can be broadly categorized into three major lines: knowledge distillation, model pruning, and parameter sharing. for knowledge distillation-based methods @xcite @xcite , plms were distilled into student networks with much fewer parameters. for pruningbased methods, they tried to remove less important components @xcite or very small weights @xcite . moreover, the parameter-sharing method was further proposed by sharing all parameters @xcite or incorporating specific auxiliary components @xcite . different from these works, we design an mpobased architecture that can reduce the model size and enable adaptation flexibility, by decomposing the original matrix. ## method in this section, we describe the proposed mpobert approach for building deep plms via a highly parameter-efficient architecture. our ap-proach follows the classic weight sharing paradigm while introducing a principled mechanism for sharing informative parameters across layers and also enabling layer-specific weight adaptation. ## experiments in this section, we first set up the experiments and then evaluate the efficiency of mpobert on a variety of tasks with different model settings. ## conclusion we develop mpobert, a parameter-efficient pretrained language model that allows for the efficient scaling of deep models without the need for additional parameters or computational resources. we achieve this by introducing an mpo-based transformer layer and sharing the central tensors across layers. during training, we propose initialization methods for the central and auxiliary tensors, which are based on theoretical analysis to address training stability issues. the effectiveness of mpobert is demonstrated through various experiments, such as supervised, multitasking, and few-shot where it consistently outperforms other competing models.