bert2 BERT : Towards Reusable Pretrained Language Models (2022)

Abstract: In recent years, researchers tend to pre-train ever-larger language models to explore the upper limit of deep models. However, large language model pre-training costs intensive computational resources, and most of the models are trained from scratch without reusing the existing pre-trained models, which is wasteful. In this paper, we propose bert2BERT, which can effectively transfer the knowledge of an existing smaller pre-trained model to a large model through parameter initialization and significantly improve the pre-training efficiency of the large model. Specifically, we extend the previous function-preserving method proposed in computer vision on the Transformer-based language model, and further improve it by proposing a novel method, advanced knowledge for large model’s initialization. In addition, a two-stage learning method is proposed to further accelerate the pre-training. We conduct extensive experiments on representative PLMs (e.g., BERT and GPT) and demonstrate that (1) our method can save a significant amount of training cost compared with baselines including learning from scratch, StackBERT and MSLT; (2) our method is generic and applicable to different types of pre-trained models. In particular, bert2BERT saves about 45% and 47% computational cost of pre-training BERT _{\rm BASE} and GPT _{\rm BASE} by reusing the models of almost their half sizes.

## introduction recently, there is a trend of training extremely large models to explore the upper limits of plms. for example, large pre-trained models, including gpt-3 (brown et al., 2020) (175b), panguα @xcite and switch transformers @xcite (1571b), have been proved promising in language understanding and generation. however, these models are all pre-trained from scratch independently without utilizing the knowledge of smaller ones that have already been trained. on the other hand, our empirical studies show that the pre-trained models of different scales could share similar knowledge, for example in figure 2 , the attention patterns of the two plms with different sizes are similar. l2 h1 l2 h2 l2 h4 l2 h5 l12 h4 l12 h5 l12 h10 l12 h11 l2 h8 l2 h2 l2 h4 l2 h5 l12 h3 l12 h8 l12 h5 l12 h7 figure 2 : the comparisons of attention patterns between small and large plms. the upper ones are the attention patterns of bert base model whose architecture is {@xmath0, @xmath1, and the lower ones are the attention patterns of one small bert model whose architecture is {@xmath2, @xmath3. we find that there are a large number of similar attention patterns in the same layer of the two models, indicating the possibility of reusing parameters of trained small plms to speed up the pre-training of large plms. the attention maps of plms with different layers are also similar, which is visualized in previous work @xcite . to demonstrate the superiority of our method, we conduct extensive experiments on two representative plms: bert and gpt, with different source model sizes. the results show that: (1) our method can save a significant amount of computation in pre-training compared to the traditional way of learning from scratch and progressive stacking methods such as stackbert @xcite and mslt @xcite ; (2) our method is model-agnostic, which can be applied on a wide range of transformer-based plms. one typical example is that, when using a small pre-trained model with half the size of bert base for initialization, bert2bert saves 45% computation cost of the original bert base pre-training. in general, our contributions are summarized as follows: (1) we explore a new direction for the efficient pre-training by reusing the trained parameters of small models to initialize the large model; (2) we successfully extend function preserving method @xcite on bert and further propose advanced knowledge initialization, which can effectively transfer the knowledge of the trained small model to the big model and improve the pre-training efficiency; (3) the proposed method outperforms other training methods and achieves 45% computation reduction on bert base ; (4) our method is generic, effective for both the bert and gpt models, and have great potential to become an energy-efficient solution for pre-training super large-scale language models. ## related work efficient pre-training in nlp. the efficiency of pre-training has been explored by previous work. some works @xcite @xcite propose progressive learning to accelerate the pre-training, which are motivated by the fact that different layers have some similar knowledge (e.g., attention patterns). they start pre-training a small model with fewer transformer layers, and then iteratively expand the model by stacking the already trained layers on the top. another line of work proposes to "back distill" the knowledge of the small models into large models, which is termed as knowledge inheritance @xcite . some works focus on the data effi-ciency @xcite and take notes for rare words during the pre-training process to help the model understand them when they occur next. electra @xcite proposes a task of replaced token detection to predict whether each token in the input was replaced or not, which improves the pre-training efficiency. our method is orthogonal to this kind of work and the combination of electra and bert2bert could achieve better efficiency. in addition, there are several other orthogonal techniques for efficient pre-training: mixed-precision training @xcite , large batch optimization @xcite , model architecture innovation @xcite , layer dropping technique @xcite , etc. ## preliminary bert consists of one embedding layer and multiple transformer @xcite layers. ## conclusion and future work this paper proposes an efficient pre-training method, bert2bert, which reuses the parameters of the small trained model as the initialization parameters of the large model. we employ the proposed method in bert and gpt under different settings of model sizes. the extensive results show that bert2bert is generic to transformerbased models and saves a significant amount of computation cost. moreover, the detailed analysis shows that our techniques, function-preserving, advanced knowledge initialization, and two-stage pre-training, are all effective. in the future, we will apply bert2bert on training super largescale language models (e.g., use the 10b source model to train the 100b target model) and extends its scope to other plms such as electra and bart @xcite .