Knowledge Inheritance for Pre-trained Language Models (2022)

Abstract: Recent explorations of large-scale pre-trained language models (PLMs) have revealed the power of PLMs with huge amounts of parameters, setting off a wave of training ever-larger PLMs. However, it requires tremendous computational resources to train a large-scale PLM, which may be practically unaffordable. In addition, existing large-scale PLMs are mainly trained from scratch individually, ignoring that many well-trained PLMs are available. To this end, we explore the question how could existing PLMs benefit training large-scale PLMs in future. Specifically, we introduce a pre-training framework named “knowledge inheritance” (KI) and explore how could knowledge distillation serve as auxiliary supervision during pre-training to efficiently learn larger PLMs. Experimental results demonstrate the superiority of KI in training efficiency. We also conduct empirical analyses to explore the effects of teacher PLMs’ pre-training settings, including model architecture, pre-training data, etc. Finally, we show that KI could be applied to domain adaptation and knowledge transfer.

## introduction recently, it has become a consensus in the nlp community to use pre-trained language models (plms) as the backbone for various downstream tasks @xcite . despite the great follow-up efforts of exploring various pre-training techniques and model architectures, researchers find that simply enlarging the model capacity, data size and training steps can further improve the performance of plms @xcite . this discovery sets off a wave of training large-scale plms @xcite @xcite . although huge plms have shown awesome performance @xcite , it requires tremendous computational resources to train largescale plms @xcite , raising severe environmental concerns on the prohibitive computational costs. moreover, existing plms are generally trained from scratch individually, ignoring that many well-trained plms are available. this leaves us an important question: how could existing plms benefit training larger plms in future? considering that humans can leverage the knowledge summarized by their predecessors to learn new tasks, so that the learning process could become efficient; similarly, it is worth inheriting the implicit knowledge distributed in existing plms. in this sense, we could distill the knowledge summarized by an existing small plm during pretraining to efficiently learn larger plms. we dub the above process as knowledge inheritance (ki). this intuition is similar to reversed kd @xcite in the field of computer vision. they indicate that a delicate student model could still benefit from a teacher with an inferior architecture for a specific downstream task. however, the success of reversed kd in supervised downstream tasks does not guarantee its feasibility under the scenario of large-scale selfsupervised pre-training. therefore, in this paper, we strive to answer the following research questions: (rq1) could distilling knowledge from an existing trained plm benefit large plms' training from scratch? (rq2) considering human beings are able to hand down knowledge from generation to generation, could ki similarly be sequentially performed among a series of plms with growing sizes? (rq3) as more and more plms with different pre-training settings (model architectures, training data, training strategies, etc) emerge, how would different settings affect the performance of ki? (rq4) besides training a large plm from scratch, when adapting an already trained large plm to a new domain, how could smaller domain teachers benefit such a process? in conclusion, the contributions of this paper are summarized as follows: (1) we are the first to formulate the problem of knowledge inheritance, and demonstrate the feasibility of inheriting the knowledge from previously trained plms for efficiently training larger ones; (2) we show that the learned knowledge in plms could accumulate and further be passed down from generation to generation; (3) we systematically conduct empirical analyses to show the effects of various teacher pretraining settings, which may indicate how to select the most appropriate plm as the teacher for ki; (4) we further show that during domain adaptation, an already trained large plm could benefit from multiple small plms of different domains under the ki framework. the above empirical studies indicate that ki can well support cross-model knowledge transfer, providing a promising direction to share the knowledge learned by different plms and continuously promote their performance. another line of methods @xcite @xcite proposes to pre-train larger plms progressively. they first train a small plm, and then gradually increase the depth or width of the network based on parameter recycling (pr). although pr could be used for the goal of ki, these methods typically have strict requirements on the architectures of both models, which is not flexible for practical uses; instead, we resort to kd as the solution for ki without architecture constraints. in addition, different from ki, pr is not applicable for absorbing knowledge from multiple teacher models and domain adaptation. more detailed comparisons between ki and pr are discussed in appendix e. knowledge distillation for plms. knowledge distillation (kd) @xcite aims to compress a large model into a fast-to-execute one. kd has renewed a surge of interest in plms recently. some explore kd at different training phases, e.g., pre-training @xcite , downstream finetuning @xcite , or both of them @xcite ; others explore distilling not only the final logits output by the large plm, but also the intermediate hidden representations @xcite @xcite . conventional kd presumes that teacher models play pivotal roles in mastering knowledge, and student models generally cannot match their teachers in performance. when it comes to the scenario of ki, since student models have larger capacities, the performance of teacher models is no longer an "upper bound" of student models. outside nlp, researchers recently demonstrate that a student model could also benefit from a poor teacher for a specific downstream task @xcite (reversed kd). based on the prior explorations, in this paper, we investigate the application of reversed kd in pre-training. ## knowledge inheritance task formulation. given a textual input @xmath0, . . . , x n } and the corresponding label @xmath1, where k is the number of classes for the specific pre-training task, e.g., the vocabulary size for masked language modeling (mlm) @xcite , a plm m converts each token x @xmath2, ..., z j k ]. z j is then converted to a probability distribution p(x j ; τ ) = [p 1 (x j ; τ ), ..., p k (x j ; τ )] using a softmax function with temperature τ . m is pre-trained with the objective l self (x, y) = h(y, p(x; τ )), where h is the loss function, e.g., cross-entropy for mlm. assume that we have a well-trained small plm m s optimized with the self learning objective l self (such as mlm), our goal is leveraging m s 's knowledge to efficiently train a larger plm m l on the corpora investigated methodology. specifically, imparting m s 's knowledge to m l on d l is implemented by minimizing the kullback-leibler (kl) divergence between two probability distributions output by m s and m l on the same input in addition, m l is also encouraged to conduct self-learning by optimizing l self (x i , y i ). both l self and l ki are balanced with an inheritance rate α: ## empirical analysis in this section, we answer our research questions proposed before. specifically, (1) we first demonstrate the effectiveness of ki in § 4.1. (2) then we show plms can accumulate knowledge over generations in § 4.2. (3) we also investigate the effects of different pre-training settings of the teacher models in § 4.3. (4) finally, we show that ki could benefit domain adaptation, and a trained plm can learn more efficiently with the help of multiple domain teachers in § 4.4. detailed pre-training hyper-parameters are listed in appendix b. ## conclusion and future work in this work, we propose a general knowledge inheritance (ki) framework that leverages previously trained plms for training larger ones. we conduct sufficient empirical studies to demonstrate its feasibility. in addition, we show that ki could well support knowledge transfer over a series of plms with growing sizes. we also comprehensively analyze various pre-training settings of the teacher model that may affect ki's performance, the results shed light on how to choose the most appropriate teacher plm for ki. finally, we ex-tend ki and show that, during domain adaptation, an already trained large plm could benefit from smaller domain teachers. in general, we provide a promising direction to share and exchange the knowledge learned by different models and continuously promote their performance. in future, we aim to explore the following directions: (1) the efficiency of ki, i.e., given limited computational budget and pre-training corpus, how to more efficiently absorb knowledge from teacher models. potential solutions include denoising teacher models' predictions and utilizing more information from the teacher. how to select the most representative data points for ki is also an interesting topic; (2) the effectiveness of ki under different settings, i.e., how can ki be applied if the teachers and the students are pre-trained on different vocabularies, languages, pre-training objectives and modalities. finally, we believe it is vital to use fair benchmarking that can accurately and reliably judge each ki algorithm. thus, we suggest future work to: (1) conduct all experiments under the same computation environment and report the pre-training hyper-parameters and hardware deployments in detail, (2) evaluate the downstream tasks with multiple different random seeds and choose tasks that give relatively stable and consistent results, which could serve as better indicators for plms' effectiveness. in addition, it is also essential that plms are tested on diverse downstream tasks which evaluate plms' different abilities, (3) save the checkpoint more frequently during pre-training and evaluate the downstream performance, which can better indicate the trend of plms' effectiveness, and (4) open-source all the codes and model parameters for future comparisons.