Knowledge Inheritance for Pre-trained Language Models
Abstract
Recent explorations of large-scale pre-trained language models (PLMs) have revealed the power of PLMs with huge amounts of parameters, setting off a wave of training everlarger PLMs.
However, it requires tremendous computational resources to train a largescale PLM, which may be practically unaffordable.
In addition, existing large-scale PLMs are mainly trained from scratch individually, ignoring that many well-trained PLMs are available.
To this end, we explore the question how could existing PLMs beneﬁt training large-scale PLMs in future.
Speciﬁcally, we introduce a pre-training framework named “knowledge inheritance” (KI) and explore how could knowledge distillation serve as auxiliary supervision during pre-training to efﬁciently learn larger PLMs.
Experimental results demonstrate the superiority of KI in training efﬁciency.
We also conduct empirical analyses to explore the effects of teacher PLMs’ pre-training settings, including model architecture, pre-training data, etc.
Finally, we show that KI could be applied to domain adaptation and knowledge transfer.
The implementation is publicly avail - able at https : github.com/ thunlp Knowledge-Inheritance
Introduction
Recently, it has become a consensus in the NLP community to use pre-trained language models (PLMs) as the backbone for various downstream tasks (Han et al., 2021;
Min et al., 2021).
Despite the great follow-up efforts of exploring various pre-training techniques and model architectures, researchers ﬁnd that simply enlarging the model capacity, data size and training steps can further improve the performance of PLMs (Kaplan et al., 2020; Li et al., 2020b).
This discovery sets off a wave of training large-scale PLMs (Raffel et al., 2019;
Brown et al., 2020; Fedus et al., 2021).
Although huge PLMs have shown awesome performance (Bommasani et al., 2021), it requires tremendous computational resources to train largescale PLMs (Schwartz et al., 2019), raising severe environmental concerns on the prohibitive computational costs.
Moreover, existing PLMs are generally trained from scratch individually, ignoring that many well-trained PLMs are available.
This leaves us an important question: how could existing PLMs beneﬁt training larger PLMs in future?
Considering that humans can leverage the knowledge summarized by their predecessors to learn new tasks, so that the learning process could become efﬁcient; similarly, it is worth inheriting the implicit knowledge distributed in existing PLMs.
In this sense, we could distill the knowledge summarized by an existing small PLM during pretraining to efﬁciently learn larger PLMs.
We dub the above process as knowledge inheritance (KI).
This intuition is similar to reversed KD (Yuan et al., 2020) in the ﬁeld of computer vision.
They indicate that a delicate student model could still beneﬁt from a teacher with an inferior architecture for a speciﬁc downstream task.
However, the success of reversed KD in supervised downstream tasks does not guarantee its feasibility under the scenario of large-scale selfsupervised pre-training.
Therefore, in this paper, we strive to answer the following research questions: ( RQ1 ) could distilling knowledge from an existing trained PLM beneﬁt large PLMs’ training from scratch?
( RQ2 )
Considering human beings are able to hand down knowledge from generation to generation, could KI similarly be sequentially performed among a series of PLMs with growing sizes?
( RQ3 )
As more and more PLMs with different pre-training settings (model architectures, training data, training strategies, etc) emerge, how would different settings affect the performance of
KI?
( RQ4 )
Besides training a large PLM from scratch, when adapting an already trained large
PLM to a new domain, how could smaller domain teachers beneﬁt such a process?
In conclusion, the contributions of this paper are summarized as follows: (1) we are the ﬁrst to formulate the problem of knowledge inheritance, and demonstrate the feasibility of inheriting the knowledge from previously trained PLMs for efﬁciently training larger ones; (2) we show that the learned knowledge in PLMs could accumulate and further be passed down from generation to generation; (3) we systematically conduct empirical analyses to show the effects of various teacher pretraining settings, which may indicate how to select the most appropriate PLM as the teacher for KI; (4) we further show that during domain adaptation, an already trained large PLM could beneﬁt from multiple small PLMs of different domains under the KI framework.
The above empirical studies indicate that KI can well support cross-model knowledge transfer, providing a promising direction to share the knowledge learned by different PLMs and continuously promote their performance.
Related Work
Efﬁcient Pre-training for NLP.
Recently, researchers ﬁnd that the performance of PLMs can be simply improved by increasing the model size, data size and training steps (Liu et al., 2019; Raffel et al., 2019; Kaplan et al., 2020), sparking a wave of training ever-larger PLMs.
For instance, the revolutionary GPT-3 (Brown et al., 2020), which contains 175 billion parameters, shows strong capabilities for language understanding and generation.
This means that utilizing PLMs with huge parameters for downstream tasks may greatly relieve the cost of manual labeling and model training for new tasks.
However, larger models require greater computational demands (Patterson et al., 2021).
To this end, researchers propose to accelerate pre-training by mixed-precision training (Shoeybi et al., 2019), distributed training (Shoeybi et al., 2019), large batch optimization (You et al., 2020), etc.
Another line of methods (Gong et al., 2019; Gu et al., 2021; Chen et al., 2022; Qin et al., 2022) proposes to pre-train larger PLMs progressively.
They ﬁrst train a small PLM, and then gradually increase the depth or width of the network based on parameter recycling (PR).
Although PR could be used for the goal of KI, these methods typically have strict requirements on the architectures of both models, which is not ﬂexible for practical uses; instead, we resort to KD as the solution for KI without architecture constraints.
In addition, different from KI, PR is not applicable for absorbing knowledge from multiple teacher models and domain adaptation.
More detailed comparisons between KI and PR are discussed in appendix E.
Knowledge Distillation for PLMs. Knowledge
Distillation (KD) (Hinton et al., 2015) aims to compress a large model into a fast-to-execute one.
KD has renewed a surge of interest in PLMs recently.
Some explore KD at different training phases, e.g., pre-training (Sanh et al., 2019), downstream ﬁnetuning (Sun et al., 2019; Krishna et al., 2020), or both of them (Jiao et al., 2020); others explore distilling not only the ﬁnal logits output by the large
PLM, but also the intermediate hidden representations (Sanh et al., 2019; Jiao et al., 2020; Sun et al., 2020).
Conventional KD presumes that teacher models play pivotal roles in mastering knowledge, and student models generally cannot match their teachers in performance.
When it comes to the scenario of KI, since student models have larger capacities, the performance of teacher models is no longer an “upper bound” of student models.
Outside NLP, researchers recently demonstrate that a student model could also beneﬁt from a poor teacher for a speciﬁc downstream task (Yuan et al., 2020) (reversed KD).
Based on the prior explorations, in this paper, we investigate the application of reversed KD in pre-training.
Knowledge Inheritance
Task Formulation.
Given a textual input x = { x 1 , . . .
, x n } and the corresponding label y ∈ R K , where K is the number of classes for the speciﬁc pre-training task, e.g., the vocabulary size for masked language modeling (MLM) (Devlin et al., 2019), a PLM M converts
each token x j ∈ x to task-speciﬁc logits z j = [ z j 1 , ..., z jK ] .
z j is then converted to a probability distribution
P ( x j ; τ ) =
[ p 1 ( x j ; τ ) , ..., p K ( x j ; τ )] using a softmax function with temperature τ .
M is pre-trained with the objective L SELF ( x , y )
=
H ( y , P ( x ; τ )) , where H is the loss function, e.g., cross-entropy for
MLM.
Assume that we have a well-trained small
PLM M S optimized with the self learning objective L SELF (such as MLM), our goal is leveraging
M S ’s knowledge to efﬁciently train a larger PLM
M L on the corpora D L = { ( x i , y i ) } |D L | i =1 .
Investigated Methodology
Speciﬁcally, imparting M S ’s knowledge to M L on D L is implemented by minimizing the Kullback-Leibler (KL) divergence between two probability distributions output by M S and M L on the same input x i ∈ D L , i.e., L KI ( x i ; M S ) [Equation]
In addition,
M L is also encouraged to conduct self-learning by optimizing L SELF ( x i , y i ) .
Both L SELF and L KI are balanced with an inheritance rate α :
[Equation]
Since larger models generally converge faster and can achieve better ﬁnal performance (Li et al., 2020b), M L becomes more and more knowledgeable during the learning process, and would surpass the teacher eventually.
Thus, it is necessary to encourage M L increasingly learning knowledge on its own, not only following the teacher’s instructions.
Additionally, after M L has surpassed its teacher, it no longer needs the guidance from
M S and should conduct pure self-learning from then on.
Therefore, different from reversed KD, we dynamically change the inheritance rate α .
Speciﬁcally, for a total training steps of T , we linearly decay α t with a slope of α T T .
The student only inherits knowledge from the teacher for Tα T steps, and then conducts pure self-learning, i.e., α t = max(1 − α T × tT , 0) .
Formally, at step t , the loss function for inheriting knowledge of M S on
D L is formulated as:
[Equation]
Note the logits of M S on D L can be precomputed and saved ofﬂine so that we do not need to re-compute the logits of M S when training M L .
This process is done once and for all.
Empirical Analysis
In this section, we answer our research questions proposed before.
Speciﬁcally, (1) we ﬁrst demonstrate the effectiveness of KI in § 4.1.
Then we show PLMs can accumulate knowledge over generations in § 4.2.
(3) We also investigate the effects of different pre-training settings of the teacher models in § 4.3.
(4) Finally, we show that KI could beneﬁt domain adaptation, and a trained PLM can learn more efﬁciently with the help of multiple domain teachers in § 4.4.
Detailed pre-training hyper-parameters are listed in appendix B.
4. RQ1: How Could Knowledge Inheritance Beneﬁt Large PLMs’ Training? Setting.
Our KI framework is agnostic to the speciﬁc self-supervised pre-training task and the
PLM architecture.
Without loss of generality, we mainly focus on the representative MLM task and use the model architecture of RoBERTa (Liu et al., 2019).
Speciﬁcally, we ﬁrst choose RoBERTa BASE (denoted as BASE ) as the teacher ( M S ) and
RoBERTa LARGE (denoted as LARGE ) as the student ( M L ).
We also experiment on auto-regressive language modeling using GPT (Radford et al., 2018) to show KI is model-agnostic.
For pre-training data, we use the concatenation of Wikipedia and BookCorpus (Zhu et al., 2015) same as BERT (Devlin et al., 2019), with roughly 3 , 400 M tokens in total.
All models ( M S and M L ) are trained for 125 k steps, with a batch size of 2 , 048 and a sequence length of 512 .
Note the whole training computations are comparable with those of BERT.
We pre-train M L by inheriting
M S ’s knowledge under KI (denoted as “ BASE →
LARGE ”).
We compare it with “ LARGE ” that only conducts self-learning from beginning to end.
For performance evaluation, we report the validation perplexity (PPL) during pre-training and the downstream performance on development sets of eight GLUE (Wang et al., 2019) tasks.
Note compared with the self-learning baseline, in KI, the logits output by M L are additionally used to calculate
L KI , we empirically ﬁnd that the additional computations caused by it are almost negligible compared with the cumbersome computations in Transformer blocks.
Therefore, it requires almost the same computational cost between KI and the baseline for [Figure] each step.
Hence, report the performance training step (Li et al., 2020a), while the performance w.r.t.
FLOPs (Schwartz et al., 2019) and wall-clock time (Li et al., 2020b) can be roughly obtained by stretching the ﬁgure horizontally.
Overall Results.
As shown in Figure 1 (a), we conclude that: (1) training M L under KI converges faster than the self-learning baseline , indicating that inheriting the knowledge from an existing teacher is far more efﬁcient than solely learning such knowledge.
That is, to achieve the same level of validation PPL, KI requires fewer computational costs.
Speciﬁcally, under the guidance of M S , whose validation PPL is 4 . 18 , BASE →
LARGE achieves a validation PPL of 3 .
41 at the end of pre-training, compared with baseline ( LARGE ) 3 .
After BASE → LARGE stops learning from the teacher at the 40 k-th step, it improves the validation PPL from 4 .
60 ( LARGE ) to 4 .
28 , which is almost the performance when the baseline LARGE conducts self-learning for 55 k steps, thus saving roughly 27 .
3% computational costs 1 .
The results in Table 1 show that (2) M L trained under KI achieves better performance than the baseline on downstream tasks at each step .
We also found empirically that, under the same setting (e.g., data, hyper-parameters and model architectures), lower validation PPL generally indicates better downstream performance.
Since the performance gain in downstream tasks is consistent with that reﬂected in PPL, we only show the latter for the remaining experiments.
Concerning the energy cost, for the remaining experiments, unless otherwise speciﬁed, we choose MEDIUM ( 9 layers, 576 hidden size) as M S and BASE as M L .
Effects of Inheritance Rate.
We set α t in Eq.
(2) to be linearly decayed (denoted as Linear ) to gradually encourage M L exploring knowledge on its own.
We analyze whether this design is necessary by comparing it with two other strategies: the ﬁrst is to only learn from the teacher at ﬁrst and change to pure self-learning (denoted as
Heviside ) at the 35 k-th step; the second is to use a constant ratio ( 1 : 1 ) between L SELF and L KI throughout the whole training process (denoted as
Constant ).
We can conclude from Figure 1 (b) that: (1) annealing at ﬁrst is necessary .
The validation PPL curve of Linear converges the fastest, while Heviside tends to increase after M L stops learning from the teacher, indicating that, due to the difference between learning from the teacher and self-learning, annealing at ﬁrst is necessary so that the performance won’t decay at the transition point (the 35 k-th step).
(2) Supervision from the teacher is redundant after M L surpasses M S .
Although Constant performs well in the beginning, its PPL gradually becomes even worse than the other two strategies.
This indicates that after
M L has already surpassed M S , it will be encumbered by keeping following guidance from M S .
Saving Storage Space with Top-K Logits.
Loading the teacher M S repeatedly for KI is cumbersome, and an alternative way is to pre-compute and save the predictions of M S ofﬂine once and for all.
We show that using the information of top- K logits (Tan et al., 2019) can reduce the memory footprint without much performance decrease.
Speciﬁcally, we save only top- K probabilities of
P S ( x j ; τ ) followed by re-normalization, instead of the full distribution over all tokens.
For RoBERTa, [Table] the dimension of P S ( x j ; τ ) is decided by its vocabulary size, which is around 50 , 000 .
We thus vary K in { 10 , 50 , 100 , 1000 } to see its effects in
Figure 1 (c), from which we observe that: top-K logits contain the vast majority of information .
Choosing a relatively small K (e.g., 10 ) is already good enough for inheriting knowledge from the teacher without much performance decrease.
Previous work also indicates the relation between KD and label smoothing (Shen et al., 2021), however, we show in appendix F that the improvements of
KI are not because of beneﬁting from optimizing smoothed targets, which impose regularization.
Experiments on GPT.
To demonstrate that KI is model-agnostic, we conduct experiments on auto-regressive language modeling and choose
GPT (Radford et al., 2018) architecture with growing sizes of { 73 M , 124 M , 209 M , 354 M , 773 M , 1 B } parameters in total, respectively.
The detailed architectures are speciﬁed in Table 6.
All the teacher models are pre-trained for 62 .
5 k steps with a batch size of 2 , 048 .
As reﬂected in Figure 2 (a), training larger GPTs under our KI framework converges faster than the self-learning baseline, which demonstrates KI is agnostic to the speciﬁc pretraining objective and PLM architecture .
4. RQ2: Could Knowledge Inheritance be Performed over Generations?
Human beings can inherit the knowledge from their antecedents, reﬁne it and pass it down to their offsprings, so that knowledge can gradually accumulate over generations.
Inspired by this, we investigate whether PLMs also have this kind of pattern.
Speciﬁcally, we experiment with the knowledge inheritance among three generations of RoBERTa with roughly 1 .
7 x growth in model size: G 1 ( BASE , 125 M), G 2 ( BASE_PLUS , 211 M) and G 3 ( LARGE , 355 M), whose architectures are listed in Table 6.
All models are trained from scratch for 125 k steps with a batch size of 2 ,on the same corpus.
We compare the differences among (1) self-learning for each generation (denoted as G 1 , G 2 and G 3 ), (2) KI over two generations (denoted as G 1 → G 2 , G 1 → G 3 and
G 2 → G 3 ), and (3) KI over three generations (denoted as G 1 → G 2 → G 3 ), where G 2 ﬁrst inherits the knowledge from G 1 , reﬁnes it by additional self-exploring and passes its knowledge down to
G 3 .
The results are drawn in Figure 2 (b).
Comparing the performance of G 2 and G 1 → G 2 , Gand G 1 → G 3 , or G 3 and G 2 → G 3 , we can again demonstrate the superiority of KI over self-training as concluded before.
Comparing the performance of G 1 → G 3 and G 1 → G 2 → G 3 , or G 2 → Gand G 1 → G 2 → G 3 , it is observed that the performance of G 3 beneﬁts from the involvements of both G 1 and G 2 , which means knowledge could be accumulated through more generations’ involvements.
4. RQ3: How Could M S ’s Pre-training Setting Affect Knowledge Inheritance?
Existing PLMs are typically trained under quite different settings, and it is unclear how these different settings will affect the performance of KI.
Formally, we have a series of well-trained smaller PLMs
M S =
{M 1 S , ..., M N S S } , each having been optimized on D S
= {D 1 S , ..., D N S S } , respectively.
Considering that the PLMs in M S , consisting of varied model architectures, are pre-trained on different corpora of various sizes and domains with arbitrary strategies, thus the knowledge they master is also manifold.
In addition, M L ’s pre-training data D L may also consist of massive, heterogeneous corpora from multiple sources, i.e., D L = {D 1 L , ..., D N L L } .
Due to the difference between D L and D S , M S [Figure] may be required to transfer its knowledge on instances unseen during its pre-training.
Ideally, we want M S to teach the courses it is skilled in.
Therefore, it is essential to choose the most appropriate teacher for each composition D ∗ L ∈ D L .
To this end, we conduct thorough experiments to analyze the effects of several representative factors: model architecture, pre-training data, M S ’s pre-training step (appendix A.2) and batch size (appendix A.3).
Effects of Model Architecture.
Large PLMs generally converge faster and achieve lower PPL, thus serving as more competent teachers.
We experiment with two widely chosen architecture variations, i.e., depth (number of layers) and width (hidden size), to explore the effects of M S ’s model architectures.
We choose BASE ( 12 layer,hidden size) as M L ’s architecture, and choose the architecture of M S to differ from M L in either depth or width.
Speciﬁcally, for M S , we vary the depth in { 4 , 6 , 8 , 10 } , and the width in { 384 , 480 , 576 , 672 } , respectively, and pre-train
M S under the same setting as M L .
The PPL curve for each teacher model is shown in appendix A.7, from which we observe that deeper / wider teachers with more parameters converge faster and achieve lower PPL.
After that, we pre-train M L under KI leveraging these teacher models.
As shown in Figure 2 (c) and appendix A.4, choosing a deeper / wider teacher accelerates M L ’s convergence , demonstrating the beneﬁts of learning from a more knowledgeable teacher.
Since the performance of
PLMs is weakly related to the model shape but highly related to the model size (Li et al., 2020b), it is always a better strategy to choose the larger teacher if other settings are kept the same.
In experiments, we also ﬁnd empirically that, the optimal duration of learning from the teacher is longer for larger teachers, which means it takes more time to learn from a more knowledgeable teacher.
Effects of Pre-training Data.
In previous experiments, we assume M L is pre-trained on the same corpus as M S , i.e., D L = D S .
However, in realworld scenarios, it may occur that the pre-training corpus used by both M L and M S is mismatched, due to three main factors: (1) data size.
When training larger models, the pre-training corpus is often enlarged to improve downstream performance, i.e., |D S | (cid:28) |D
L | ; (2) data domain.
PLMs are trained on heterogeneous corpora from various sources (e.g., news articles, literary works, etc.), i.e., P D S (cid:54) =
P D L .
The different knowledge contained in each domain may affect PLMs’ generalization in downstream tasks; (3) data privacy.
Even if both size and domain of D S and D L are ensured to be the same, it may be hard to retrieve the pre-training corpus used by M S due to privacy concerns, with an extreme case: D L ∩ D S = ∅ .
The gap between D S and D L may hinder M S ’s successful knowledge transfer.
We thus design experiments to analyze the effects of these factors, with three observations concluded:
• Obs.
1: PLMs can image the big from the small for in-domain data .
To evaluate the effects of data size, we ﬁrst pre-train teacher models on different partitions of the original training corpus under the same setting by randomly sampling { 116 , 18 , 14 , 12 , 11 } of it, resulting in teacher models with ﬁnal validation PPL of { 5 . 43 , 5 . 15 , 5 . 04 , 4 . 98 , 4 . 92 } , respectively.
The ﬁnal validation PPL increases as we shrink the size of M S ’s pre-training corpus, which implies that training with less data weakens the teacher’s ability.
Next, we compare the differences when their knowledge is inherited by M L .
As reﬂected in Figure 3 (a), however, the performance of KI is not substantially undermined until only 116 of the original data is leveraged by the teacher.
This indicates that PLMs can well image the overall data distribution even if it only sees a small part.
Hence,
[Figure]
when training larger PLMs, unless the data size is extensively enlarged, its impact can be ignored.
• Obs.
2: Inheriting on similar domain im-
proves performance .
To evaluate the effects of data domain, we experiment on the cases where D S and D L have domain mismatch.
Speciﬁcally, keeping data size the same, we mix Wikipedia and BookCorpus (WB) used before with computer science (CS) papers from S2ORC (Lo et al., 2020), whose domain is distinct from WB, using different proportions, i.e., WB : CS = { 1 : 2 , 2 : 1 , 3 : 1 , 4 : 1 } , respectively.
We pre-train M S on the constructed corpora, then test the performance when M L inherits these teachers’ knowledge on the WB domain data.
As shown in Figure 3 (b), with the domain of the constructed corpus M S is trained on becoming gradually similar to WB, the beneﬁts from KI become more obvious, which means it is essential that both M
S and M L are trained on similar domain of data, so that M S can successfully impart knowledge to M L by teaching the “right” course.
• Obs.
3: Data privacy is not that important if the same domain is ensured .
To evaluate the effects of data privacy, we experiment in an extreme case where D S and D L have no overlap at all.
To avoid the inﬂuences of size and domain, we randomly split the WB domain training corpus D into two halves ( D A and D B ) and pre-train two teacher models (denoted as MEDIUM A and MEDIUM B ) on them.
After pre-training, both of them achieve almost the same ﬁnal PPL ( 4 . 99 ) on the same validation set.
They are then inherited by the student model BASE on D B (denoted as MEDIUM A → BASE B and MEDIUM B → BASE B ), which is exactly the pre-training corpus of MEDIUM B and has no overlap with that of MEDIUM A .
We also choose M L that conducts pure self-learning on D B as the baseline (denoted as BASE B ).
It is observed from Figure 3 (c) that, there is little difference between the validation PPL curves of MEDIUM A → BASE B and MEDIUM B → BASE B , indicating that whether the pre-training corpus of M S and M L has data overlap or not is not a serious issue as long as they share the same domain.
This is meaningful when organizations aim to share the knowledge of their PLMs without exposing either the pre-training data or the model parameters due to privacy concerns.
4. RQ4: How Could Knowledge Inheritance Beneﬁt Domain Adaptation?
With streaming data of various domains continuously emerging, training domain-speciﬁc PLMs and storing the model parameters for each domain can be prohibitively expensive.
To this end, researchers recently demonstrated the feasibility of adapting PLMs to the target domain through continual pre-training (Gururangan et al., 2020).
In this section, we further extend KI and demonstrate that domain adaptation for PLM can beneﬁt from inheriting knowledge of existing domain experts.
Speciﬁcally, instead of training large PLMs from scratch, which is the setting used before, we focus on adapting BASE WB , which has been well-trained on the WB domain for 125 k steps, to two target domains, i.e., computer science (CS) and biomedical (BIO) papers from S2ORC (Lo et al., 2020).
The proximity (vocabulary overlap) of three domains is listed in appendix D. We assume there exist two domain experts, i.e., MEDIUM CS and MEDIUM BIO . Each model has been trained on CS / BIO domain for 125 k steps.
Note their training computation is far less than BASE WB due to fewer model parameters.
Hence, either MEDIUM CS or MEDIUM BIO is no match for BASE WB in WB domain but has richer knowledge in CS / BIO domain.
For evaluation, we compare both (1) the validation PPL on the target domain and (2) the performance (test F1) on downstream tasks, i.e. ACL-ARC (Jurgens et al., 2018) for CS domain and CHEMPROT (Kringelum et al., 2016) for BIO domain.
Before adaptation,
[Table]
[Table]
CS domain
We observe that SL exhibits severe when is relatively
BASE WB achieves a PPL of 5 .
41 / 4 . 86 and F1 of 68 . 5 / 81 .
6 on CS / BIO domain, while MEDIUM CS achieves 2 .
95 (PPL) and 69 .
4 (F1) on CS domain,
MEDIUM BIO achieves 2 .
55 (PPL) and 83 .
6 (F1) on
BIO domain.
This demonstrates the superiority of two teachers over the student in their own domain despite their smaller model capacity.
We compare two strategies for domain adaptation: (1) only conducting self-learning on the target domain and (2) inheriting knowledge from welltrained domain teachers.
Speciﬁcally, BASE WB is post-trained for additional 4 k steps on either CS or
BIO domain to learn new knowledge.
In addition, considering that in real-world scenarios, it can be hard to retrieve enough pre-training data for a speciﬁc domain, due to some privacy issues.
Hence, we conduct experiments with different sizes of domain corpus.
In Table 2, BASE WB is post-trained on either CS or BIO domain while in Table 3, it is trained on synthetic domain data (BIO : CS = 1 : 1 ) to absorb knowledge from two domains simultaneously (we assume M L is trained with the optimal teacher selection strategy, i.e., each teacher imparts the knowledge on its own domain data).
It can be concluded from Table 2 and Table 3 that: (1) KI is more training-efﬁcient .
Compared with self-learning, inheriting knowledge from do-
main teachers achieves lower ﬁnal PPL and improved performance in domain-speciﬁc downstream tasks, indicating that, for domain adaptation, KI is more training
efﬁcient
so that an already
trained
large
PLM
could absorb more knowledge
from new
domain
with the
same training budget.
(2) KI is more data-efﬁcient .
The PPL gap be-
tween
KI and SL is further enlarged when there is less domain-speciﬁc data available for adaptation, which means KI is more data-efﬁcient especially under the low-resource setting, where domain data is scarce.
In other words, only providing a small portion of domain-speciﬁc data is enough for satisfactory adaptation performance under KI, while self-learning exhibits overﬁtting to some extent.
Large PLMs can simultaneously absorb knowledge from multiple domains and thus become omnipotent .
From Table 3, we observe BASE WB achieves improved performance on both domains after being taught by two teachers simultaneously.
KI shows superiority over self-learning.
However, simultaneous learning overﬁts training data more easily and its performance on either domain is no match for learning only one domain at a time.
Conclusion and Future Work
In this work, we propose a general knowledge inheritance (KI) framework that leverages previously trained PLMs for training larger ones.
We conduct sufﬁcient empirical studies to demonstrate its feasibility.
In addition, we show that KI could well support knowledge transfer over a series of PLMs with growing sizes.
We also comprehensively analyze various pre-training settings of the teacher model that may affect KI’s performance, the results shed light on how to choose the most appropriate teacher PLM for KI.
Finally, we ex- tend KI and show that, during domain adaptation, an already trained large PLM could beneﬁt from smaller domain teachers.
In general, we provide a promising direction to share and exchange the knowledge learned by different models and continuously promote their performance.
In future, we aim to explore the following directions: (1) the efﬁciency of KI, i.e., given limited computational budget and pre-training corpus, how to more efﬁciently absorb knowledge from teacher models. Potential solutions include denoising teacher models’ predictions and utilizing more information from the teacher. How to select the most representative data points for KI is also an interesting topic; (2) the effectiveness of KI under different settings, i.e., how can KI be applied if the teachers and the students are pre-trained on different vocabularies, languages, pre-training objectives and modalities. Finally, we believe it is vital to use fair benchmarking that can accurately and reliably judge each
KI algorithm.
Thus, we suggest future work to:
(1) conduct all experiments under the same computation environment and report the pre-training hyper-parameters and hardware deployments in detail, (2) evaluate the downstream tasks with multiple different random seeds and choose tasks that give relatively stable and consistent results, which could serve as better indicators for PLMs’ effective-
ness
In
addition, it is also essential that PLMs are tested on diverse downstream tasks which evaluate PLMs’ different abilities, (3) save the checkpoint more frequently during pre-training and evaluate the downstream performance, which can better indicate the trend of PLMs’ effectiveness, and (4) open-source all the codes and model parameters for future
comparisons.
Acknowledgments
This work is supported by the National Key R&D
Program of China (No. 2020AAA0106502), Institute Guo Qiang at Tsinghua University, NExT++ project from the National Research Foundation,
Prime Minister’s Ofﬁce, Singapore under its
IRC@Singapore Funding Initiative, and Beijing
Academy of Artiﬁcial Intelligence (BAAI).
This work is also supported by the Pattern Recognition Center, WeChat AI, Tencent Inc. Yujia Qin and Yankai Lin designed the methods and the experiments.
Yujia Qin, Jing Yi and Jiajie Zhang conducted the experiments.
Yujia Qin, Yankai Lin and
Xu Han wrote the paper.
Zhiyuan Liu, Peng Li,
Maosong Sun and Jie Zhou advised the project.
All the authors participated in the discussion.
A Additional Experiments and Analysis A. Effects of Model Size
We experiment on four PLMs with roughly 1 .
7 x growth in model size: M 1 (RoBERTa MEDIUM , 73 .
5 M),
M(RoBERTa BASE , 125 M),
M(RoBERTa BASE_PLUS , 211 M) and
M(RoBERTa LARGE , 355 M), whose architectures are listed in Table 6.
We ﬁrst pre-train a teacher
PLM M i ( M S ) for 125 k steps with a batch size of 2 , 048 under the same setting then train a larger one M i +
( M L ) by inheriting M i ’s knowledge under KI framework (denoted as
M i → M i +1 , i ∈ { 1 , 2 , 3 } ).
We compare
M i → M i +1 with M
i +1 that conducts selflearning from beginning to end.
As shown in
Figure 4, the superiority of KI is observed across all models.
In addition, with the overall model size of M S and M L gradually increasing, the beneﬁts of KI become more evident, reﬂected in the broader absolute gap between the PPL curve of
M i → M i +
and M i +1 when i gradually grows.
This implies that with the advance of computing power in future, training larger PLMs will beneﬁt more and more from our KI framework.
A. Effects of M S ’s Pre-training Steps
Longer pre-training has been demonstrated as an effective way for PLMs to achieve better performance (Liu et al., 2019) and thus become more knowledgeable.
To evaluate the beneﬁts of more pre-training steps for M S , we ﬁrst vary RoBERTa MEDIUM ’s pre-training steps in { 62 .
5 k, 125 k, 250 k, 500 k } , and keep all other settings the same.
After pre-training, these teacher models achieve the ﬁnal validation PPL of { 5 . 25 , 4 . 92 , 4 . 72 , 4 . 51 } , respectively.
Then we compare the performances when RoBERTa BASE learn from these teacher models and visualize the results in Figure 4, from which we can conclude that, inheriting knowledge from teachers with longer pre-training time (steps) helps M L converge faster.
However, such a beneﬁt is less and less obvious as M S ’s pre-training steps increase, which means after enough training computations are invested, the teacher model enters a plateau of convergence in validation PPL, and digging deeper in knowledge becomes even harder.
The bottleneck lies in other factors, e.g., the size and diversity of pre-training data, which hinder M S from becoming more knowledgeable.
We also found empirically that, after being pre-trained for 125 k steps on the corpus with a batch size of 2 , 048 , all the models used in this paper have well converged, and longer pre-training only results in limited performance gain in either PPL or downstream performance.
A. Effects of M L ’s Batch Size
Batch size is highly related to PLM’s training efﬁciency, and previous work (Liu et al., 2019;
Li et al., 2020b; You et al., 2019) found that slow-butaccurate large batch sizes can bring improvements to model training, although the improvements become marginal after increasing the batch size beyond a certain point (around 2 , 048 ).
BERT (Devlin et al., 2019) is pre-trained for 1 , 000 k steps with a batch size of 256 , and the computational cost is equivalent to training for 125 k steps with a batch size of 2 , 048 (Liu et al., 2019), which is the pretraining setting chosen in our main paper.
Choosing RoBERTa MEDIUM as the teacher model and
RoBERTa BASE as the student model, in Figure 4 we compare the validation PPL as we vary the batch size in { 256 , 512 , 1024 , 2 , 048 } , controlling for the number of passes through the pre-training corpus.
We also vary the peak learning rate in { 1 .
× 10 − 4 , 2 . 5 × 10 − 4 , 3 . 8 × 10 − 4 , 5 .
× 10 − 4 } and pretrain for { 1 , 000 k , 500 k , 250 k , 125 k } steps, respectively, when increasing the batch size.
We observe that increasing the batch size results in improved ﬁnal validation PPL, which is aligned with previous ﬁndings (Liu et al., 2019).
When adjusting batch size, KI accelerates the convergence unanimously, and its beneﬁts become more evident when training with a smaller batch size, reﬂected in the absolute improvement in ﬁnal validation PPL.
We hypothesize that this is because learning from the smoothed target probability of KI, containing rich secondary information (Yang et al., 2019) or dark knowledge (Furlanello et al., 2018), makes the pretraining process more stable.
The student PLM is prevented from ﬁtting to unnecessarily strict distributions and can thus learn faster.
A. Additional Experiments of the Effects of Teacher Model M S ’s architecture (width)
We show in Figure 5 the validation PPL of M L when choosing the teacher PLM M S with different hidden sizes ( { 384 , 480 , 576 , 672 } ).
As mentioned in our main paper, choosing a wider teacher
[Figure] [Figure]
model improves the training efﬁciency of the student PLM.
A. Additional Experiments of Knowledge Inheritance for Domain Adaptation
[Table]
Different Number of Post-training Steps.
In the main paper, we adapt RoBERTa BASE_WB to either CS or BIO domain by post-training it for 4 k steps.
We further vary the number of training steps in { 1 k , 2 k , 3 k , 4 k , 5 k } and visualize the validation
PPL in Figure 5.
We also experiment on different sizes of domain corpus, i.e., 3 , 400 M, 200 M, 100 M, 40 M tokens, respectively, as done in the main paper.
We observe that generally the validation PPL on each domain decreases with the training step growing, and the performance of KI is always better than self-learning.
The improvement of KI over self-learning is further enlarged when there is less target domain data available, demonstrating that
KI is more data-efﬁcient and can work well in lowresource settings.
In addition, self-learning exhibits overﬁtting problems when the data size of the target domain is relatively small, which is not observed under our KI framework, which means KI can mitigate overﬁtting under low-resource settings.
Catastrophic Forgetting on the Source Domain.
Table 4 lists the validation PPL on the source domain (WB) after RoBERTa BASE_WB is post-trained on the target domain (CS / BIO) with self-learning (SL) and knowledge inheritance (KI) for 4 k steps.
We show the results w.r.t.
different sizes of domain corpus ( 3 , 400 M, 200 M, 100 M and 40 M tokens).
We observe that after domain adaptation, the validation PPL on the source domain increases, which means PLMs may forget some key knowledge on the source domain when learning new knowledge
[Figure]
in the target domain, i [Figure] e., the catastrophic forgetting problem.
In addition [Figure] we ﬁnd that the problem is more evident for KI than self-learning.
We expect future work to further explore how to mitigate the catastrophic forgetting [Figure]
A.
Detailed Downstream Performances on GLUE Tasks
Figure 6 visualizes the downstream performance of RoBERTa LARGE and RoBERTa BASE →
RoBERTa LARGE on the dev sets of six GLUE tasks at different pre-training steps with an interval of 5 k. It can be observed that the downstream performance of RoBERTa BASE → RoBERTa LARGE rises faster than the baseline, which means it takes fewer pre-training steps for our KI framework to get a high score in downstream tasks.
Aligned with previous ﬁndings (Li et al., 2020b), we found MNLI and SST-2 to be the most stable tasks in GLUE, whose variances are lower.
We also list the average GLUE performance for
RoBERTa BASE → RoBERTa LARGE and the baseline RoBERTa LARGE in Table 5, from which we observe that the baseline at 70 k-th step achieves almost the same GLUE performance as our method at 40 k-th step, which means our framework saves around 42 .
9 % FLOPs, much higher than the reported 27 .
3 % FLOPs saved based on the pretraining PPL metric in the main paper.
In addition, our method achieves almost the same GLUE performance as the baseline at the ﬁnal step ( 125 k) with only 70 k steps, which means our framework saves 44 % FLOPs in total.
Both the perplexity in the pre-training stage and performance in downstream tasks can be chosen as the evaluation metric for measuring the computational cost savings.
However, in this paper, we choose the former because it is more stable and accurate than the latter.
We ﬁnd empirically that some GLUE tasks like CoLA have higher variances than others, which might make the measurement inaccurate.
Besides, when discussing the effects of model architectures in the main paper, we only show the validation PPL of each model during pre-training, we visualize the corresponding downstream performance (MNLI) in Figure 7, from which it can be observed that learning from teacher models with more parameters helps achieve better downstream performance at the same pre-training step.
In general, we observe that, under our setting, the performance gain in downstream tasks is aligned with that reﬂected in validation PPL during pre-training
. A. Teacher Models’ Validation PPL Curves during Pre-training for “Effects of Model Architecture”
Figure 7 visualizes the validation PPL curves for all the teacher models used in the experiments [Figure] [Figure] [Table]
on the effects of model architecture.
The teacher models differ from RoBERTa BASE in either the depth or width.
Speciﬁcally, we vary the depth in { 4 , 6 , 8 , 10 } (denoted as { RoBERTa H_4 ,
RoBERTa H_6 ,
RoBERTa H_8 ,
RoBERTa H_10 } ), and the width in { 384 , 480 , 576 , 672 } (denoted as { RoBERTa D_384 ,
RoBERTa D_480 ,
RoBERTa D_576 , RoBERTa D_672 } ).
Generally,
PLMs with larger model parameters converge faster and achieve better ﬁnal performance.
B Pre-training Hyper-parameters
In Table 6, we list the architectures we used for all models, covering the details for the total number of trainable parameters ( n params ), the total number of layers ( n layers ), the number of units in each bottleneck layer ( d model ), the total number of attention heads ( n heads ), the inner hidden size of FFN layer ( d FFN ) and the learning rate when batch size is set to 2 , 048 (lr).
The training-validation ratio of pretraining data is set to 199 : 1 .
We set the weight decay to 0 .
01 , dropout rate to 0 . 1 , and use linear learning rate decay.
Adam is chosen as the optimizer. The learning rate is warmed up for the ﬁrst 10 % steps.
The hyper-parameters for Adam optimizer is set to 1 × 10 − 6 , 0 . 9 , 0 .
98 for (cid:15), β 1 , β 2 , respectively.
For a fair comparison, all experiments are done in the same computation environment with 8 NVIDIA 32GB V100 GPUs.
Table 7 describes the total number of pre-training steps for each ( M L , M S ) pair chosen in our experiments.
C Fine-tuning Hyper-parameters
Table 8 describes the hyper-parameters for ACLARC, CHEMPROT and GLUE tasks.
The selection of these hyper-parameters closely follows (Liu et al., 2019) and (Gururangan et al., 2020).
D Domain Proximity of WB, CS and BIO
Table 9 lists the domain proximity (vocabulary overlap) of WB, CS and BIO used in this paper.
E Comparison between Knowledge Inheritance and Parameter Recycling
Parameter recycling (i.e., progressive training) ﬁrst trains a small PLM, and then gradually increases the depth or width of the network based on parameter initialization.
It is an orthogonal research direction against our KI, and has many limitations as follows:
Architecture Mismatch
Existing parameter recycling methods (Gong et al., 2019; Gu et al., 2021) require that the architectures of both small PLMs [Table]
[Table] and large PLMs are matched to some extent, however, our KI does not have such a requirement.
For example, Gong et al.
; Gu et al.
(2021) either requires the number of layers, or the hidden size/embedding size of a large PLM to be the integer multiples of that of a small PLM.
Hence, it is not ﬂexible to train larger PLMs with arbitrary architectures, making parameter recycling hard to be implemented practically.
Besides, there are more and more advanced non-trivial Transformer modiﬁcations appearing (we refer to Lin et al.
(2021) for details), e.g., pre-normalization, relative embedding, sparse attention, etc.
It is non-trivial to directly transfer the parameters between two PLMs if they have different inner structures.
Nevertheless, our KI framework will not be inﬂuenced by such architectural mismatches.
Inability for Multi-to-one Knowledge Inheritance.
It is non-trivial to support absorbing knowledge from multiple teacher models by jointly recycling their model parameters.
Instead, it is easy to implement for KI.
As shown in our experiments, we demonstrate that under our framework, large PLMs can simultaneously absorb knowledge from multiple teachers.
Inability of Knowledge Inheritance for Domain
Adaptation.
Parameter recycling is hard to support continual learning, which makes large PLMs absorb knowledge from small ones in a lifelong manner.
In real-world scenarios, numerous PLMs of different architectures are trained locally with HyperParam [Table] -ARC &
[Table] GLUE [Table] [Table]
among three domains (WB, CS, BIO) discussed in this paper.
Following (Gururangan et al., 2020), we create the vocabulary for each domain by considering the top 10 k most frequent words (excluding stopwords).
different data.
These small PLMs can be seen as domain experts, and it is essential that larger PLMs can continuously beneﬁt from these existing PLMs efﬁciently by incorporating their knowledge so that larger PLMs can become omnipotent.
As described before, it is easy to implement for our framework and we have demonstrated the effectiveness.
Model Privacy.
Parameter recycling requires the availability of the parameters of an existing PLM, which may be impractical due to some privacy issues, e.g., GPT-3 only provides API access for prediction instead of the model parameters.
Instead, our KI framework does not presume access to an existing model parameter since the predictions of the small model can be pre-computed and saved ofﬂine.
This superiority will further make it possible for API-based online knowledge transfer.
F Comparing Label Smoothing and Knowledge Inheritance
Previous work shows the relation between label smoothing and knowledge distillation to some extent (Shen et al., 2021).
To demonstrate that the success of our KI is not because of learning from a more smoothed target, we conduct experiments comparing both label smoothing and our KI in Table 10.
Speciﬁcally, for label smoothing, PLMs optimize a smoothed target y Si
= (1 − α ) ∗ y
i + α ∗ (cid:126) 1 / ( K − 1) , where α = 0 denotes learning from scratch with no label smoothing, larger α means a more smoothed target for PLMs to learn
[Table]
from, K denotes the vocabulary size.
Speciﬁcally, we choose α from { 0 . 1 , 0 . 2 , 0 . 3 } .
It can be concluded from the results in Table 10 that adding label smoothing into the pre-training objectives of PLMs leads to far worse performance than the vanilla baseline, which shows that the improvements of our knowledge inheritance framework are non-trivial: larger PLMs are indeed inheriting the “knowledge” from smaller ones, instead of beneﬁting from optimizing a smoothed target, which imposes regularization.