Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting (2020)

Abstract: Deep pretrained language models have achieved great success in the way of pretraining first and then fine-tuning. But such a sequential transfer learning paradigm often confronts the catastrophic forgetting problem and leads to sub-optimal performance. To fine-tune with less forgetting, we propose a recall and learn mechanism, which adopts the idea of multi-task learning and jointly learns pretraining tasks and downstream tasks. Specifically, we introduce a Pretraining Simulation mechanism to recall the knowledge from pretraining tasks without data, and an Objective Shifting mechanism to focus the learning on downstream tasks gradually. Experiments show that our method achieves state-of-the-art performance on the GLUE benchmark. Our method also enables BERT-base to achieve better average performance than directly fine-tuning of BERT-large. Further, we provide the open-source RecAdam optimizer, which integrates the proposed mechanisms into Adam optimizer, to facility the NLP community.

## introduction deep pretrained language models (lms), such as elmo @xcite and bert @xcite , have significantly altered the landscape of natural language processing (nlp), and a wide range of nlp tasks has been promoted by these pretrained language models. these successes are mainly achieved through sequential transfer learning @xcite : pretrain a language model on large-scale unlabeled data and then adapt it to downstream tasks. the adaptation step is usually conducted in two manners: fine-tuning or freezing pretrained weights. in practice, fine-tuning is adopted more widely due to its flexibility @xcite @xcite ). instead of learning pretraining tasks and downstream tasks in sequence, multi-task learning learns both of them simultaneously, thus can inherently avoid the catastrophic forgetting problem. @xcite tackled forgetting in automatic speech recognition by jointly training the model with previous and target tasks. @xcite proposed elastic weight consolidation (ewc) to overcome catastrophic forgetting when continuous learning multiple tasks by adopting the multi-task learning paradigm. ewc regularizes new task training by constraining the parameters which are important for previous tasks and adapt more aggressively on other parameters. thanks to the appealing effects on catastrophic forgetting, ewc has been widely applied in various domains, such as game playing @xcite , neural machine translation @xcite and reading comprehension @xcite . however, these multi-task learning methods cannot be directly applied to the sequential transferring regime of deep pretrained lms. firstly, multi-task learning methods require to use the data of pre-training tasks during adaptation, but the pretraining data of lms is often inaccessible or too large for the adaptation. secondly, we only care about the downstream task's performance, while multi-task learning also aims to promote performance on pretraining tasks. in this paper, we propose a recall and learn mechanism to cope with the forgetting problem of fine-tuning the deep pretrained lms. to achieve this, we take advantage of multi-task learning by adopting lms pretraining as an auxiliary learning task during fine-tuning. specifically, we introduce two mechanisms for the two challenges mentioned above, respectively. as for the challenge of data obstacles, we introduce the pretraining simulation to achieve multi-task learning without accessing to pretraining data. it helps the model recall previously learned knowledge by simulating the pretraining objective using only the pretrained parameters. as for the challenge of learning objective difference, we introduce the objective shifting to balance new task learning and pretrained knowledge recalling. it allows the model to focus gradually on the new task by shifting the multi-task learning objective to the new task learning. we also provide recall adam (recadam) optimizer to integrate the proposed recall and learn mechanism into adam optimizer @xcite . we release the source code of the recadam optimizer implemented in pytorch @xcite . it is easy to use and can facilitate the nlp community for better fine-tuning of deep pretrained lms. experiments on the glue benchmark with the bert-base model show that the proposed method can significantly outperform the vanilla fine-tuning method. our method with the bert-base model can even achieve better average results than directly fine-tuning the bertlarge model. in addition, thanks to the effectiveness of pretrained knowledge recalling, we can initialize the model with random parameters and gain better performance with larger parameter search space than the pretrained initialization. finally, we achieve state-of-the-art performance on the glue benchmark with the albert-xxlarge model. our contributions can be summarized as follows: ## background in this section, we present two transfer learning settings: sequential transfer learning and multi-task learning. they both aim to improve the learning performance by transferring knowledge across multiple tasks, but apply to different scenarios. ## methodology in this section, we introduce pretraining simulation ( § 3.1) and objective shifting ( § 3.2) to overcome the two challenges (discussed in § 2.2) respectively. pretraining simulation allows the model to learn source tasks without pretraining data, and objective shifting allows the model to focus on target tasks gradually. we also introduce the recadam optimizer ( § 3.3) to integrate these two mechanisms into the common-used adam optimizer. ## related works catastrophic forgetting has been observed as a great challenge issue in sequential transfer learning, especially in the continuous learning paradigm @xcite @xcite . many methods have been proposed to avoid catastrophic forgetting @xcite @xcite . we focus on regularization-based methods @xcite which recall the previous knowledge with an regularization term, because they don't require the storage of the pretraining data, and are flexible on the new tasks. regularization-based methods can be further divided into data-focused and prior-focused methods. data-focused methods regularize the new task learning by knowledge distillation from the pretrained model @xcite @xcite . prior-focused methods regard the distribution of the pretrained parameters as prior when learning the new task @xcite @xcite . we adopted the idea of prior-focused methods because they enable the model to learn more general knowledge from the pretrained parameters more efficiently. while the prior-focused methods, such as ewc @xcite and its variants @xcite , don't directly access to the pretraining data, they need some pretraining knowledge which is not available in our setting. therefore, we further approximate to a quadratic penalty which is independent with the pretraining data given the pretrained parameters. catastrophic forgetting in nlp has raised increased attention recently @xcite @xcite . many approaches have been proposed to overcome the forgetting problem in various domains, such as machine translation @xcite and reading comprehension @xcite . as sequential transfer learning widely used for nlp tasks @xcite @xcite , previous works explore many finetuning tricks to reduce catastrophic forgetting for adaptation of the deep pretrained lms @xcite @xcite @xcite . in this paper, we bring the idea of multi-task learning which can inherently avoid catastrophic forgetting, and achieve consistent improvement with the proposed recadam optimizer. ## conclusion in this paper, we propose to tackle the catastrophic forgetting in transferring deep pretrained language models by bridging two transfer learning paradigms: sequential fine-tuning and multi-task learning. to cope with the absence of pretraining data during the joint learning of the pretraining task, we introduce a pretraining simulation mechanism to learn the pretraining task without data. then we introduce the objective shifting mechanism to better balance the learning of the pretraining and downstream tasks. experiments demonstrate the superiority of our method in transferring deep pretrained language models, and we provide the open-source recadam optimizer by integrating the proposed mechanisms into adam optimizer to facilitate better usage of deep pretrained language models.