Continual Training of Language Models for Few-Shot Learning (2022)

Abstract: Recent work on applying large language models (LMs) achieves impressive performance in many NLP applications. Adapting or posttraining an LM using an unlabeled domain corpus can produce even better performance for end-tasks in the domain. This paper proposes the problem of continually extending an LM by incrementally post-train the LM with a sequence of unlabeled domain corpora to expand its knowledge without forgetting its previous skills. The goal is to improve the few-shot end-task learning in these domains. The resulting system is called CPT (Continual PostTraining), which to our knowledge, is the first continual post-training system. Experimental results verify its effectiveness.

## introduction recent work has shown that large lms have the ability to perform few-shot (or even zero-shot) learning well @xcite @xcite . post-training (a.k.a., domain-adaptive pre-training or pre-finetuning) an lm with a large unlabeled domain corpus before end-task fine-tuning in the domain achieves better results @xcite than directly fine-tuning the lm. this paper goes a step further to study the problem of improving an lm's ability to handle new and ever emerging domains. for this, one needs to continually posttrain the lm with a sequence of domains. a key issue associated with this problem is catastrophic forgetting (cf). 2 this paper thus investigates how to continually extend the lm's knowledge without suffering from cf. from a broader perspective, since training a large lm from scratch is extremely expensive and computation intensive, incrementally updating the lm with the latest language data reflecting the ever changing development of the language itself, social events and the knowledge from different fields is becoming more and more critical. as humans are very effective at incremental learning, if we can imitate this human capability with little or no forgetting, we will be pushing the ai research forward significantly. the proposed system, called cpt, is a continual learning (cl) system for post-training. starting from a pre-trained lm (e.g., roberta @xcite ), it incrementally post-trains the lm with a sequence of domains using their unlabeled corpora. once a task (a domain in our case) foot_2 is trained, its data is no longer accessible. at any time, the resulting continually post-trained lm can be used by end-tasks in the trained domains. this is in the task-incremental learning (til) setting of cl, where the task id (domain id in our case) is provided when the learned model of a task needs to be used later (the use of domain id is discussed in sec. 2.1). 4 this paper proposes an effective approach called cpt and focuses on the challenging and practical scenario of few-shot end-task learning after post-training a sequence of domains. continual post-training is different from conventional cl @xcite . the key difference is that in conventional cl, each task is an end-task, but in our case the end-task involves fine-tuning the continual post-trained lm (called p-lm). this causes major forgetting, which we call the catastrophic butterfly effect (cbe) and does not happen in conventional cl. our proposed system, cpt, can solve both cf and cbe, based on a novel hard masking mechanism (sec. 2.2) and can achieve no forgetting. as shown in sec. 3.3, naively ap-plied existing cl systems cannot effectively prevent cf (even though some existing techniques have shown almost perfect cf prevention ability in conventional cl). experiments in 4 domains and their corresponding end-tasks demonstrate the effectiveness of the proposed cpt system. ## proposed cpt system cpt continually post-trains roberta @xcite . this is achieved by two continual learning plug-in (called cl-plugin) modules inserted into each transformer layer of roberta. clplugin is inspired by adapters in @xcite . while adapters can isolate different tasks, one needs to allocate a new adapter for each task and no knowledge can be shared among different tasks' adapters. the cl-plugin, however, is a cl system that learns a sequence of tasks with adapters shared by all domains. figure 1 gives the cpt architecture with two cl-plugins added to roberta. sequential vs. parallel cl-plugin. instead of following the original sequential adapter (houlsby et al., 2019), cl-plugin adopts the parallel adapter idea in @xcite . the difference is that the former inserts an adapter after the ffn/attention layer while the latter inserts it before ffn/attention layer (see fig. 1 ). we choose the parallel version as it performs better (see sec. 3.3). in post-training, only the two cl-plugins are trained. the components of the original pre-trained roberta are fixed. in end-task fine-tuning, all components are trainable. a cl-plugin is a twolayer fully connected network with a task mask mechanism. it takes two inputs: (1) hidden states h (t) from the feed-forward layer in a transformer layer and (2) task id t needed by task incremental learning (til). inside a cl-plugin, task masks (tms), which indicate task-specific neurons, are used to deal with cf. since tms is differentiable, the whole cpt can be trained end-to-end. ## experiments the proposed paradigm uses a different evaluation from that of conventional continual learning (cl). after unsupervised continual post-training of an lm (roberta in our case) with a sequence of domains, the resulting p-lm is used to fine-tune an end few-shot classification task from any posttrained domain. there is no cl during end-task fine-tuning. each fine-tuning task is done separately. ## conclusion this paper proposed to continually post-train an lm with a sequence of domains using their unlabeled domain corpora. an effective method (cpt) is also proposed. an end-task from any post-trained domain can fine-tune the resulting lm. experimental results demonstrate the effectiveness of cpt.