Fine-tuned Language Models are Continual Learners (2022)

Abstract: Recent work on large language models relies on the intuition that most natural language processing tasks can be described via natural language instructions and that models trained on these instructions show strong zero-shot performance on several standard datasets. However, these models even though impressive still perform poorly on a wide range of tasks outside of their respective training and evaluation sets.To address this limitation, we argue that a model should be able to keep extending its knowledge and abilities, without forgetting previous skills. In spite of the limited success of Continual Learning, we show that Fine-tuned Language Models can be continual learners .We empirically investigate the reason for this success and conclude that Continual Learning emerges from self-supervision pre-training. Our resulting model Continual-T0 (CT0) is able to learn 8 new diverse language generation tasks, while still maintaining good performance on previous tasks, spanning in total of 70 datasets. Finally, we show that CT0 is able to combine instructions in ways it was never trained for, demonstrating some level of instruction compositionality.

## introduction recent work has shown that large language models have the ability to perform zero-shot and fewshot learning reasonably well @xcite @xcite . a particularly successful line of work relies on the intuition that most natural language processing tasks can be described via natural language instructions. for example, a summarization task can be reformatted as a response to a natural language input as shown in the picture appeared on the wall of a poundland store on whymark avenue [...] how would you rephrase that in a few words? graffiti artist banksy is believed to be behind [....] however impressive, these models are still limited to simple instructions and mainly natural language understanding (nlu) tasks. these models perform poorly on a wide range of tasks that differ from their respective evaluation sets. to improve their ability on new and diverse tasks, one needs to fine-tune these models again. however, one key problem associated with fine-tuning is catastrophic forgetting @xcite . so, how can we extend these models knowledge and abilities, without suffering from catastrophic forgetting? in this paper, we study continual learning of large language models fine-tuned on natural language instructions and investigate their ability to adapt to diverse tasks, while avoiding catastrophic forgetting on the older tasks. for this purpose, we propose continual-t0 (ct0), a t0 model that uses continual learning with rehearsal @xcite , i.e. using a memory buffer containing a small portion of previous data replayed during training (section 3). we start from t0, a model trained jointly on 50 datasets, resulting in a good zero-shot performance on 12 completely different datasets. we are then able to teach progressively 8 new diverse tasks, while maintaining almost 100% of the initial performance on all the previous datasets. this result is obtained by using only 1% of data for memory buffer. notably, we also maintain the performance for the t0 zero-shot evaluation datasets, even though no rehearsal for those was done, the first of its kind setup for cl (section 4). our final model, continual-t0 (ct0) in addition to performing as well as t0 on all the different t0 datasets, can also understand instructions about the newly introduced tasks focused on language generation problems such as writing a haiku, generating empathetic responses in a dialogue, simplifying text, generating a headline with decoding constraints, generating natural language explanations for natural language inferece (nli) tasks, generating a tweet on a given topic in the style of a given author, or question answering for new domain/concepts such as covid-19. we also conduct an extensive analysis and show that our newly learned instructions can be composed in ways never seen during training, leading to better generalization (section 4.3). given the surprising performance of a simple continual learning strategy, we empirically investigate the reason for this success. why transformer models like t0 are continual learners? is it because of their multi-task nature or the instruction tuning paradigm? or does the large scale parameterization of language models contribute to this success? our experimental analysis show that the easy adaptability and continual learning capabilities actually emerge from pre-training and not the above, including scale ( ## related work continual learning current models are limited in continuously learning without forgetting any previously acquired knowledge and abilities. research in this direction has investigated various strategies such as external memory, constraints and model plasticity @xcite . external memory methods often simply use rehearsal with a replay during training @xcite . through the lens of nlp tasks, biesialska et al. (2020) look at the problem of cl and discuss major challenges involved. @xcite show cl algorithms are effective for knowledge preservation. their study also infer that continual pretraining im-proves temporal generalization. @xcite proposed a a dynamic expansion of special tokens with a transformer architecture. mi et al. (2020) and madotto et al. (2021 ) perform cl for task oriented dialog systems by using replay based strategy. cao et al. (2021) propose a new cl framework for nmt models, while ke et al. (2021) proposes a novel capsule network based model called b-cl (bert based cl) for sentiment classification tasks. jin et al. (2020) show how existing cl algorithms fail at learning compositional phrases. lin et al. (2022) propose a benchmark and highlight key challenges for continual model refinement in outof-distribution data streams. more recently, sun et al. (2019) propose a lifelong learning method lamol that is capable of continually learning new tasks by replaying pseudo-samples of previous tasks that require no extra memory or model capacity. to the best of our knowledge, lamol corresponds to the state-of-the-art for cl in nlp. most similar to our work is that of yin et al. (2022) who also study continual learning from task instructions based on the natural-instruction benchmark @xcite .finally instead of limiting to vision-only and language-only tasks @xcite study the challenge of learning multimodal tasks in a cl setting, and systematically evaluate how upstream continual learning can rapidly generalize to new multimodal and unimodal tasks most of the aforementioned works fall into the 2 scenarios differentiated by @xcite : 1) learning new data of known classes (online learning), and 2) learning new classes (classincremental learning). thus, the study are often limited to a narrow domain, or a specific task. in our work, we propose to address continual learning more broadly: learning a diverse set of new tasks different from the ones used for training. for this, we leverage the idea of instruction tuning @xcite , that enables us to frame any nlp task as a response to a natural language input and use rehearsal as a mechanism to avoid catastrophic forgetting @xcite . ## conclusion we explored for the first time continual learning for instruction-based models. our results indicate that fine-tuned language models are efficient continual learners: 1% rehearsal is enough to maintain a high performance on previously learned tasks, while learning new ones. additionally, we show that our model ct0 is able to comprehend new instructions obtained via instruction composition. the current technique to learn multiple tasks is to train a model from scratch. we hope this work paves the way toward a new paradigm where models do not have to be retrained all over again. we believe our experimental findings will contribute to the effectiveness of large language models, enabling them to progressively adapt to new concepts and acquire more and more abilities. as an analogy with software development, this could be seen as learning new features.