Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learning (2023)

Abstract: Real-life multilingual systems should be able to efficiently incorporate new languages as data distributions fed to the system evolve and shift over time. To do this, systems need to handle the issue of catastrophic forgetting, where the model performance drops for languages or tasks seen further in its past. In this paper, we study catastrophic forgetting, as well as methods to minimize this, in a massively multilingual continual learning framework involving up to 51 languages and covering both classification and sequence labeling tasks. We present LR ADJUST, a learning rate scheduling method that is simple, yet effective in preserving new information without strongly overwriting past knowledge. Furthermore, we show that this method is effective across multiple continual learning approaches. Finally, we provide further insights into the dynamics of catastrophic forgetting in this massively multilingual setup.

## introduction standard supervised nlp methods perform well when training on enough data from a uniform distribution. however, they fail to retain knowledge learnt in the past when sudden shifts occur in training data distributions. this effect of dropping performance on data from past distributions is commonly referred to as catastrophic forgetting @xcite @xcite , where stability or preservation of knowledge is traded off for increased plasticity or the ability to acquire new knowledge. to tackle this issue, continual learning (cl) methods were proposed under various settings, such as limited compute or ability to store past data @xcite . the data shifts commonly studied are obtained by training over a sequence of non-iid partitions @xcite , 2018), different tasks @xcite , or by training on various domains such as in task-oriented dialogue @xcite , named entity recognition @xcite , part-of-speech @xcite , and intent detection @xcite . lifelong learning is key to the success of deployed multilingual systems, enabling the system to incorporate annotated data for new languages as they become available without costly retraining and redeployment of the entire system. this sequential availability of data for new languages is a common case of training data shift (see figure 1 for the task setup). yet, the effect of catastrophic forgetting was not yet systematically studied for multi-lingual models with multiple diverse languages. @xcite study continual learning in a crosslingual setting limited to just six languages. the cross-lingual abilities of pre-trained models were found to drop when performing fine-tuning for a target language @xcite , although applying continual learning approaches can effectively reduce the magnitude of the effect @xcite . in this paper, we systematically study the effect of catastrophic forgetting and mitigation strategies in a massively multilingual setting covering up to 51 languages on three different tasks. we start by quantifying the extent to which forgetting happens when languages are presented to the model in sequence, identifying an up to 16% f1 drop compared to training using all the data mixed. next, we propose lr adjust, a simple, yet effective, method to preserve the learned knowledge by adjusting the learning rate over time to alleviate the knowledge overwriting from the new language and preserve the previous learned knowledge. this method is orthogonal to continual learning methods and thus can be handily combined with any of these. we find that across three different cl methods, lr adjust helps further reduce the gap between a fully trained model and the cl setup. we conduct analysis on the aspect of cross-lingual transfer in backward and forward directions to measure the influence of the cl on previous tasks and its ability in zero-shot learning respectively. finally, we conduct analyses on the effects of catastrophic forgetting when first training on multiple languages jointly and when using a curriculum learning approach informed by language similarity. ## conclusion we present the first study of catastrophic forgetting in a massively multilingual setting involving up to 51 languages on named entity recognition and natural language understanding tasks. we investigate continual learning methods and present a learning rate scheduling method that is simple yet effective in reducing the effects of catastrophic forgetting. furthermore, we show that this method is effective across multiple continual learning methods. finally, we provide analysis and further insights into the dynamics of catastrophic forgetting. afshin rahimi, yuan li, and trevor cohn. 2019. massively multilingual transfer for ner. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 151-164. david rolnick, arun ahuja, jonathan schwarz, timothy lillicrap, and gregory wayne. 2019. experience replay for continual learning. advances in neural information processing systems, 32. andrei a rusu, neil c rabinowitz, guillaume desjardins, hubert soyer, james kirkpatrick, koray kavukcuoglu, razvan pascanu, and raia hadsell. 2016. progressive neural networks. arxiv preprint arxiv:1606.04671. tongtong wu, massimo caccia, zhuang li, yuan-fang li, guilin qi, and gholamreza haffari. 2021. pretrained language model in continual learning: a comparative study. in international conference on learning representations.