Few-Shot Regularization to Tackle Catastrophic Forgetting in Multilingual Machine Translation (2022)

Abstract: Increasing the number of tasks supported by a machine learning model without forgetting previously learned tasks is the goal of any lifelong learning system. In this work, we study how to mitigate the effects of the catastrophic forgetting problem to sequentially train a multilingual neural machine translation model using minimal past information. First, we describe the catastrophic forgetting phenomenon as a function of the number of tasks learned (language pairs) and the ratios of past data used during the learning of the new task. Next, we explore the importance of applying oversampling strategies for scenarios where only minimal amounts of past data are available. Finally, we derive a new loss function that minimizes the forgetting of previously learned tasks by actively re-weighting past samples and penalizing weights that deviate too much from the original model. Our work suggests that by using minimal amounts of past data and a simple regularization function, we can significantly mitigate the effects of the catastrophic forgetting phenomenon without increasing the computational costs.

## introduction the catastrophic forgetting is the phenomenon whereby a neural network forgets previously learned information after learning new one @xcite . given the ubiquity nature of machine learning models in our lives, tackling the catastrophic forgetting phenomenon is a problem of particular interest for the industry as machine learning models tend to lose performance over time due to the changing nature of our world. to counteract this problem, researchers and engineers must periodically re-train these models. however, despite the inefficiency of re-training a large model from scratch and the carbon footprint that this practice entails in the long run, previous training data is not always available due to privacy issues, licensing, data losses, or simply, because the training data is not available. this problem is incredibly challenging since any learning system with a limited amount of memory will, at some point, have to forget past information in order to keep learning new information @xcite . fortunately, we can develop mechanisms so that our machine learning models can selectively forget as little information as possible by penalizing changes in weights that deviate too much from a reference model @xcite , designing dynamic architectures that grow linearly with the number of tasks @xcite , or using complementary learning systems (cls) that, inspired by how the human brain work, generate synthetic data to control the forgetting @xcite . from a practical point of view, these approaches tend to be quite hard to implement and often are very computationally intensive. in addition, most of these strategies are not specifically designed for natural language tasks, making their implementation even more difficult. therefore, we decided to tackle the catastrophic forgetting problem in machine translation, framed as a sequential learning problem for a multilingual machine translation system, where each new task is a different language pair (english-spanish, english-french, english-german, and english-czech). the contributions of this work are the following: ## related work the catastrophic forgetting (cf) phenomenon has been widely studied since it was introduced for the first time by @xcite . however, despite the numerous works that have delved into the root causes that produce it @xcite , these findings could be reduced to the stability-plasticity dilemma, whereby there is a trade-off between the ability of a model to preserve past knowledge (stability) and the ability to learn new information effectively (plasticity). given this dilemma, most approaches are based on adjusting the network weights during training to control the forgetting of the model, expanding the model's capacity to support new tasks, or using some refreshing mechanism to remember past tasks. for example, @xcite presented a model with shared parameters across tasks and task-specific parameters; @xcite identified which weights were important for the past tasks so that they could penalize the updates on those weights; @xcite penalized changes in the final hidden layer; @xcite introduced the concept of intelligent synapses that accumulate task-relevant information; @xcite trained a model with a set of parameters that was shared by all tasks and the second set of parameters that were dynamically generated to adapt the model to each new task. however, despite the number of works, these strategies are constrained by the model's capacity @xcite . to deal with this issue, many researchers decided to focus their efforts on linearly expanding the model's capacity as the number of tasks grows. accordingly, @xcite retained a pool of pre-trained models throughout training to learn lateral connections for the new task; @xcite , which was inspired by the neurogenesis in the hippocampus of the brain decided to add new neurons to deep layers so that novel information could be acquired more efficiently; and @xcite introduced an architecture that dynamically controls the network capacity. similarly, other researchers have addressed this problem by using data from past tasks during the training of new tasks, such as lopez-paz and ranzato (2017), who proposed a model that alleviates the catastrophic forgetting problem by storing a subset of the observed examples from an old task (episodic memory), and @xcite , who instead of storing actual training data from past tasks, trained a deep generative model that replayed past data (synthetically) during training to prevent forgetting. in addition to these works, there are others worth to mention due to their results and original approaches, such as icarl @xcite , pathnet @xcite , fear-net @xcite , imm @xcite or mas @xcite . nonetheless, despite the progress made on lifelong learning strategies and the recent breakthroughs in the natural language field @xcite @xcite , the catastrophic forgetting problem has not been so widely studied in the field of machine translation. along these lines, @xcite proposed a metalearning method that exploits knowledge from past domains to generate improved embeddings for a new domain; @xcite showed that pre-trained embeddings could be effective in low-resource scenarios; @xcite learned corpus-dependent features by sequentially updating sentence encoders (previously initialized with the help of corpus-independent features) using boolean operations of conceptor matrices; @xcite presented a method to adapt the embeddings between domains by projecting the target embeddings into the source space, and then fine-tuning them on the target domain; @xcite introduced a vocabulary adaptation scheme to extend the language capacity of multilingual machine translation models; and more recently, @xcite adapted the elastic weight consolidation method @xcite to mitigate the drop in general-domain performance of nmt models. ## conclusions this work has studied the catastrophic forgetting problem in machine translation framed as a sequential learning problem for a multilingual machine translation system, where each new language pair is considered a new task. from studying the effects of the catastrophic forgetting problem as a function of the number of learned tasks and the ratios of past data used during the learning of the new task, we discovered that even with minimal amounts of past data, we could retain up a 95% of the performance in past tasks. then, we tried to boost the performance in past tasks through an oversampling strategy. however, this approach was primarily beneficial for scenarios where only minimal amounts of past data were available (<1%). finally, we derived a new loss function based on actively re-weighting past tasks and penalizing weights that deviate too much from the original model to minimize forgetting past tasks while learning the new one. this approach has practically no extra cost and shines by simplicity when compared to other popular but more complex and resource-hungry approaches. this work suggests that to easily mitigate the effects of the catastrophic forgetting in machine translation with no extra cost, we only need a minimal amount of past data and a simple regularization function that exploits past knowledge information.