Self-generated Replay Memories for Continual Neural Machine Translation (2024)

Abstract: Modern Neural Machine Translation systems exhibit strong performance in several different languages and are constantly improving. Their ability to learn continuously is, however, still severely limited by the catastrophic forgetting issue. In this work, we leverage a key property of encoder-decoder Transformers, i.e. their generative ability, to propose a novel approach to continually learning Neural Machine Translation systems. We show how this can effectively learn on a stream of experiences comprising different languages, by leveraging a replay memory populated by using the model itself as a generator of parallel sentences. We empirically demonstrate that our approach can counteract catastrophic forgetting without requiring explicit memorization of training data. Code will be publicly available upon publication.

## introduction neural machine translation (nmt) systems have achieved remarkable performance on numerous language pairs, particularly those with abundant resources. the substantial growth in model parameters and the availability of large crawled corpora have greatly contributed to the adoption of advanced techniques like back-translation @xcite and denoising pre-training @xcite @xcite , further increasing the translation quality of these models. consequently, even low-resource languages have benefited from the increased multilingual capabilities of these models @xcite . despite their impressive quality, modern nmt systems exhibit limited continual learning (cl) capabilities and are susceptible to catastrophic forgetting @xcite . while the cl community has extensively investigated this phenomenon over the years, it has 1 https://github.com/m-resta/sg-rep figure 1 : a scheme of the cill setting. a model is trained incrementally on a stream of experiences comprising training data for various language pairs. received relatively less attention from the natural language processing (nlp) community. previous works in this area have primarily focused on domain adaptation, and only recently have turned to different continual learning settings such as incremental language learning @xcite . in this paper, we present sg-rep: a novel approach to continually train an nmt system on a stream of experiences comprising several language pairs while mitigating the detrimental effects of catastrophic forgetting. our method leverages a replay memory populated by synthetic parallel sentences generated by the model itself. we evaluate the effectiveness of our approach across different translation directions and demonstrate its ability to alleviate cf without the need for explicit memorization of the training data. this aspect is crucial where is not possible to store real training samples for privacy reasons or data retention policies. ## related works continual learning or lifelong learning (ll) research focuses on developing computational systems that can gradually acquire, refine, and transfer knowledge over extended periods, imitating biolog-ical systems. the primary challenge arises from the inherent plasticity of neural networks, which leads to catastrophic forgetting. this refers to a situation where performance on previously learned tasks deteriorates as the network parameters are updated, resulting in the loss of acquired knowledge. in addition to mitigating cf, lifelong learning methods should also promote knowledge transfer across tasks in both forward and backward directions. research work in the field can be broadly categorized into architecture, regularization, or databased approaches. ## the self-generated replay method the goal of our method,which we abbreviate as sg-rep, is to incrementally learn a single model that is able to translate into many directions. the training data is not available as a whole but is presented to the model in incremental steps, as a stream of experiences that comprises one or more language directions. the key challenge is preserving translation performance on past experience while adapting effectively to new data. to this end, we use the encoder-decoder model learned in a given experience as a generator of synthetic training samples. the obtained pseudo-samples will populate a fixedsize replay memory that will be used in future tasks to mitigate catastrophic forgetting. ## experimental results the translation output of the models is scored using the 4-gram case-sensitive bleu @xcite with the sacrebleu tool @xcite using the default tokenization scheme foot_2 based on mosestokenizer foot_3 . comet scores are reported in appendix g. each model is evaluated at the end of training on the last experience on all translation directions. in addition to the average bleu, we also report a metric for comparative analysis to compensate for the diversity of the different test sets for each language. it quantifies the performance delta between the chosen system and the upper bound (i.e. joint training) for each language pair. specifically, we define the language pair delta as ## conclusions we proposed a simple yet effective continual learning method for nmt that uses a replay memory to mitigate catastrophic forgetting. differently from other data-based approaches, we do not memorize training samples explicitly and instead use the model itself as a generator of parallel sentences. the experimental results prove that our method can achieve significant improvement over several strong continual learning baselines.