Conditions for Catastrophic Forgetting in Multilingual Translation (2025)

Abstract: Fine-tuning multilingual foundation models on specific languages often induces catastrophic forgetting, degrading performance on languages unseen in fine-tuning. While this phenomenon is widely-documented, the literature presents fragmented results about when forgetting occurs. To address this ambiguity, we conduct a systematic empirical study using machine translation as a testbed to identify the conditions that trigger catastrophic forgetting in multilingual fine-tuning. Through controlled experiments across different model architectures, data scales, and fine-tuning approaches, we reveal that the relative scale between model and data size is a primary determinant of forgetting. Moreover, we demonstrate that a model’s instruction-following ability is more critical for retaining multilingual knowledge than its architecture. Contrary to assumptions, parameter-efficient fine-tuning offers no clear advantage over full fine-tuning in mitigating forgetting. Lastly, we show that cross-lingual alignment can mitigate forgetting while also facilitating positive transfer to unseen target languages.

## introduction foundation models pretrained on vast amounts of multilingual data have become the standard backbone for modern natural language processing systems. to achieve optimal performance, however, these models typically require fine-tuning on downstream tasks. this specialization introduces a critical trade-off: while performance on the target task improves, the model may suffer from catastrophic forgetting @xcite , a substantial degradation of capabilities on tasks or languages not present in the fine-tuning data. a common use case is to fine-tuning multilingual models to focus on specific languages or language pairs. ideally, this process would not harm, and might even improve, performance on unseen languages through positive transfer, as illustrated on the left of figure 1 . however, empirical evidence often shows the opposite. models frequently lose proficiency in languages they were not fine-tuned on @xcite @xcite , as shown on right of figure 1 . machine translation (mt) serves as a compelling testbed for studying multilingual catastrophic forgetting. first, a model supporting n languages encompasses n(n -1) directed translation pairs, offering a large and structured space to analyze forgetting patterns. second, languages can be "unseen" in different roles. for example, a language may be present only as a source language, only as a target language, or in specific source-target pairs that were never explicitly trained. this enables finegrained analysis of how different types of exposure during fine-tuning affect retention. moreover, forgetting can occur asymmetrically, where a model may retain the ability to translate from language a to b while losing the reverse direction. despite its practical importance, the literature presents a fragmented and sometimes contradictory picture of when catastrophic forgetting occurs in mt. on one hand, studies on traditional nmt models trained from scratch @xcite and some large pretrained models @xcite @xcite report severe forgetting after standard fine-tuning, where the ability to translate unseen directions is almost entirely lost. these findings suggest that catastrophic forgetting is an inevitable consequence of selective specialization. on the other hand, recent works on large language models (llms) provided mixed evidence. @xcite demonstrated that fine-tuning llama 2 @xcite and tower @xcite models on specific language pairs could improve performance on unseen pairs, indicating positive transfer. conversely, @xcite found that their fine-tuned llama 2 models performed very poorly on unseen directions, again indicating issues with forgetting. these conflicting results raise fundamental questions about the factors leading to catastrophic forgetting. has the emergence of large language models altered the dynamics of catastrophic forgetting? to what extent do model architecture (encoderdecoder versus decoder-only), scale, or fine-tuning methodology determine whether a model forgets or generalizes? how do factors like the volume of fine-tuning data, the use of parameter-efficient fine-tuning (peft), or instruction-following capabilities influence the retention of multilingual abilities? to resolve these ambiguities, we conduct a systematic study to identify the conditions that trigger catastrophic forgetting in multilingual mt. we systematically control for key variables, including model architecture and size, fine-tuning data composition and scale, full-parameter vs. parameterefficient fine-tuning, and instruction-following versus standard fine-tuning approaches. with a series of controlled experiments, we demonstrate that: • the relative scale between pre-trained model parameters and fine-tuning data volume is a critical factor in catastrophic forgetting, with smaller models fine-tuned on larger datasets being most vulnerable ( §4.1). ## controlled setting to study catastrophic forgetting in mt our controlled experiments are structured along two dimensions, namely the choice of base model and the characteristics of the training dataset. ## gain-forgetting analyses we investigate the trade-off between performance gains on fine-tuned language pairs and potential catastrophic forgetting on those unseen during finetuning. to visualize this relationship, we create scatter plots (figure 2 and figure 3 ) where each point represents a language pair's performance before (x-axis) and after (y-axis) fine-tuning. the diagonal line (@xmath0, where points below indicate catastrophic forgetting, while those above indicate performance improvement. ## evaluating cross-lingual alignment for forgetting mitigation having identified the architectural and training factors that impact catastrophic forgetting, we pose a question about mitigation strategies: do established forgetting mitigation methods primarily restore lost performance, or do they also improve cross-lingual transfer? we focus on cross-lingual alignment methods, as they encourage similar representations for semantically equivalent content across languages, which could mitigate forgetting. ## conclusion in this work, we aim to resolve ambiguities in the literature regarding when catastrophic forgetting occurs for multilingual fine-tuning for mt. based on our findings, we provide the following practical recommendations: 1) consider the relative scale between model size and fine-tuning data. larger datasets may require larger base models to pre-vent forgetting. 2) prioritize models with strong instruction-following abilities over specific architectural choices. 3) do not rely solely on parameterefficient fine-tuning methods as a forgetting mitigation strategy. 4) for models exhibiting forgetting, cross-lingual alignment is promising for unseen pairs where both source and target languages have been separately seen in fine-tuning. for instructionfollowing models, we recommend training with inlanguage instructions as an initial data-oriented approach before proceeding with cross-lingual alignment approaches.