Learn and Consolidate: Continual Adaptation for Zero-Shot and Multilingual Neural Machine Translation (2023)

Abstract: Although existing multilingual neural machine translation (MNMT) models have demonstrated remarkable performance to handle multiple translation directions in a single model and achieved zero-shot translation between language pairs unseen in training, they still suffer from relatively poor translation qualities for some language pairs. A practical scenario is that how to continually update MNMT models for both supervised and zero-shot translations when limited new data arrives. To this end, we propose a two-stage approach that encourages original models to acquire language-agnostic multilingual representations from new data, and preserves the model architecture without introducing parameters. Experimental results and further analysis demonstrate that our method can efficiently improve performance of existing MNMT models in translation directions where they are initially weak, and mitigates the degeneration in the original well-performing translation directions, offering flexibility in the real-world scenario.

## introduction existing multilingual neural machine translation (mnmt) models, such as mbart @xcite and m2m-100 @xcite , have showcased significant advancements to handle multiple translation directions in a single model, especially enabling zero-shot translation directions between languages not encountered during training with implicit cross-lingual transfer @xcite @xcite . however, they still perform poorly on language pairs without sufficient parallel corpus @xcite . fortunately, new parallel sentence pairs will continually emerge between high-resource languages, e.g., german↔chinese, which can be used to facilitate translation directions with poor performance through supervised learning @xcite . in addition to updated data, obtaining parallel data between some languages, e.g., tamil↔chinese, encounters difficulties in the world @xcite . we aspire to achieve many-to-one translation with cross-lingual transferability of mnmt models in a zero-shot manner @xcite , when the "one" target language related data is updated. as shown in figure 1 , we assume three translation directions for continual adaptation: new supervised translations, new zero-shot translations, and original wellperforming translations (typically english-centric). in this scenario, we aim to continually improve performance for both new supervised and zero-shot translations while retaining previously acquired knowledge in the other translation directions using only new data. in accordance with this scenario, an intuitive solution is to introduce additional parameters for continual adaptation, regarded as parameter-isolation based methods @xcite . the drawback of these methods is that they increase the size of the base models, and as new data continues to accumulate, the total number of parameters may expand indefinitely @xcite . to preserve the model architecture, another solution is to leverage the continual learning paradigm @xcite @xcite . although these methods propose multi-objective learning to alleviate the issue of catastrophic forgetting, they necessitate full parameter fine-tuning @xcite , resulting in relatively low training efficiency @xcite . besides, a significant pitfall of the above-mentioned methods is that they lack the ability to promote zero-shot translation. in this work, we propose a two-stage method consisting of a learning stage and a consolidation stage for continual adaptation (lcca), which efficiently adapts mnmt models to diverse translation directions. we first introduce a flexible pluggable module in the penultimate encoder and decoder layer, respectively, as the multilingual adaptation space. and the introduced multilingual space is optimized with contrastive learning to make representation language-agnostic for zero-shot translations, realizing the learning stage. then we attempt to compress the additional parameters to the same size of base mnmt models, regarded as a consolidation stage. the second stage adopts the information matrix and collaborative distillation to facilitates original components to learn introduced modules in a specific range. furthermore, aside from the related components, all the parameters of the original model remain fixed, allowing for a parameter-efficient manner to large-scale mnmt models. the two stages are also an isolated process designed to adapt to diverse application requirements in real-world scenarios. to sum up, our contributions are as follows: ## related work zero-shot translation with mnmt models mnmt models have demonstrated their ability to facilitate knowledge transfer across languages and enable zero-shot translations between language pairs that are not covered in training data @xcite @xcite . to further enhance the performance of zero-shot translation, some studies investigate language-agnostic representations @xcite @xcite and language-specific features @xcite for zero-shot translations. for instance, @xcite align cross-lingual representations using additional dictionaries and contrastive learning. this work continues to explore the potential of multilingual representations, but focusing on addressing the continual adaptation for new zero-shot translations within an wellperformed representation space while avoid catastrophic forgetting. continual learning for mnmt some previous methods of continual learning attempt to address the issue of catastrophic forgetting when only the new data is accessible @xcite , including replay-based methods @xcite @xcite , regularization-based methods @xcite @xcite , and parameter-efficient transfer (pet) methods @xcite @xcite . on the one hand, the first two methods adopt the multi-objective learning @xcite to balance performance between old and new tasks in continual learning through full parameters tuning, which is inefficient and time-consuming. on the other hand, pet methods introduce additional task-specific parameters and freeze all original parameters to completely retain performance on previous tasks @xcite @xcite . however, they introduce extra parameters to adapt to new tasks and lack the ability to improve zero-shot performance, limiting their application in practice @xcite . differing from these approaches, our method mitigates the issue of zero-shot translations only using new data, and achieves efficient continual adaptation without introducing additional parameters. ## method our scenario is to efficiently improve performance for some particular translation directions without compromising previous well-performing translations. to achieve this, we propose a two-stage method to alleviate the issues of continual adaptation in particular translation directions at different stages, as shown in figure 2 . specifically, we first introduce an additional space to capture languageagnostic features for new target languages, as the learning stage. then we compress the additional space back into the original models with a collaboration, as the consolidation stage. and we calculate an information matrix to measure the parameters optimization in a smooth region across multiple languages. ## conclusion in this work, we propose a two-stage method of learning and consolidation to improve performance of pre-trained mnmt models when new data arrives in both supervised and zero-shot translations. it extends multilingual language-agnostic space with the contrastive learning for new data adaptation in the learning stage, and then adopts a collaborative distillation with an information matrix to consolidate both previously and newly learned knowledge for module integration. as a result, our proposed method, lcca, achieves continual adaptation to multiple translation directions while keeping the model architecture intact. experimental results demonstrate that the learning stage encourages original models to learn new knowledge from updated parallel data and the consolidation stage mitigates the performance degradation on previous well-performing translation directions when the model compresses. further analyses reveal that our method effectively captures linguistic features and bridges the gap of shared representation space for comprehensive continual adaptation.