Robust Domain Adaptation for Pre-trained Multilingual Neural Machine Translation Models (2022)

Abstract: Recent literature has demonstrated the potential of multilingual Neural Machine Translation (mNMT) models. However, the most efficient models are not well suited to specialized industries. In these cases, internal data is scarce and expensive to find in all language pairs. Therefore, fine-tuning a mNMT model on a specialized domain is hard. In this context, we decided to focus on a new task: Domain Adaptation of a pre-trained mNMT model on a single pair of language while trying to maintain model quality on generic domain data for all language pairs. The risk of loss on generic domain and on other pairs is high. This task is key for mNMT model adoption in the industry and is at the border of many others. We propose a fine-tuning procedure for the generic mNMT that combines embeddings freezing and adversarial loss. Our experiments demonstrated that the procedure improves performances on specialized data with a minimal loss in initial performances on generic domain for all languages pairs, compared to a naive standard approach (+10.0 BLEU score on specialized data, -0.01 to -0.5 BLEU on WMT and Tatoeba datasets on the other pairs with M2M100).

## introduction building a nmt model supporting multiple language pairs is an active and emerging area of research @xcite @xcite . multilingual nmt(mnmt) uses a single model that supports translation in multiple language pairs. multilingual models have several advantages over their bilingual counterparts @xcite . this modeling proves to be both efficient and effective as it reduces the operational cost (a single model is deployed for all language pairs) and improves translation performances, especially for low-resource languages. all these advantages make mnmt models interesting for real-world applications. however, they are not suitable for specialized industries that require domain-specific translation. training a model from scratch or fine-tuning all the pairs of a pretrained mnmt model is almost impossible for most companies as it requires access to a large number of resources and specialized data. that said, finetuning a single pair of a pre-trained mnmt model in a specialized domain seems possible. ideally this domain adaptation could be learned while sharing parameters from old ones, without suffering from catastrophic forgetting @xcite . this is rarely the case. the risk of degrading performance on old pairs is high due to the limited available data from the target domain and to the extremely high complexity of the pre-trained model. in our case, overfitting on fine-tuning data means that the model might not even be multilingual anymore in this context, this article focuses on a new real-world oriented task fine-tuning a pre-trained mnmt model in a single pair of language on a specific domain without losing initial performances on the other pairs and generic data. our research focuses on fine-tuning two state-of-the-art pre-trained multilingual mnmt freely available: m2m100 @xcite and mbart50 @xcite which both provide high performing bleu scores and translate up to 100 languages. we explored multiple approaches for this domain adaptation. our experiments were made on english to french data from medical domain foot_0 . this paper shows that fine-tuning a pre-trained model with initial layers freezing, for a few steps and with a small learning rate is the best performing approach. it is organized as follows : firstly, we introduce standard components of modern nmt, secondly we describe related works, thirdly we present our methods. we finally systematically study the impact of some state-of-the-art fine-tuning methods and present our results. our main contributions can be separated into 2 parts: ## methods our new real-world oriented task being at the crossboard of many existing task, we applied ideas from current literature and tried to combine different approaches to achieve the best results. ## conclusion and discussion in this paper, we propose a study of robust domain adaptation approaches on mnmt models where indomain data is available only for a single language pair. best performing approach combines embedding freezing and simple fine-tuning with good hyperparameters. this approach shows good improvements with few in-domain data on all language pairs. the framework effectively avoids overfitting and aggressive forgetting on out-of-domain generic data while quickly adapting to in-domain data. we demonstrate that this could be a solution for incremental adaptation of mnmt models. finally our work is a call for more research in domain adaptation for multilingual models as it is key for real-world applications.