Strategies for Adapting Multilingual Pre-training for Domain-Specific Machine Translation (2022)

Abstract: Pretrained multilingual sequence-to-sequence models have been successful in improving translation performance for mid- and lower-resourced languages. However, it is unclear if these models are helpful in the domain adaptation setting, and if so, how to best adapt them to both the domain and translation language pair. Therefore, in this work, we propose two major fine-tuning strategies: our language-first approach first learns the translation language pair via general bitext, followed by the domain via in-domain bitext, and our domain-first approach first learns the domain via multilingual in-domain bitext, followed by the language pair via language pair-specific in-domain bitext. We test our approach on 3 domains at different levels of data availability, and 5 language pairs. We find that models using an mBART initialization generally outperform those using a random Transformer initialization. This holds for languages even outside of mBART’s pretraining set, and can result in improvements of over +10 BLEU. Additionally, we find that via our domain-first approach, fine-tuning across multilingual in-domain corpora can lead to stark improvements in domain adaptation without sourcing additional out-of-domain bitext. In larger domain availability settings, our domain-first approach can be competitive with our language-first approach, even when using over 50X less data.

## introduction recent pretrained multilingual sequence-to-sequence (seq2seq) models have provided a basis to easily create neural machine translation (mt) systems via the pretrain and fine-tune paradigm ubiquitous throughout nlp @xcite . due to the fact that fine-tuning these models generally requires less data than is needed for from-scratch translation models, pretrained models are great candidates for mt domain adaptation tasks, where domain-specific bitext is generally less available as compared to general bitext. however, these models have seldom been studied in domain-specific settings. for mt domain adaptation, pretrained multilingual seq2seq models must be adapted to both 1) the language pair and 2) the domain of interest. previous work has introduced several methods for adapting general translation models to domains, including training first on general bitext to bolster the total amount of bitext available, followed by training on smaller domainspecific bitext @xcite . however, in the case of multilingual sequence models, the initial pretraining objective differs significantly from the task of machine translation, which suggests that alternative adaptation approaches are necessary. additionally, approaches involving additional general bitext may not even benefit these models as they were already initially trained on large amounts of general pretraining data. therefore, it is currently unclear 1) if pretrained multilingual seq2seq models have useful properties for mt domain adaptation and 2) how to best adapt them to both the translation language pair and domain. as a result, in this work, we aim to systematically compare finetuning approaches for applying mbart to domain adaptation @xcite . we choose to focus on mbart as it has previously shown the most promising results in the mt setting among comparable models @xcite . by framing language pair and domain as decoupled entities to learn during the fine-tuning process, we can compare two major approaches to the adaptation process. the first fine-tunes mbart on general bitext, followed by in-domain bitext. the second uses multilingual finetuning on in-domain bitext across several language pairs followed by bilingual fine-tuning on in-domain bitext in the language pair of interest @xcite . in other words, the first approach adapts to the language pair first, and the second approach adapts to the domain first, before both eventually fine-tune on the small amount of in-domain, language pair-specific bitext. we emphasize the importance of a multi-staged approach as we find that they are consistently better than naively fine-tuning mbart only on domain-specific bitext, especially when this data is limited. figure 1 : summaries of our two major approaches. our language pair first approach first finetunes the multilingual pretrained model on language-pair specific translation, and then on the domain. our domain first approach first fine-tunes the multilingual pretrained model on the domain of interest, followed by the specific language pair. we test our approach on 5 language pairs and 3 domains: ted talks, microblogs, and covid-19 related information. we note that the amount of available in-domain bitext varies greatly across these domains. because we want our method to be broadly applicable to new domains where data may be very limited and/or expensive to procure, we test our approaches on a small, fixed amount of domain data, as well as on the entirety of the domain data available. we find that in comparable approaches, those with an mbart initialization outperform those with a vanilla transformer initialization in a majority of our language pairs and domains, and across our two domain availability settings. this holds even in the cases of higher-resourced language pairs originally unhelped by mbart's multilingual pretraining, and in language pairs outside of mbart's pretraining set. for our out-of-mbart persian-english language pair, simply using an mbart initialization leads to +4.8 to 12.8 bleu points across our domains. in addition, we find that our domain-first approach provides an efficient alternative to using additional general bitext by leveraging available multilingual in-domain corpora via multilingual fine-tuning. we show that in our whole domain availability setting, which is still several times smaller than the data needed for our language-first approach, our domain-first approach consistently shows improvements over baselines, and is sometimes competitive with our more data-heavy language-first approach. this paper makes the following contributions: ## approach because mbart is a multilingual denoising autoencoder, it is trained only to reconstruct text in the source language given. we detail our two major approaches to domain adaptation using mbart, focusing on translation language pair and domain. our approaches propose two different ways to learn these competencies. we summarize our approaches in figures 2 and 3 . ## conclusion in this paper, we demonstrate that multilingual pretraining can be very effective in the domain adaptation setting, and we propose two methods of adaptation that are more useful than a naive adaptation approach. we also find that between our methods, our language-first approach where models are first customized to a specific bilingual setting, is consistently our best system, especially in limited domain scenarios. however, we also find that when we first customize our models to a domain, as in our domain-first approach, we achieve considerable translation quality at a fraction of the data needed in our language-first approach. interestingly, we are also able to show that multilingual pretraining and fine-tuning continue to be effective domain adaptation techniques even when the pretrained model has not seen the language pair before.