m ^4 Adapter: Multilingual Multi-Domain Adaptation for Machine Translation with a Meta-Adapter (2022)

Abstract: Multilingual neural machine translation models (MNMT) yield state-of-the-art performance when evaluated on data from a domain and language pair seen at training time. However, when a MNMT model is used to translate under domain shift or to a new language pair, performance drops dramatically. We consider a very challenging scenario: adapting the MNMT model both to a new domain and to a new language pair at the same time. In this paper, we propose m^4Adapter (Multilingual Multi-Domain Adaptation for Machine Translation with a Meta-Adapter), which combines domain and language knowledge using meta-learning with adapters. We present results showing that our approach is a parameter-efficient solution which effectively adapts a model to both a new language pair and a new domain, while outperforming other adapter methods. An ablation study also shows that our approach more effectively transfers domain knowledge across different languages and language information across different domains.

## introduction multilingual neural machine translation (mnmt; @xcite @xcite , uses a single model to handle translation between multiple language pairs. there are two reasons why mnmt is appealing: first, it has been proved to be effective on transferring knowledge from high-resource languages to low-resource languages, especially in zero-shot scenarios @xcite ; second, it significantly reduces training and inference cost, as it requires training only a single multilingual model, instead of a separate model for each language pair. adapting mnmt models to multiple domains is still a challenging task, particularly when domains are distant to the domain of the training cor-pus. one approach to address this is fine-tuning the model on out-of-domain data for nmt @xcite . another approach is to use lightweight, learnable units inserted between transformer layers, which are called adapters @xcite for each new domain. similarly, there is research work on adapting mnmt models to a new language pair using fine-tuning @xcite and adapters @xcite @xcite . although effective, the above approaches have some limitations: i) fine-tuning methods require updating the parameters of the whole model for each new domain, which is costly; ii) when finetuning on a new domain, catastrophic forgetting @xcite reduces the performance on all other domains, and proves to be a significant issue when data resources are limited. iii) adapter-based approaches require training domain adapters for each domain and language adapters for all languages, which also becomes parameterinefficient when adapting to a new domain and a new language because the parameters scale linearly with the number of domains and languages. in recent work, cooper stickland et al. (2021a) compose language adapters and domain adapters in mnmt and explore to what extent domain knowledge can be transferred across languages. they find that it is hard to decouple language knowledge from domain knowledge and that adapters often cause the 'off-target' problem (i.e., translating into a wrong target language @xcite ) when new domains and new language pairs are combined together. they address this problem by using additional in-domain monolingual data to generate synthetic data (i.e., back-translation; @xcite and randomly dropping some domain adapter layers (adapterdrop; @xcite . ## method our goal is to efficiently adapt an mnmt model to new domains and languages. we propose a novel approach, m 4 adapter, which formulates the multilingual multi-domain adaptation task as a multitask learning problem. to address it, we propose a 2-step approach, which combines meta-learning and meta-adaptation with adapters. our approach permits sharing parameters across different tasks. the two steps are explained in subsections 3.1 and 3.2. ## experiments datasets. we split the datasets in two groups: metatraining or training dataset (used in step 1, § 3.1) and meta-adapting or adapting dataset (used in step 2, § 3.2). we first meta-learn the adapters on the training dataset (that contains dlps different to the ones we will evaluate on), then fine-tune to new domains and language pairs on the adapting dataset (a small dataset of the dlps we will evaluate on). we list the datasets used, each treated as a different domain: eubookshop, kde, opensubtitles, qed, ted, ubuntu, bible, un, tanzil, infopankki. the datasets cover the following languages (iso 639-1 language code foot_2 ): en, de, fr, mk, sr, et, hr, hu, fi, uk, is, lt, ar, es, ru, zh and are publicly available on opus foot_3 @xcite . data preprocessing. for each training dataset, we strictly limit the corpus of each dlp to a maxi-mum of 5000 sentences to simulate a low-resource setting. for each adapting dataset, we use 500 sentences in each dlp to fine-tune the mnmt model, simulating a few-shot setting. for the validation and test set, we select 500 sentences and avoid overlap with the adapting dataset by de-duplication. we filter out sentences longer than 175 tokens and preprocess all data using sentencepiece foot_4 @xcite . more details for the data used in this paper can be found in the appendix a.1. baselines. we compare m 4 adapter with the following baselines: i) m2m: using the original m2m model @xcite to generate the translations. ii) m2m + ft: fine-tuning m2m on all dlps. iii) m2m + tag: fine-tuning m2m with domain tags @xcite on all dlps. iv) agnosticadapter: mixing the data from all dlps to train the adapters @xcite , to obtain language and domain-agnostic adapters. v) stack-adapter: training two adapters for each language pair and domain, then stacking both adapters @xcite . taking 'ubuntuen-sr' as an example, this approach first trains a language pair adapter for 'en-sr' using all data containing 'en-sr' in all domains (also including the 'ubuntu' domain) and a domain adapter for 'ubuntu' using all data covering all language pairs in the 'ubuntu' domain. then, the two adapters are stacked together. vi) meta-learning: traditional meta-learning methods using the maml algorithm @xcite ) on all dlps. implementation. we use m2m, released in the huggingface repository foot_5 @xcite . for adapter training, we use the implementation of the adapterhub repository foot_6 @xcite . we use deepspeed foot_7 @xcite to accelerate the pre-training of big models. note that all baseline systems except stack-adapter train a single mnmt model or a single adapter on all dlps in the training datasets and then fine-tune to a specific dlp on a single adapting dataset. for stackadapter, the number of language pair adapters and domain adapters to be trained is proportional to the number of language pairs and the number of domains (see more details in appendix a.2). evaluation. we measure case-sensitive detokenized bleu with sacrebleu foot_8 @xcite chinese we use the sacrebleu tokenizer (tok zh) and convert all traditional characters generated by the model to simplified characters using hanziconv. 10 we also evaluate our models using chrf @xcite due to the recent criticism of bleu score @xcite ; the results are listed in the appendix a.3.1. ## results our goal is to evaluate the adaptability of m 4 adapter on a variety of new domains and new language pairs simultaneously. in the meta-training stage, we perform meta-learning of the model on 180 dlps, which contain 6 domains (eubookshop, kde, opensubtitles, qed, ted, ubuntu) and 30 language pairs (en, et, mk, sr, hr, hu), comparing our approach to different baseline systems. in the metaadaptation stage, we fine-tune both our model and the baselines to 3 domains (un, tanzil, infopankki) and 30 language pairs (using ar, en, es, fr, ru, zh) of the same dataset simultaneously. as shown in because of the unpredictability of the baseline systems' performance, it is difficult to draw reliable conclusions. for example, in the un domain, metalearning is on par with the original m2m model. however, performance on tanzil and infopankki is much worse than the one of the original m2m model. the agnostic-adapter also performs comparably with the original m2m model in the same domains, which shows that it is a robust model. still, it obtains much worse performance on un. in contrast, m 4 adapter has a more stable performance when adapting to new domains and language pairs. in addition, m 4 adapter has the ability to improve the performance of some dlps on which baseline models obtain extremely low bleu scores, especially in some distant domains. for example, in tanzil-ar-ru, the traditional metalearning method only gets 1.70 bleu score, while m 4 adapter gets 4.33. ## analysis in this section, we conduct additional experiments to better understand the strengths of m 4 adapter. we first investigate the benefits in terms of speed in m 4 adapter training and adapting (section 6.1), then investigate the cross-lingual domain transfer and cross-domain language transfer through an ablation study (section 6.2). ## conclusion we present m 4 adapter, a novel multilingual multidomain nmt adaptation framework which combines meta-learning and parameter-efficient finetuning with adapters. m 4 adapter is effective on adapting to new languages and domains simultaneously in low-resource settings. we find that m 4 adapter also transfers language knowledge across domains and transfers domain information across languages. in addition, m 4 adapter is efficient in training and adaptation, which is practical for online adaptation @xcite to complex scenarios (new languages and new domains) in the real world.