Monolingual Adapters for Zero-Shot Neural Machine Translation (2020)

Abstract: We propose a novel adapter layer formalism for adapting multilingual models. They are more parameter-efficient than existing adapter layers while obtaining as good or better performance. The layers are specific to one language (as opposed to bilingual adapters) allowing to compose them and generalize to unseen language-pairs. In this zero-shot setting, they obtain a median improvement of +2.77 BLEU points over a strong 20-language multilingual Transformer baseline trained on TED talks.

## introduction of the many virtues of multilingual neural machine translation (mnmt), arguably the most attractive is the promise of improving performance in the low resource setting @xcite @xcite . these models even allow for the extreme of these cases, namely to translate in language pair directions which are unseen at training time (zero-shot setting in this paper). unfortunately, while performance in the low-resource setting indeed increases significantly, their zero-shot performance remains very low @xcite . in this paper, we propose a neural architecture that allows to translate from any of the source languages towards any of the target languages seen in the training data, regardless of the presence of that specific language direction during training. for that, we build upon the recently proposed adapter layers for nmt @xcite , by using monolingual (language-specific) adapter layers, instead of bilingual (language-pair specific) ones. this design difference improves their compositionality, permitting to combine any encoder adapter with other decoder adapters. monolingual adapter layers perform as good as bilingual adapter layers in * work done during an internship at naver labs europe. the non-zero-shot setting, are effective in the zeroshot setting and have the additional advantage of requiring fewer parameters. ## related work zero-shot translation is direct translation in a language pair unseen during training. @xcite analyze the zero-shot performance of mnmt models as a function of the number of language pairs. they observe that having more languages results in better zero-shot performance. however, several artifacts arise, as described by @xcite ; @xcite ; @xcite ; @xcite , like offtarget translation and insufficient modeling capacity of the mnmt models. @xcite use language-aware layer normalization and linear transformation to improve some drawbacks of mnmt; they also rely massively on backtranslation to improve zero-shot translation. adaptation to a new language pair may be addressed by training a multilingual model then fine-tuning it with parallel data in the language pair of interest @xcite @xcite . @xcite propose plug-and-play encoders and decoders per language, which take advantage of a single representation in each language but at the cost of larger model sizes. in order to add only a few trainable parameters per task, adapter modules -initially introduced for computer vision @xcite -were proposed for language modeling by @xcite . @xcite used them for parameter-efficient adaptation in mnmt. the parameters of the original mnmt network (the parent model) remain fixed, which permits a high degree of parameter sharing. the final multilingual model (the adapted model) is just slightly larger than the original one. (bapna @xcite . we use languages as the tasks in the encoder and in the decoder. xx and yy denote source and target languages respectively. and firat, 2019) show that adapters mitigate one major problem of mnmt models: performance drop in high-resource languages. the motivation of that work was not zero-shot, and it is not obvious how to use them in such a scenario as the adapter layers are language-pair specific. while in section 5 we propose a way of using those adapters through pivoting adapter layers, the main contribution of this paper is monolingual adapters which allow combining any encoder adapter with other decoder adapters. ## monolingual adapters adapter modules @xcite were formulated for nmt by @xcite : lightweight adapter layers are transplanted between the layers of a pre-trained network and fine-tuned on the adaptation corpus. as shown in figure 1 (left), an adapter layer is a down projection to a bottleneck dimension followed by an up projection to the initial dimension. the bottleneck allows to limit the number of parameters of the adapter module. the residual connection coupled with a near-identity initialization enables a pass-through and allows keeping at least the performance of the parent model. in their initial formulation, @xcite proposed adapters for each language pair (bilingual adapters), while we propose monolingual adapters. we illustrate the mechanism in figure 1 : our monolingual-adapter layers are inserted into each of the transformer encoder and decoder layers. when translating from language xx to language yy, we only activate the encoder adapter layers for xx, denoted by θ e xx ; and the decoder adapter layers for yy, denoted by θ d yy . our formulation is different from @xcite , who propose adapter layers for each language direction (θ @xmath0. in a multiparallel setting (i.e., where parallel data is available for all language pairs), this requires training n(n -1) sets of layers, where n is the number of languages. our monolingual (language-specific) adapters only require 2n layers. ## results and discussion we evaluate the effectiveness of monolingual adapters in two settings: adaptation, where multiparallel training data is available for the adapters; and in the zero-shot setting where translation is done on unseen language pairs. 2019 )" is a 58-language model. "parent" is our mnmt model trained on englishcentric data. "parent adaptation" is the same model fine-tuned for 10 epochs on the full multiparallel corpus (similar setting as "mono-1024 adaptation", but without adapters). "bilingual baselines" are models trained on one language direction only, with the same architecture as "parent". pivot-translating through english with "parent" gives an average zero-shot performance (@xmath1.39 bleu. figure 2 : comparison of bilingual adapters and monolingual adapters on the ted test sets in the adaptation setting, over 380 language pairs (sorted by available data). the y-axis shows the absolute bleu differences with the parent model (trained on english-centric data only). the trendlines are obtained by interpolating a polynomial of degree 7 over the individual points. ## ablation study we investigate the individual contribution of the encoder and decoder adapter layers at inference time. we compare the full model using mono-1024 adapters against the two options of activating (1) only encoder adapters (2) only decoder adapters. the interpolated curves for all language pairs are in figure 4 for the adaptation setting and in figure 5 for the zero-shot setting. in the adaptation setting, enabling only the decoder layers brings a median improvement of +1.03 over the parent model, while enabling only the encoder gives -7.00 bleu (versus +5.59 when both encoder and decoders are enabled). in the zero-shot setting, the contribution of the encoder is larger (+1.69) than the decoder (+0.63), compared to +2.77 when both encoder and decoder adapters are enabled. we have seen that in the adaptation case, using the encoder adapter layers alone leads to a severe drop in performance. this might indicate that -at least during the adaptation -important information is captured in the encoder's adapter layer (in line with previous reports by @xcite or that the decoder adaptation grows dependent on the encoder adapters, to the point where dropping the latter degrades the system. however, further analysis would be needed to confirm either of these hypotheses. ## conclusion this work investigated adapter modules and their compositionality for mnmt, in particular in the zero-shot setting. we introduced monolingual adapters and compared them to bilingual adapters, which we also applied to zero-shot translation. our adaptation experiments show the potential of the proposed monolingual adapters, which outperform bilingual adapters while having fewer parameters. in a zero-shot setting, we naturally compose our monolingual adapters and obtain a median improvement of +2.77 bleu points over a strong mnmt model. future work will investigate the compositional capability of these adapters, and combine domain and monolingual adapters for nmt. more generally, this work adds to the growing evidence of the flexibility of adapter layers @xcite , and their potential for lightweight fine-tuning, including in zero-shot scenarios @xcite and in a variety of tasks @xcite .