Multilingual Unsupervised Neural Machine Translation with Denoising Adapters (2021)

Abstract: We consider the problem of multilingual unsupervised machine translation, translating to and from languages that only have monolingual data by using auxiliary parallel language pairs. For this problem the standard procedure so far to leverage the monolingual data is _back-translation_, which is computationally costly and hard to tune. In this paper we propose instead to use _denoising adapters_, adapter layers with a denoising objective, on top of pre-trained mBART-50. In addition to the modularity and flexibility of such an approach we show that the resulting translations are on-par with back-translating as measured by BLEU, and furthermore it allows adding unseen languages incrementally.

## introduction two major trends have in the last years provided surprising and exciting new avenues in neural machine translation (nmt). first, multilingual neural machine translation @xcite @xcite has achieved impressive results on large-scale multilingual benchmarks with diverse sets of language pairs. it has the advantage of resulting in only one model to maintain, as well as benefiting from cross-lingual knowledge transfer. second, unsupervised neural machine translation (unmt) @xcite allows to train translation systems from monolingual data only. training bilingual unmt systems @xcite often assume high-quality in-domain monolingual data and is mostly limited to resource-rich languages. in addition to the pretraining and the denoising autoencoding, they require one or more expensive steps of back-translation @xcite in order to create an artificial parallel training corpus. in this paper, we propose a 2-step approach based on denoising adapters that enable modular multilingual unsupervised nmt without backtranslation. our approach combines monolingual denoising adapters with multilingual transfer learning on auxiliary parallel data. more precisely our denoising adapters are lightweight adapter mod-ules inserted into multilingual bart @xcite and trained with a denoising objective on monolingual data for each language separately. the first step, i.e. monolingual training, allows learning of language-specific encoding and decoding through adapter modules which can easily be composed with other languages' adapters for translation. the second step transfers mbart to multilingual unmt by plugging in our denoising adapters and then fine-tuning cross-attention with auxiliary parallel data. our approach also allows extending mbart with new languages which are not included in pretraining as shown in sect. 6.1. this means that denoising adapters can be trained incrementally after mbart fine-tuning to add any new language to the existing setup. in our experiments, we train denoising adapters for 17 diverse unsupervised languages together with 20 auxiliary languages and evaluate the final model on ted talks @xcite . our results show that our approach is on par with backtranslation for a majority of languages while being more modular and efficient. moreover, using denoising adapters jointly with back-translation further improves unsupervised translation performance. contributions in summary, we make the following contributions: 1) we propose denoising adapters, monolingually-trained adapter layers to leverage monolingual data for unsupervised machine translation. 2) we introduce a 2-step approach for multilingual unmt using denoising adapters and multilingual fine-tuning of mbart's cross-attention with auxiliary parallel data. 3) we conduct experiments on a large set of language pairs showing effectiveness of denoising adapters with and without back-translation. 4) finally, we provide further analysis to the use of denoising adapters such as extending mbart with completely new languages. ## denoising adapters for multilingual unsupervised mt we address the limitations of existing methods mentioned above by proposing denoising adapters for multilingual unsupervised mt. denoising adapters are monolingually-trained language adapters, therefore eliminating the dependence on parallel data. they allow learning and localizing general-purpose language-specific representations on top of pretrained models such as mbart. these denoising adapters can then easily be used for multilingual mt, including unsupervised machine translation without back-translation. architecture for our denoising adapters, following @xcite , we use a simple feedforward network with a relu activation. each adapter module also includes a parametrized normalization layer that acts on the input of the adapter and allows learning the activation pattern of transformer layers. figure 2 shows the architecture of an adapter layer. more formally, a denoising adapter module d i at layer i consists of a layernormalization ln of the input z @xmath0, followed by a down-projection w @xmath1, a non-linear function and a up projection w @xmath2: bias terms are omitted for clarity. for simplicity, we denote as similarly to @xcite , we insert an adapter module into each layer of the transformer encoder and decoder, after the feed-forward self attention feed forward feed forward mbart decoder (xn) mbart encoder (xn) en e fr e de e en d fr d de d [en] <@xmath3: denoising autoencoding with monolingual data self attention feed forward feed forward mbart decoder (xn) mbart encoder (xn) [en] adapters are so @xmath4: multilingual mt training with parallel data in 3b, all denoising adapters that are trained in 3a are frozen, and only the cross-attention of mbart @xcite is updated with auxiliary parallel data. ## experimental setup dataset we use ted talks @xcite to create an english-centric (en) multilingual dataset by picking 20 languages with different training size ranging from 214k (ar) to 18k (hi) parallel sentences. for multilingual unmt evaluation, in addition to the 20 training languages, we select 17 "unsupervised" languages, 6 of which are unknown to mbart @xcite . to train the denoising adapters, we use wikipedia foot_1 and news crawl foot_2 with maximum 20m sentences per language. details of languages and training datasets are given in appendix a.1 baselines we compare our approach with the following baselines: (1) bilingual, baseline bilingual models trained on ted talks. these are small transformer models trained separately on each language direction, using the same settings as @xcite . note that these models do not have any pretraining and they are trained from scratch. the bilingual models and all the mbart variants are fine-tuned on the same english-centric multilingual parallel data. ## results for @xmath6, the two baselines mbart-ft and task adapters are quite decent: the ability of mbart to encode the unsupervised source languages and its transfer to nmt using auxiliary parallel data provide good multilingual unsupervised nmt performance. among the two baselines, task-specific mt adapters better mitigate catastrophic forgetting, ensuring the model does not overfit to the supervised languages and to benefit more from multilingual fine-tuning which results in +5.4 bleu compared to standard finetuning. our approach, however, outperforms the two mbart baselines and the bilingual models: denoising adapters are superior for all languages compared to mbart-ft and task adapters and result in respectively +8.6 and +3.2 bleu on average. finally, it even performs better than the supervised bilingual models for most languages (all but es and nl). for the @xmath7, the two baselines mbart-ft and task adapters are ineffective, showing the limitation of mbart pretraining for multilingual unmt when translating from english. a possible explanation for this is the fact that these models have learnt to encode english with only auxiliary target languages; and the transfer from mbart to nmt has made the decoder forget how to generate text in the 11 unsupervised languages we are interested in. fig. 4 shows unsupervised translation performance for @xmath8. as opposed to our approach, the low start in mbart-ft and the quick drop in task adapters confirm the forgetting in impact of back-translation 3rd blocks (3) in for @xmath9, although back-translation slightly improves the results, the overall impact of backtranslation is very limited for all models including our approach. interestingly, for ur the backtranslation decreased the performance. we relate this to the domain difference between test (ted talks) and back-translated data (wikipedia/news). here, denoising adapters without back-translation still provide superior unsupervised translation quality compared to baselines even after the backtranslation. for @xmath10, the back-translation significantly increased translation results: +15.0, +16.2 and +3.0 bleu for mbart-ft, task adapters and de-noising adapters respectively. we hypothesize that the huge boost in the baselines scores is due to the fact that training on the back-translated parallel data allows these models to recover generation ability in the target languages. however, our approach outperforms baselines in all languages, showing that denoising adapters can be used jointly with back-translation for further improvements. finally, denoising adapters without back-translation (2) are still competitive with the mbart baselines. 6 analysis and discussion 6.1 denoising adapters for languages unknown to mbart all the languages considered so far (in ## conclusion we have presented denoising adapters, adapter modules trained on monolingual data with a denoising objective, and a 2-step approach to adapt mbart by using these adapters for multilingual unsupervised nmt. our experiments conducted on a large number of languages show that denoising adapters are very effective for unsupervised translation even without the need of back-translation. moreover, denoising adapters are complementary with back-translation; using them jointly improves the translation quality even further. we have also demonstrated that for a language new to mbart, 5 sacrebleu @xcite signature: bleu+c.mixed+#.1+s.exp+tok.13a+v.1.5.0 6 comet model: wmt20-comet-da 7 bert score hash code: roberta-large_l17_no-@xmath13.3.10 (@xmath14.10.0)-rescaled_fast-tokenizer denoising adapters offer an efficient way to extend mbart incrementally. finally, although it is designed for unsupervised nmt, our approach still reaches competitive performance in supervised translation in a multilingual nmt setup. for the future direction, translating between two unseen languages may be considered as a natural extension of our work. as preliminary experiment, we addressed a language pair including two languages of the unsupervised setup: spanish (es) and dutch (nl). we inserted denoising adapters of those languages to encoder/decoder and directly used this model without further training for @xmath15. although our auxilliary language pairs with parallel data are english-centric, these two models perform at a decent level (15.4, 7.2 bleu respectively) and they could be a good starting point for further improvements. another direction is to apply denoising adapters to domain adaptation, a use-case where back-translation is a standard solution to leverage monolingual data. we provide supplementary material to facilitate future research. 8 we use the fairseq library @xcite to conduct our experiments. the hyperparameters used for fairseq are given in