Naver Labs E urope’s Participation in the Robustness, Chat, and Biomedical Tasks at WMT 2020 (2020)

Abstract: This paper describes Naver Labs Europe’s participation in the Robustness, Chat, and Biomedical Translation tasks at WMT 2020. We propose a bidirectional German-English model that is multi-domain, robust to noise, and which can translate entire documents (or bilingual dialogues) at once. We use the same ensemble of such models as our primary submission to all three tasks and achieve competitive results. We also experiment with language model pre-training techniques and evaluate their impact on robustness to noise and out-of-domain translation. For German, Spanish, Italian, and French to English translation in the Biomedical Task, we also submit our recently released multilingual Covid19NMT model.

## introduction we participate in three german ↔ english tasks: robustness, chat, and biomedical. because these tasks allow the use of the same german-english data, we are able to submit a single model to all of them. we use adapter layers (bapna and firat, 2019) to specialize this common model on the provided in-domain data, and obtain a single multidomain model. ## our model we explore several techniques to train a model that should be able to cover all tasks with minimal adaptation. we want our model to be bidirectional, robust to noise and new domains, and to be able to translate full bilingual documents at once. ## conclusion we find that, if given enough capacity (e.g., transformer big), a single bidirectional model can give similar performance to mono-directional models of the same size. like showed by bapna and firat (2019), it is possible to perform lightweight domain adaptation using adapter layers, and achieve comparable performance to fine-tuning of the whole model. thanks to adapter layers added to our bidirectional model, we achieve competitive results on all 3 tasks with one model. mlm pre-training results for bidirectional models are inconclusive. the pre-trained model seems to be slightly more robust in some aspects, but not as robust to domain shift as one would hope. this may be due to fewer training epochs compared to our previous experiments (clinchant et al., 2019). roberta pre-training gives promising results in terms of noise robustness; it also seems to bring slight improvements in terms of domain robustness. note that the models initialized with roberta have fewer parameters than the transformer big nmt architecture. finally, document-level fine-tuning gives document-level decoding abilities to a bidirectional nmt model without degrading its sentence-level decoding performance. however, document-level decoding does not improve translation quality as measured by bleu. we also find that generating documents with the right number of sentences (i.e., same length as the input) can be challenging on some test sets.