Deep Blue Sonics’ Submission to IWSLT 2020 Open Domain Translation Task (2020)

Abstract: We present in this report our submission to IWSLT 2020 Open Domain Translation Task. We built a data pre-processing pipeline to efficiently handle large noisy web-crawled corpora, which boosts the BLEU score of a widely used transformer model in this translation task. To tackle the open-domain nature of this task, back- translation is applied to further improve the translation performance.

## introduction neural machine translation (nmt) is a wellstudied problem boosted in recent years by powerful transformer models @xcite , which find their ways to various sequence-tosequence tasks, such as automatic speech recognition (asr) @xcite , speech translation @xcite ), text-to-speech @xcite , to name a few. nevertheless, many challenges remain in training an efficient transformer nmt model in practice, such as the following ones raised in iwslt 2020 open domain translation task: handling noisy dataset hassan et al. pointed out that machine translation models are vulnerable to noise even in small quantity. in practice, manual correction of a massive corpora is prohibitive, thus calling for an automatic data cleaning pipeline. leveraging monolingual data compared to parallel corpora, monolingual data can be acquired at a much lower cost. common ways of using monolingual data include language modeling (c ¸aglar @xcite , back-translation @xcite , and dual learning @xcite , all exhibiting promising results; furthermore, they could be adopted in a complementary way when carefully designed @xcite . last but not least, massive pre-trained language models like bert perform strongly in nlp tasks like question answering, reading comprehension and text classification @xcite , motivating our attempts to incorporate them in our nmt system. domain mismatch nmt systems trained with data from specific domains may translate poorly in other domains @xcite . training the model with all available corpora, and fine-tuning it on a specific domain generally achieves best results in this domain @xcite . in open domain cases, it's impractical to keep a dedicated model or to obtain enough training data for every single domain. hence multi-domain nmt, where a single model generalizes to multiple domains, is gaining interest in recent research. for example, @xcite injected domain information into model input, leading to convincing and consistent improvements, in which domain information may be derived in both supervised and unsupervised manners. our work consists of establishing an efficient data pre-processing pipeline for large web-crawled corpora to train a transformer model for nmt and exploiting large amount of monolingual data with back-translation and language modeling. this report is organized as follows: section 2 depicts the different techniques applied to improve the official baseline model, whereas in section 3 the experiments and results are described in greater details. finally, we conclude our work and suggest a few future work directions in section 4.