Tencent AI Lab Machine Translation Systems for WMT 20 Chat Translation Task (2020)

Abstract: This paper describes the Tencent AI Lab’s submission of the WMT 2020 shared task on chat translation in English-German. Our neural machine translation (NMT) systems are built on sentence-level, document-level, non-autoregressive (NAT) and pretrained models. We integrate a number of advanced techniques into our systems, including data selection, back/forward translation, larger batch learning, model ensemble, finetuning as well as system combination. Specifically, we proposed a hybrid data selection method to select high-quality and in-domain sentences from out-of-domain data. To better capture the source contexts, we exploit to augment NAT models with evolved cross-attention. Furthermore, we explore to transfer general knowledge from four different pre-training language models to the downstream translation task. In general, we present extensive experimental results for this new translation task. Among all the participants, our German-to-English primary system is ranked the second in terms of BLEU scores.

## introduction although neural machine translation @xcite @xcite has achieved great progress in recent years, translating conversational text is still a challenging task due to its inherent characteristics such as discourse awareness @xcite , informality @xcite and personality @xcite . this is a task-oriented chat translation task @xcite , which aims to translating conversations between customers and agents. as a customer and an agent can respectively natively speak in german and english, the systems should translate the customer's utterances in german⇒english (de⇒en) while the agent's in german⇐english (de⇐en). in this paper, we present our submission to the novel task in de⇔en. we explore a breadth of established techniques for building chat nmt systems. specifically, our systems are based on the self-attention networks including both sentenceand document-level transformer @xcite . besides, we investigated non-autoregressive translation (nat) models augmented with our recently proposed evolved cross-attention @xcite . technically, we used the most recent effective strategies including back/forward translation, data selection, domain adaptation, batch learning, finetuning, model ensemble and system combination. particularly, we proposed a multi-feature data selection on large general-domain data. we not only use three language models (i.e. n-gram, transformer and bert based lms) to filter low-quality sentences, but also employ feature decay algorithms (fda, @xcite to select domain-relevant data. in addition, we explore large batching @xcite for this task and found that it can significantly outperform models with regular batching settings. to alleviate the low-resource problem, we employ large scale pre-training language models including monolingual bert @xcite , bilingual xlm @xcite and multilingual mbart @xcite , of which knowledge can be transferred to chat translation models. 1 for better finetuning, we investigate homogenous and heterogeneous strategies (e.g. from sentence-level to document-level architectures). simultaneously, we conduct fully-adapted data processing, model ensemble, back/forward translation and system combination. according to the official evaluation results, our systems in de⇒en and de⇐en are respectively ranked 2nd and 4th. 2 furthermore, a number of advanced technologies reported in this paper are also adapted to our systems for biomedical translation @xcite and news translation @xcite tasks, which respectively achieve up to 1st and 2nd ranks in terms of bleu scores. though our empirical experiments, we gain some interesting findings on the chat translation task: the rest of this paper is organized as follows. section 2 introduces data statistics and our processing methods. in section 3, we present our system with four different models: sentence-level nmt, document-level nmt, non-autoregressive nmt and nmt with pre-training lms. section 4 describes advanced technique integrated into our systems such as data selection and system combination. in section 5, we reports ablation study and experimental results, which is followed by our conclusion in section 6. ## models we adopt four different model architectures namely: sent, doc, nat and pretrain. ## approaches we integrated advanced techniques into our systems, including data selection, model ensemble, back/forward translation, larger batch learning, finetuning, and system combination. ## experimental results unless otherwise specified, reported bleu scores are calculated based on combined and tokenized validation set by muti-bleu.perl, which is different from the official evaluation method. ## conclusion the paper is a system description for the tencent ai lab's entry into the wmt2020 chat translation task. we explore a breadth of established techniques for building chat translation systems. the paper includes numerous models making use of sentence-level, document-level, non-autoregressive nmt. it also investigates a number of advanced techniques including data selection, model ensemble, finetuing, back/forward translation and initialization using a pretrained lms. we present extensive experimental results and hope that this work could help both mt researchers and industries to boost the performance of discourse-aware mt systems @xcite .