Tencent Translation System for the WMT 21 News Translation Task (2021)

Abstract: This paper describes Tencent Translation systems for the WMT21 shared task. We participate in the news translation task on three language pairs: Chinese-English, English-Chinese and German-English. Our systems are built on various Transformer models with novel techniques adapted from our recent research work. First, we combine different data augmentation methods including back-translation, forward-translation and right-to-left training to enlarge the training data. We also apply language coverage bias, data rejuvenation and uncertainty-based sampling approaches to select content-relevant and high-quality data from large parallel and monolingual corpora. Expect for in-domain fine-tuning, we also propose a fine-grained “one model one domain” approach to model characteristics of different news genres at fine-tuning and decoding stages. Besides, we use greed-based ensemble algorithm and transductive ensemble method to further boost our systems. Based on our success in the last WMT, we continuously employed advanced techniques such as large batch training, data selection and data filtering. Finally, our constrained Chinese-English system achieves 33.4 case-sensitive BLEU score, which is the highest among all submissions. The German-English system is ranked at second place accordingly.

## introduction in this year's news translation task, our translation team at tencent ai lab & cloud xiaowei participated in three shared tasks, including chinese⇒english, english⇒chinese and german⇒english. we used the same data strategies, model architectures and corresponding techniques for all tasks. we hypothesized that different models have their own strengths and characteristics, and they can complement each other. thus, we built various advanced nmt models which mainly differ in training data and model architectures. these models (i.e. deep, large and large-ffn) are empirically designed based on transformer-deep which has proven more effective than the transformer-big models @xcite . in addition to the original multi-head self-attention, we also proposed a mixed attention strategy by combining relative position with the original one, which extends the self-attention to efficiently consider representations of the relative positions. we use a variation of relative position, the random attention (ran) @xcite . as a results, we combined these models at transductive fine-tuning stage. in terms of data augmentation, we adapt backtranslation (bt) @xcite , forwardtranslation (ft) @xcite and rightto-left (r2l) @xcite techniques to generate large-scale synthetic training data. different from the standard back-translation, we add noise to the synthetic source sentence in order to take advantage of large-scale monolingual text. in addition, we used tagged bt mechanism (i.e. add a special token to the synthetic source sentence) to help the model better distinguish the originality of data. all the parallel data and a large amount of monolingual data are used in corresponding data augmentation methods, and finally we combine them together to build strong baseline models. to enhance the domain-specific knowledge, we introduced approaches at both data and model levels. first, we employed a hybrid data selection method @xcite to produce different finetuning datasets. more specifically, we apply language coverage bias @xcite , data rejuvenation @xcite and uncertaintybased sampling @xcite to select content-relevant and high-quality data from parallel and monolingual corpora. the news texts contain a number of sub-genres such as covid-19 and government report. thus, we fine-tuned a domainspecific model translate each sub-genre of text in the test set (i.e. "one domain one model"). we take advantage of the combination methods to further improve the translation quality. the "greedy search ensemble algorithm" @xcite is used to select the best combinations from single models. furthermore, we propose an multi-model & multi-iteration transductive ensemble (m 2 te) method based on the translation results of the ensemble models. first, we divided models into two parts. second, each part produced syntactic parallel testsets which is used to fine-tune another part of models. we repeated this procedure for n times. this paper is structured as follows: section 2 describes our advanced model architectures. we then present the data statistics and processing methods in section 3. the methods and ablation study are detailed in section 4 followed by final experimental results in section 5. finally, we conclude our work in section 6. ## model architecture in this section, we mainly introduced three model architectures, which are empirically adapted from transformer @xcite . ## method and ablation study in this section, we conducted a comprehensive ablation study of the techniques used in this competition. we reported results on the chinese⇒english task using the constrained data. ## final results in this section, we combined all the presented methods and techniques (detailed in section 4) together and showed the final results in ## conclusion this paper presents the tencent translation systems for wmt2021 news translation tasks. we investigate various deep architectures to build strong baseline models. then popular data augmentation