W e C hat Neural Machine Translation Systems for WMT 21 (2021)

Abstract: This paper introduces WeChat AI’s participation in WMT 2021 shared news translation task on English->Chinese, English->Japanese, Japanese->English and English->German. Our systems are based on the Transformer (Vaswani et al., 2017) with several novel and effective variants. In our experiments, we employ data filtering, large-scale synthetic data generation (i.e., back-translation, knowledge distillation, forward-translation, iterative in-domain knowledge transfer), advanced finetuning approaches, and boosted Self-BLEU based model ensemble. Our constrained systems achieve 36.9, 46.9, 27.8 and 31.3 case-sensitive BLEU scores on English->Chinese, English->Japanese, Japanese->English and English->German, respectively. The BLEU scores of English->Chinese, English->Japanese and Japanese->English are the highest among all submissions, and that of English->German is the highest among all constrained submissions.

## introduction we participate in the wmt 2021 shared news translation task in three language pairs and four language directions, @xmath0, english↔japanese, and @xmath1. in this year's translation tasks, we mainly improve the final ensemble model's performance by increasing the diversity of both the model architecture and the synthetic data, as well as optimizing the ensemble searching algorithm. diversity is a metric we are particularly interested in this year. to quantify the diversity among different models, we compute self-bleu @xcite from the translations of the models on the valid set. to be precise, we use the translation of one model as the hypothesis and the translations of other models as references to calculate an aver-age bleu score. a higher self-bleu means this model is less diverse. for model architectures @xcite @xcite , we exploit several novel transformer variants to strengthen model performance and diversity. besides the pre-norm transformer, the post-norm transformer is also used as one of our baselines this year. we adopt some novel initialization methods @xcite to alleviate the gradient vanishing problem of the post-norm transformer. we combine the average attention transformer (aan) @xcite and multi-head-attention @xcite to derive a series of effective and diverse model variants. furthermore, talking-heads attention @xcite ) is introduced to the transformer and shows a significant diversity from all the other variants. for the synthetic data generation, we exploit the large-scale back-translation @xcite ) method to leverage the target-side monolingual data and the sequence-level knowledge distillation @xcite to leverage the source-side of bilingual data. to use the source-side monolingual data, we explore forward-translation by ensemble models to get general domain synthetic data. we also use iterative in-domain knowledge transfer @xcite to generate in-domain data. furthermore, several data augmentation methods are applied to improve the model robustness, including different token-level noise and dynamic top-p sampling. for training strategies, we mainly focus on scheduled sampling based on decoding steps @xcite , the confidence-aware scheduled sampling @xcite @xcite , the target denoising @xcite ) method and the graduated label smoothing @xcite for in-domain finetuning. for model ensemble, we select high-potential candidate models based on two indicators, namely model performance (bleu scores on valid set) and model diversity (self-bleu scores among all other models). furthermore, we propose a search algorithm based on the self-bleu scores between the candidate models with selected models. we observed that this novel method can achieve the same bleu score as the brute force search while saving approximately 95% of search time. this paper is structured as follows: sec. 2 describes our novel model architectures. we present the details of our systems and training strategies in sec. 3. experimental settings and results are shown in sec. 4. we conduct analytical experiments in sec. 5. finally, we conclude our work in sec. 6. ## model architectures in this section, we describe the model architectures used in the four translation directions, including several different variants for the transformer @xcite . ## system overview in this section, we describe our system used in the wmt 2021 news shared task. we depicts the overview of our nmt system in figure 2 , which can be divided into four parts, namely data filtering, large-scale synthetic data generation, in-domain finetuning, and ensemble. the synthetic data generation part further includes the generation of general domain and in-domain data. next, we proceed to illustrate these four parts. ## analysis to verify the effectiveness of our approach, we conduct analytical experiments on model variants, finetune methods, and ensemble strategies in this section. ## conclusion we investigate various novel transformer based architectures to build robust systems. our systems are also built on several popular data augmentation methods such as back-translation, knowledge distillation and iterative in-domain knowledge transfer. we enhance our system with advanced finetuning approaches, i.e., target denoising, graduated label smoothing and confidence-aware scheduled sampling. a boosted self-bleu based model ensemble is also employed which plays a key role in our systems. our constrained systems achieve 36.9, 46.9, 27.8 and 31.3 case-sensitive bleu scores on @xmath2, @xmath3, @xmath4, respectively. the bleu scores of @xmath5, @xmath6, and that of @xmath7.