Tencent AI Lab Machine Translation Systems for the WMT 21 Biomedical Translation Task (2021)

Abstract: This paper describes the Tencent AI Lab submission of the WMT2021 shared task on biomedical translation in eight language directions: English-German, English-French, English-Spanish and English-Russian. We utilized different Transformer architectures, pretraining and back-translation strategies to improve translation quality. Concretely, we explore mBART (Liu et al., 2020) to demonstrate the effectiveness of the pretraining strategy. Our submissions (Tencent AI Lab Machine Translation, TMT) in German/French/Spanish⇒English are ranked 1st respectively according to the official evaluation results in terms of BLEU scores.

## introduction this paper describes the tencent ai lab submission of the wmt2021 shared task on biomedical translation. last year, we participated in three translation tasks: news @xcite , chat @xcite , and biomedical @xcite . in biomedical translation, we adopt deep transformer @xcite , hybrid transformer @xcite and data rejuvenation 1 @xcite . this year, we participated in eight language directions: english-german (en-de), english-french (en-fr), english-spanish (en-es) and english-russian (en-ru) in the biomedical translation. in this paper, we also apply the pre-train and fine-tune paradigm for the biomedical translation task. the pre-train model is first trained on the the large-scale monolingual data in a self-supervised manner, then is fine-tuned on downstream bilingual data. specifically, we adopt the encoder-decoder pre-trained model mbart @xcite to implement the pre-training strategy. ## system our systems are implemented with transformer @xcite and the pre-trained model mbart. the training details of theses models are described in section 4. ## data in this section, we present the training and validation data used in our system. besides the in-domain data provided by organisers, we collect the out-of-domain bilingual data from wmt news translation shared task. ## experiments for the corpus filtering, we follow @xcite to filter duplicate sentence pairs (khayrallah and koehn, 2018), sentence pairs with wrong language @xcite or length problem @xcite . for the synthetic bilingual data generation, we adopt iterative knowledge distillation @xcite to improve the translation quality. our iterative knowledge distillation is performed with 3 big transformer teachers and 3 iterations. we also try to use the right-to-left (r2l) training @xcite but fail in achieving significant improvements on the test sets. we follow @xcite to train the big and large transformer models. specifically, we first use the combination of the out-of-domain data and the in-domain data to train the teacher model. then we use the teacher model to generate the synthetic bilingual data. finally, we train the student model on the combination of the synthetic and real bilingual data @xcite . the learning rate is set to 0.0007. all models are trained for 600k steps on 8 tesla v100 gpus where each is allocated with a batch size of 8192 tokens. ## official results the official automatic evaluation results of our submissions for wmt 2021 biomedical translation task are shown in ## conclusion in this paper, we present tencent ai lab machine translation systems for the wmt21 biomedical translation shared task. we participated in eight language directions: english-german (en-de), english-french (en-fr), english-spanish (en-es) and english-russian (en-ru). our systems german/french/spanish⇒english are ranked 1st according to the official evaluation results in terms of bleu scores. it is worth mentioning that most advanced technologies reported in this paper are also adapted to our systems for news translation task @xcite , which achieve the 1st rank in chinese⇒english task. in the future, we plan to explore non-autoregressive machine translation (nat) models to improve the system performance @xcite @xcite and will integrate these advanced techniques in our tencent transmart system @xcite 21 .