Huawei’s Submissions to the WMT 20 Biomedical Translation Task (2020)

Abstract: This paper describes Huawei’s submissions to the WMT20 biomedical translation shared task. Apart from experimenting with finetuning on domain-specific bitexts, we explore effects of in-domain dictionaries on enhancing cross-domain neural machine translation performance. We utilize a transfer learning strategy through pre-trained machine translation models and extensive scope of engineering endeavors. Four of our ten submissions achieve state-of-the-art performance according to the official automatic evaluation results, namely translation directions on English<->French, English->German and English->Italian.

## introduction neural machine translation (nmt) models built upon the transformer architecture @xcite start to dominate the leader board of wmt biomedical shared tasks in recent years @xcite . in-domain data (parallel and monolingual corpora) have been widely used in finetuning general domain nmt models. despite ongoing improvements on the translation quality observed from recent biomedical shared tasks, domain adaptation remains an open problem. the in-domain data is hard to obtain and, as a consequence, greatly limits the cross-domain translation capability an nmt model can offer. domain terminologies, on the other hand, are regarded as critical resources to improve the quality of machine translation by mitigating effects of scarce in-domain bitexts @xcite . however, few research works leverage domain-specific terminologies (or dictionaries) in training cross-domain nmt systems. in this paper, we present the system architecture and research approaches underpinning huawei's submissions to the wmt20 biomedical translation task. we implement two nmt systems to maximize the performances of the shared task. the system i is an in-house nmt system built upon the transformer-big architecture @xcite and trained using general domain data. we explore means to enhance cross-domain coverage of an nmt model by finetuning the nmt model with in-domain bitexts. we also investigate the effects of domain dictionaries in this domain adaptation process. reusing pre-trained models has been regarded as an efficient way of transfer learning. pre-trained nmt models @xcite ) are adopted in the system ii to this end. all nmt systems are evaluated against the test set released in the wmt19 biomedical shared task. we submitted translated results for a total of ten language directions between english (en) and other five languages including french (fr), german (de), italian (it), russian (ru) and chinese (zh). four of the submissions achieve the best bleu scores according to the official automatic evaluation results. substantial increases in bleu scores are recorded in translation directions of @xmath0.9 bleu), @xmath1.5 bleu), and @xmath2.8 bleu) compared to our submissions last year @xcite . the improvements on en⇔de can be ascribed to strong pre-trained nmt baseline models and a series of optimization techniques, for example, in-domain data augmentation and a reranking method with strong language models. high-quality in-domain data and large-scale back-translation contribute to the improvements of the @xmath3. wmt20. 1 the in-domain data consist of bitexts from emea @xcite , ufal, foot_1 pubmed, and medline. 3 the data is processed by methods in the next section. the test data for the system i are from the wmt19 shared task. ## the data the data used for finetuning the system ii are different from those for the system i. the system ii only focuses on medline as we discovered it is the most effective ind data for this shared task. the development (dev.) set for the system ii is the ok-aligned test data from the wmt19 biomedical shared task. a batch of monolingual medline data in english dated before july 2018 has been extracted to provide a basis for data augmentation and noisy channel model reranking @xcite . it produces the augmented ind data for the @xmath4." in ## the approaches the proposed systems are finetuned and enhanced using the following methods. all models are trained on tesla v100 gpus. systems i and ii use batch sizes of 6,144 and 8,000 tokens respectively in the finetuning process. ## experimental results the systems are trained with ood data and finetuned using ind data to produce the submitted results. we benchmarked the submissions using wmt19 test data. the bleu scores are calculated using the mteval script from moses @xcite . results are shown in ## conclusion this paper depicts huawei's submissions to the wmt20 biomedical shared task. for all ten translation directions, we have explored the effects of using ind bitexts and dictionaries on enhancing the performances of cross-domain nmt. we have demonstrated the benefits of the transfer learning strategy of reusing pre-trained nmt models. four of our ten submissions achieve the best records according to the released wmt20 official results.