Summer: W e C hat Neural Machine Translation Systems for the WMT 22 Biomedical Translation Task (2022)

Abstract: This paper introduces WeChat’s participation in WMT 2022 shared biomedical translationtask on Chinese→English. Our systems are based on the Transformer(Vaswani et al., 2017),and use several different Transformer structures to improve the quality of translation. In our experiments, we employ data filtering, data generation, several variants of Transformer,fine-tuning and model ensemble. Our Chinese→English system, named Summer, achieves the highest BLEU score among all submissions.

## introduction this article describes the wechat's participation in wmt 2022 shared biomedical translation task on @xmath0. we improve the translation quality of the system by increasing the diversity of model structure and data, fine-tuning the model with in-domain data, inserting tags at the beginning of each source sentence and selecting models with high diversity and good performance for ensemble. for model architectures, our system adopt big and deep transformer models which contain 10layer and 20-layer encoders, 10240 and 4096 filter sizes, respectively, with transformer-big setting @xcite . in order to increase the diversity of the model, we use structures such as average attention transformer (aan) @xcite and mixed-aan transformer architecture @xcite in the decoder part. for data generation, we use backtranslation @xcite , knowledge distillation @xcite , and forwardtranslation @xcite to improve data quality. and we use some data augmentation methods to improve the model robustness, such as adding synthetic noise and dynamic top-p sampling @xcite . furthermore, according to the different sources of the corpora, we add tags at the beginning of the source sentence to perform domain adaptation. for fine-tuning, we use in-domain bilingual corpus to fine-tune models from the general domain to the biomedical domain, and use target denoising @xcite to improve the diversity of models and mitigate training-generation discrepancy. for model ensemble, we use self-bleu @xcite to evaluate the similarity between models. we take the prediction of one model as the reference and use the prediction of the other model to calculate the bleu score. the higher the self-bleu score, the lower the diversity of the models. in the remainder of this paper, we start with presenting the data strategy in section 2. then we describe our system details in section 3. section 4 presents the experimental results. finally, we conclude our work in section 5. ## data in this section, we introduce the details of bilingual and monolingual data used in this shared task. ## system overview in this section, we introduce the details of our system used in the wmt 2022 shared biomedical translation task. our system adopts data filtering, data generation, model architectures, fine-tuning and ensemble. ## conclusion we introduce wechat's participation in wmt 2022 shared biomedical translation task on @xmath1. our system is based on the transformer @xcite , and uses several different transformer structures such as average attention and mixed-aan to improve the performance. we use several data augmentation methods such as iterative back-translation, knowledge distillation, forward-translation and synthetic noise. we use tags to assist the model in domain learning and use in-domain fine-tuning with target denoising to domain transfer. finally a self-bleu based ensemble method is used for model ensemble. overall, our system achieves 46.91 bleu score on wmt21 ok-aligned biomedical test set, and we achieve the highest bleu score among all submissions.