The Path to Continuous Domain Adaptation Improvements by HW - TSC for the WMT 23 Biomedical Translation Shared Task (2023)

Abstract: This paper presents the domain adaptation methods adopted by Huawei Translation Service Center (HW-TSC) to train the neural machine translation (NMT) system on the English↔German (en↔de) language pair of the WMT23 biomedical translation task. Our NMT system is built on deep Transformer with larger parameter sizes. Based on the biomedical NMT system trained last year, we leverage Curriculum Learning, Data Diversification, Forward translation, Back translation, and Transductive Ensemble Learning to further improve system performance. Overall, we believe our submission can achieve highly competitive result in the official final evaluation.

## introduction machine translation (mt) @xcite refers to the automatic translation of text from one language to another. the wmt23 biomedical translation task aims to evaluate the performance of mt systems in the biomedical domain. due to the lack of sufficient in-domain data, domain adaptation @xcite has naturally become the main research direction of this task. this paper presents the domain adaptation methods adopted by hw-tsc to train the nmt @xcite system on en↔de language pair of the wmt23 biomedical translation task. our method is mainly based on previous works @xcite @xcite . we try to train a domain classifier to select biomedical data from general data, then perform multi-step data cleaning on the selected in-domain data and keep only a high-quality subset for training. based on the biomedical nmt system trained last year, we leverage curriculum learning @xcite , data diversification @xcite , forward translation @xcite , back translation @xcite , and transductive ensemble learning @xcite ) to further improve system performance. our system report includes four parts. section 2 focuses on our data processing strategies while section 3 describes our training details. section 4 explains our experiment settings and training processes, and section 5 presents the results. ## experiment settings we use the open-source fairseq @xcite for training, then we use sacrebleu @xcite and multi-eval tool foot_1 to measure system performances. the main parameters are as follows: each model is trained using 8 a100 gpus, batch size is 6144, parameter update frequency is 1, and learning rate is 5e-4. the number of warmup steps is 4000, and model is saved every 1000 steps. the architecture we used is described in section 3.1. we adopt dropout @xcite , and the rate varies across different training phases. when the training data is higher than tens of millions, the dropout ratio is set to 0.1, otherwise it is set to 0.3. ## results regarding en↔de, we use curriculum learning (cl), data diversification (dd), forward translation (ft), back translation (bt), and transductive ensemble learning (tel). the evaluation results of @xmath0. we see that cl can stably bring 3 sacrebleu and multi-eval improvement, while dd, ft & st and tel can further slightly improve sacrebleu and multi-eval. our final @xmath1.48 and 48.75 sacrebleu, 41.22 and 49.91 multi-eval respectively. @xmath2.11 37.80 44.45 45.50 + cl 40.11 40.89 47.77 48.89 + dd, ft & st 40.23 41.00 48.60 49.76 + tel 40.48 41.22 48.75 49.91 ## conclusion this paper presents the submission of hw-tsc to the wmt23 biomedical translation task. we participate in en↔de language pair and perform a series of domain adaptation experiments based on the biomedical nmt system trained last year. the effectiveness of each domain adaptation method is demonstrated. our experiments show that domain adaptation methods are effective for model training.