HW - TSC ’s Submissions to the WMT 2022 General Machine Translation Shared Task (2022)

Abstract: This paper presents the submissions of Huawei Translate Services Center (HW-TSC) to the WMT 2022 General Machine Translation Shared Task. We participate in 6 language pairs, including Zh↔En, Ru↔En, Uk↔En, Hr↔En, Uk↔Cs and Liv↔En. We use Transformer architecture and obtain the best performance via multiple variants with larger parameter sizes. We perform fine-grained pre-processing and filtering on the provided large-scale bilingual and monolingual datasets. For medium and highresource languages, we mainly use data augmentation strategies, including Back Translation, Self Training, Ensemble Knowledge Distillation, Multilingual, etc. For low-resource languages such as Liv, we use pre-trained machine translation models, and then continue training with Regularization Dropout (R-Drop). The previous mentioned data augmentation methods are also used. Our submissions obtain competitive results in the final evaluation.

## introduction this paper introduces our submissions to the wmt 2022 general machine translation shared task. we participate in 6 language pairs including chinese/english (zh↔en), russian/english (ru↔en), ukrainian/english (uk↔en), croatian/english (@xmath0, ukrainian/czech(uk↔cs), and livonian/english (liv↔en). for zh↔en translation, we use additional in-house in-domain data, so the final submission for this language pair is unconstrained. for liv↔en translation, although we did not use additional data, we used m2m-100 @xcite as the pretrained model, and the final submission is also unconstrained. all other languages pair participate in the constrained evaluation. our method is mainly based on previous works @xcite @xcite but with fine-grained data cleansing techniques and language-specific optimizations. for each language pair, we perform multi-step data cleansing on the provided dataset and only keep a high-quality subset for training. at the same time, several strategies are tested in a pipeline, including backward @xcite and forward @xcite translation, multilingual translation @xcite , iterative joint training @xcite , r-drop, pretrained nmt model, ensemble knowledge distillation @xcite , fine-tuning @xcite , ensemble @xcite , and post-processing. our system report includes four parts. section 2 focuses on our data processing strategies while section 3 describes our training details. section 4 explains our experiment settings and training processes and section 5 presents the results. ## system overview our method basically follows our previous training strategies @xcite , such as commonly used back-translation @xcite , iterative joint training @xcite , multilingual enhancement @xcite @xcite , data diversification @xcite (for details, please refer to our previous work @xcite al. (2021) ), ensemble and fine-tuning, etc. we will not detail these strategies in this report. the following paper focuses on new strategies used in this year. ## experiment settings we use the open-source fairseq @xcite for training and sacrebleu @xcite to measure system performances. the main parameters are as follows: each model is trained using 8 v100 gpus. the size of each batch is set as 2048, parameter update frequency as 4, and learning rate as 5e-4 @xcite . the number of warmup steps is 4000, and model is saved every 1000 steps. the architecture we used is described in section 3.1. we adopt dropout, and the rate varies across different language pairs. r-drop is used in model training, and we set parameter λ to 5 for all language pairs. ## conclusion this paper presents the submissions of hw-tsc to the wmt 2022 general machine translation task. we participate in six language pairs and perform experiments with a series of pre-processing and training strategies. the effectiveness of each strategy is demonstrated. our experiments show that in very low-resource scenarios, fine-tuning on pre-trained nmt models can significantly improve system performance. r-drop also brings stable improvement across languages. certainly, commonlyused data augmentation strategies are still effective for model training. our submissions finally achieve competitive results in the evaluation.