HW - TSC ’s Participation in the WMT 2021 Large-Scale Multilingual Translation Task (2021)

Abstract: This paper presents the submission of Huawei Translation Services Center (HW-TSC) to the WMT 2021 Large-Scale Multilingual Translation Task. We participate in Samll Track #2, including 6 languages: Javanese (Jv), Indonesian (Id), Malay (Ms), Tagalog (Tl), Tamil (Ta) and English (En) with 30 directions under the constrained condition. We use Transformer architecture and obtain the best performance via multiple variants with larger parameter sizes. We train a single multilingual model to translate all the 30 directions. We perform detailed pre-processing and filtering on the provided large-scale bilingual and monolingual datasets. Several commonly used strategies are used to train our models, such as Back Translation, Forward Translation, Ensemble Knowledge Distillation, Adapter Fine-tuning. Our model obtains competitive results in the end.

## introduction this paper introduces our submission to the wmt 2021 large-scale multilingual translation task. we participate in small track #2, including 6 languages: javanese (jv), indonesian (id), malay (ms), tagalog (tl), tamil (ta) and english (en) with 30 directions. we consider that the officially provided dataset has the acceptable size and quality and therefore only participate in the constrained evaluation. our method is mainly based on previous works but with fine-grained data cleaning techniques and a multi-step multilingual training strategy. for each language pair, we perform multi-step data cleaning on the provided dataset and only keep a high-quality subset for training. at the same time, several training strategies are tested in a pipeline, including backward @xcite and forward @xcite translation, multilingual translation @xcite , iterative joint training @xcite , ensemble knowledge distillation @xcite , adapter fine-tuning @xcite , and ensemble @xcite . based on the task requirements, we train a single multilingual model that translates all 30 directions. we refer to @xcite and employ language tags @xcite . by combining multiple strategies, our model achieves considerable quality improvements in all directions. section 2 focuses on our data processing strategies while section 3 describes our training techniques, including model architecture and the iterative training strategy, etc. section 4 explains our experiment settings and training processes and section 5 presents our experiment results. ## results and analysis we use methods described in section 2.2 for data processing. model architecture mentioned in section 3.1 is employed to increase system diversity. on the basis of @xmath0, we use ftst data augmentation to further enhance model performance. in our experiment, due to the inference problem mentioned above, we have not seen much performance improvements. the low quality of model inference leads to poor ft results, which made no contributions to the model. and even worse, it offsets the gain brought by bt results to the model. we also found that the @xmath2, which is the same as the results observed by the industry. ## conclusion this paper presents the submissions of hw-tsc to the wmt 2021 large-scale multilingual translation task. we perform experiments with a series of pre-processing and training strategies. the effectiveness of each strategy is demonstrated. we finally achieve competitive results.