HW - TSC Systems for WMT 22 Very Low Resource Supervised MT Task (2022)

Abstract: This paper describes the submissions of Huawei translation services center (HW-TSC) to the WMT22 Very Low Resource Supervised MT task. We participate in all 6 supervised tracks including all combinations between Upper/Lower Sorbian (Hsb/Dsb) and German (De). Our systems are build on deep Transformer with a large filter size. We use multilingual transfer with German-Czech (De-Cs) and German-Polish (De-Pl) parallel data. We also utilize regularized dropout (R-Drop), back translation, fine-tuning and ensemble to improve the system performance. According to the official evaluation results on OCELoT, our supervised systems of all 6 language directions get the highest BLEU scores among all submissions. Our pre-trained multilingual model for unsupervised De2Dsb and Dsb2De translation also gain highest BLEU.

## introduction in this paper, we describe our very low resource supervised mt systems for all combinations between hsb, dsb and de. we first select a base pre-trained multilingual model and then fine-tune it. as we focus primarily on the supervised task, we only apply our pre-trained multilingual system with zero-shot for unsupervised task submissions. as show in wmt21 shared task @xcite , most participants use de-cs for transfer or combine de-cs with the low resource pairs to build a multilingual system. finetuning based on a multilingual pre-trained @xcite model has shown very promising results for low resource tasks. we add de-pl data and train our multilingual pre-trained model to transfer the low resource pairs. this paper is structured as follows: we describe our data source and data pre-processing method in section 2. we detail the model structure and 1 https://ocelot-wmt22.mteval.org method we used in section 3. we then present the final experiments in section 4 and section 5, and finally we conclude our work in section 6. ## experimental settings during the training phase, we use pytorch-based fairseq foot_0 @xcite open-source framework. each model is trained using 8-v100 with a batch size of 2048 tokens for each gpu. dropout was set to 0.1 for pre-train multilingual model, and 0.3 for fine-tuning model. the label smoothing rate @xcite drop training strategy. for pre-training the multilingual model, the update frequency, the learning rate and warm-up steps are 4, 5e-4 and 4000 respectively; for fine-tuning the model, the update frequency and the learning rate is 1 and 1e-4 without warm-up. in the evaluation phase, we use marian foot_1 (junczys-dowmunt et al., 2018) for decoding and then calculate the sacrebleu foot_2 (post, 2018) on the wmt21 dev set, test set and wmt22 dev test to measure the performance of each model. ## experimental result first of all, we test pre-trained multilingual models without fine-tune and get quiet low scores. next, we fine-tune all of our pre-trained models with bitext and then select a best one according to the bleu scores for every task. ## analysis 6.1 pre-trained model due to the availability of large amount of de-cs and de-pl data and the similarities between hsb/dsb and cs/pl, we pre-train several multilingual models with different strategies, and then finetune with very low resource bilingual data. we choose the best strategy for every language direction for further fine-tuning. specifically, we design three pre-trained multilingual models: a one-tomany model trained with the de2cs and de2pl bilingual data, a many-to-one model trained with the cs2de and pl2de bilingual data, and a many-tomany model trained with the cs2de, de2cs, pl2de and de2pl bilingual data. for each corpus we use a different tag to differentiate. furthermore, we train a de-cs model as done by last year's participants, in order to get better comparison results. ## conclusion this paper presents our translation systems for the wmt22 very low resource supervised mt task. during the experiment, we use multilinguals to build our base translation system, and then use forward translation and back translation methods to expand the size of training data for a better translation system. we also adopt test set fine-tuning and ensemble to further improve the system performance. finally, according to the official evaluation results on ocelot, our submission achieves the highest bleu scores in all 6 language directions in the supervised task, and our submission of de2dsb and dsb2de also gains the highest bleu in unsupervised task.