Ensembling of Distilled Models from Multi-task Teachers for Constrained Resource Language Pairs (2021)

Abstract: This paper describes the Microsoft Egypt Development Center (EgDC) submission to the constrained track of WMT21 shared news translation task. We focus on the three relatively low resource language pairs Bengali ↔ Hindi, English ↔ Hausa and Xhosa ↔ Zulu. To overcome the limitation of relatively low parallel data we train a multilingual model using a multitask objective employing both parallel and monolingual data. In addition, we augment the data using back translation. We also train a bilingual model incorporating back translation and knowledge distillation then combine the two models using sequence-to-sequence mapping. We see around 70% relative gain in BLEU point for En ↔ Ha and around 25% relative improvements for Bn ↔ Hi and Xh ↔ Zu compared to bilingual baselines.

## introduction neural machine translation (nmt) witnessed a lot of success in the past few years especially for high resource languages @xcite . improving the quality of low resource languages is still challenging. some of the popular techniques are adding high resource helper languages as in multilingual neural machine translation (mnmt) @xcite @xcite @xcite , using monolingual data including pre-training @xcite , multi-task learning @xcite , back translation @xcite or any combination of these methods @xcite and system combination of multiple systems @xcite . this paper describes the microsoft egypt development center (egdc) submission to the wmt21 shared news translation task for three low resource language pairs (six directions), bengali ↔ hindi (bn ↔ hi), english ↔ hausa (en ↔ ha) and xhosa ↔ zulu (xh ↔ zu). we focus on the constrained track because it is easier to compare different systems and it is always possible to improve performance by adding more data. the main features of our approach are as follows: the paper is organized as follows. section 2 gives an overview of the data used in the constrained scenario, followed by section 3 that gives a detailed description of our approach. section 4 presents our experimental evaluation. finally, our findings are summarized in section 5. ## data following the constrained track, we use bitext data provided in wmt21 for the following pairs: bengali ↔ hindi, english ↔ hausa, xhosa ↔ zulu and english ↔ german. statistics of the parallel data used for the three pairs in addition to the german helper are shown in ## system architecture the final mt system in each direction is an ensemble of two nmt models comprising a bilingual model (one for each of the six primary directions) and a multilingual model trained to provide translations for 8 directions (the six primary directions plus english ↔ german). the multilingual system uses a recently proposed multitask framework for training @xcite . we describe the individual systems in subsection 3.1. this is followed by presenting our system combination techniques in subsection 3.2. finally we present the architecture of the submitted system highlighting our design decisions in subsection 3.3. ## experimental results in this section, we describe the results of our intermediate and final systems. we report sacrebleu @xcite on the validation set released in wmt21, and both sacrebleu and comet @xcite using the available implementation foot_2 on the official test set released in wmt21. the results for the six submitted language pairs are in tables 3 4 5 . the first row in each table shows the bilingual baseline which performs relatively poor due to the limited amount of parallel data for each pair. this is followed by the four multilingual systems with different objectives. it is clear that adding a monolingual objective brings nice improvements for all language pairs. the m t + dae and m t + m lm + dae perform closely for all language pairs indicating that target monolingual data is most important. the next two rows show the results of adding back-translated data to the multilingual model and a bilingual baseline using back-translated and knowledge distilled data generated from the best multilingual model. as expected adding back translation brings significant improvement to all language pairs. also using the multilingual model to create data for a bilingual model shows excellent results that outperform the multilingual model. finally, the ensemble, as expected, performs better than the individual models. the significant difference between reported improvements in ha ↔ en and other directions shows the effectiveness of adding de ↔ en parallel and monolingual data that helps english centric directions more than other directions. we evaluated the final submitted systems on the official test set released in wmt21 as shown in