Maastricht University’s Large-Scale Multilingual Machine Translation System for WMT 2021 (2021)

Abstract: We present our development of the multilingual machine translation system for the large-scale multilingual machine translation task at WMT 2021. Starting form the provided baseline system, we investigated several techniques to improve the translation quality on the target subset of languages. We were able to significantly improve the translation quality by adapting the system towards the target subset of languages and by generating synthetic data using the initial model. Techniques successfully applied in zero-shot multilingual machine translation (e.g. similarity regularizer) only had a minor effect on the final translation performance.

## introduction this paper describes maastricht university's participation in the large-scale multilingual machine translation task of wmt 2021. we participate in small track #2. in this track, the task is to build a translation system between english and 5 southeast asian languages. the evaluation is performed on all 30 possible translation directions between these languages. we are provided with parallel data extracted from wikipedia and other sources for all language pairs, as well as a large-scale multilingual machine translation model pretrained on 124 languages @xcite including the languages in this task. starting from the provided baseline models, we investigate several directions in order to improve the performance on the 30 target language directions. as the first step, we focus on methods to adapt the model to these language directions. specifically, we investigate different strategies to fine-tune the model on the proposed parallel training data. since the provided parallel data is extremely limited for several translation directions, we investigate the use of synthetic parallel data. we focus on using pivot languages in order to use well performing language direction to generate training data for worse performing language directions. finally, we investigate the usefulness of techniques to promote the similarity of representation between different languages. while these techniques were shown essential for models to perform zero-shot machine translation @xcite @xcite , in our experiments the impact of these methods is only limited. ## data we start by introducing the training data and preprocessing steps. ## techniques our efforts to improve upon the provided baselines can be categorized into three directions: finetuning (subsection 3.1), utilizing synthetic data (subsection 3.2), encouraging similarities between languages (subsection 3.3). ## results in this section, we report the results of the three directions we explored: finetuning on the focus languages (subsection 5.1), utilizing synthetic data (subsection 5.2), and encouraging similarities between languages (subsection 5.3). ## conclusion this paper summarizes our participation in the wmt 2021 large-scale multilingual translation task. we focus on small track #2 for english and 5 southeast asian languages. building upon the provided baseline models, we achieved the largest gain from fine-tuning on the parallel data of all directions in this task. by further utilizing synthetic data and a combination of similarity regularization and adapters, we were able to further improve the system.