Findings of the WMT 2021 Shared Task on Large-Scale Multilingual Machine Translation (2021)

Abstract: We present the results of the first task on Large-Scale Multilingual Machine Translation. The task consists on the many-to-many evaluation of a single model across a variety of source and target languages. This year, the task consisted on three different settings: (i) SMALL-TASK1 (Central/South-Eastern European Languages), (ii) the SMALL-TASK2 (South-East Asian Languages), and (iii) FULL-TASK (all 101 x 100 language pairs). All the tasks used the FLORES-101 dataset as the evaluation benchmark. To ensure the longevity of the dataset, the test sets were not publicly released and the models were evaluated in a controlled environment on Dynabench. There were a total of 10 participating teams for the tasks, with a total of 151 intermediate model submissions and 13 final models. This year’s result show a significant improvement over the known base-lines with +17.8 BLEU for SMALL-TASK2, +10.6 for FULL-TASK and +3.6 for SMALL-TASK1.

## introduction despite recent advances in translation quality for a handful of languages and domains, mt systems still perform poorly on low-resource languages. yet, most of the world's population speak lowresource languages and would benefit from improvements in translation quality on their native languages. as a result, the field has been shifting focus towards the evaluation of mt in lowresource situations @xcite @xcite @xcite . however, these efforts have had poor coverage of low-resource languages which limits our understanding on generalization.more importantly, there has been little focus on the evaluation of true many-to-many multilingual models, which hampers the progress of the field despite all the recent excitement on this research direction @xcite . the recent release of the flores-101 @xcite benchmark made possible to evaluate massively multilingual systems in a consistent way. the benchmark consists of 3001 sentences sampled from english wikipedia and professionally translated in 101 languages. this poses a unique opportunity to understand translation across many languages with varied typology, resources, etc. in this first multilingual large-scale shared task, we use the flores-101 benchmark to evaluate the progress on massively multilingual translation, where the evaluation is performed in a non-englishcentric way. we propose 3 different tasks: two small tasks involving translation between 6 languages each (30 pairs), and a large task involving the translation across 101 languages (10k pairs). in the remainder of this paper, we describe the task setup, the participants, and the official results for the task. we also analyze the results to understand better the languages for which progress has been attained, and those where a gap in quality is still observed. finally, we propose future directions for other tasks in the future. ## shared tasks in this section, we briefly describe each of the tasks, the data, the baselines and metric used for evaluation. ## participants in this section, we list each of the task participants and briefly describe each of their submissions. for reproducibility, we link to each of the model submitted, available in the dynabench platform. samsung rph -konvergen ai @xcite the submission of the samsung research philippines/kovergen ai's team focuses on the languages in small-task2, in particular on data preprocessing. for large-scale multilingual models, the importance of preprocessing has risen as researchers focus on using web crawls or noisily aligned data to train translation models. in this submission, various different preprocessing techniques are applied while holding the model and architecture fixed. the authors have gains of more than 1 bleu point from improving preprocessing. models: model 443 (small-task2) tentrans @xcite the submission explores several techniques to improve performance. it focuses on two systems: tentrans and flo-res101, although the second one is favored in later experimentation. the authors achieve large improvements in performance by using a the pretrained m2m124 flores101 model. main benefit comes from in-domain knowledge adaptation and fine-tuning. the authors use a domain classifier based on bert. then they use gradual fine-tuning to gradually removing the least-likely in-domain sentence pairs at the later stages of training. they also explore other techniques, including model averaging that help to improve the performance of their system. models: model 460 (small-task2) telu-ku (budiwati et al., 2021) the team from telu-ku participated in small-task2. their approach explores an interesting alternative of improving nmt performance via hyper-parameter optimization (most promising for low resource languages). although simple, this approach effectively provides improvements by +1.08 bleu on top of the small baseline and opens up a promising direction for hyper-parameter optimization. models: model 465 (small-task2) umd @xcite this system build upon the baseline m2m-124 model @xcite . it includes two improvements: (i) finetuning over multiccaligned; (ii) it uses relus, which improve +0.8 bleu over gelus. in ad-dition, the final system is the result of an extensive hyper-parameter optimization. interestingly, the authors find that using the bible for finetuning improves performance over the baseline model despite its small size (only about 0.5 bleu behind multiccaligned). models: model 304 (small-task2) ## evaluation environment all models were evaluated within the dynaboard evaluation-as-a-service framework @xcite ) that is a part of the dynabench platform @xcite . this was done to ensure that the flores test set remains hidden while we evaluate many-to-many translation. moreover, the testing conditions were constrained to a p2.xlarge aws instance, which has one nvidia k80 gpu. all model submissions had to be wrapped in a torchserve foot_2 handler and were required to follow a fixed input/output specification using dynalab foot_3 . submitting a system to the task required writing some wrapper code, and often testing different configurations (e.g. batch size), to ensure that the model was able to run under the constraints. given the additional work needed to run the evaluation, participants were encouraged to test the platform and to submit models early on. to avoid fine-tuning on the devtest set, we established a submission cap of one model per day. in total, we had 81 distinct model submissions for the small task2 (south-east asian languages), 57 distinct submissions to the small task1 (central / south-east european languages), and 13 model submissions to the full task. during the evaluation period, participants were requested to mark a model as their final submission. in the end, we had 10 final submissions to the small task2, 4 to the small task 1 and 3 to the full task. in figure 1 we observe the total number of submissions per day. we can see that the total number of submissions per day remained low (less than 5) until august, where the number of submissions reached 16 per day. ## results present the analysis of the results for each of the tasks. furthermore, we analyze the progress made for each task, that is, how much improvement has 0 1 3 date of submission 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 number of submissions per day there been between the baselines and the best models. lastly, we analyze the difference between the models for the full task and each of the smaller tasks. ## conclusion and future work in this paper we presented the first iteration of the large-scale multilingual translation task. this task attracted several teams from across the globe and many models submissions. we kept the test set blind and used a platform to evaluate model submissions under a controlled environment. in this task, we observed significant progress in translation quality across tasks, but particularly in the small tasks. we observed that pre-trained language models and large amounts of back-translation (either at one go, or in iterative fashion) were important tools used by many participants. we observed that models that have to translate fewer languages trend to do better on average, and that lower resources and morphology complicate translation, particularly for decoding. we also observed that languages in certain groups, like the african languages in the bantu and nilotic families, experience little to no improvement. in the future, we want to organize shared tasks with languages for which little or no progress was achieved this time around. additionally, we want to open up the flores evaluation setup to other organizers interested groups of languages within the flores-101 set.