The U niversity of M aryland, College Park Submission to Large-Scale Multilingual Shared Task at WMT 2021 (2021)

Abstract: This paper describes the system submitted to Large-Scale Multilingual Shared Task (Small Task #2) at WMT 2021. It is based on the massively multilingual open-source model FLORES101_MM100 model, with selective fine-tuning. Our best-performing system reported a 15.72 average BLEU score for the task.

## introduction massively multilingual models such as facebook's m2m-100 @xcite model provide an attractive approach to scaling machine translation to many language pairs by sharing encoder-decoder parameters across languages. by not centering english in its training process, m2m-100 improves translation quality substantially (by over 10 bleu points) compared to the best single systems of wmt before 2020 on the "large-scale many-to-many dataset for 100 languages" @xcite . however, translation quality for low-resource languages still leaves much room for improvement. we address the large-scale multilingual machine translation shared task (small track #2) at wmt 2021, by fine-tuning the flo-res101_mm100 model for the languages in the shared task. we consider different fine-tuning configurations, with a goal to minimize the computational and data resources required. first, we consider the impact of finetuning on datasets of different sizes, and surprisingly show that finetuning with the smaller dataset gives better performance for some language pairs. second, we consider selectively dropping layers during fine-tuning to reduce the computational cost of working with a transformer model with millions of parameters. we adopt a structure dropout technique, layerdrop, which has been shown to have a regularization effect and to effectively reduce model size for inference @xcite , as well as to reduce training * these authors contributed equally to this work time while preserving decoding quality @xcite . we have used layerdrop so that our model can run on large datasets for low resource language pairs. our best performing system is fine-tuned on the large multiccaligned training data and yields a sentence-piece bleu score (the official shared task metric) of 15.72 on the shared task test set. however, a model fine-tuned on smaller amounts of data (bible-uedin) approaches that result, with a bleu score of 15.10. this paper describes the submitted models, as well as experiments with lay-erdrop configuration, which show that dropping the top layers does not help bleu. we preprocess the data using the "sentence-piece" module @xcite for tokenization and byte-pair encoding, and remove duplicate samples. evaluation sets the shared task evaluates models on three distinct datasets: dev, devtest and test. they are all drawn the flores-101 benchmark for many-to-many multilingual translation @xcite . it consists of 3001 sentences extracted from english wikipedia and covering a variety of different topics and domains. these sentences have been translated into 101 languages by professional translators through a carefully controlled process. the shared task uses a subset of six languages including english from flores-101. the languages are: javanese (jav), indonesian (ind), malay(msa), tagalog (tgl), tamil (tam), and english (eng). the dev and devtest sets are both 2.8mb in size. these datasets were evaluation test set and were therefore held out from our fine-tuning experiments. the test set were inaccessible to the shared task participants. ## model configurations this section describes our base model and the various fine-tuning configurations considered. ## submitted system configuration the submitted system is fine-tuned with the multi-ccaligned dataset for all the language pairs mentioned in ## conclusion we described the university of maryland submission to the large-scale multilingual shared task (small task #2) at wmt 2021. we considered several fine-tuning configurations on top of the massively multilingual flores101_mm100, and find that using multiccaligned data and a standard model configuration give the best result. we also show that finetuning on the much smaller bibleuedin dataset approaches our best result, with a bleu score of 15.10. selecting appropriate finetuning data thus plays a significant role in the quality of the final model, and the amount of data alone is a suboptimal selection criterion. dropping the last three layers of the encoder and decoder decreased the translation quality. future work is needed to determine how to reduce the computational needs of large-scale multilingual mt.