SM a LL -100: Introducing Shallow Multilingual Machine Translation Model for Low-Resource Languages (2022)

Abstract: In recent years, multilingual machine translation models have achieved promising performance on low-resource language pairs by sharing information between similar languages, thus enabling zero-shot translation. To overcome the “curse of multilinguality”, these models often opt for scaling up the number of parameters, which makes their use in resource-constrained environments challenging. We introduce SMaLL-100, a distilled version of the M2M-100(12B) model, a massively multilingual machine translation model covering 100 languages. We train SMaLL-100 with uniform sampling across all language pairs and therefore focus on preserving the performance of low-resource languages. We evaluate SMaLL-100 on different low-resource benchmarks: FLORES-101, Tatoeba, and TICO-19 and demonstrate that it outperforms previous massively multilingual models of comparable sizes (200-600M) while improving inference latency and memory usage. Additionally, our model achieves comparable results to M2M-100 (1.2B), while being 3.6x smaller and 4.3x faster at inference.

## introduction neural machine translation (nmt) systems are usually trained on datasets consisting of millions of parallel sentences, thus still performing poorly on low-resource languages, i.e., languages without a large amount of training data. over the past few years, previous work has proposed several approaches to improve the quality of translations in low-resource languages, e.g., multilingual neural machine translation (mnmt) models @xcite @xcite , back-translation @xcite and unsupervised machine translation @xcite . massively mnmt models are particularly interesting for low-resource languages as they benefit the most from knowledge transfer from related languages @xcite . however, it is also seen that curse of multilinguality hurts the performance of high-resource languages. so, previous work attempted to increase the model size to maintain the translation performance in both high and low-resource languages. this makes the use of these massively mnmt models challenging in real-world resource-constrained environments. to overcome this problem, we propose small-100, a shallow multilingual machine translation model for low-resource languages covering 100 languages, which is a distilled alternative of m2m-100 (12b) @xcite , the most recent and biggest available multilingual nmt model. in this paper, we focus on very-low and low-resource language pairs as there is no reasonable-size universal model that achieves acceptable performance over a great number of low-resource languages. we do so by training small-100 on a perfectly balanced dataset. 2 while this leads to lower performance on the high-resource languages, we claim that this loss is easily recoverable through further fine-tuning. we evaluate small-100 on different low-resource benchmarks, e.g., @xcite , tatoeba (tiedemann, 2020), and @xcite . to summarize, our contributions are as follows: ## conclusion we presented small-100 model, a shallow multilingual nmt model, focusing on low-resource languages. we evaluated our model on different nmt benchmarks. small-100 significantly outperforms multilingual models of comparable size on all of the tested benchmarks (flores-101, tatoeba, tico-19) and is much faster at inference. it also achieves competitive results with m2m-100 1. @xcite , while being 4.3× faster at inference and 3.6× smaller. compared to m2m-100 (12b), the biggest available mnmt model, small-100 loses nearly 1.7 spbleu on average but it is significantly faster (7.8×) and smaller (36×), which makes it a good fit for resource-constrained settings. additionally, we show that small-100 can achieve similar performance as m2m-100 (12b) with just a few steps of fine-tuning on specific language pairs.