Many-to- E nglish Machine Translation Tools, Data, and Pretrained Models (2021)

Abstract: While there are more than 7000 languages in the world, most translation research efforts have targeted a few high resource languages. Commercial translation systems support only one hundred languages or fewer, and do not make these models available for transfer to low resource languages. In this work, we present useful tools for machine translation research: MTData, NLCodec and RTG. We demonstrate their usefulness by creating a multilingual neural machine translation model capable of translating from 500 source languages to English. We make this multilingual model readily downloadable and usable as a service, or as a parent model for transfer-learning to even lower-resource languages.

## introduction neural machine translation (nmt) @xcite has progressed to reach human performance on select benchmark tasks @xcite . however, as mt research has mainly focused on translation between a small number of high resource languages, the unavailability of usable-quality translation models for low resource languages remains an ongoing concern. even those commercial translation services attempting to broaden their language coverage has only reached around one hundred languages; this excludes most of the thousands of languages used around the world today. freely available corpora of parallel data for many languages are available, though they are hosted at various sites, and are in various forms. a challenge for incorporating more languages into mt models is a lack of easy access to all of these datasets.while standards like iso 639-3 have been established to 1 demo website: http://rtg.isi.edu/many-eng . video demo: https://youtu.be/nsy0-mvo1ke . specifically, we offer three tools which can be used either independently or in combination to advance nmt research on a wider set of languages (section 2): firstly, mtdata, which helps to easily obtain parallel datasets (section 2.1); secondly, nl-codec, a vocabulary manager and storage layer for transforming sentences to integer sequences, that is efficient and scalable (section 2.2); and lastly, rtg, a feature-rich pytorch-backed nmt toolkit that supports reproducible experiments (section 2.3). we demonstrate the capabilities of our tools by preparing a massive bitext dataset with more than 9 billion tokens per side, and training a single multilingual nmt model capable of translating 500 source languages to english (section 3). we show that the multilingual model is usable either as a service for translating several hundred languages to english (section 4.1), or as a parent model in a transfer learning setting for improving translation of low resource languages (section 4.2). ## tools our tools are organized into the following sections: ## applications the model we trained as a demonstration for our tools is useful on its own, as described in the following sections. figure 2 : many-to-english bleu on opus-100 tests @xcite . despite having four times more languages on the source side, our model scores competitive bleu on most languages with the strongest system of @xcite . the tests where our model scores lower bleu have shorter source sentences (mean length of about three tokens). ## conclusion we have introduced our tools: mtdata for downloading datasets, nlcodec for processing, storing and retrieving large scale training data, and rtg for training nmt models. using these tools, we have collected a massive dataset and trained a multilingual model for many-to-english translation. we have demonstrated that our model can be used independently as a translation service, and also showed its use as a parent model for improving low resource language translation. all the described tools, used datasets, and trained models are made available to the public for free.