The Tatoeba Translation Challenge – Realistic Data Sets for Low Resource and Multilingual MT (2020)

Abstract: This paper describes the development of a new benchmark for machine translation that provides training and test data for thousands of language pairs covering over 500 languages and tools for creating state-of-the-art translation models from that collection. The main goal is to trigger the development of open translation tools and models with a much broader coverage of the World’s languages. Using the package it is possible to work on realistic low-resource scenarios avoiding artificially reduced setups that are common when demonstrating zero-shot or few-shot learning. For the first time, this package provides a comprehensive collection of diverse data sets in hundreds of languages with systematic language and script annotation and data splits to extend the narrow coverage of existing benchmarks. Together with the data release, we also provide a growing number of pre-trained baseline models for individual language pairs and selected language groups.

## introduction the tatoeba translation challenge includes shuffled training data taken from opus, foot_0 an open collection of parallel corpora @xcite , and test data from tatoeba, foot_1 a crowd-sourced collection of user-provided translations in a large number of languages. all data sets are labeled with iso-639-3 language codes using macro-languages in case when available. naturally, training data do not include sentences from tatoeba and neither from the popular wmt testsets to allow a fair comparison to other models that have been evaluated using those data sets. here, we propose an open challenge and the idea is to encourage people to develop machine translation in real-world cases for many languages. the most important point is to get away from artificial setups that only simulate low-resource scenarios or zero-shot translations. a lot of research is tested with multi-parallel data sets and high resource languages using data sets such as wit 3 @xcite or europarl @xcite simply reducing or taking away one language pair for arguing about the capabilities of learning translation with little or without explicit training data for the language pair in question (see, e.g., @xcite ; @xcite ; @xcite ). such a setup is, however, not realistic and most probably over-estimates the ability of transfer learning making claims that do not necessarily carry over towards real-world tasks. in the set we provide here we, instead, include all available data from the collection without removing anything. in this way, the data refers to a diverse and skewed collection, which reflects the real situation we need to work with and many lowresource languages are only represented by noisy or very unrelated training data. zero-shot scenarios are only tested if no data is available in any of the sub-corpora. more details about the data compilation and releases will be given below. tatoeba is, admittedly, a rather easy test set in general but it includes a wide variety of languages and makes it easy to get started with rather encouraging results even for lesser resourced languages. the release also includes medium and high resource settings and allows a wide range of experiments with all supported language pairs including studies of transfer learning and pivot-based methods. ## data releases the current release includes over 500gb of compressed data for 2,961 language pairs covering 555 languages. the data sets are released per language pair with the following structure, using deu-eng as an example (see figure 1 ). the data sets are compiled from the pre-aligned bitexts but further cleaned in various ways. first of all, we remove non-printable characters and strings that violate unicode encoding principles using regular expressions and a recoding trick using the forced encoding mode of recode (v3.7), a popular character conversion tool. 3 furthermore, we also de-escape special characters (like '&' encoded as '&amp;') that may appear in some of the corpora. for that, we apply the tools from moses @xcite . finally, we also apply automatic language identification to remove additional noise from the data. we use the compact language detect library (cld2) through its python bindings foot_3 and a python library for converting between different iso-639 standards. foot_4 cld2 supports 172 languages and we use the options for "best effort" and apply the assumed language from the original data as the "hint language code". for unsupported languages, we remove all examples that are detected to be english as this is a common problem in some corpora where english texts appear in various places (e.g. untranslated text in localization data of community efforts). in all cases, we only rely on the detected language if it is flagged as reliable by the software. all corpus data and sub-languages are merged and shuffled using terashuf foot_5 that is capable to efficiently shuffle large data sets. but we keep track of the original data set and provide labels to recognize the origin. in this way, it is possible to restrict training to specific subsets of the data to improve domain match or to reduce noise. the entire procedure of compiling the tatoeba challenge data sets is available from the project repository at https://github.com/ helsinki-nlp/tatoeba-challenge . the largest data set (english-french) contains over 180 million aligned sentence pairs and 173 language pairs are covered by over 10 million sentence pairs in our collection. altogether, there are almost bilingual 3,000 data sets and we plan regular updates to improve the coverage. below, we give some more details about the language labels, test sets and monolingual data sets that we include in the package as well. ## the translation challenge the main challenge is to develop translation models and to test them with the given test data from tatoeba. the focus is on low-resource languages and to push their coverage and translation quality. resources for high-resource are also provided and can be used as well for translation modeling of those languages and for knowledge transfer to less resourced languages. note that not all language pairs have sufficient data sets for test, @xcite and training (train) data. hence, we divided the tatoeba challenge data into various subsets based on the size of the training data available. for all those 522 selected language pairs, the data set provides at least 200 sentences per test set. 101 of them involves english as one of the languages. 288 test sets contain more than 1,000 sentence pairs of which only 68 include english. note, that everything below 1,000 sentences is probably not very reliable as a proper test set but we decided to release smaller test sets as an initial benchmark to trigger further development even for extremely under-resourced language pairs. we also decided to use very low thresholds for the division into lowresource languages. having 10,000 training examples or less is very realistic for many real-world examples and we want to encourage the work on such cases in particular. the maximum size of test sets in our collection is 10,000 sentence pairs, which is available for 76 language pairs. the test size is reduced to 5,000 if there is less than 20,000 sentence pairs in tatoeba (19 data sets). the remaining sentences are released as disjoint validation data. for 48 tatoeba language pairs with less than 10,000 sentence pairs, we keep 2,500 for the test set and the rest for validation and for 78 tateoba language pairs with less than 5,000 sentence pairs we keep 1,000 for validation and the rest for testing. finally, for language pairs with less than 2,000 sentences in tatoeba we skip validation data and use everything for test purposes. test and validation data are strictly disjoint and none of the examples from tatoeba are explicitly included in the training data. however, as it is common in realistic cases, there is a natural chance for a certain overlap between those data sets. figure 3 plots the percentage of sentence pairs in test and validation sets that can also be found in the corresponding training data we release. the average proportion is rather low around 5.5% for both with a median percentage of 2.3% and 2.9% for test and validation data, respectively. there is one clear outlier with a very high proportion of over 55% overlap and that is danish-english for some reason that is not entirely clear to us. otherwise, the values are well below that ratio. ## the data challenge the most important ingredient for improved translation quality is data. it is not only about training data but very much also about appropriate test data that can help to push the development of transfer models and other ideas of handling low-resource settings. therefore, another challenge we want to open here is the increase of the coverage of test sets for low-resource languages. our strategy is to organize the extension of the benchmarks directly through the tatoeba initiative. users who would like to contribute to further mt benchmark development are asked to register for the open service provided by tatoeba and to upload new translations in the languages of interest. from our side, we will continuously update our challenge data set to include the latest data releases coming from tatoeba including new language pairs and extended data sets for existing language pairs. we will make sure that the new test sets do not overlap with any released development data from previous revisions to enable fair comparisons of old models with new benchmarks. the extended test and validation data sets will be released as new packages and old revisions will be kept for replicability of existing scores. in order to provide information about language pairs in need, we provide a list of data sets with less than 1,000 examples per language pair. in the current release, this refers to 2,375 language pairs. 2,141 language pairs have less than 200 translation units and are, therefore, not included in the released benchmark test set. furthermore, we also provide a list of languages for which we release training data coupled with english but no test data is available from tatoeba. currently, this relates to 246 languages. we encourage users to especially contribute translations for those data sets in order to improve the language coverage even further. we hope to trigger a grass-root development that can significantly boost the availability of development and test sets as one of the crucial elements for pushing nmt development in the corresponding languages. finally, we also encourage to incorporate other test sets besides of the tatoeba data. currently, we also test with wmt news test sets for the language pairs that are covered by the released development and test sets over the years of the news translation campaign. contributions and links can be provided through the repository management interface at github. ## how to participate the goal of the data release is to enable a straightforward setup for machine translation development. everyone interested is free to use the data for their own development. a leader board for individual language pairs will be maintained. furthermore, we also intend to make models available that are listed in the challenge. this does not only support replicability but also provides a new unique resource of pre-trained models that can be inte-grated in real-world applications or can be used in further research, unrelated downstream tasks or as a starting point for subsequent fine-tuning and domain adaptation. a large number of models is already available from our side providing baselines for a large portion of the data set. more details will be provided below. for participation, there are certain rules that apply: ## baseline models along with the data, we also release baseline models that we train with state-of-the-art trans-former models @xcite using marian-nmt, foot_10 a stable production-ready nmt toolbox with efficient training and decoding capabilities @xcite . we apply a common setup with 6 self-attentive layers in both, the encoder and decoder network using 8 attention heads in each layer. the hyper-parameters follow the general recommendations given in the documentation of the software. 12 the training procedures follow the strategy implemented in opus-mt @xcite and detailed instructions are available from github. 13 we train a selection of models on v100 gpus with early-stopping after 10 iterations of dropping validation perplexities. we use sentencepiece @xcite for the segmentation into subword units and apply a shared vocabulary of a maximum of 65,000 items. language label tokens in the spirit of @xcite are used in case of multiple language variants or scripts in the target language. models for over 400 language pairs are currently available and we refer the reader to the website with the latest results. for illustration, we provide some example scores below in one of the most interesting questions is the ability of multilingual models to push the performance of low-resource machine translation. the tatoeba translation challenge provides a perfect testbed for systematic studies on the effect of transfer learning across various subsets of language pairs. we already started various experiments with a number of multilingual translation models that we evaluate on the given benchmarks. in our current work, we focus on models that include languages in established groups and for that we facilitate the iso-639-5 standard. this standard defines a hierarchy of language groups and we map our data sets accordingly to start new models that cover those sets. as an example, we look at the task of belorussian-english translation that has been included in the previous section as well. the models focus on different levels of relatedness of the languages and range from east slavic languages (zle), slavic languages (sla) to the language family of indo-european languages (ine) and the set that contains all languages (mul). each model is trained on sampled data set in order to balance between different languages. the smallest training sets are based on data that are sampled to include a maximum of one million sentence per language pair (opus1m). we use both, down-sampling and up-sampling. the latter is done by simply multiplying the existing data until the threshold is reached. we also set a threshold of 50 for the maximum of repeating the same data in order to avoid over-representing small noisy data. the onemillion models are trained first and form the basis of larger models. we continue training with data sets sampled to two million before increasing to four million sentence pairs. the table shows some interesting patterns. first of all, we can clearly see a big push in performance when adding related languages to the training data. this is certainly expected especially in the case of belorussian that is closely related to higherresource-languages such as russian and ukrainian. interesting is that the east slavic language group is not the best performing model even though it includes those two related languages. the additional information from other slavic languages pushes the performance beyond their level quite significantly. certainly, those models will see more data and this may cause the difference. the 'sla-eng' model covers 13 source languages whereas 'zle' only 5. also interesting to see is that the indo-european language model fairs quite well despite the enormous language coverage that this model has to cope with. on the other hand, the big 'mul' translation model does not manage to create the same performance and the limits of the standard model with such a massive setup become apparent. training those models becomes also extremely expensive and slow and we did not manage to start the 4-million-sentence model. currently, we look into the various models we train and many other interesting patterns can be seen. we will leave a careful analyses to future work and also encourage the community to explore this field further using the given collection and benchmark. updates about models and scores will be published on the website and we would also encourage more qualitative studies that we were not able to do yet. ## zero-shot and few-shot translation the table shows that a naive approach of throwing all languages that are part of zero-shot language pairs into one global multilingual model (tatoebazero) does not work well. this is probably not very surprising. another interesting observation is that a symmetric multilingual model with indo-european languages on both sides (ine-ine) also underperforms compared to other multilingual models that only translate into english. once again, the indo-european-language-family to english model performs quite well. note that the performance purely comes from overlaps with related languages as no awadhi language data is available during training. the performance is still very poor and needs to be taken with a grain of salt. they demonstrate, however, the challenges one faces with realistic cases of zero-shot translation. in ## comparison to the wmt news task finally, we also include a quick comparison to the wmt news translation task, see the results demonstrate that the models can achieve high quality even on a domain they are not optimized for. the best scores in the german-english case are close to the top performing model registered for this task even though the comparison is not fair for various reasons. the purpose is anyway not to provide state-of-the-art models for the news translation task but baseline models for the tatoeba case and in future work we will also ex-plore the use of our models as the basis for systems that can be developed for other benchmarks and applications. in the example we can also see that multilingual models significantly lag behind bilingual ones in high-resource cases. each increase of the language coverage (except for the move from west germanic languages (gmw) to germanic languages (gem) in the german-english case) leads to a drop in performance but note that those multilingual models are not fine-tuned for translating from and to german. ## conclusions this paper presents a new comprehensive data set and benchmark for machine translation that covers roughly 3,000 language pairs and over 500 languages and language variants. we provide training and test data that can be used to explore realistic low-resource scenarios and zero-shot machine translation. the data set is carefully annotated with standardized language labels including variations in scripts and with information about the original source. we also release baseline models and results and encourage the community to contribute to the data set and machine translation development. all tools for data preparation and training bilingual as well as multilingual translation models are provided as open source packages on github. we are looking forward to new models, extended test sets and a better coverage of the world's languages.