The F lores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation (2022)

Abstract: One of the biggest challenges hindering progress in low-resource and multilingual machine translation is the lack of good evaluation benchmarks. Current evaluation benchmarks either lack good coverage of low-resource languages, consider only restricted domains, or are low quality because they are constructed using semi-automatic procedures. In this work, we introduce the Flores-101 evaluation benchmark, consisting of 3001 sentences extracted from English Wikipedia and covering a variety of different topics and domains. These sentences have been translated in 101 languages by professional translators through a carefully controlled process. The resulting dataset enables better assessment of model quality on the long tail of low-resource languages, including the evaluation of many-to-many multilingual translation systems, as all translations are fully aligned. By publicly releasing such a high-quality and high-coverage dataset, we hope to foster progress in the machine translation community and beyond.

## introduction machine translation (mt) is one of the most successful applications in natural language processing, as exemplified by its numerous practical applications and the number of contributions on this topic at major machine learning and natural language processing venues. despite recent advances in translation quality for a handful of language pairs and domains, mt systems still perform poorly on low-resource languages, that is, languages without a lot of training data. in fact, many low-resource languages are not even supported by most popular translation engines. yet, much of the world's population speak low-resource languages and would benefit from improvements in translation quality on their native languages. as a result, the field has been increasing focus towards low-resource languages. at present, there are very few benchmarks on low-resource languages. these often have very low coverage of low-resource languages @xcite @xcite @xcite @xcite , limiting our understanding of how well methods generalize and scale to a larger number of languages with a diversity of linguistic features. there are some benchmarks that have high coverage, but these are often in specific domains, like covid-19 @xcite or religious texts @xcite @xcite ; or have low quality because they are built using automatic approaches @xcite @xcite . as a result, it is difficult to draw firm conclusions about research efforts on low-resource mt. in particular, there are even fewer benchmarks that are suitable for evaluation of many-to-many multilingual translation, as these require multi-lingual alignment (i.e., having the translation of the same sentence in multiple languages), which hampers the progress of the field despite all the recent excitement on this research direction. we present the flores-101 benchmark, consisting of 3001 sentences sampled from various topics in english wikipedia and professionally translated in 101 languages. with this dataset, we make several contributions. first, we provide the community with a high-quality benchmark that has much larger breadth of topics and coverage of low resource languages than any other existing dataset ( §4). second, flores-101 is suitable for many-to-many evaluation, meaning that it enables seamless evaluation of 10,100 language pairs. this enables the evaluation of popular multilingual mt systems as well as the evaluation of regionallyrelevant language pairs like spanish-aymara and vietnamese-thai, for example. third, we thoroughly document the annotation process we followed ( §3), helping the community build institutional knowledge about how to construct mt datasets. fourth, we release not only sentences with their translation but also rich meta-data that enables other kinds of evaluations and tasks, such as document level translation, multimodal translation, and text classification. fifth, we propose to use the bleu metric based on sentence piece tokenization @xcite ( §5) to enable evaluation of all languages in the set in a unified and extensible framework, while preserving the familiarity of bleu. finally, we publicly release both data and baselines used in our experiments ( §6), to foster research in low-resource machine translation and related areas. traditionally, the yearly workshop on machine translation (wmt) and its associated shared tasks have provided standardized benchmarks and metrics to the community, fostering progress by providing means of fair comparison among various approaches. over recent years, the main translation task at wmt has challenged participants with low-resource languages, but the evaluation has been limited to a handful of languages-for example, latvian in 2017 @xcite , kazakh in 2018 @xcite , gujarati and lithuanian in 2019 @xcite , and inuktitut, khmer, pashto, and tamil in 2020 @xcite . moreover, these tasks have considered translation to and from english only, while the field has been recently focusing on large-scale multilingual models @xcite @xcite . there are other datasets for evaluation purposes, such as flores v1.0 @xcite , lorelei @xcite , alt @xcite @xcite , and tico-19 @xcite , as well as datasets for specific languages such as igbo @xcite and fon (dossou and emezue, 2020). these are similar to flores-101 because they focus on low-resource languages. however, the language coverage of these datasets is much smaller. among these, only tico-19 is suitable for multilingual machine translation, but its content is centered around covid-19, unlike the much broader coverage of topics offered by flores-101. the tatoeba corpus @xcite covers a large number of languages and translation directions, but the low volume of sentences for many directions makes the evaluation less reliable. further, many of the tatoeba corpus sentences are very short and straightforward, making the dataset not as generalizable to a more diverse set of content. ## dataset construction we describe how flores-101 was constructed, first noting where the sentences originated from and subsequently describing the carefully designed translation process. ## flores-101 at a glance in this section, we analyze flores-101. we provide a high-level comparison of flores-101 with existing benchmarks, then discuss the sentences, languages, and translation quality in detail. ## multilingual evaluation automatic evaluation of translation quality is an active research field. each year, the wmt figure 2 : translation quality score across languages. we require the final translation quality score to be above 90% before the translation is of sufficient quality to include in flores-101. metrics shared task seeks to determine the metrics that better correlate with human evaluations @xcite . while many metrics have been proposed through the years, most have not gained traction in the community. 5 in fact, 99% of mt publications in the last decade still report bleu, and 74% do so exclusively @xcite . through the years, researchers continue to use bleu to compare different models. as a result, the community has developed strong intuitions on the significance of the effects when looking at bleu. unfortunately, using word-level bleu as-is is suboptimal in a multilingual context, as n-gram overlap heavily depends on the particular tokenization used, which is not well defined in many low-resource languages. ## evaluating baselines on flores-101 in this section, we present the evaluation of three baseline models on flores-101. ## conclusion the potential to develop translation for languages globally is hindered by lack of reliable, high-quality evaluation. we create and open-source flores-101, a benchmark with 101 languages. it supports many-to-many evaluation, meaning all 10,100 directions can be evaluated. unlike many other datasets, flores-101 is professionally translated, including human evaluation during dataset creation. beyond translation, flores-101 can be used to evaluate tasks such as sentence classification, language identification, and domain adaptation.