Findings of the WMT ’22 Shared Task on Large-Scale Machine Translation Evaluation for A frican Languages (2022)

Abstract: We present the results of the WMT’22 SharedTask on Large-Scale Machine Translation Evaluation for African Languages. The shared taskincluded both a data and a systems track, alongwith additional innovations, such as a focus onAfrican languages and extensive human evaluation of submitted systems. We received 14system submissions from 8 teams, as well as6 data track contributions. We report a largeprogress in the quality of translation for Africanlanguages since the last iteration of this sharedtask: there is an increase of about 7.5 BLEUpoints across 72 language pairs, and the average BLEU scores went from 15.09 to 22.60.

## introduction a large portion of the world's population speak low-resource languages, and would benefit from improvements in translation quality on their native languages. recent advances in translation quality, particularly from massively multilingual models @xcite , have enabled progress in the translation quality of low-resource languages. however, many languages have seen little to no progress. this is particularly true for african languages. for example, in the 2021 large scale multilingual evaluation shared task @xcite , there was a progress 2 of +19.3 avg. bleu when translating into languages like irish and welsh (included in the other indo-european grouping), but only a progress of +3.5 avg. bleu when translating into languages like fula and igbo (included in the nilotic/atlantic congo grouping). the african continent is home to a rich diversity of languages. around a third of the world's living languages are from africa and only a small fraction author names are sorted in alphabetical order. 2 progress was determined by comparing the average bleu score between the best system in the task vs. the baseline. to bring attention of the research community to the challenges of translating african languages, this year we focus on the multilingual evaluation of 24 african languages together with french and english. we base our evaluation on the benchmark provided by flores @xcite , and its recent expansion to more african languages @xcite . in this second multilingual large-scale shared task, we evaluate the progress on massively multilingual translation for african languages in a non-english-centric way. we propose 100 evaluation language pairs, based on regional clusters (south/south-east africa, horn of africa and central/east africa, nigeria and gulf of guinea, and central africa; and pivot languages @xcite . this year is characterized by two further innovations: first, a data track is included, in which participants share corpora to be used during the shared task; second, we perform human evaluation for a subset of the language pairs. in the remainder of this paper, we describe the task setup, the participants, and the official results for the task. we also analyze the results to understand better the languages for which progress has been attained, and those where a gap in quality is still observed. finally, we propose future directions for other tasks in the future. the human and automatic evaluation is based around 100 language directions, selected based on translator and annotator availability: ## data track the data track focused on the contribution of novel corpora. participants were welcomed to opensource and share monolingual, bilingual or multilingual datasets relevant to the training of mt models for this year's set of languages. there were seven submissions in this track: lava foot_0 lava corpus contains millions of parallel bilingual sentences, which are mined from common crawl. it covers five african languages. mafand-mt @xcite foot_1 contains a few thousand high-quality and human translated parallel sentences for 21 african languages in the news domain. each language has between 1,400 -34,500 parallel sentences for training and/or evaluation. the languages covered are amharic, bambara, ghomala, ewe, fon, hausa, igbo, kinyarwanda, luganda, dholuo, mossi, chichewa, nigerian-pidgin, chishona, swahili, setswana, twi, wolof, yorùbá, isixhosa, and isizulu. these languages include 7 languages which were not present in the shared task (and which are not reported therefore in kentrans 5 this project produced a parallel corpus between swahili and 2 other kenya languages: dholuo and luhya. the luhya language has several dialects. in the project 3 dialects were chosen as a start: lumarachi, logooli and lubukusi. a total of 12,400 sentences were translated to kiswahili from a sample of dholuo and luhya (1500 dholuo-kiswahili sentence pairs and 10,900 luhya-kiswahili sentence pairs). this corpus has an extension to speech in the version of @xcite 6 . ## system track we provided a selection of training corpora, i.e. parallel sentences, to enable training of the mt systems. submissions in the constrained translation track were only allowed to use data from the following sources: participants who used other resources, had to submit to the unconstrained translation track. publicly available resources for african languages are very limited, for some of them less than fifty thousand sentences of bitexts are available. in addition to human translated sentences, several approaches were proposed to automatically mine parallel sentence from large collections of monolingual data. unfortunately, recent approaches like paracrawl or ccmatrix @xcite cover only few african languages. we extended the basic idea of mining based on a similarity measure in an multilingual embedding space @xcite and developed sentence encoders for all african languages of this evaluation. we then performed bitext mining against 21.5 billion english sentences from common crawl, and 3.3 million sentences in french, respectively. these resources as well as the sentence encoders were made available to the participants of this evaluation. a detailed description of this mining approach can be found in @xcite . ## results in this section we analyze both the human scores and automatic metrics for all the participants. we present the results and official ranking for the main task; the analysis of the performance from/to english, an to/from african languages; and the progress in performance w.r.t. to the previous year's task. ## conclusion and future work in this paper, we presented the results of the second shared task on large-scale machine translation evaluation. in this edition of the shared task, we evaluate the progress on massively multilingual translation for african languages in a non-englishcentric way. from our findings, we observe that data is still an important factor in translation performance, and that systems that used more data, either in an unconstrained way, or through data augmentation techniques made the most progress.the quality of the data is also important, as most participants developed a set of heuristics to clean it. we also observed the popularity of pre-trained translation models such as: deltalm, m2m-100 and mt5, as most systems used a version of these when developing their final model. we observe that there we also observed that there has been significant progress translating into and out of northern sotho, hausa, and somali. maja popović. 2017. chrf++: words helping character n-grams. in proceedings of the second conference on machine translation, pages 612-618, copenhagen, denmark. association for computational linguistics. xian qian, kai hu, jiaqiang wang, yifeng liu, xingyuan pan, jun cao, and mingxuan wang. 2022. the volctrans system for wmt22 multilingual machine translation task. in proceedings of the seventh conference on machine translation: shared task papers. holger schwenk, guillaume wenzek, sergey edunov, edouard grave, armand joulin, and angela fan. 2021. ccmatrix: mining billions of high-quality parallel sentences on the web. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 6490-6500, online. association for computational linguistics. jörg tiedemann. 2012. parallel data, tools and interfaces in opus. in proceedings of the eighth international conference on language resources and evaluation (lrec'12), pages 2214-2218, istanbul, turkey. european language resources association (elra). pavanpankaj vegi, sivabhavani j, biswajit paul, prasanna kumar k r, and chitra viswanathan. 2022a. anvita multilingual neural machine translation system for african languages. in proceedings of the seventh conference on machine translation: shared task papers. pavanpankaj vegi, sivabhavani j, biswajit paul, abhinav mishra, prashant banjare, prasanna kumar k r, and chitra viswanathan. 2022b. webcrawl african: a multilingual parallel corpora for african languages. in proceedings of the seventh conference on machine translation: shared task papers. barack wamkaya wanjawa, lilian d. a. wanzare, florence indede, owen mconyango, edward ombui, and lawrence muchemi. 2022. kencorpus: a kenyan language corpus of swahili, dholuo and luhya for natural language processing tasks. arxiv, abs/2208.12081. guillaume wenzek, vishrav chaudhary, angela fan, sahir gomez, naman goyal, somya jain, douwe kiela, tristan thrush, and francisco guzmán. 2021. findings of the wmt 2021 shared task on large-scale multilingual machine translation. in proceedings of the sixth conference on machine translation, pages 89-99, online. association for computational linguistics. jian yang, shuming ma, haoyang huang, dongdong zhang, li dong, shaohan huang, alexandre muzio, saksham singhal, hany hassan, xia song, and furu wei. 2021. multilingual machine translation systems from microsoft for wmt21 shared task. in proceedings of the sixth conference on machine translation, pages 446-455, online. association for computational linguistics.