A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for A frican News Translation (2022)

Abstract: Recent advances in the pre-training for language models leverage large-scale datasets to create multilingual models. However, low-resource languages are mostly left out in these datasets. This is primarily because many widely spoken languages that are not well represented on the web and therefore excluded from the large-scale crawls for datasets. Furthermore, downstream users of these models are restricted to the selection of languages originally chosen for pre-training. This work investigates how to optimally leverage existing pre-trained models to create low-resource translation systems for 16 African languages. We focus on two questions: 1) How can pre-trained models be used for languages not included in the initial pretraining? and 2) How can the resulting translation models effectively transfer to new domains? To answer these questions, we create a novel African news corpus covering 16 languages, of which eight languages are not part of any existing evaluation dataset. We demonstrate that the most effective strategy for transferring both additional languages and additional domains is to leverage small quantities of high-quality translation data to fine-tune large pre-trained models.

## introduction enormous efforts have been invested in making language and translation models more multilingual while leveraging the maximal amount of data for training, most prominently large crawls of monolingual and parallel data from the web @xcite @xcite . the resulting models are now capable of translating between hundreds of languages, including language pairs that in isolation do not have large collections of parallel data @xcite @xcite . for example, m2m-100 @xcite can translate (with low accuracy) between hausa and yorùbá, two of the most widely spoken languages in nigeria, even though there is barely any parallel data available for training. for languages that are not included in the set of training languages, the model would have no knowledge on how to generate translations. does this mean there is no hope for languages that do not have large presence on the web and are therefore not included in these pre-trained models? we investigate how large-scale pre-trained models can be leveraged for the translation of unseen low-resource languages and domains. we address this question by studying 16 african languages that are largely underrepresented in nlp research @xcite and further have little to no training data available ( §3). these languages provide an ideal testbed for two challenging knowledge transfer tasks: (1) how can pre-trained models create translations for languages unseen at train-ing time? and (2) since training data may only exist in single domain (i.e. religious texts), how can a model be trained in one domain and translate another effectively at test time? these questions are extremely relevant for our chosen languages because they all have millions of native speakers and a massive need for translation technologies. for example, news concerning the african continent are almost exclusively published in english, french, or arabic, and thereby inaccessible for speakers of only native african languages. this creates a bottleneck for information transmission, which becomes even more critical in times of crises @xcite @xcite . furthermore, the task of translating news has historically played a central role in translation research, e.g. in shared tasks since 2008 @xcite and as a test for determining human parity @xcite @xcite . to spur the development of dedicated news translation models for africa, we construct a benchmark of news translation for translating between 16 native african languages and english or @xcite . this allows us to compare three approaches to leveraging large-scale multilingual models for the translation of previously unseen languages: (1) zero-shot transfer, (2) continual pre-training on monolingual data, and (3) multi-domain fine-tuning on parallel data ( §5). we find that fine-tuning pretrained models on a few thousand sentences of high quality bitext is remarkably effective, and can be further augmented with continual pre-training on african languages and fine-tuning on news domain data ( §6). our contributions are the following: 1 1. we create a new african news corpus for machine translation (following principles of participatory research ∀ et al. ( 2020 )) covering 16 african languages. we find that having a targeted collection of translations is surprisingly effective, showcasing the power of local knowledge in so-called "zeroresource" scenarios @xcite . this paints a promising picture for the development of nlp technology for understudied languages: being able to customize these models for new language of interest with as little as 2k sentences and a few fine-tuning steps, mt developers and users from any language community are less dependent on choices and monetary interest of industry powerhouses from the global north @xcite . ## focus languages and their data focus languages. we focus on 16 african languages with varying quantities of available data @xcite , including moderately lowresource languages such as swahili and hausa, and very low-resource languages such as ghomálá' 2 with the bible being its largest available corpus. language characteristics. all languages are written in latin script, using letters of the basic latin alphabet with a few omissions (e.g "c", "q", "x", "z") and additions (e.g. "e", "o", "n", "o . ", including digraphs like "gb", "kp", "gh", and sometimes more than two-character letters). 13 of the languages are tonal, and about nine make use of diacritics. many african languages are morphologically rich. for example, all bantu languages are agglutinative. fon, mossi, and yorùbá are highly isolating. all languages follow the subject-verb-object sentence structure like english and french. table c provides more details. existing parallel corpora. we curate publicly available parallel data for our focus languages, which consists primarily of text in the religious domain. for most african languages, the largest available parallel corpora is jw300 @xcite , sourced from jw.org, which publishes biblical texts as well as lifestyle and opinion columns. varying quantities of data are available for 11 of the 16 focus languages. éwé, igbo, swahili, setswana, twi, yorùbá, and isizulu have over 400k parallel sentences. hausa and mossi have slightly more than 200k parallel sentences, while fon and naija have around 30k sentences. for the remaining five languages that are not in the jw300 corpus, foot_1 we make use of the bible. 4 we aligned the sentences automatically by the verses (around 31k in total). ghomálá' only has the new testament with 8k verses. bambara and wolof are missing some verses and books, leading to a total size of 28k and 22k. ## results and discussion we successfully adapt several multilingual pretrained models to previously unseen african languages and quantify the effectiveness of small indomain translation datasets. we discuss the effects of domain shift and analyze mitigation strategies. ## conclusion we have created mafand-mt, a corpus of 16 african languages to study translation systems for low-resource languages in the news domain. we investigate how to most effectively adapt large-scale pre-trained models to incorporate new languages and new domains. our findings suggest that as little as 2k sentences are sufficient for fine-tuning, with an improved performance, paving the way for others to create new translation systems without relying on large collections of web-sourced text. this has strong implications for languages that are spoken by millions but lack presence on the web. in the future, we hope to expand our coverage to additional under-resourced languages, and to develop even more effective fine-tuning objectives. currently, we are extending our corpus to chichewa, kinyarwanda, shona, and isixhosa, including an expansion of the hausa corpus, they will be released under mafand-mt dataset name.