Preparing the Vuk’uzenzele and ZA -gov-multilingual S outh A frican multilingual corpora (2023)

Abstract: This paper introduces two multilingual government themed corpora in various South African languages. The corpora were collected by gathering South African government speeches (ZA-gov-multilingual), as well as the South African Government newspaper (Vuk’uzenzele), that are translated into all 11 South African official languages. The corpora can be used for a myriad of downstream NLP tasks. The corpora were created to allow researchers to study the language used in South African government publications, with a focus on understanding how South African government officials communicate with their constituents. In this paper we highlight the process of gathering, cleaning and making available the corpora. We create parallel sentence corpora for Neural Machine Translation tasks using Language-Agnostic Sentence Representations (LASER) embeddings. With these aligned sentences we then provide NMT benchmarks for 9 indigenous languages by fine-tuning massively multilingual pre-trained language model.

## introduction the advancement of natural language processing (nlp) research in africa is impeded due to the scarcity of data for training models for various nlp tasks @xcite as well as availability of benchmarks and ways to reproduce them @xcite . for many south african languages there are still challenges finding easily available textual datasets @xcite even if there are many speakers for those languages @xcite . there is a need to focus on development of local language @xcite nlp resources. this paper builds upon the work of autshumato @xcite by creating automatically aligned parallel corpora from government textual data in the 11 official languages of south africa. while the autshumato project focused on creating machine translation tools for five indigenous languages, the resulting corpora lacked information about its origin or context, limiting its usefulness for other nlp tasks such as categorisation, topic modelling over time and other tasks that require contextual information of the content. our approach provides more comprehensive data that can support a wider range of nlp applications. our belief is that there is a significant opportunity to create a more user-friendly data collection process that can be easily maintained and provide extraction tools for others. it is essential to preserve the data source and structure it in a way that enables extensions. our goal is to enhance neural machine translation (nmt) resources in the government data domain by including all indigenous languages and broadening the translation directions beyond english as the source language. additionally, we recognise the importance of providing aligned data across all south african languages beyond english. further, this paper introduces parallel corpora datasets in the 11 official languages of south africa, created from text data obtained from the government. these datasets are designed to facilitate the development of nmt models. the corpora are automatically aligned, and are expected to serve as a valuable resource for researchers and practitioners working in the field of machine learning. the parallel corpora were generated using laser encoders @xcite , facilitating the one-to-one alignment of tokenised sentence data. the data was sourced from credible sources such as newspapers and academic journals and covers diverse topics including health, finance, and politics. we also provide nmt benchmarks for the parallel corpora by fine-tuning a massively multilingual model (m2m100 @xcite ) building on the work of @xcite . this paper is structured as follows. in the fol-lowing section, we detail the datasets that we have compiled, including their compilation methodology and the information they contain. we then describe how we have aligned and created parallel corpora using these datasets. the subsequent section presents our nmt experiments and provides an analysis of the results obtained. finally, we conclude the paper with our findings and make recommendations for future research. ## methods for processing and compilation the datasets are a two way parallel corpus of the 11 official languages of south africa, which are listed in the datasets contain texts written in the official languages of south africa, including afrikaans, english, isindebele, isixhosa, isizulu, siswati, sepedi, xitsonga, siswati, tshivenda, and setswana. as such, there are 55 ways of combining these languages into pairs, producing 55 distinct corpora in each of the datasets. the dataset uses the iso 639-2 language codes in its naming convention, i.e., 'aligned-afr-zul.csv'. by nature of compilation, some datasets have more observations than others, which could lead to varying results, i.e., if used for nmt, then a better model can be produced for two languages from a dataset with more observations as opposed to one with fewer observations. this compilation of data allows for further exploration into the complexities of south african language and discourse, creating a multi-dimensional representation of how language is used and interpreted in south africa. through these datasets, the range of language usage in south africa can be explored, providing insights into how different languages interact. ## nmt benchmarks minimal aligned sentence corpora, for lowresourced african languages, hinder the quality of nmt models trained from scratch @xcite @xcite . recently @xcite approached this problem by fine-tuning massively multilingual models, including the m2m100 model @xcite , on a small number of aligned sentences. the m2m100 model is a many-to-many non-english centric language model trained to translate directly between 100 languages, including five south african official languages @xcite . @xcite demonstrated how to effectively leverage this model for small quantities of data, to create nmt systems for languages and domains not included in pre-training. building on their work, we create baseline translation benchmarks for the vuk'uzenzele and zagov-multilingual datasets, in the government publication domain, by fine-tuning the m2m100 model. to provide our results in context and for comparison purposes we also fine-tune the m2m100 model on subsets of the existing autshumato parallel corpora obtained from the south african centre for digital language resources (mckellar, (2021,2,2,2,0,2) ( https://sadilar.org ). we focus our efforts on providing nmt benchmarks for the low resource african languages in the datasets, as such afrikaans translations are not included due to the relatively high availability of digital datasets in this language, and we leave this for future work. ## conclusion finally, this paper presented two multilingual corpora that are automatically aligned to facilitate the translation of texts between languages. these datasets contribute to an expanding collection of corpora for training african language nmt models that are left out or under-resourced in contemporary nlp research. it is the hope of the authors that these datasets will aid in creating nmt models for low-resource african languages and that the models can be used to facilitate access to translation services, empowering african speakers and writers to communicate more effectively in their native languages. it is also hoped that the datasets will further nlp research into the multilingualism of african languages and contribute to an understanding of the various dialects present in africa.