Building Resources for Emakhuwa: Machine Translation and News Classification Benchmarks (2024)

Abstract: This paper introduces a comprehensive collection of NLP resources for Emakhuwa, Mozambique’s most widely spoken language. The resources include the first manually translated news bitext corpus between Portuguese and Emakhuwa, news topic classification datasets, and monolingual data. We detail the process and challenges of acquiring this data and present benchmark results for machine translation and news topic classification tasks. Our evaluation examines the impact of different data types—originally clean text, post-corrected OCR, and back-translated data—and the effects of fine-tuning from pre-trained models, including those focused on African languages.Our benchmarks demonstrate good performance in news topic classification and promising results in machine translation. We fine-tuned multilingual encoder-decoder models using real and synthetic data and evaluated them on our test set and the FLORES evaluation sets. The results highlight the importance of incorporating more data and potential for future improvements.All models, code, and datasets are available in the https://huggingface.co/LIACC repository under the CC BY 4.0 license.

## introduction natural language processing (nlp) has witnessed significant advances in recent years. however, addressing low-resource languages, including those spoken in africa, is a persistent challenge. these languages often lack the extensive datasets and evaluation sets readily available for high-resource languages. this scarcity significantly hinders nlp progress in these languages. emakhuwa, spoken in northern and central mozambique, is no exception. while it is the most spoken language in the country, it has received limited attention in nlp research. this lack of resources hinders the development of nlp applications that could benefit emakhuwa speakers. we present a valuable collection of emakhuwa resources for the nlp community to address this gap. our contributions include: ## emakhuwa emakhuwa (also known as makua, macua, or makhuwa) is a bantu language spoken in northern and central mozambique, i.e., niassa, cabo delgado, and nampula, as well as in some parts of the zambezia province. it is estimated that approximately 25% of the country's population of 30 million people make use of the language daily as an alternative to portuguese (ronaldo rodrigues de paula, 2016). the language is spoken in neighbouring countries such as tanzania and malawi, with relatively few speakers. there are eight variants of emakhuwa, with emakhuwa-central (iso-639 code vmw) being the standard variety @xcite . emakhuwa is an svo (subject-verb-object) language written in latin script. unlike many other languages, it has no grammatical gender. like other bantu languages, it is complex and rich, featuring agglutinative morphology and tonal attributes. ## related works evaluation datasets and benchmarks are crucial for developing nlp models, especially for lowresource languages such as those spoken in africa. the scarcity of evaluation datasets and benchmarks for low-resource african languages has significantly hindered the progress of nlp in this region. without these essential tools, it is difficult to assess the performance of nlp models for these languages, making it harder to identify areas for improvement and track progress. despite these challenges, a growing movement within the research community aims to address this gap. researchers and organizations are actively working to create and disseminate evaluation datasets and benchmarks for low-resource african languages. one prominent group in this effort is the masakhane research community. this community has established a robust network through which many resources have been developed and made available to the research community. some key resources include masakhanews @xcite , mafand-mt @xcite , masakhaner @xcite , or afrisent (muhammad et al., 2023), among others. the impact of these efforts is becoming increasingly evident. for example, in a study by @xcite , researchers demonstrated that their pre-trained models surpassed the benchmarks for news topic classification established by @xcite . similarly, the study by @xcite showed that their language models improved machine translation across multiple language pairs, thus outperforming the benchmarks set by @xcite . nevertheless, the existing evaluation datasets and benchmarks for african languages still cover a limited number of languages. for instance, although emakhuwa is an african language that has seen some development in nlp, the data and evaluation tools are not widely available in the public domain, highlighting the need for further resource development and accessibility. for example, @xcite compiled what appears to be the first parallel dataset for machine translation between emakhuwa and portuguese. despite their efforts in dataset construction, the reproducibility of their results is compromised due to the non-reproducible nature of their evaluation sets. ## data collection our data collection includes (1) the first manually translated parallel news data, (2) data extracted from printed books, and (3) monolingual data. a notable aspect of our collected data is its diverse range of sources, which introduces significant variations in writing styles. this diversity includes different genres, contexts, and periods, providing a multifaceted linguistic resource. for example, the manually translated parallel data reflects the contemporary language, while the data extracted from printed books captures more of traditional narratives, dialogues, and idiomatic expressions. monolingual data, on the other hand, spans content from modern publications and digital platforms and adds further variety. this heterogeneity is crucial for creating nlp models that can handle the complexity and richness of emakhuwa in real-world applications. below, we describe in detail the process of creating each of the types of data collected. ## datasets we gathered approximately 18,540 aligned parallel sentences, 1,897 labeled news articles, 1,911 post-corrected ocr bitext sentences, and 14,463 monolingual sentences. this data was meticulously cleaned and prepared for machine translation and news topic classification. while the dataset holds potential for other nlp tasks, such as ocr postcorrection, loanword detection, and language modeling, these applications fall outside the scope of the benchmarks presented in this study. once cleaned, the data was divided into training (train), validation (dev), and test (test) sets. for the machine translation task, we allocated 17k sentences for training. the dev and test sets were meticulously constructed by extracting sentences from the compiled parallel corpus. to ensure high quality and wide domain coverage in the dev and test set, we also included a small set of sentence pairs from the ali et al. (2021) test set. the following steps were then applied to select sentences for the test and dev sets: (1) source sentences had to contain between 5 and 150 tokens; (2) each sentence needed to start with a capital letter and end with appropriate punctuation; (3) sentence pairs were chosen if the length ratio between the source and target language sentences fell between 0.66 and 1.5, as recommended by @xcite . this helps reduce potential issues caused by significant discrepancies in sentence lengths between the two languages. this ultimately resulted in 964 sentences for validation and 993 for testing. for the news classification dataset, the split included 1,337 articles for training, 185 for validation, and 375 for testing corresponding to a 70%, 10%, and 20% split ratio respectively. ## benchmarks and experiments to promote reproducibility, we provide detailed descriptions of the experiments and benchmarks conducted, particularly focusing on machine translation and news classification tasks. ## results in this section, we discuss the benchmark results of both machine translation and news topic classification. ## conclusion this paper presents a valuable collection of nlp resources for the emakhuwa language, including parallel text corpora, news topic classification datasets, and monolingual data. we also establish benchmark results for machine translation and news topic classification tasks, demonstrating the potential of these resources to advance nlp research in emakhuwa. our benchmarks for the respective tasks are quite challenging. for news topic classification, classical machine learning approaches yield better results, achieving an f1 score of 72.83. we observe relatively strong performance in machine translation, particularly when translating from portuguese to emakhuwa. our model, fine-tuned from nllb using a combination of real and synthetic data generated through back-translation, achieves a bleu score of 40.90 and a chrf score of 74.04. by publicly making these resources available, we aim to foster further research and development in nlp for african languages, ultimately benefiting the emakhuwa-speaking community.