W eb C rawl A frican : A Multilingual Parallel Corpora for A frican Languages (2022)

Abstract: WebCrawl African is a mixed domain multilingual parallel corpora for a pool of African languages compiled by ANVITA machine translation team of Centre for Artificial Intelligence and Robotics Lab, primarily for accelerating research on low-resource and extremely low-resource machine translation and is part of the submission to WMT 2022 shared task on Large-Scale Machine Translation Evaluation for African Languages under the data track. The corpora is compiled through web data mining and comprises 695K parallel sentences spanning 74 different language pairs from English and 15 African languages, many of which fall under low and extremely low resource categories. As a measure of corpora usefulness, a MNMT model for 24 African languages to English is trained by combining WebCrawl African corpora with existing corpus and evaluation on FLORES200 shows that inclusion of WebCrawl African corpora could improve BLEU score by 0.01-1.66 for 12 out of 15 African to English translation directions and even by 0.18-0.68 for the 4 out of 9 African to English translation directions which are not part of WebCrawl African corpora. WebCrawl African corpora includes more parallel sentences for many language pairs in comparison to OPUS public repository. This data description paper captures creation of corpora and results obtained along with datasheets. The WebCrawl African corpora is hosted on github repository.

## introduction parallel corpus play a vital role in the progress of data driven machine translation research and development. availability of parallel corpora is still a concern for a large collection of world languages. africa alone is home to an estimated 1200 to 2100 spoken languages 2 and more than 34 languages 1 https://github.com/pavanpankaj/web-crawl-african 2 https://en.wikipedia.org/wiki/languages_of_africa with 1 million plus speakers. many of these 34 languages and associated language pairs fall under the low and extremely low resource categories and machine translation researchers face setbacks due to unavailability of parallel corpus in public domain. webcrawl african corpora is a little step put forward towards addressing this issue. languages covered in webcrawl african corpora include (1) afrikaans(afr), (2) amharic(amh), (3) chichewa(nya), (4) hausa(hau), ( 5 ) igbo(ibo), ( 6 ) lingala(lin), ( 7 ) luganda(lug), ( 8 ) oromo(orm), ( 9 ) swahili(swh), ( 10 ) swati(ssw), (11) tswana/setswana(tsn), ( 12 ) xhosa(xho), ( 13 ) xitsonga(tso), ( 14 ) yoruba(yor) (15) zulu(zul) and ( 16 ) english and language pairs include african-english and african-african pairs. webcrawl african is submitted as a part of large-scale machine translation evaluation for african languages shared task(data track) of wmt22 @xcite . rest of the paper is organized as follows. section-2 briefly covers related work on parallel corpora compilation through web data mining. section-3 covers content collection process followed for webcrawl african corpora compilation, section-4 details its alignment processes, section-5 presents results and analysis. finally section-6 presents the datasheet capturing responses to bunch of critical questions capturing many relevant facets of webcrawl african corpora ranging from motivation, composition, collection process, processing, users, distribution, maintenance and section-7 conclusion. ## related work a good amount of translated text are available on the web. however compilation of parallel corpora from web which involves suitable source discovery, sentence extraction, sentence alignment and quality assessment, control is not trivial. sentence alignment is the most critical part and alignment techniques range from simple heuristics to neural sentence embedding. @xcite al. (2020) ## content collection through web crawling creation of parallel corpora through web data mining, by making use of sources of multilingual translated text present on the web has almost became the de-facto technique for its cost effectiveness and scaling advantages. webcrawl african corpora creation followed similar strategy. as a first step, search has been carried out to discover potential websites having the following characteristics. we ended-up finding four websites namely (1) south african government foot_0 comprising of government communication, (2) nalibali foot_1 comprising of multi-genre short stories, (3) gotquestions foot_2 comprising of spiritual q&a and (4) african gospel foot_3 comprising of song lyrics. text content is mined from these four identified websites following four step process. african and english language web-pages are available in the websites selected, the same is exploited for web-page level alignment and consequently search space for sentence alignment reduced significantly. two different strategies are employed for sentence alignment duly leveraging the source websites information structure. algorithm 1 is used for english-african parallel sentence alignment using cross-lingual embeddings and algorithm 2 3 is used for fast african-african parallel sentence alignment based on common english sentences without using computationally expensive cross-lingual embeddings approach. ## results and analysis we propose to evaluate the compiled webcrawl african corpora in three ways. first, we present the distribution of extracted parallel sentences across language pairs. we then assess its usefulness by training a mnmt system for 24 @xmath0. ## conclusion this paper presented detailed description of we-bcrawl african corpora. the paper also describes approach and design choices to systematically create parallel corpora and extend the webcrawl african corpora through web data mining and alignment. webcrawl african corpora compiled comprises 695k parallel sentences spanning 74 different language pairs from english and 15 african languages, many of which fall under low and extremely low resource categories. webcrawl african corpora is hosted at https://github. com/pavanpankaj/web-crawl-african for noncommercial, not-for-profit and fair use. this corpora comprises sentences from multiple domains and includes government communication, short children stories, religious text and lyrics. though human verification of the corpora was not carried out but favourable characteristics of selected source websites aided to address some of the quality concerns relatively better. experiments and evaluation of results show that inclusion of webcrawl african corpora with wmt 2022 corpus has improved bleu score by 0.01-1.66 for 12 out of 15 @xmath1.18-0.68 for the 4 out of 9 @xmath2. webcrawl african corpora is primarily intended for machine translation tasks, specially for accelerating research on low resource and extremely low resource machine translation. it can also be used as monolingual corpora for tasks such as language modeling, corpus based language studies and few other nlp tasks with additional annotations.