A fro LID : A Neural Language Identification Tool for A frican Languages (2022)

Abstract: Language identification (LID) is a crucial precursor for NLP, especially for mining web data. Problematically, most of the world’s 7000+ languages today are not covered by LID technologies. We address this pressing issue for Africa by introducing AfroLID, a neural LID toolkit for 517 African languages and varieties. AfroLID exploits a multi-domain web dataset manually curated from across 14 language families utilizing five orthographic systems. When evaluated on our blind Test set, AfroLID achieves 95.89 F_1-score. We also compare AfroLID to five existing LID tools that each cover a small number of African languages, finding it to outperform them on most languages. We further show the utility of AfroLID in the wild by testing it on the acutely under-served Twitter domain. Finally, we offer a number of controlled case studies and perform a linguistically-motivated error analysis that allow us to both showcase AfroLID’s powerful capabilities and limitations

## introduction language identification (lid) is the task of identifying the human language a piece of text or speech segment belongs to. the proliferation of social media have allowed greater access to multilingual data, making automatic lid an important first step in processing human language appropriately @xcite . this includes applications in speech, sign language, handwritten text, and other modalities of language. it also includes distinguishing languages in codemixed datasets @xcite . unfortunately, for the majority of languages in the world, including most african languages, we do not have the resources for developing lid tools. in addition to resource creation, lack (or poor performance) of lid tools negatively impacts preprocessing of lr languages since lid can be a prerequisite for determining, e.g., appropriate tokenization. @xcite . furthermore, some preprocessing approaches may be necessary for certain languages, but may hurt perforrmance in other languages @xcite . developing lid tools is thus vital for all nlp. in this work, we focus on lid for african languages and introduce afrolid. afrolid is a neural lid tool that covers 517 african languages and language varieties 2 across 14 language families. the languages covered belong to 50 african countries and are written in five diverse scripts. we show the countries covered by afrolid in figure 1 . examples of the different scripts involved in the 517 languages are displayed in figure 2 . to the best of our knowledge, afrolid supports the largest subset of african languages to date. afrolid is also usable without any end-user training, and it exploits data from a variety of domains to ensure robustness. we manually curate our clean training data, which is of special significance in low resource settings. we show the utility of afrolid in the wild by applying it on two twitter datasets and compare its performance with existing lid tools that cover any number of african languages such as cld2 @xcite , cld3 @xcite , franc, langdetect @xcite , and langid.py @xcite . our results show that afrolid consistently outperforms all other lid tools for almost all languages, and serves as the new sota for language identification for african languages. to summarize, we offer the following main con2 our dataset involves different forms that can arguably be viewed as varieties of the same language such as twi and akan. figure 2 : examples from the five scripts in our data. to facilitate nlp research, we make our models publicly available. the rest of the paper is organized as follows: in section 2 we discuss a number of typological features of our supported languages. we describe afrolid's training data in section 3. next, we introduce afrolid in 4. this includes our experimental datasets and their splits, preprocessing, vocabulary, implementation and training details, and our evaluation settings. we present performance of afrolid in section 5 and compare it to other lid tools. our analysis show that afrolid outperforms other models for most languages. in the same section, we also describe the utility of afrolid on non-latin scripts, creole languages, and languages in close geographical proximity. although afrolid is not trained on twitter data, we experiment with tweets in section 6 in order to investigate performance of afrolid in out of domain scenarios. through two diagnostic studies, we demonstrate afrolid's robustness. we provide an overview of related work in section 7. we conclude in section 8, and outline a number of limitations for our work in section 9. ## typological information language families. we experiment with 517 african languages and language varieties across 50 african countries. these languages belong to 14 language families (eberhard et al., 2021) as follows: afro-asiatic, austronesian, creole (english based), creole (french based), creole (kongo based), creole (ngbadi based), creole (portuguese based), indo-european, khoe-kwadi (hainum), khoe-kwadi (nama), khoe-kwadi (southwest), niger-congo, and nilo-saharan. the large and typologically diverse data we exploit hence endow our work with wide coverage. we show in figure 1 a map of africa with the countries afrolid covers. we also show the number of languages we cover, per country, in figure e in the appendix. table e.1, table e.2, and table e. 3 in the appendix also provide a list of the languages afrolid handles. we represent the languages using iso-3 codes 3 for both individual languages and macro-languages. we use a macro-language tag when the language is known but the specific dialect is unknown. for this reason we specify that afrolid supports 517 african languages and language varieties. sentential word order. there are seven categories of word order across human languages around the world. these are subject-verb-object (svo), subject-object-verb (sov), object-verbsubject (ovs), object-subject-verb (osv), verbobject-subject (vos), verb-subject-object (vso), and languages lacking a dominant order (which often have a combination of two or more orders within its grammar) @xcite . again, our dataset is very diverse: we cover five out of these seven types of word order. ## curating an african language dataset afrolid is trained using a multi-domain, multiscript language identification dataset that we manually curated for building our tool. to collect the dataset, we perform an extensive manual analysis of african language presence on the web, identifying as much publicly available data from the 517 language varieties we treat as is possible. we adopt this manual curation approach since there are only few african languages that have any lid tool coverage. in addition, available lid tools that treat african languages tend to perform unreliably @xcite . we therefore consult research papers focusing on african languages, such as @xcite , or provide language data @xcite , sifting through references to find additional african data sources. moreover, we search for newspapers across all 54 african countries. 4 we also collect data from social media such as blogs and web fora written in african languages as well as databases that store african language data. these include lanafrica, sadi-lar, masakhane, niger-volta-lti, and alti. our resulting multi-domain dataset contains religious texts, government documents, health documents, crawls from curated web pages, news articles, and existing human-identified datasets for african languages. as an additional sanity check, we ask a number of native speakers from a subset of the languages to verify the correctness of the self-labels assigned in respective sources within our collections. 5 our manual inspection step gave us confidence about the quality of our dataset, providing near perfect agreement by native speakers with labels from data sources. in total, we collect 100 million sentences in 528 languages across 14 language families in africa and select 517 languages which had at least 2000 sentences. again, the dataset has various orthographic scripts, including 499 languages in latin scripts, eight languages in ethiopic scripts, four languages in arabic scripts, one language in vai scripts, and one in coptic scripts. ## afrolid experimental dataset and splits. from our manually-curated dataset, we randomly select 5, 000, 50, and 100 sentences for train, development, and test, respectively, for each language. 6 overall, afrolid data comprises 2, 496, 980 sentences for training (train), 25, 850 for development (dev), and 51, 400 for test (test) for 517 languages and language varieties. preprocessing. we ensure that our data represent naturally occurring text by performing only minimal preprocessing. specifically, we tokenize our data into character, byte-pairs, and words. we do not remove diacritics and use both precomposed and decomposed characters to cater for the inconsistent use of precomposed and decomposed characters by many african languages in digital media. 7 we create our character level tokenization scripts and generate our vocabulary using fairseq. we use sentencepiece tokenizer for the word level and byte-pair tokens before we preprocess in fairseq. vocabulary. we experiment with byte-pair (bpe), word, and character level encodings. we used vocabulary sizes of 64k, 100k, and 2, 260 for the bpe, word, and character level models across the 517 language varieties. the characters included both letters, diacritics, and symbols from other non-latin scripts for the respective languages. implementation. afrolid is built using a transformer architecture trained from scratch. we use 12 attention layers with 12 heads in each layer, 768 hidden dimensions, making up ∼ 200m parameters. 8 hyperparameter search and training. to identify our best hyperparameters, we use a subset of our training data and the full development set for our hyperparameter search. namely, we randomly sample 200 examples from each language in our training data to create a smaller train set, foot_5 while using our full dev set. we train for up to 100 epochs, with early stopping. we search for the following hyperparameter values, picking bolded ones as our best: dropout rates from the set {0.1, 0.2, 0.3, 0.4, 0.5}, learning rates from {5e-5, 5e-6}, and patience from {10, 20, 30}. other hyperparameters are similar to those for xml-r @xcite . we perform hyperparameter search only with our character level model and use identified values with both the bpe and word models. evaluation. we report our results in both macro f 1 -score and accuracy, selecting our best model on into u + 00e4. in unicode, they are included primarily to aid computer systems with incomplete unicode support, where equivalent decomposed characters may render incorrectly. dev based on f 1 . for all our models, we report the average of three runs. ## diagnostic case studies although afrolid is not trained on twitter data, we evaluate its performance on twitter to investigate the robustness of our models in out of domain scenarios. namely, we carry out two diagnostic case studies using twitter data. in the first study, which we refer to as twitter in the wild, we use unannotated tweets crawled from the web. in the the second, we use annotated tweets. we now turn to the details of these studies. ## related work lid tools are often used to select data to pre-train language models (buck et al., 2014a) and, more generally, develop multilingual corpora @xcite @xcite . for many languages, including african languages, lid tools are either not available or perform poorly @xcite . a few works, however, have already focused on african language identification. for example ## conclusion we introduced our novel african language identification tool, afrolid. to the best of our knowledge, afrolid is the first publicly available tool that covers a large number of african languages and language varieties. afrolid also has the advantages of wide geographical coverage (50 african countries) and linguistic diversity. we demonstrated the utility of afrolid on non-latin scripts, creoles, and languages with close geographical proximity. we also empirically showed afrolid's superiority to five available tools, including in performance in the wild as applied to the much-needed twitter domain. in the future, we plan to extend afrolid to cover the top 100 most popular languages of the world as well as code-switched texts.