From N-grams to Pre-trained Multilingual Models For Language Identification (2024)

Abstract: In this paper, we investigate the use of N-gram models and Large Pre-trained Multilingual models for Language Identification (LID) across 11 South African languages. For N-gram models, this study shows that effective data size selection remains crucial for establishing effective frequency distributions of the target languages, that efficiently model each language, thus, improving language ranking. For pre-trained multilingual models, we conduct extensive experiments covering a diverse set of massively pre-trained multilingual (PLM) models – mBERT, RemBERT, XLM-r, and Afri-centric multilingual models – AfriBERTa, Afro-XLMr, AfroLM, and Serengeti. We further compare these models with available large-scale Language Identification tools: Compact Language Detector v3 (CLD V3), AfroLID, GlotLID, and OpenLID to highlight the importance of focused-based LID. From these, we show that Serengeti is a superior model across models: N-grams to Transformers on average. Moreover, we propose a lightweight BERT-based LID model (za_BERT_lid) trained with NHCLT + Vukzenzele corpus, which performs on par with our best-performing Afri-centric models.

## introduction automatic language identification (lid) is the task of determining the underlying natural language used in a written or spoken corpus @xcite . this is a challenging problem, especially for languages with insufficient training examples and closely related languages, particularly lowresourced languages @xcite . for south african languages, building quality lid technologies is significantly important for sourcing internet data, which has served as a de-facto repository for many low-resourced languages, especially from public domains such as news websites @xcite @xcite @xcite . statistical approaches for automatic lid such as n-grams @xcite , and more classical machine learning models such as logistic regression, naive bayes, random forest, boosting machines, support vector machines, and clustering techniques (e.g k nearest neighbors) have been proposed @xcite . moreover, contemporary neural-based architectures such as deep neural networks and convolutional neural networks have also been tested. in all cases, not enough work for the south african languages is reported. on the other hand, recent algorithmic advancements such as transformer architectures have made a significant impact on the natural language processing landscape @xcite . with this sudden shift in perspective, many works have proposed automatic lid using large pre-trained multilingual models, derived from attention mechanisms @xcite . large pre-trained multilingual models are transformerbased architectures simultaneously trained on multiple languages (hence multi-lingual) using various techniques such as token (s) masking training technique, where tokens from a given sentence example are hidden and the objective of the training transformer is to predict the hidden word (s). in this work, we make use of the recently released vuk'zenzele crawled corpus @xcite and the nchlt dataset @xcite to develop and experiment on automatic language identification models on 10 low-resourced south african languages: northern sesotho (nso), setswana (tsn), sesotho (sot), isizulu (zul), isixhosa (xho), isiswati (ssw), isindebele (nbl), tshivenda (ven), xitsonga (tso), and afrikaans (af). additionally, we included the highresource south african english (eng) to ensure representation of all 11 official languages in south africa. we conduct extensive experiments on ngram models, large pre-trained multilingual models -xlm-r, mbert, and afri-centric multilingual models -afriberta, afro-xlmr, afrolm, and serengeti. we shed light on the limitations and robustness of n-gram-based approaches and the significant improvement boost of pre-trained multilingual models, especially, for those pre-exposed to low-resourced south african languages during pre-training. ## related work large pre-trained multilingual models have shown astonishing state-of-the-art results on various natural language processing (nlp) tasks such as machine translation, question answering, and sentiment analyses @xcite @xcite . a precursor of these tasks is the crawling of large volumes of internet data and categorizing the data into different languages (i.e. language identification) for pretraining. for language identification, many works have used pre-trained multilingual models to expand monolingual datasets using the internet. @xcite conducted a comparative study between adaptive naive bayes, heli2.0, multilingual bert, and xlm-r models for dravidian language identification in a code-switched context (i.e. a conventional modus operandi for communication on the internet). @xcite developed a transformer-based lid model aside from basic filtering techniques such as tunableprecision-based filtering using a created wordlist, tf-idf filtering, and a percent-threshold filtering threshold proposed in their study to filter noisy web-crawled content. although they were able to collect corpora for over 212 languages, their set-up for their best-performing transformer model was unclear. similar to our work, @xcite conducted a comparative study on distil-bert, albert, and xlm-r and showed that a lightweight version of distilbert delivers comparable results to resource-intense models. @xcite , on the other hand, implemented a massive transformer-based lid model with 12 attention layers and heads. they then trained this model on 512 languages with close to 2 million sentences across 14 language families (south african languages included). their model achieved over 95 % f1 score on a left-out test sets, outperforming available lid tools: cld version 2, langid, fast- corpora no. sent voc unq. voc train dev test vuk 33k 690k 132k 3395 -728 nchlt + vuk 74k 16m 258k 6790 -1454 ## methodology the methodology employed in this study uses language-identifiable monolingual corpora from reliable sources as training examples for language identification and compares various pre-trained multilingual models for the task of discriminating between languages. ## discussions ensuring the development of robust lid detection systems remains a critical research area with implications on many nlp tasks. importantly, the availability of reliable lid systems ensures accurate reporting on the state of low-resourced languages @xcite . on the side of model performance, baseline techniques such as naive bayes, support vector machines, and logistic regression seem to be performing quite well on the task of sentence-level language identification. we recommend these models for further research for high-level lid, compared to large pre-trained multi-lingual models which require specialized computing resources such as gpus, to accelerate training. however, we deem such trade-offs to require more research, especially in complex lid subtasks such as code-switching, or similar language discrimination. we also, highlight the importance of evaluation metric selection as we have observed that most of the lid tools explored in this study are not completely wrong, but rather have challenges discriminating among closely related languages. therefore, we recommend precision as an evaluation metric for lid to be further investigated. ## conclusion language identification remains a critical study area for the widespread inclusion of many lowresourced languages into the booming technology space. in this study, we experimented with statistical approaches, traditional machine learning techniques, the recent advanced pre-trained multilingual models, as well as lid tools publicly available (covering a wide range of african languages) on the task of lid for 11 south african language discrimination. we were able to shed light on the approaches showing promising results in the south african language context and made suggestions for future directions. concretely, we showed that the naive bayes algorithm performs surprisingly well for lid and warrants further exploration and exploitation, especially given its cheap-compute advantage. finally, we compared publicly available pre-trained models and showed that context-exposed models have an edge over other contextoblivious multilingual models, where context refers to the language. we released our models on hug-gingface and code with datasets on github.