G lot LID : Language Identification for Low-Resource Languages (2023)

Abstract: Several recent papers have published good solutions for language identification (LID) for about 300 high-resource and medium-resource languages. However, there is no LID available that (i) covers a wide range of low-resource languages, (ii) is rigorously evaluated and reliable and (iii) efficient and easy to use. Here, we publish GlotLID-M, an LID model that satisfies the desiderata of wide coverage, reliability and efficiency. It identifies 1665 languages, a large increase in coverage compared to prior work. In our experiments, GlotLID-M outperforms four baselines (CLD3, FT176, OpenLID and NLLB) when balancing F1 and false positive rate (FPR). We analyze the unique challenges that low-resource LID poses: incorrect corpus metadata, leakage from high-resource languages, difficulty separating closely related languages, handling of macrolanguage vs varieties and in general noisy data. We hope that integrating GlotLID-M into dataset creation pipelines will improve quality and enhance accessibility of NLP technology for low-resource languages and cultures. GlotLID-M model, code, and list of data sources are available: https://github.com/cisnlp/GlotLID.

## introduction the nlp community should create technology that covers as many languages as possible, not only medium-resource and high-resource languages. this goal can only be achieved if corpora for low-resource languages are available. web-mined datasets -including cc100 @xcite , mc4 @xcite and oscar @xcite -have made important contributions to low-resource nlp. in particular, they lay the ground for multilingual neural models like xlm-r @xcite , mt5 @xcite and @xcite . however, existing web-mined datasets have systematic quality issues @xcite and insufficient coverage of low-resource languages. low-quality datasets cause poor performance for downstream applications. they can also give rise to a misleading perception of progress when coverage of a low-resource language is claimed based on noisy data. nlp for low-resource languages requires high-quality datasets and high-quality datasets require high-quality lid (language identification). for this reason, high-quality lid for low-resource languages is paramount. to address this need, in this paper we present glotlid-m, a high-quality lid that covers 1665 languages. we use iso 639-3 to individuate languages. when expanding the scope of lid from a few hundred to 1665 languages, the problem of granularity becomes severe. in real-world settings, lid needs to support both macrolanguages and their varieties; it also needs to be robust against out-ofmodel cousins @xcite . we pay particular attention to this issue. while low-resource is our main focus, blevins and zettlemoyer (2022) point out that low-quality lid also affects high-resource corpora through contamination, resulting in claims of successful crosslingual transfer that are due to unrecognized coverage of low-resource languages. we also address this issue, e.g., we improve english f1 on the "universal declaration of human rights" corpus (udhr) to .85 compared to .43 for openlid. contributions. (i) we curate glotlid-c, a comprehensive dataset covering 1665 languages, most of them low-resource, from a diverse set of domains. (ii) we train glotlid-m on glotlid-c, an open-source lid covering these 1665 languages. (iii) in our experiments, glotlid-m outperforms several baselines by more than 12% absolute f1 on udhr, which we take as the best benchmark for our focus on low-resource languages. (iv) when balancing f1 and false positive rate (fpr), glotlid-m also outperforms baselines on flores-200, which is dominated by high-/medium-resource languages. broad coverage of languages, minimize out-ofmodel cousin errors. we strive for as broad a coverage as is possible given available datasets. this has two benefits. first, it reduces "out-of-model cousin" errors @xcite , i.e., it reduces the risk that a language not covered is misclassified as a closely related covered language. second, having lids that discriminate many low-resource languages is a pre-requisite for developing nlp technologies for the largest possible number of languages. yet many existing lids only cover a few hundred languages. in this study, we therefore focus on lids having a broad coverage, excluding cld2 @xcite , @xcite , langdetect @xcite and langid.py @xcite . these lids cover less than 100 languages or are outperformed by the models we compare with. ## dataset curation we now describe glotlid-c, a corpus for lid training that covers 1665 languages. source selection. we choose sources that we deem trustworthy (i.e., high chance of correct language label). to address the domain sensitivity of lid and broaden language coverage, we curate a diverse set of text domains. we review sources referenced by @xcite ; @xcite ; @xcite ; @xcite ; adebara and @xcite . in each case, we consider the collection methodology, selecting sources whose language labels are trustworthy. we generally do not use web-crawled sources to avoid the associated problems @xcite . most selected sources are derived from wikipedia, religious texts, collaborative translations, storybooks, and news sites. this gives us a coverage of 1832 languages, more than any other public lid. for a list of data sources, see §a. preprocessing. we ensure that each sentence is written in the correct script, based on the writing system databases of @xcite and @xcite . we use the glotscript @xcite python library to determine scripts. 2 we also eliminate duplicate sentences. statistics. our final corpus, glotlid-c, comprises 289 million sentences (i.e., lines of data) totaling 40gb and spans 1832 languages (identified by their iso 639-3 code). 1677 languages have more than 1000 sentences. refer to §d for the total number of sentences per language. train/test split. we designate 85% of the data as glotlid-c train. let n l be the number of sentences from language l in the remaining 15%. then we sample min(1000, n l ) sentences from it. we refer to the resulting dataset as glotlid-c test. contamination. to make sure our evaluation data (especially udhr, refer to §5.1) do not overlap with our sources, we compute contamination of udhr in glotlid-c train. we count a udhr test sentence as occurring in the training set if all of its word four-grams occur in one sentence of glotlid-c. most of these contaminations are due to two resources: wikipedia and tatoeba. 3 glotlid-c train shares 374 languages with udhr. for 292 languages, we find that none of the udhr test sentences occurs in the training data. for 57 languages, less than 10% of udhr test sentences occur in the training data. the remaining 25 languages with a contamination rate over 10% are all high/medium resource languages. in our experiments, we decided against remov-ing any sentences from glotlid-c, as there is little contamination of udhr for low-resource languages. we follow here most prior work which has the problem of contamination of udhr for high-resource languages. we will however remove from glotlid-c train the sentences causing contamination as part of our next release. ## glotlid-m we select fasttext @xcite as the architecture for glotlid-m, because it satisfies all requirements outlined in §2 as we will explain now. we train our fasttext model glotlid-m on glotlid-c train with 1832 languages. fasttext can easily handle the large number of languages in the corpus. because of this broad coverage, outof-model cousin errors are reduced. although we restrict the number of classes to 1665 for some experiments (e.g., in fasttext provides an open-source codebase for training, which supports customization and extension of glotlid-m. fasttext is easy to use: it offers a number of language bindings, making it compatible with multiple programming languages (including c++, python, java, node.js, rust, ruby, r) and reducing dependency, incompatibility and other software issues. fasttext meets the requirement of uncertainty assessment because it provides confidence scores that can serve as thresholds to effectively mitigate noise in the data. for the same reason, fasttext also supports granularity flexibility: we can accumulate probabilities over language varieties to get a good estimate of the probability of the macrolanguage. to this end, we simply add to the macrolanguage probability the probabilities of its varieties. this way, the system can return appropriate estimates at various levels of granularity. as a professionally designed and implemented linear classifier, fasttext is efficient: it had the best throughput of the candidate solutions we tested and can process large corpora with high speed. as a linear model, fasttext has the additional advantage of delivering explainable classification decisions. fasttext is a multinomial logistic classifier. the input sentence is represented as an average of n-gram embeddings. this allows us to visualize how much each n-gram contributed to the final prediction. see nllb @xcite , fig. 8 , for details. taking all these requirements together (and its good lid performance demonstrated in §6 and acceptable calibration in §f), glotlid-m, based on fasttext, is, in our opinion, an excellent tool for supporting our use case, the creation of highquality low-resource corpora. ## experimental setup we train glotlid-m on glotlid-c train using the hyperparameters in @xcite and otherwise fasttext defaults (see §b). following @xcite , @xcite and @xcite , we perform up-sampling for low resource languages. sentences from a language l representing p l of the dataset are sampled proportionally to p 1 t l where t is the temperature. following nllb @xcite and @xcite , we set 1 @xmath0.3. ## results focusing on the "subset" results for θ = .5, f1 is .973 on glotlid-c and .924 on flores; and fpr is .0002 on glotlid-c and .0010 on flores. this is a very good performance, in particular for the use case of low-resource corpus creation because low fpr means that the resulting corpora will be less contaminated. on udhr, again for the "subset" results for θ = .5, f1 is .770 and fpr .0006. this is again an encouragingly low fpr, but f1 is quite a bit lower than for glotlid-c and flores. the reason is that we have a domain shift (compared to glotlid-c) and many more languages (compared to flores), resulting in lower f1. although the udhr results should be improved further, we will now show that they outperform the state of the art. for the set? setting, glotlid-m consistently outperforms cld3 by a large margin. taking into account that f1 and fpr should be balanced, we also take it to outperform ft176. even though glotlid-m's fpr is slightly higher in some cases, its f1 is better by a large margin, so that it is clearly the better performing system. on udhr, glotlid-m also clearly outperforms openlid and nllb for f1 and fpr by large margins. on flores, f1 is slightly worse and fpr slightly better compared with openlid and nllb. we point out that this comparison is not entirely fair since openlid and nllb were designed with flores in mind. more importantly, our use case is the creation of low-resource corpora for which udhr is the more appropriate benchmark. comparing results for different thresholds, we observe that increasing θ lowers f1 (because recall is hurt) and lowers fpr (because precision is increased). this suggests that a higher thresh- flores-200 udhr cld3 ft176 openlid nllb cld3 ft176 openlid nllb |l| = 96 |l| = 108 |l| = 195 |l| = 188 |l| = 100 |l| = 124 |l| = 159 |l| = 172 lid model θ f1↑ fpr↓ f1↑ fpr↓ f1↑ fpr↓ f1↑ fpr↓ f1↑ fpr↓ f1↑ fpr↓ f1↑ fpr↓ f1↑ fpr↓ set? baselines .0 .753 .0098 .775 .0090 .923 .0051 .947 .0053 .544 .0099 .566 .0079 .645 .0056 .641 .0051 baselines θ 1 .779 .0081 .816 .0033 .923 .0050 .948 .0051 .576 .0081 .644 .0025 .676 .0046 .677 .0040 baselines θ 2 .799 .0060 .796 .0021 .923 .0044 .947 .0047 .618 .0060 .647 .0014 .718 .0034 .717 .0030 glotlid-m .0 .978 .0051 .987 .0042 .916 .0043 .947 .0035 .868 .0033 .868 .0030 .848 .0020 .847 .0019 glotlid-m .3 .980 .0042 .987 .0037 .898 .0020 .927 .0019 .881 .0028 .879 .0026 .846 .0015 .844 .0015 glotlid-m .5 .980 .0031 .987 .0029 .886 .0014 .916 .0013 .903 .0023 .890 .0021 .847 .0012 .846 .0011 set! baselines .0 .952 .0104 .881 .0093 .923 .0051 .950 .0053 .922 .0101 .739 .0081 .881 .0063 .854 .0058 glotlid-m .0 .983 .0104 .991 .0093 .922 .0051 .954 .0053 .952 .0100 .927 .0081 .926 .0064 .925 .0060 for the less realistic set! setting, glotlid-m performs better than cld3 and ft176 and comparably to openlid and nllb. overall, glotlid-m clearly outperforms all baselines for the lowresource corpus creation use case. to analyze variance of results, we ran three glotlid experiments with different initial seeds on the 200 languages with the most data, splitting the data into 80% train and 20% test. the f1 score was .991 each time. this indicates that the variance of fasttext in this task (and by extension glotlid) is negligible. ## analysis in this section, we analyze the glotlid-m results summarized in most errors. we first analyze languages with a high number of errors. most of the fifteen pairs of "conflated" languages shown in the table are closely related languages: varieties of arabic (standard, najdi, egyptian and levantine), persian (iranian, dari), chinese (mandarin, yue, wu, hakka), english (standard, liberian), quechua (huallaga huánuco, huamalíes-dos de mayo huánuco), finnic (finnish, karelian), slavic (russian, church slavic), bihari (bihari, bhojpuri) and hindi (standard, awadhi). in many of these cases, speakers of arabic, chinese and english (and perhaps also hindi, persian and bihari) are diglossic linguistic communities. there may be a lack of clear separation between the two conflated varieties in the available corpora because speakers switch back and forth between more formal and less formal ways of speaking depending on factors like context, audience and subject. this type of fluid switching between languages often occurs in a single sentence or conversation, i.e., it manifests as code switching. as a result, much of the text (and speech) produced in one language may be mixed with the other language. new methods will have to be developed to deal with these quite complex challenges of creating training corpora for language identification; see also @xcite . apart from these related languages, at least four conflated language pairs in glotlid-m confuses mandarin and cherokee because our cherokee training data do not cover the cherokee syllabary script. sentences written in this script are returned with a close to uniform distribution over several other scripts, including chinese, japanese and thai, which explains the confusion. the dinka test set is noisy. in a manual inspection, we found 377 sentences that are clearly english, not dinka. because glotlid-m did not learn very well to discriminate english and liberian english, 174 of these 377 sentence were classified as liberian english. ## conclusion we create glotlid-c, an lid resource that covers 1832 languages, several times more than prior work. we introduce glotlid-m, an open-source lid that covers 1665 languages with good results. the comparison of glotlid-m against four lid baselines shows superior performance for the lowresource use case. in future research, we would like to improve quality of our training corpora and add more low-resource languages in to glotlid. we hope glotlid will be a valuable resource in creating higher-quality corpora for low-resource languages.