Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus (2020)

Abstract: Large text corpora are increasingly important for a wide variety of Natural Language Processing (NLP) tasks, and automatic language identification (LangID) is a core technology needed to collect such datasets in a multilingual context. LangID is largely treated as solved in the literature, with models reported that achieve over 90% average F1 on as many as 1,366 languages. We train LangID models on up to 1,629 languages with comparable quality on held-out test sets, but find that human-judged LangID accuracy for web-crawl text corpora created using these models is only around 5% for many lower-resource languages, suggesting a need for more robust evaluation. Further analysis revealed a variety of error modes, arising from domain mismatch, class imbalance, language similarity, and insufficiently expressive models. We propose two classes of techniques to mitigate these errors: wordlist-based tunable-precision filters (for which we release curated lists in about 500 languages) and transformer-based semi-supervised LangID models, which increase median dataset precision from 5.5% to 71.2%. These techniques enable us to create an initial data set covering 100K or more relatively clean sentences in each of 500+ languages, paving the way towards a 1,000-language web text corpus.

## introduction thousands of languages are spoken in our world @xcite , but technologies like machine translation (mt) and automatic speech recognition (asr) are only available in about 100 of them. as internet access becomes increasingly common with the spread of smartphones @xcite , bringing technologies that can help lower language and literacy barriers to more languages is ever more important. unfortunately, bringing language technologies to more languages is costly, as for many technologies, extending to an additional language has generally required the use of large parallel labeled datasets. for example, asr systems are usually trained on large sets of audio recordings and transcriptions, while mt systems have historically needed a set of bilingual sentence pairs. increasingly, small parallel datasets do exist for many languages @xcite @xcite , but those resources were either produced at high cost, or are restricted to narrow domains. parallel resources, which rarely occur naturally, remain scarce for most languages. monolingual text data, which is more commonly produced, is also used in building out language technologies: for example, in training language models, which are used in many applications ranging from next-word prediction in keyboard input software @xcite to asr and mt @xcite . historically, though, a monolingual text corpus by itself has not been sufficient to build asr and mt systems in a new language: at least some parallel data was typically necessary. recently, however, significant progress has been made in cross-lingual learning for nlp tasks @xcite @xcite : for example, some approaches appear capable of extending machine translation models to new languages with only monolingual data @xcite @xcite , and similar findings have been reported for other nlp tasks @xcite . for asr it is possible to combine a target-language language model with an acoustic model from a phonologically similar language, with no need for parallel datasets of audio recordings and transcriptions @xcite . such approaches are likely to get even more effective with nearly-universal acoustic models @xcite and more scalable grapheme-to-phoneme modeling approaches @xcite @xcite @xcite . even if more work is needed to establish when such approaches will work well @xcite @xcite , having useful monolingual text corpora across languages is clearly a prerequisite to exploring such approaches further. additionally, using techniques such as labse @xcite , parallel corpora can also be constructed from monolingual corpora. unfortunately, it has proven challenging to derive highly multilingual text corpora from the web @xcite . one commonly cited reason is that most web content is written in widely-spoken languages like english and mandarin foot_0 . still, previous work has shown that the web contains labeled and unlabeled data in thousands of languages @xcite . since most web pages do not have any language labels attached, previous efforts to build web text corpora often rely at least in part on crawling selected urls and top-level domains in each language, or use popular n-gram language identification (langid) models like fasttext @xcite to target a limited number of languages @xcite . however, previous work (section 2) has shown that it is possible to build highly accurate langid systems covering 1,000+ languages. thus, aiming to build a 1,000-language web text corpus, we trained a similar large-coverage langid model, and used it in a large web crawl. however, we found that such langid systems do not deliver useful results in a real-world web-crawl scenario. to address this, we make the following contributions: this work focuses on monolingual corpora, but the problems described also apply to parallel texts, and it is straightforward to extend the improvements described here to parallel data crawling. ## langid approaches for web corpora to create text corpora in as many languages as possible, we needed a broad-coverage, accurate langid model for our web crawl. we cover existing work and describe our model, built along similar lines. ## failure modes of langid models on web text despite our langid models performing well on the held-out test sets, when applied on real-life web data, the models were not as accurate as we had expected. we performed an initial limited crawl with a 648language model, but some quick evaluations showed that the results were highly noisy, so we performed a full crawl on ≈100b documents with a 224-language model to isolate the problems for closer analysis. this model had comparable performance to the models in while we expected some accuracy loss due to the domain mismatch between clean training data and ## improving langid precision on web text monolingual web-text corpora afflicted by the issues described in section 3 will likely prove unusable for practical purposes. we report on two distinct approaches we found helpful in improving precision. ## web-crawled dataset and comparison with other public datasets using the above methods foot_3 , we performed a deep crawl of the web (@xmath0. using percent-threshold filtering foot_4 we made a recall-focused dataset, then post-filter with a ss-lid model for high precision, yielding a larger, cleaner set than is found in similar corpora. more details and comparisons to public corpora (oscar, ccnet) are in appendices e and d. ## future work our approach yielded usable monolingual text corpora in ≈600 languages. internal user experience research suggests the web may now contain at least some amount of monolingual text in thousands of languages, so we plan to scale up with more multilingual langid models, like our 1,629-language model. truly covering the linguistic richness of the web will also need crawling approaches to be fine-tuned further. text for some languages may only be found in pdf files @xcite , and some scripts are commonly represented in non-unicode fonts-such as kruti dev for devanagari, requiring separate detection for conversion into unicode-encoded devanagari @xcite . applying ocr may also help handle non-unicode text, and can uncover textual content within images. and many languages that are not officially written in the latin alphabet have informal transliterated orthographies @xcite ; our models can identify the most common ones, but we could cover more. finally, our work focused on a web crawl, but many new internet users primarily use their language online on social media platforms and in chat messages @xcite . other work has looked at applying langid to social media @xcite @xcite . our techniques should help improve langid accuracy in this challenging domain, too. ## conclusion language identification (langid) is by no means a solved problem, and n-gram models are much worse than popularly believed. we trained langid models covering up to 1,629 languages, but found that even seemingly high-quality models (> 95 f1) were nearly unusable in practice for low-resource languages. we described and analyzed several major issues encountered in applying langid to a real-life web crawl. these practical problems included large amounts of noise, much of which appears to be natural language and can't be easily filtered out; insufficient expressiveness of n-gram models; issues with related languages; and a massive class imbalance problem, meaning that even 99% f1 can be insufficient. to solve these issues, we developed two major improvements to our langid system: tunable-precision filtering methods (for which we release wordlists in about 500 languages) and semi-supervised neural models. these allowed us to create usable monolingual text corpora across hundreds of languages based on our deep web crawl, with much more and cleaner data per language than previously published approaches. such corpora hold great promise for bringing technologies like mt and asr to more languages, and we believe it should be possible to use the approaches we outlined to create monolingual corpora in many more languages, which should help extend language technology even further.