Identifying Rare Languages in C ommon C rawl Data is a Needles-in-a-Haystack Problem (2025)

Abstract: Automatic language identification is frequently framed as a multi-class classification problem. However, when creating digital corpora for less commonly written languages, it may be more appropriate to consider it a data mining problem. For these varieties, one knows ahead of time that the vast majority of documents are of little interest. By minimizing resources spent on classifying such documents, we can create corpora covering previously overlooked languages faster than existing pipelines. To demonstrate the effectiveness of the targeted mining perspective, we introduce a new pipeline that can filter a single snapshot in two hours. We also provide web corpora for several French-based Creoles.

## introduction as natural language processing (nlp) technologies gain prominence, so does the demand for corpora. filtered versions of common crawl data, such as oscar @xcite @xcite , madlad-400 @xcite , @xcite , and fineweb-2 @xcite , are one solution to this demand. for english and mandarin, this approach has yielded terabyte-sized datasets. yet, even for widely written languages like romanized arabic and hindi, filtering common crawl has yielded modest results. as such, language identification (lid) for the thousands of varieties with little representation in web corpora has become a prominent research agenda @xcite . for example, @xcite document some types of noise that affect less common languages. similarly, openlid @xcite , glotlid @xcite , and @xcite , try to improve the coverage of lid models. at the same time, creole languages have garnered considerable attention @xcite . they generally have strong lexical overlap with certain widely-spoken languages, such as english, french, and portuguese, but differ in morphosyntax. although often lumped into a broad "low-resource" category, contact varieties bring unique challenges and opportunities for nlp @xcite . for french-based creoles (fcs), a group of 10-20 closely-related languages foot_0 spoken by approximately 15 million people, nlp in general remains challenging. these languages share many linguistic traits with french and even more with each other, which greatly limits the effectiveness of existing lid solutions. at the same time, these languages exist in distinct sociolinguistic contexts. at one end of the spectrum lies haitian creole, which accounts for roughly 2/3 of fc speakers. it is the national language of haiti and spoken by sizable diaspora communities in countries like the united states. as such, it has a robust online presence, and is wellrepresented in lid benchmarks like @xcite . at the other end, the creoles of louisiana and trinidad are critically endangered and have limited online presence. in the middle, lesser antillean, 3 french guianese creoles, mauritian and réunionese creoles are partly institutionalized. although models like glotlid and afrolid @xcite , try to take more fcs into language text ## related work due to its important role in multilingual nlp, lid has a long history, which @xcite resume in depth. by the 90s, approaches based on n-gram frequencies, including @xcite had achieved 99% accuracy on monolingual documents of sufficient length in several common languages. however, many subproblems like closely-related varieties, short texts, and code-switching remained open @xcite . during the late 2000s and early-mid 2010s, alternative approaches were explored, culminating in the adoption of sentence-level linear classifiers as a de facto standard. in this section, we briefly review the usage of linear classifiers for language identification, and then explore how keyword methods provide a useful alternative. ## multilevel feature mining the phrase "needle in a haystack" is a longstanding metaphor for search scenarios where one or more target "needle(s)" is/are embedded in a very large number of distracting "hay" records @xcite . the heart of a needle(s)in-a-haystack problem is to eliminate hay quickly. for language identification, this means ignoring documents that lack (quasi)-unique features of our target language(s). to find the few documents that do contain clusters of such features, we create a kind of search engine that uses lexicons as queries and returns ranked document-and line-level corpora. to implement this data triage, we introduce an indexing-scoring system that operates in two phases: first at the document-level, and then at the sentence level. the main components of this method are shown in figure 1 ## experiments and results comparing our method with other systems is difficult for two reasons. firstly, building lid test sets for long-tail languages is complicated in general, since the data available are often overly clean, skewed to a few reliable sources, and/or themselves collected with lid. secondly, because our primary focus is document-level filtration, sentence-level datasets like flores-200 are likely to severely underestimate our performance. these problems are further aggravated when seeking to compare our method to some other list-based filtering approaches mentioned in sections 2 and 3, as scannell (2007) and @xcite further rely on the indexing of external search engines, while @xcite and @xcite use wordlists as secondary filtering to support multinomial classifiers, rather than first-pass models. with these concerns in mind, we adopt a fourpronged approach. first, we first benchmark the speed, recall, and false positive rate of feature mining and glotlid. to get an idea of how much content we should expect to find in a snapshot, we then apply our document-level filtration system to subcorpora from three recent, large-scale filtered corpora that feature our target languages, namely madlad-400, glotcc, and fineweb-2. for all three corpora, we report our recall on the 'clean' portions of these corpora, as well as our ability to find usable data in the discard portion of the largest, fineweb-2. after that, we test documentlevel filtering on a full common crawl snapshot with different wordlists at two thresholds. lastly, we briefly explore the potential of type-based filtering for the second pass, and qualitatively estimate corpora quality. for now, we focus on improving lid for lesser antillean creole(s) acf and gcf, 8 french guianese gcr, and mauritian mfe, seychellois crs, and réunionese rcf. we do not include endangered varieties because the reference corpora do not address them. as haitian creole is already covered by many systems, we include it mainly as a point of reference. ## conclusion we have introduced multilevel feature mining as an efficient approach to web corpora creation for less commonly written languages. taking advantage of the large class imbalance among varieties, we are able to eliminate the overwhelming majority of documents by counting how many distinctive types appear in the document. for french-based creoles, such types can be identified quickly using whitespace tokenization. rapid lid facilitates new kinds of exploration of crawled data, including but not limited to training language models.