An Open Dataset and Model for Language Identification (2023)

Abstract: Language identification (LID) is a fundamental step in many natural language processing pipelines. However, current LID systems are far from perfect, particularly on lower-resource languages. We present a LID model which achieves a macro-average F1 score of 0.93 and a false positive rate of 0.033% across 201 languages, outperforming previous work. We achieve this by training on a curated dataset of monolingual data, which we audit manually to ensure reliability. We make both the model and the dataset available to the research community. Finally, we carry out detailed analysis into our model’s performance, both in comparison to existing open models and by language class.

## introduction language identification (lid) is a foundational step in many natural language processing (nlp) pipelines. it is used not only to select data in the relevant language, but also to exclude 'noise'. for this reason, accurate and precise lid systems are key for building useful and representative nlp applications. despite their importance, recent work has found that existing lid algorithms perform poorly in practice compared to test performance @xcite . the problem is particularly acute for low-resource languages: @xcite found a positive spearman rank correlation between quality of data and size of language for all of the lid-filtered multilingual datasets they studied. in addition, for a significant fraction of the language corpora they studied, less than half of the sentences were in the correct language. they point out that such low-quality data not only leads to poor performance in downstream tasks, but that it also contributes to 'representation washing', where the community is given a false view of the actual progress of low-resource nlp. for applications such as corpus filtering, lid systems need to be fast, reliable, and cover as many languages as possible. there are several open lid models offering quick classification and high language coverage, such as cld3 or the work of @xcite . however, to the best of our knowledge, none of the commonly-used scalable lid systems make their training data public. we address this gap by releasing an open and curated dataset for lid, which we audit by hand to assure quality. we also train a high-performing lid model on this dataset to show its utility. 1 the contributions of this paper are as follows: ## background there is a long history of research into lid using a plethora of methods @xcite . for high-coverage lid, dunn (2020) presents a model which achieves an f1 score above 0.95 for 464 languages, with training data sourced from common crawl. the highest coverage lid system to our knowledge is that developed by @xcite which covers as many as 1366 language varieties. unlike our work, the training data in both cases has not been manually checked for quality. recent work by @xcite presents a lid system covering 517 african languages and varieties where the training data has been curated manually. unfortunately, this data is not easily available. @xcite released a substantial piece of research aiming to improve machine translation coverage for over 200 languages. as part of this, they provided several professionally-translated datasets for use as test and development sets for multilingual tasks. for this reason, we use their system as our benchmark. however, whilst they did release scripts to recreate their parallel data, 2 they did not provide-or even document-the monolingual data used to train their lid system, saying only that they use "publicly available datasets" supplemented with their own dataset nllb-seed. we present a curated, open monolingual dataset covering 201 languages to facilitate further research. ## model and hardware we use our open dataset to train a fasttext lid model using the command-line tool @xcite . it embeds character-level n-grams from the input text, and then uses these as input to a multiclass linear classifier. we follow @xcite and train with softmax loss over two epochs with a learning rate of 0.8 and embedding dimension of 256. we discard words with less than 1000 occurences after upsampling and generate ngrams between 2 and 5, setting word n-grams to one and bucket size to 1,000,000. we set number of threads to 68. we train our model on one ice lake node of the csd3 hpc service. each node has 76 cpus and 256gib of ram. our model takes c. 1hr 45mins to train and contains 60.5 million parameters. inference over the 206,448 lines of the test set takes 22.4 secs (9216.4 lines/sec). ## results and analysis our results are given in we achieve an f1 score of 0.927 and a fpr of 0.033 on flores-200 * . we also outperform both nllb and cld3 on the subsets of flores-200 * which are covered by all models. since nllb and our model share the same architecture and the same parameters, we attribute our success to our training data selection and manual audit process. notably, our f1 score jumps to 0.959 and fpr falls to 0.020 when we exclude the eight arabic dialects from the test set to compare with nllb. the 95 languages covered by cld3, nllb, and our model are mostly high resource, and so it is unsurprising that we achieve the highest f1 score (0.989) and lowest fpr (0.011) on this subset. we notice that the pearson correlation between lines of training data and f1 score for each language is only 0.0242. this is not unexpected: some of the least resourced languages achieve perfect scores on the test set due to high domain overlap, whereas the higher-resourced languages might get lower scores on the test set but have better robustness across domains. full results by language are available in appendix b. ## conclusion we present an open dataset covering 201 languages, which we curate and audit manually to ensure high confidence in its data and language labels. we demonstrate the quality of our dataset by using it to train a high-performing and scalable lid model. finally, we provide detailed analysis into its performance by class. we make both our model and our dataset available to the research community.