license: gpl-3.0
library_name: fasttext
tags:
- text-classification
- language-identification
metrics:
- f1
- precision
- recall
datasets:
- laurievb/OpenLID-v2
OpenLID-v2
- Developed by: Laurie Burchell, Alexandra Birch, Nikolay Bogoychev, Kenneth Heafield
- Model type: Text classification (language identification)
- Language(s) (NLP): en
- License: gpl-3.0
- Resources for more information: OpenLID paper and OpenLID v2 blog
Model description
OpenLID-v2 is a high-coverage, high-performance language identification model. It is an improved version of OpenLID.
The original model and training data are described in Burchell et al. (2023). The changes made to produce OpenLID-v2 and the rationale behind them are described in this blog post.
How to use
Here is how to use this model to detect the language of a given text. For best results, text should be cleaned and normalised with openlid.clean_text() prior to classification (available in this repository):
>>> import fasttext
>>> from openlid import clean_text
>>> from huggingface_hub import hf_hub_download
>>> model_path = hf_hub_download(repo_id="laurievb/OpenLID-v2", filename="model.bin")
>>> model = fasttext.load_model(model_path)
>>> input_text = clean_text("Hello, world!")
>>> model.predict(input_text)
(('__label__eng_Latn',), array([0.81148803]))
>>> model.predict("Hello, world!", k=5)
(('__label__eng_Latn', '__label__vie_Latn', '__label__nld_Latn', '__label__pol_Latn', '__label__deu_Latn'),
array([0.61224753, 0.21323682, 0.09696738, 0.01359863, 0.01319415]))
Limitations and bias
The dataset and model only covers 189 languages. In addition, because the FLORES-200 test set consists of sentences from a single domain (wiki articles), performance on this test set may not reflect how well our classifier works in other domains.
Our work aims to broaden NLP coverage by allowing practitioners to identify relevant data in more languages. However, we note that LID is inherently a normative activity that risks excluding minority dialects, scripts, or entire microlanguages from a macrolanguage. Choosing which languages to cover may reinforce power imbalances, as only some groups gain access to NLP technologies. In addition, errors in LID can have a significant impact on downstream performance, particularly (as is often the case) when a system is used as a ‘black box’. The performance of our classifier is not equal across languages which could lead to worse downstream performance for particular groups. We mitigate this by providing metrics by class.
Training data
The model was trained on the OpenLID-v2 dataset. Classes were up/downsampled with temperature sampling prior to training; code to do this can be found in the scripts directory.
Training procedure
The model was trained using fastText with the following hyperparameters set. All other hyperparameters were set to their default values.
- loss: softmax
- epochs: 2
- learning rate: 0.8
- minimum number of word occurances: 1000
- embedding dimension: 256
- character n-grams: 2-5
- word n-grams: 1
- bucket size: 1,000,000
- threads: 68
Evaluation datasets
The model was evaluated using the FLORES-200 benchmark provided by Costa-jussà et al. (2022) using normalised language labels. Further information is available in the OpenLID paper and OpenLID v2 blog.
BibTeX entry and citation info
ACL citation (preferred)
@inproceedings{burchell-etal-2023-open,
title = "An Open Dataset and Model for Language Identification",
author = "Burchell, Laurie and
Birch, Alexandra and
Bogoychev, Nikolay and
Heafield, Kenneth",
editor = "Rogers, Anna and
Boyd-Graber, Jordan and
Okazaki, Naoaki",
booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)",
month = jul,
year = "2023",
address = "Toronto, Canada",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.acl-short.75",
doi = "10.18653/v1/2023.acl-short.75",
pages = "865--879",
abstract = "Language identification (LID) is a fundamental step in many natural language processing pipelines. However, current LID systems are far from perfect, particularly on lower-resource languages. We present a LID model which achieves a macro-average F1 score of 0.93 and a false positive rate of 0.033{\%} across 201 languages, outperforming previous work. We achieve this by training on a curated dataset of monolingual data, which we audit manually to ensure reliability. We make both the model and the dataset available to the research community. Finally, we carry out detailed analysis into our model{'}s performance, both in comparison to existing open models and by language class.",
}