|
--- |
|
license: mit |
|
language: |
|
- fr |
|
- zh |
|
- fa |
|
- ky |
|
- ru |
|
- lt |
|
- uz |
|
- en |
|
- pt |
|
- bg |
|
- th |
|
- pl |
|
- ur |
|
- sw |
|
- tr |
|
- es |
|
- ar |
|
- it |
|
- hi |
|
- de |
|
- el |
|
- nl |
|
- vi |
|
- ja |
|
pipeline_tag: text-classification |
|
tags: |
|
- pytorch |
|
- mt0 |
|
--- |
|
# language identification mt0 |
|
|
|
This model is a fine-tuned version of encoder from [bigscience/mt0-small](https://huggingface.co/bigscience/mt0-small) on the [Language Identification](https://huggingface.co/datasets/papluca/language-identification#additional-information) dataset as well as some private data. |
|
|
|
## Limitations |
|
|
|
Currently, it supports the following 20 languages: |
|
|
|
arabic (ar), bulgarian (bg), german (de), modern greek (el), english (en), spanish (es), french (fr), hindi (hi), italian (it), kyrgyz (ky), uzbek (uz), persian (fa), lithuanian (lt), japanese (ja), dutch (nl), polish (pl), portuguese (pt), russian (ru), swahili (sw), thai (th), turkish (tr), urdu (ur), vietnamese (vi), and chinese (zh) |
|
|
|
## Inference |
|
|
|
First you will need to have this library installed |
|
|
|
```python |
|
pip install bert-for-sequence classfication |
|
``` |
|
|
|
|
|
```python |
|
from bert_clf import EncoderCLF |
|
|
|
model = EncoderCLF("whitefoxredhell/language_identification") |
|
|
|
text = "London is the capital of Great Britain" |
|
|
|
model.predict(text) |
|
# 'en' |
|
|
|
model.predict_proba(text) |
|
# { |
|
# 'fr': 3.022890814463608e-05, |
|
# 'zh': 2.328997834410984e-05, |
|
# 'fa': 5.344639430404641e-05, |
|
# 'ky': 3.5296812711749226e-05, |
|
# 'ru': 2.3277720174519345e-05, |
|
# 'lt': 0.00021786204888485372, |
|
# 'uz': 3.461417873040773e-05, |
|
# 'en': 0.999232292175293, |
|
# 'pt': 1.2590448022820055e-05, |
|
# 'bg': 1.5775613064761274e-05, |
|
# 'th': 9.429674719285686e-06, |
|
# 'pl': 2.4624938305350952e-05, |
|
# 'ur': 3.982995986007154e-05, |
|
# 'sw': 4.8921840061666444e-05, |
|
# 'tr': 2.6844283638638444e-05, |
|
# 'es': 2.325668538105674e-05, |
|
# 'ar': 2.4103366740746424e-05, |
|
# 'it': 1.8611381165101193e-05, |
|
# 'hi': 1.4575023669749498e-05, |
|
# 'de': 2.210299498983659e-05, |
|
# 'el': 1.3880739061278291e-05, |
|
# 'nl': 2.767637124634348e-05, |
|
# 'vi': 1.3878144272894133e-05, |
|
# 'ja': 1.3629408385895658e-05 |
|
# } |
|
``` |
