File size: 2,072 Bytes
4471600 9e07931 4471600 1f8179c |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 |
---
license: mit
language:
- fr
- zh
- fa
- ky
- ru
- lt
- uz
- en
- pt
- bg
- th
- pl
- ur
- sw
- tr
- es
- ar
- it
- hi
- de
- el
- nl
- vi
- ja
pipeline_tag: text-classification
tags:
- pytorch
- mt0
---
# language identification mt0
This model is a fine-tuned version of encoder from [bigscience/mt0-small](https://huggingface.co/bigscience/mt0-small) on the [Language Identification](https://huggingface.co/datasets/papluca/language-identification#additional-information) dataset as well as some private data.
## Limitations
Currently, it supports the following 20 languages:
arabic (ar), bulgarian (bg), german (de), modern greek (el), english (en), spanish (es), french (fr), hindi (hi), italian (it), kyrgyz (ky), uzbek (uz), persian (fa), lithuanian (lt), japanese (ja), dutch (nl), polish (pl), portuguese (pt), russian (ru), swahili (sw), thai (th), turkish (tr), urdu (ur), vietnamese (vi), and chinese (zh)
## Inference
First you will need to have this library installed
```python
pip install bert-for-sequence classfication
```
```python
from bert_clf import EncoderCLF
model = EncoderCLF("whitefoxredhell/language_identification")
text = "London is the capital of Great Britain"
model.predict(text)
# 'en'
model.predict_proba(text)
# {
# 'fr': 3.022890814463608e-05,
# 'zh': 2.328997834410984e-05,
# 'fa': 5.344639430404641e-05,
# 'ky': 3.5296812711749226e-05,
# 'ru': 2.3277720174519345e-05,
# 'lt': 0.00021786204888485372,
# 'uz': 3.461417873040773e-05,
# 'en': 0.999232292175293,
# 'pt': 1.2590448022820055e-05,
# 'bg': 1.5775613064761274e-05,
# 'th': 9.429674719285686e-06,
# 'pl': 2.4624938305350952e-05,
# 'ur': 3.982995986007154e-05,
# 'sw': 4.8921840061666444e-05,
# 'tr': 2.6844283638638444e-05,
# 'es': 2.325668538105674e-05,
# 'ar': 2.4103366740746424e-05,
# 'it': 1.8611381165101193e-05,
# 'hi': 1.4575023669749498e-05,
# 'de': 2.210299498983659e-05,
# 'el': 1.3880739061278291e-05,
# 'nl': 2.767637124634348e-05,
# 'vi': 1.3878144272894133e-05,
# 'ja': 1.3629408385895658e-05
# }
``` |