---
language: lt
datasets:
- common_voice
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
widget:
- example_title: Common Voice sample 11
  src: https://huggingface.co/m3hrdadfi/wav2vec2-large-xlsr-lithuanian/resolve/main/sample11.flac
- example_title: Common Voice sample 74
  src: https://huggingface.co/m3hrdadfi/wav2vec2-large-xlsr-lithuanian/resolve/main/sample74.flac
model-index:
- name: XLSR Wav2Vec2 Lithuanian by Mehrdad Farahani
  results:
  - task:
      name: Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Common Voice lt
      type: common_voice
      args: lt
    metrics:
       - name: Test WER
         type: wer
         value: 34.66

---

# Wav2Vec2-Large-XLSR-53-Lithuanian

Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) in Lithuanian using [Common Voice](https://huggingface.co/datasets/common_voice). When using this model, make sure that your speech input is sampled at 16kHz.

## Usage
The model can be used directly (without a language model) as follows:

**Requirements**
```bash
# requirement packages
!pip install git+https://github.com/huggingface/datasets.git
!pip install git+https://github.com/huggingface/transformers.git
!pip install torchaudio
!pip install librosa
!pip install jiwer
```

**Normalizer**
```bash
!wget -O normalizer.py https://huggingface.co/m3hrdadfi/wav2vec2-large-xlsr-lithuanian/raw/main/normalizer.py
```

**Prediction**
```python
import librosa
import torch
import torchaudio
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
from datasets import load_dataset

import numpy as np
import re
import string

import IPython.display as ipd

from normalizer import normalizer

def speech_file_to_array_fn(batch):
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    speech_array = speech_array.squeeze().numpy()
    speech_array = librosa.resample(np.asarray(speech_array), sampling_rate, 16_000)

    batch["speech"] = speech_array
    return batch

def predict(batch):
    features = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)

    input_values = features.input_values.to(device)
    attention_mask = features.attention_mask.to(device)

    with torch.no_grad():
        logits = model(input_values, attention_mask=attention_mask).logits

    pred_ids = torch.argmax(logits, dim=-1)

    batch["predicted"] = processor.batch_decode(pred_ids)[0]
    return batch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
processor = Wav2Vec2Processor.from_pretrained("m3hrdadfi/wav2vec2-large-xlsr-lithuanian")
model = Wav2Vec2ForCTC.from_pretrained("m3hrdadfi/wav2vec2-large-xlsr-lithuanian").to(device)

dataset = load_dataset("common_voice", "lt", split="test[:1%]")
dataset = dataset.map(
    normalizer,
    fn_kwargs={"remove_extra_space": True},
    remove_columns=list(set(dataset.column_names) - set(['sentence', 'path']))
)

dataset = dataset.map(speech_file_to_array_fn)
result = dataset.map(predict)

max_items = np.random.randint(0, len(result), 20).tolist()
for i in max_items:
    reference, predicted =  result["sentence"][i], result["predicted"][i]
    print("reference:", reference)
    print("predicted:", predicted)
    print('---')
```

**Output:**
```text
reference: jos tikslas buvo rasti kelią į ramųjį vandenyną šiaurės amerikoje
predicted: jos tikstas buvo rasikelia į ramų į vandenyna šiaurės amerikoje
---
reference: pietrytinėje dalyje likusių katalikų kapinių teritorija po antrojo pasaulinio karo dar padidėjo
predicted: pietrytinė daljelikusių gatalikų kapinių teritoriją pontro pasaulnio karo dar padidėjo
---
reference: koplyčioje pakabintas aušros vartų marijos paveikslas
predicted: koplyčioje pakagintas aušos fortų marijos paveikslas
---
reference: yra politinių debatų vedėjas
predicted: yra politinių debatų vedėjas
---
reference: žmogui taip pat gali būti mirtinai pavojingi
predicted: žmogui taip pat gali būti mirtinai pavojingi
---
reference: tuo pačiu metu kijeve nuverstas netekęs vokietijos paramos skoropadskis
predicted: tuo pačiu metu kiei venų verstas netekės vokietijos paramos kropadskis
---
reference: visos dvylika komandų tarpusavyje sužaidžia po dvi rungtynes
predicted: visos dvylika komandų tarpuso vysų žaidžia po dvi rungtynės
---
reference: kaukazo regioną sudaro kaukazo kalnai ir gretimos žemumos
predicted: kau kazo regioną sudaro kaukazo kalnai ir gretimos žemumus
---
reference: tarptautinių ir rusiškų šaškių kandidatas į sporto meistrus
predicted: tarptautinio ir rusiškos šaškių kandidatus į sporto meistrus
---
reference: prasideda putorano plynaukštės pietiniame pakraštyje
predicted: prasideda futorano prynaukštės pietiniame pakraštyje
---
reference: miestas skirstomas į senamiestį ir naujamiestį
predicted: miestas skirstomas į senamėsti ir naujamiestė
---
reference: tais pačiais metais pelnė bronzą pasaulio taurės kolumbijos etape komandinio sprinto rungtyje
predicted: tais pačiais metais pelnį mronsa pasaulio taurės kolumbijos etape komandinio sprento rungtyje
---
reference: prasideda putorano plynaukštės pietiniame pakraštyje
predicted: prasideda futorano prynaukštės pietiniame pakraštyje
---
reference: moterų tarptautinės meistrės vardas yra viena pakopa žemesnis už moterų tarptautinės korespondencinių šachmatų didmeistrės
predicted: moterų tarptautinės meistrės vardas yra gana pakopo žymesnis už moterų tarptautinės kūrespondencinių šachmatų didmesčias
---
reference: teritoriją dengia tropinės džiunglės
predicted: teritorija dengia tropinės žiunglės
---
reference: pastaroji dažnai pereina į nimcovičiaus gynybą arba bogoliubovo gynybą
predicted: pastaruoji dažnai pereina nimcovičiaus gynyba arba bogalių buvo gymyba
---
reference: už tai buvo suimtas ir tris mėnesius sėdėjo butyrkų kalėjime
predicted: užtai buvo sujumtas ir tris mėne susiedėjo butirkų kalėjime
---
reference: tai didžiausias pagal gyventojų skaičių regionas
predicted: tai didžiausias pagal gyventojų skaičių redionus
---
reference: vilkyškių miške taip pat auga raganų eglė
predicted: vilkiškimiškė taip pat auga ragano eglė
---
reference: kitas gavo skaraitiškės dvarą su palivarkais
predicted: kitas gavos karaitiškės dvarą spolivarkais
---
```

## Evaluation

The model can be evaluated as follows on the test data of Common Voice.

```python
import librosa
import torch
import torchaudio
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
from datasets import load_dataset, load_metric

import numpy as np
import re
import string

from normalizer import normalizer

def speech_file_to_array_fn(batch):
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    speech_array = speech_array.squeeze().numpy()
    speech_array = librosa.resample(np.asarray(speech_array), sampling_rate, 16_000)

    batch["speech"] = speech_array
    return batch

def predict(batch):
    features = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)

    input_values = features.input_values.to(device)
    attention_mask = features.attention_mask.to(device)

    with torch.no_grad():
        logits = model(input_values, attention_mask=attention_mask).logits

    pred_ids = torch.argmax(logits, dim=-1)

    batch["predicted"] = processor.batch_decode(pred_ids)[0]
    return batch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
processor = Wav2Vec2Processor.from_pretrained("m3hrdadfi/wav2vec2-large-xlsr-lithuanian")
model = Wav2Vec2ForCTC.from_pretrained("m3hrdadfi/wav2vec2-large-xlsr-lithuanian").to(device)

dataset = load_dataset("common_voice", "lt", split="test")
dataset = dataset.map(
    normalizer,
    fn_kwargs={"remove_extra_space": True},
    remove_columns=list(set(dataset.column_names) - set(['sentence', 'path']))
)

dataset = dataset.map(speech_file_to_array_fn)
result = dataset.map(predict)

wer = load_metric("wer")

print("WER: {:.2f}".format(100 * wer.compute(predictions=result["predicted"], references=result["sentence"])))
```
]

**Test Result**:
- WER: 34.66%

## Training & Report
The Common Voice `train`, `validation` datasets were used for training.

You can see the training states [here](https://wandb.ai/m3hrdadfi/wav2vec2_large_xlsr_lt/reports/Fine-Tuning-for-Wav2Vec2-Large-XLSR-53-Lithuanian--Vmlldzo1OTM1MTU?accessToken=kdkpara4hcmjvrlpbfsnu4s8cdk3a0xeyrb84ycpr4k701n13hzr9q7s60b00swx)

The script used for training can be found [here](https://colab.research.google.com/github/m3hrdadfi/notebooks/blob/main/Fine_Tune_XLSR_Wav2Vec2_on_Lithuanian_ASR_with_%F0%9F%A4%97_Transformers_ipynb.ipynb)

## Questions?
Post a Github issue on the [Wav2Vec](https://github.com/m3hrdadfi/wav2vec) repo.


---
language: fa
license: apache-2.0
---

# ALBERT Persian

A Lite BERT for Self-supervised Learning of Language Representations for the Persian Language

> میتونی بهش بگی برت_کوچولو

[ALBERT-Persian](https://github.com/m3hrdadfi/albert-persian) is the first attempt on ALBERT for the Persian Language. The model was trained based on Google's ALBERT BASE Version 2.0 over various writing styles from numerous subjects (e.g., scientific, novels, news) with more than 3.9M documents, 73M sentences, and 1.3B words, like the way we did for ParsBERT.

Please follow the [ALBERT-Persian](https://github.com/m3hrdadfi/albert-persian) repo for the latest information about previous and current models.

## Persian Text Classification [DigiMag, Persian News]

The task target is labeling texts in a supervised manner in both existing datasets `DigiMag` and `Persian News`.

### Persian News

A dataset of various news articles scraped from different online news agencies' websites. The total number of articles is 16,438, spread over eight different classes.

1. Economic
2. International
3. Political
4. Science Technology
5. Cultural Art
6. Sport
7. Medical

|        Label       |   #  |
|:------------------:|:----:|
|       Social       | 2170 |
|      Economic      | 1564 |
|    International   | 1975 |
|      Political     | 2269 |
| Science Technology | 2436 |
|    Cultural Art    | 2558 |
|        Sport       | 1381 |
|       Medical      | 2085 |

**Download**
You can download the dataset from [here](https://drive.google.com/uc?id=1B6xotfXCcW9xS1mYSBQos7OCg0ratzKC)

## Results

The following table summarizes the F1 score obtained as compared to other models and architectures.

|      Dataset      | ALBERT-fa-base-v2 | ParsBERT-v1 | mBERT |
|:-----------------:|:-----------------:|:-----------:|:-----:|
|    Persian News   |       97.01       |    97.19    | 95.79 |

### BibTeX entry and citation info

Please cite in publications as the following:

```bibtex
@misc{ALBERTPersian,
  author = {Mehrdad Farahani},
  title = {ALBERT-Persian: A Lite BERT for Self-supervised Learning of Language Representations for the Persian Language},
  year = {2020},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/m3hrdadfi/albert-persian}},
}

@article{ParsBERT,
    title={ParsBERT: Transformer-based Model for Persian Language Understanding},
    author={Mehrdad Farahani, Mohammad Gharachorloo, Marzieh Farahani, Mohammad Manthouri},
    journal={ArXiv},
    year={2020},
    volume={abs/2005.12515}
}
```

## Questions?
Post a Github issue on the [ALBERT-Persian](https://github.com/m3hrdadfi/albert-persian) repo.


---
language: fa
license: apache-2.0
---

# ParsBERT + Sentence Transformers

Please follow the [Sentence-Transformer](https://github.com/m3hrdadfi/sentence-transformers) repo for the latest information about previous and current models.

```bibtex
@misc{SentenceTransformerWiki,
  author = {Mehrdad Farahani},
  title = {Sentence Embeddings with ParsBERT},
  year = {2020},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {https://github.com/m3hrdadfi/sentence-transformers},
}
```


---
pipeline_tag: feature-extraction
tags:
- sentence-transformers
- feature-extraction
- sentence-similarity
- transformers
---

# Sentence Embeddings with `albert-zwnj-wnli-mean-tokens`

## Usage (Sentence-Transformers)
Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:

```
pip install -U sentence-transformers
pip install -U sentencepiece
```

Then you can use the model like this:

```python
from sentence_transformers import SentenceTransformer

sentences = [
    'اولین حکمران شهر بابل کی بود؟',
    'در فصل زمستان چه اتفاقی افتاد؟',
    'میراث کوروش'
]
model = SentenceTransformer('m3hrdadfi/albert-zwnj-wnli-mean-tokens')
embeddings = model.encode(sentences)
print(embeddings)
```

## Usage (HuggingFace Transformers)
Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.

```python
from transformers import AutoTokenizer, AutoModel
import torch

# Max Pooling - Take the max value over time for every dimension.
def max_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    token_embeddings[input_mask_expanded == 0] = -1e9  # Set padding tokens to large negative value
    return torch.mean(token_embeddings, 1)[0]

# Sentences we want sentence embeddings for
sentences = [
    'اولین حکمران شهر بابل کی بود؟',
    'در فصل زمستان چه اتفاقی افتاد؟',
    'میراث کوروش'
]

# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('m3hrdadfi/albert-zwnj-wnli-mean-tokens')
model = AutoModel.from_pretrained('m3hrdadfi/albert-zwnj-wnli-mean-tokens')

# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)
# Perform pooling. In this case, max pooling.
sentence_embeddings = max_pooling(model_output, encoded_input['attention_mask'])

print("Sentence embeddings:")
print(sentence_embeddings)
```

## Questions?
Post a Github issue from [HERE](https://github.com/m3hrdadfi/sentence-transformers).


---
language: fa
datasets:
- common_voice
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
widget:
- example_title: Common Voice sample 687
  src: https://huggingface.co/m3hrdadfi/wav2vec2-large-xlsr-persian/resolve/main/sample687.flac
- example_title: Common Voice sample 1671
  src: https://huggingface.co/m3hrdadfi/wav2vec2-large-xlsr-persian/resolve/main/sample1671.flac
model-index:
- name: XLSR Wav2Vec2 Persian (Farsi) by Mehrdad Farahani
  results:
  - task:
      name: Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Common Voice fa
      type: common_voice
      args: fa
    metrics:
       - name: Test WER
         type: wer
         value: 32.20

---

# Wav2Vec2-Large-XLSR-53-Persian

Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) in Persian (Farsi) using [Common Voice](https://huggingface.co/datasets/common_voice). When using this model, make sure that your speech input is sampled at 16kHz.

## Usage
The model can be used directly (without a language model) as follows:

**Requirements**
```bash
# requirement packages
!pip install git+https://github.com/huggingface/datasets.git
!pip install git+https://github.com/huggingface/transformers.git
!pip install torchaudio
!pip install librosa
!pip install jiwer
!pip install hazm
```

**Prediction**
```python
import librosa
import torch
import torchaudio
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
from datasets import load_dataset

import numpy as np
import hazm
import re
import string

import IPython.display as ipd

_normalizer = hazm.Normalizer()
chars_to_ignore = [
    ",", "?", ".", "!", "-", ";", ":", '""', "%", "'", '"', "�",
    "#", "!", "؟", "?", "«", "»", "ء", "،", "(", ")", "؛", "'ٔ", "٬",'ٔ', ",", "?",
    ".", "!", "-", ";", ":",'"',"“", "%", "‘", "”", "�", "–", "…", "_", "”", '“', '„'
]

# In case of farsi
chars_to_ignore = chars_to_ignore + list(string.ascii_lowercase + string.digits)

chars_to_mapping = {
    'ك': 'ک', 'دِ': 'د', 'بِ': 'ب', 'زِ': 'ز', 'ذِ': 'ذ', 'شِ': 'ش', 'سِ': 'س', 'ى': 'ی',
    'ي': 'ی', 'أ': 'ا', 'ؤ': 'و', "ے": "ی", "ۀ": "ه", "ﭘ": "پ", "ﮐ": "ک", "ﯽ": "ی",
    "ﺎ": "ا", "ﺑ": "ب", "ﺘ": "ت", "ﺧ": "خ", "ﺩ": "د", "ﺱ": "س", "ﻀ": "ض", "ﻌ": "ع",
    "ﻟ": "ل", "ﻡ": "م", "ﻢ": "م", "ﻪ": "ه", "ﻮ": "و", "ئ": "ی", 'ﺍ': "ا", 'ة': "ه",
    'ﯾ': "ی", 'ﯿ': "ی", 'ﺒ': "ب", 'ﺖ': "ت", 'ﺪ': "د", 'ﺮ': "ر", 'ﺴ': "س", 'ﺷ': "ش",
    'ﺸ': "ش", 'ﻋ': "ع", 'ﻤ': "م", 'ﻥ': "ن", 'ﻧ': "ن", 'ﻭ': "و", 'ﺭ': "ر", "ﮔ": "گ",
    "\u200c": " ", "\u200d": " ", "\u200e": " ", "\u200f": " ", "\ufeff": " ",
}

def multiple_replace(text, chars_to_mapping):
    pattern = "|".join(map(re.escape, chars_to_mapping.keys()))
    return re.sub(pattern, lambda m: chars_to_mapping[m.group()], str(text))

def remove_special_characters(text, chars_to_ignore_regex):
    text = re.sub(chars_to_ignore_regex, '', text).lower() + " "
    return text

def normalizer(batch, chars_to_ignore, chars_to_mapping):
    chars_to_ignore_regex = f"""[{"".join(chars_to_ignore)}]"""
    text = batch["sentence"].lower().strip()

    text = _normalizer.normalize(text)
    text = multiple_replace(text, chars_to_mapping)
    text = remove_special_characters(text, chars_to_ignore_regex)

    batch["sentence"] = text
    return batch

def speech_file_to_array_fn(batch):
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    speech_array = speech_array.squeeze().numpy()
    speech_array = librosa.resample(np.asarray(speech_array), sampling_rate, 16_000)

    batch["speech"] = speech_array
    return batch

def predict(batch):
    features = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)

    input_values = features.input_values.to(device)
    attention_mask = features.attention_mask.to(device)

    with torch.no_grad():
        logits = model(input_values, attention_mask=attention_mask).logits

    pred_ids = torch.argmax(logits, dim=-1)

    batch["predicted"] = processor.batch_decode(pred_ids)[0]
    return batch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
processor = Wav2Vec2Processor.from_pretrained("m3hrdadfi/wav2vec2-large-xlsr-persian")
model = Wav2Vec2ForCTC.from_pretrained("m3hrdadfi/wav2vec2-large-xlsr-persian").to(device)

dataset = load_dataset("common_voice", "fa", split="test[:1%]")
dataset = dataset.map(
    normalizer,
    fn_kwargs={"chars_to_ignore": chars_to_ignore, "chars_to_mapping": chars_to_mapping},
    remove_columns=list(set(dataset.column_names) - set(['sentence', 'path']))
)

dataset = dataset.map(speech_file_to_array_fn)
result = dataset.map(predict)

max_items = np.random.randint(0, len(result), 20).tolist()
for i in max_items:
    reference, predicted =  result["sentence"][i], result["predicted"][i]
    print("reference:", reference)
    print("predicted:", predicted)
    print('---')
```

**Output:**
```text
reference: اطلاعات مسری است
predicted: اطلاعات مسری است
---
reference: نه منظورم اینه که وقتی که ساکته چه کاریه خودمونه بندازیم زحمت
predicted: نه منظورم اینه که وقتی که ساکت چی کاریه خودمونو بندازیم زحمت
---
reference: من آب پرتقال می خورم لطفا
predicted: من آپ ارتغال می خورم لطفا
---
reference: وقت آن رسیده آنها را که قدم پیش میگذارند بزرگ بداریم
predicted: وقت آ رسیده آنها را که قدم پیش میگذارند بزرگ بداریم
---
reference: سیم باتری دارید
predicted: سیم باتری دارید
---
reference: این بهتره تا اینکه به بهونه درس و مشق هر روز بره خونه شون
predicted: این بهتره تا اینکه به بهمونه درسومش خرروز بره خونه اشون
---
reference: ژاکت تنگ است
predicted: ژاکت تنگ است
---
reference: آت و اشغال های خیابان
predicted: آت و اشغال های خیابان
---
reference: من به این روند اعتراض دارم
predicted: من به این لوند تراج دارم
---
reference: کرایه این مکان چند است
predicted: کرایه این مکان چند است
---
reference: ولی این فرصت این سهم جوانی اعطا نشده است
predicted: ولی این فرصت این سحم جوانی اتان نشده است
---
reference: متوجه فاجعهای محیطی میشوم
predicted: متوجه فاجایهای محیطی میشوم
---
reference: ترافیک شدیدیم بود و دیدن نور ماشینا و چراغا و لامپهای مراکز تجاری حس خوبی بهم میدادن
predicted: ترافیک شدید ی هم بودا دیدن نور ماشینا و چراغ لامپهای مراکز تجاری حس خولی بهم میدادن
---
reference: این مورد عمل ها مربوط به تخصص شما می شود
predicted: این مورد عملها مربوط به تخصص شما میشود
---
reference: انرژی خیلی کمی دارم
predicted: انرژی خیلی کمی دارم
---
reference: زیادی خوبی کردنم تهش داستانه
predicted: زیادی خوبی کردنم ترش داستانه
---
reference: بردهای که پادشاه شود
predicted: برده ای که پاده شاه شود
---
reference: یونسکو
predicted: یونسکو
---
reference: شما اخراج هستید
predicted: شما اخراج هستید
---
reference: من سفر کردن را دوست دارم
predicted: من سفر کردم را دوست دارم
```

## Evaluation

The model can be evaluated as follows on the Persian (Farsi) test data of Common Voice.

```python
import librosa
import torch
import torchaudio
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
from datasets import load_dataset, load_metric

import numpy as np
import hazm
import re
import string

_normalizer = hazm.Normalizer()
chars_to_ignore = [
    ",", "?", ".", "!", "-", ";", ":", '""', "%", "'", '"', "�",
    "#", "!", "؟", "?", "«", "»", "ء", "،", "(", ")", "؛", "'ٔ", "٬",'ٔ', ",", "?",
    ".", "!", "-", ";", ":",'"',"“", "%", "‘", "”", "�", "–", "…", "_", "”", '“', '„'
]

# In case of farsi
chars_to_ignore = chars_to_ignore + list(string.ascii_lowercase + string.digits)

chars_to_mapping = {
    'ك': 'ک', 'دِ': 'د', 'بِ': 'ب', 'زِ': 'ز', 'ذِ': 'ذ', 'شِ': 'ش', 'سِ': 'س', 'ى': 'ی',
    'ي': 'ی', 'أ': 'ا', 'ؤ': 'و', "ے": "ی", "ۀ": "ه", "ﭘ": "پ", "ﮐ": "ک", "ﯽ": "ی",
    "ﺎ": "ا", "ﺑ": "ب", "ﺘ": "ت", "ﺧ": "خ", "ﺩ": "د", "ﺱ": "س", "ﻀ": "ض", "ﻌ": "ع",
    "ﻟ": "ل", "ﻡ": "م", "ﻢ": "م", "ﻪ": "ه", "ﻮ": "و", "ئ": "ی", 'ﺍ': "ا", 'ة': "ه",
    'ﯾ': "ی", 'ﯿ': "ی", 'ﺒ': "ب", 'ﺖ': "ت", 'ﺪ': "د", 'ﺮ': "ر", 'ﺴ': "س", 'ﺷ': "ش",
    'ﺸ': "ش", 'ﻋ': "ع", 'ﻤ': "م", 'ﻥ': "ن", 'ﻧ': "ن", 'ﻭ': "و", 'ﺭ': "ر", "ﮔ": "گ",
    "\\u200c": " ", "\\u200d": " ", "\\u200e": " ", "\\u200f": " ", "\\ufeff": " ",
}

def multiple_replace(text, chars_to_mapping):
    pattern = "|".join(map(re.escape, chars_to_mapping.keys()))
    return re.sub(pattern, lambda m: chars_to_mapping[m.group()], str(text))

def remove_special_characters(text, chars_to_ignore_regex):
    text = re.sub(chars_to_ignore_regex, '', text).lower() + " "
    return text

def normalizer(batch, chars_to_ignore, chars_to_mapping):
    chars_to_ignore_regex = f"""[{"".join(chars_to_ignore)}]"""
    text = batch["sentence"].lower().strip()

    text = _normalizer.normalize(text)
    text = multiple_replace(text, chars_to_mapping)
    text = remove_special_characters(text, chars_to_ignore_regex)

    batch["sentence"] = text
    return batch

def speech_file_to_array_fn(batch):
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    speech_array = speech_array.squeeze().numpy()
    speech_array = librosa.resample(np.asarray(speech_array), sampling_rate, 16_000)

    batch["speech"] = speech_array
    return batch

def predict(batch):
    features = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)

    input_values = features.input_values.to(device)
    attention_mask = features.attention_mask.to(device)

    with torch.no_grad():
        logits = model(input_values, attention_mask=attention_mask).logits

    pred_ids = torch.argmax(logits, dim=-1)

    batch["predicted"] = processor.batch_decode(pred_ids)[0]
    return batch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
processor = Wav2Vec2Processor.from_pretrained("m3hrdadfi/wav2vec2-large-xlsr-persian")
model = Wav2Vec2ForCTC.from_pretrained("m3hrdadfi/wav2vec2-large-xlsr-persian").to(device)

dataset = load_dataset("common_voice", "fa", split="test")
dataset = dataset.map(
    normalizer,
    fn_kwargs={"chars_to_ignore": chars_to_ignore, "chars_to_mapping": chars_to_mapping},
    remove_columns=list(set(dataset.column_names) - set(['sentence', 'path']))
)
dataset = dataset.map(speech_file_to_array_fn)
result = dataset.map(predict)

wer = load_metric("wer")
print("WER: {:.2f}".format(100 * wer.compute(predictions=result["predicted"], references=result["sentence"])))
```

**Test Result:**
- WER: 32.20%

## Training
The Common Voice `train`, `validation` datasets were used for training.
The script used for training can be found [here](https://colab.research.google.com/github/m3hrdadfi/notebooks/blob/main/Fine_Tune_XLSR_Wav2Vec2_on_Persian_ASR_with_%F0%9F%A4%97_Transformers_ipynb.ipynb)


---
language: multilingual
license: apache-2.0
datasets:
- wili_2018
---

# Zabanshenas - Language Detector

Zabanshenas is a Transformer-based solution for identifying the most likely language of a written document/text. Zabanshenas is a Persian word that has two meanings:

- A person who studies linguistics.
- A way to identify the type of written language.

## How to use

Follow [Zabanshenas repo](https://github.com/m3hrdadfi/zabanshenas) for more information!

## Evaluation
The following tables summarize the scores obtained by model overall and per each class.

### By Paragraph

|                language                | precision |  recall  | f1-score |
|:--------------------------------------:|:---------:|:--------:|:--------:|
|             Achinese (ace)             |  1.000000 | 0.982143 | 0.990991 |
|             Afrikaans (afr)            |  1.000000 | 1.000000 | 1.000000 |
|         Alemannic German (als)         |  1.000000 | 0.946429 | 0.972477 |
|              Amharic (amh)             |  1.000000 | 0.982143 | 0.990991 |
|            Old English (ang)           |  0.981818 | 0.964286 | 0.972973 |
|              Arabic (ara)              |  0.846154 | 0.982143 | 0.909091 |
|             Aragonese (arg)            |  1.000000 | 1.000000 | 1.000000 |
|          Egyptian Arabic (arz)         |  0.979592 | 0.857143 | 0.914286 |
|             Assamese (asm)             |  0.981818 | 0.964286 | 0.972973 |
|             Asturian (ast)             |  0.964912 | 0.982143 | 0.973451 |
|               Avar (ava)               |  0.941176 | 0.905660 | 0.923077 |
|              Aymara (aym)              |  0.964912 | 0.982143 | 0.973451 |
|         South Azerbaijani (azb)        |  0.965517 | 1.000000 | 0.982456 |
|            Azerbaijani (aze)           |  1.000000 | 1.000000 | 1.000000 |
|              Bashkir (bak)             |  1.000000 | 0.978261 | 0.989011 |
|             Bavarian (bar)             |  0.843750 | 0.964286 | 0.900000 |
|           Central Bikol (bcl)          |  1.000000 | 0.982143 | 0.990991 |
| Belarusian (Taraschkewiza) (be-tarask) |  1.000000 | 0.875000 | 0.933333 |
|            Belarusian (bel)            |  0.870968 | 0.964286 | 0.915254 |
|              Bengali (ben)             |  0.982143 | 0.982143 | 0.982143 |
|             Bhojpuri (bho)             |  1.000000 | 0.928571 | 0.962963 |
|              Banjar (bjn)              |  0.981132 | 0.945455 | 0.962963 |
|              Tibetan (bod)             |  1.000000 | 0.982143 | 0.990991 |
|              Bosnian (bos)             |  0.552632 | 0.375000 | 0.446809 |
|            Bishnupriya (bpy)           |  1.000000 | 0.982143 | 0.990991 |
|              Breton (bre)              |  1.000000 | 0.964286 | 0.981818 |
|             Bulgarian (bul)            |  1.000000 | 0.964286 | 0.981818 |
|              Buryat (bxr)              |  0.946429 | 0.946429 | 0.946429 |
|              Catalan (cat)             |  0.982143 | 0.982143 | 0.982143 |
|             Chavacano (cbk)            |  0.914894 | 0.767857 | 0.834951 |
|             Min Dong (cdo)             |  1.000000 | 0.982143 | 0.990991 |
|              Cebuano (ceb)             |  1.000000 | 1.000000 | 1.000000 |
|               Czech (ces)              |  1.000000 | 1.000000 | 1.000000 |
|              Chechen (che)             |  1.000000 | 1.000000 | 1.000000 |
|             Cherokee (chr)             |  1.000000 | 0.963636 | 0.981481 |
|              Chuvash (chv)             |  0.938776 | 0.958333 | 0.948454 |
|          Central Kurdish (ckb)         |  1.000000 | 1.000000 | 1.000000 |
|              Cornish (cor)             |  1.000000 | 1.000000 | 1.000000 |
|             Corsican (cos)             |  1.000000 | 0.982143 | 0.990991 |
|           Crimean Tatar (crh)          |  1.000000 | 0.946429 | 0.972477 |
|             Kashubian (csb)            |  1.000000 | 0.963636 | 0.981481 |
|               Welsh (cym)              |  1.000000 | 1.000000 | 1.000000 |
|              Danish (dan)              |  1.000000 | 1.000000 | 1.000000 |
|              German (deu)              |  0.828125 | 0.946429 | 0.883333 |
|               Dimli (diq)              |  0.964912 | 0.982143 | 0.973451 |
|              Dhivehi (div)             |  1.000000 | 1.000000 | 1.000000 |
|           Lower Sorbian (dsb)          |  1.000000 | 0.982143 | 0.990991 |
|              Doteli (dty)              |  0.940000 | 0.854545 | 0.895238 |
|              Emilian (egl)             |  1.000000 | 0.928571 | 0.962963 |
|           Modern Greek (ell)           |  1.000000 | 1.000000 | 1.000000 |
|              English (eng)             |  0.588889 | 0.946429 | 0.726027 |
|             Esperanto (epo)            |  1.000000 | 0.982143 | 0.990991 |
|             Estonian (est)             |  0.963636 | 0.946429 | 0.954955 |
|              Basque (eus)              |  1.000000 | 0.982143 | 0.990991 |
|           Extremaduran (ext)           |  0.982143 | 0.982143 | 0.982143 |
|              Faroese (fao)             |  1.000000 | 1.000000 | 1.000000 |
|              Persian (fas)             |  0.948276 | 0.982143 | 0.964912 |
|              Finnish (fin)             |  1.000000 | 1.000000 | 1.000000 |
|              French (fra)              |  0.710145 | 0.875000 | 0.784000 |
|              Arpitan (frp)             |  1.000000 | 0.946429 | 0.972477 |
|          Western Frisian (fry)         |  0.982143 | 0.982143 | 0.982143 |
|             Friulian (fur)             |  1.000000 | 0.982143 | 0.990991 |
|              Gagauz (gag)              |  0.981132 | 0.945455 | 0.962963 |
|          Scottish Gaelic (gla)         |  0.982143 | 0.982143 | 0.982143 |
|               Irish (gle)              |  0.949153 | 1.000000 | 0.973913 |
|             Galician (glg)             |  1.000000 | 1.000000 | 1.000000 |
|              Gilaki (glk)              |  0.981132 | 0.945455 | 0.962963 |
|               Manx (glv)               |  1.000000 | 1.000000 | 1.000000 |
|              Guarani (grn)             |  1.000000 | 0.964286 | 0.981818 |
|             Gujarati (guj)             |  1.000000 | 0.982143 | 0.990991 |
|           Hakka Chinese (hak)          |  0.981818 | 0.964286 | 0.972973 |
|          Haitian Creole (hat)          |  1.000000 | 1.000000 | 1.000000 |
|               Hausa (hau)              |  1.000000 | 0.945455 | 0.971963 |
|          Serbo-Croatian (hbs)          |  0.448276 | 0.464286 | 0.456140 |
|              Hebrew (heb)              |  1.000000 | 0.982143 | 0.990991 |
|            Fiji Hindi (hif)            |  0.890909 | 0.890909 | 0.890909 |
|               Hindi (hin)              |  0.981481 | 0.946429 | 0.963636 |
|             Croatian (hrv)             |  0.500000 | 0.636364 | 0.560000 |
|           Upper Sorbian (hsb)          |  0.955556 | 1.000000 | 0.977273 |
|             Hungarian (hun)            |  1.000000 | 1.000000 | 1.000000 |
|             Armenian (hye)             |  1.000000 | 0.981818 | 0.990826 |
|               Igbo (ibo)               |  0.918033 | 1.000000 | 0.957265 |
|                Ido (ido)               |  1.000000 | 1.000000 | 1.000000 |
|            Interlingue (ile)           |  1.000000 | 0.962264 | 0.980769 |
|               Iloko (ilo)              |  0.947368 | 0.964286 | 0.955752 |
|            Interlingua (ina)           |  1.000000 | 1.000000 | 1.000000 |
|            Indonesian (ind)            |  0.761905 | 0.872727 | 0.813559 |
|             Icelandic (isl)            |  1.000000 | 1.000000 | 1.000000 |
|              Italian (ita)             |  0.861538 | 1.000000 | 0.925620 |
|          Jamaican Patois (jam)         |  1.000000 | 0.946429 | 0.972477 |
|             Javanese (jav)             |  0.964912 | 0.982143 | 0.973451 |
|              Lojban (jbo)              |  1.000000 | 1.000000 | 1.000000 |
|             Japanese (jpn)             |  1.000000 | 1.000000 | 1.000000 |
|            Karakalpak (kaa)            |  0.965517 | 1.000000 | 0.982456 |
|              Kabyle (kab)              |  1.000000 | 0.964286 | 0.981818 |
|              Kannada (kan)             |  0.982143 | 0.982143 | 0.982143 |
|             Georgian (kat)             |  1.000000 | 0.964286 | 0.981818 |
|              Kazakh (kaz)              |  0.980769 | 0.980769 | 0.980769 |
|             Kabardian (kbd)            |  1.000000 | 0.982143 | 0.990991 |
|           Central Khmer (khm)          |  0.960784 | 0.875000 | 0.915888 |
|            Kinyarwanda (kin)           |  0.981132 | 0.928571 | 0.954128 |
|              Kirghiz (kir)             |  1.000000 | 1.000000 | 1.000000 |
|           Komi-Permyak (koi)           |  0.962264 | 0.910714 | 0.935780 |
|              Konkani (kok)             |  0.964286 | 0.981818 | 0.972973 |
|               Komi (kom)               |  1.000000 | 0.962264 | 0.980769 |
|              Korean (kor)              |  1.000000 | 1.000000 | 1.000000 |
|          Karachay-Balkar (krc)         |  1.000000 | 0.982143 | 0.990991 |
|            Ripuarisch (ksh)            |  1.000000 | 0.964286 | 0.981818 |
|              Kurdish (kur)             |  1.000000 | 0.964286 | 0.981818 |
|              Ladino (lad)              |  1.000000 | 1.000000 | 1.000000 |
|                Lao (lao)               |  0.961538 | 0.909091 | 0.934579 |
|               Latin (lat)              |  0.877193 | 0.943396 | 0.909091 |
|              Latvian (lav)             |  0.963636 | 0.946429 | 0.954955 |
|             Lezghian (lez)             |  1.000000 | 0.964286 | 0.981818 |
|             Ligurian (lij)             |  1.000000 | 0.964286 | 0.981818 |
|             Limburgan (lim)            |  0.938776 | 1.000000 | 0.968421 |
|              Lingala (lin)             |  0.980769 | 0.927273 | 0.953271 |
|            Lithuanian (lit)            |  0.982456 | 1.000000 | 0.991150 |
|              Lombard (lmo)             |  1.000000 | 1.000000 | 1.000000 |
|           Northern Luri (lrc)          |  1.000000 | 0.928571 | 0.962963 |
|             Latgalian (ltg)            |  1.000000 | 0.982143 | 0.990991 |
|           Luxembourgish (ltz)          |  0.949153 | 1.000000 | 0.973913 |
|              Luganda (lug)             |  1.000000 | 1.000000 | 1.000000 |
|         Literary Chinese (lzh)         |  1.000000 | 1.000000 | 1.000000 |
|             Maithili (mai)             |  0.931034 | 0.964286 | 0.947368 |
|             Malayalam (mal)            |  1.000000 | 0.982143 | 0.990991 |
|          Banyumasan (map-bms)          |  0.977778 | 0.785714 | 0.871287 |
|              Marathi (mar)             |  0.949153 | 1.000000 | 0.973913 |
|              Moksha (mdf)              |  0.980000 | 0.890909 | 0.933333 |
|           Eastern Mari (mhr)           |  0.981818 | 0.964286 | 0.972973 |
|            Minangkabau (min)           |  1.000000 | 1.000000 | 1.000000 |
|            Macedonian (mkd)            |  1.000000 | 0.981818 | 0.990826 |
|             Malagasy (mlg)             |  0.981132 | 1.000000 | 0.990476 |
|              Maltese (mlt)             |  0.982456 | 1.000000 | 0.991150 |
|          Min Nan Chinese (nan)         |  1.000000 | 1.000000 | 1.000000 |
|             Mongolian (mon)            |  1.000000 | 0.981818 | 0.990826 |
|               Maori (mri)              |  1.000000 | 1.000000 | 1.000000 |
|           Western Mari (mrj)           |  0.982456 | 1.000000 | 0.991150 |
|               Malay (msa)              |  0.862069 | 0.892857 | 0.877193 |
|             Mirandese (mwl)            |  1.000000 | 0.982143 | 0.990991 |
|              Burmese (mya)             |  1.000000 | 1.000000 | 1.000000 |
|               Erzya (myv)              |  0.818182 | 0.964286 | 0.885246 |
|            Mazanderani (mzn)           |  0.981481 | 1.000000 | 0.990654 |
|            Neapolitan (nap)            |  1.000000 | 0.981818 | 0.990826 |
|              Navajo (nav)              |  1.000000 | 1.000000 | 1.000000 |
|         Classical Nahuatl (nci)        |  0.981481 | 0.946429 | 0.963636 |
|            Low German (nds)            |  0.982143 | 0.982143 | 0.982143 |
|        West Low German (nds-nl)        |  1.000000 | 1.000000 | 1.000000 |
|      Nepali (macrolanguage) (nep)      |  0.881356 | 0.928571 | 0.904348 |
|              Newari (new)              |  1.000000 | 0.909091 | 0.952381 |
|               Dutch (nld)              |  0.982143 | 0.982143 | 0.982143 |
|         Norwegian Nynorsk (nno)        |  1.000000 | 1.000000 | 1.000000 |
|              Bokmål (nob)              |  1.000000 | 1.000000 | 1.000000 |
|               Narom (nrm)              |  0.981818 | 0.964286 | 0.972973 |
|          Northern Sotho (nso)          |  1.000000 | 1.000000 | 1.000000 |
|              Occitan (oci)             |  0.903846 | 0.839286 | 0.870370 |
|          Livvi-Karelian (olo)          |  0.982456 | 1.000000 | 0.991150 |
|               Oriya (ori)              |  0.964912 | 0.982143 | 0.973451 |
|               Oromo (orm)              |  0.982143 | 0.982143 | 0.982143 |
|             Ossetian (oss)             |  0.982143 | 1.000000 | 0.990991 |
|            Pangasinan (pag)            |  0.980000 | 0.875000 | 0.924528 |
|             Pampanga (pam)             |  0.928571 | 0.896552 | 0.912281 |
|              Panjabi (pan)             |  1.000000 | 1.000000 | 1.000000 |
|            Papiamento (pap)            |  1.000000 | 0.964286 | 0.981818 |
|              Picard (pcd)              |  0.849057 | 0.849057 | 0.849057 |
|        Pennsylvania German (pdc)       |  0.854839 | 0.946429 | 0.898305 |
|          Palatine German (pfl)         |  0.946429 | 0.946429 | 0.946429 |
|          Western Panjabi (pnb)         |  0.981132 | 0.962963 | 0.971963 |
|              Polish (pol)              |  0.933333 | 1.000000 | 0.965517 |
|            Portuguese (por)            |  0.774648 | 0.982143 | 0.866142 |
|              Pushto (pus)              |  1.000000 | 0.910714 | 0.953271 |
|              Quechua (que)             |  0.962963 | 0.928571 | 0.945455 |
|      Tarantino dialect (roa-tara)      |  1.000000 | 0.964286 | 0.981818 |
|              Romansh (roh)             |  1.000000 | 0.928571 | 0.962963 |
|             Romanian (ron)             |  0.965517 | 1.000000 | 0.982456 |
|               Rusyn (rue)              |  0.946429 | 0.946429 | 0.946429 |
|             Aromanian (rup)            |  0.962963 | 0.928571 | 0.945455 |
|              Russian (rus)             |  0.859375 | 0.982143 | 0.916667 |
|               Yakut (sah)              |  1.000000 | 0.982143 | 0.990991 |
|             Sanskrit (san)             |  0.982143 | 0.982143 | 0.982143 |
|             Sicilian (scn)             |  1.000000 | 1.000000 | 1.000000 |
|               Scots (sco)              |  0.982143 | 0.982143 | 0.982143 |
|            Samogitian (sgs)            |  1.000000 | 0.982143 | 0.990991 |
|              Sinhala (sin)             |  0.964912 | 0.982143 | 0.973451 |
|              Slovak (slk)              |  1.000000 | 0.982143 | 0.990991 |
|              Slovene (slv)             |  1.000000 | 0.981818 | 0.990826 |
|           Northern Sami (sme)          |  0.962264 | 0.962264 | 0.962264 |
|               Shona (sna)              |  0.933333 | 1.000000 | 0.965517 |
|              Sindhi (snd)              |  1.000000 | 1.000000 | 1.000000 |
|              Somali (som)              |  0.948276 | 1.000000 | 0.973451 |
|              Spanish (spa)             |  0.739130 | 0.910714 | 0.816000 |
|             Albanian (sqi)             |  0.982143 | 0.982143 | 0.982143 |
|             Sardinian (srd)            |  1.000000 | 0.982143 | 0.990991 |
|              Sranan (srn)              |  1.000000 | 1.000000 | 1.000000 |
|              Serbian (srp)             |  1.000000 | 0.946429 | 0.972477 |
|          Saterfriesisch (stq)          |  1.000000 | 0.964286 | 0.981818 |
|             Sundanese (sun)            |  1.000000 | 0.977273 | 0.988506 |
|      Swahili (macrolanguage) (swa)     |  1.000000 | 1.000000 | 1.000000 |
|              Swedish (swe)             |  1.000000 | 1.000000 | 1.000000 |
|             Silesian (szl)             |  1.000000 | 0.981481 | 0.990654 |
|               Tamil (tam)              |  0.982143 | 1.000000 | 0.990991 |
|               Tatar (tat)              |  1.000000 | 1.000000 | 1.000000 |
|               Tulu (tcy)               |  0.982456 | 1.000000 | 0.991150 |
|              Telugu (tel)              |  1.000000 | 0.920000 | 0.958333 |
|               Tetum (tet)              |  1.000000 | 0.964286 | 0.981818 |
|               Tajik (tgk)              |  1.000000 | 1.000000 | 1.000000 |
|              Tagalog (tgl)             |  1.000000 | 1.000000 | 1.000000 |
|               Thai (tha)               |  0.932203 | 0.982143 | 0.956522 |
|              Tongan (ton)              |  1.000000 | 0.964286 | 0.981818 |
|              Tswana (tsn)              |  1.000000 | 1.000000 | 1.000000 |
|              Turkmen (tuk)             |  1.000000 | 0.982143 | 0.990991 |
|              Turkish (tur)             |  0.901639 | 0.982143 | 0.940171 |
|               Tuvan (tyv)              |  1.000000 | 0.964286 | 0.981818 |
|              Udmurt (udm)              |  1.000000 | 0.982143 | 0.990991 |
|              Uighur (uig)              |  1.000000 | 0.982143 | 0.990991 |
|             Ukrainian (ukr)            |  0.963636 | 0.946429 | 0.954955 |
|               Urdu (urd)               |  1.000000 | 0.982143 | 0.990991 |
|               Uzbek (uzb)              |  1.000000 | 1.000000 | 1.000000 |
|             Venetian (vec)             |  1.000000 | 0.982143 | 0.990991 |
|               Veps (vep)               |  0.982456 | 1.000000 | 0.991150 |
|            Vietnamese (vie)            |  0.964912 | 0.982143 | 0.973451 |
|              Vlaams (vls)              |  1.000000 | 0.982143 | 0.990991 |
|              Volapük (vol)             |  1.000000 | 1.000000 | 1.000000 |
|               Võro (vro)               |  0.964286 | 0.964286 | 0.964286 |
|               Waray (war)              |  1.000000 | 0.982143 | 0.990991 |
|              Walloon (wln)             |  1.000000 | 1.000000 | 1.000000 |
|               Wolof (wol)              |  0.981481 | 0.963636 | 0.972477 |
|            Wu Chinese (wuu)            |  0.981481 | 0.946429 | 0.963636 |
|               Xhosa (xho)              |  1.000000 | 0.964286 | 0.981818 |
|            Mingrelian (xmf)            |  1.000000 | 0.964286 | 0.981818 |
|              Yiddish (yid)             |  1.000000 | 1.000000 | 1.000000 |
|              Yoruba (yor)              |  0.964912 | 0.982143 | 0.973451 |
|              Zeeuws (zea)              |  1.000000 | 0.982143 | 0.990991 |
|           Cantonese (zh-yue)           |  0.981481 | 0.946429 | 0.963636 |
|         Standard Chinese (zho)         |  0.932203 | 0.982143 | 0.956522 |
|                accuracy                |  0.963055 | 0.963055 | 0.963055 |
|                macro avg               |  0.966424 | 0.963216 | 0.963891 |
|              weighted avg              |  0.966040 | 0.963055 | 0.963606 |

### By Sentence

|                language                | precision |  recall  | f1-score |
|:--------------------------------------:|:---------:|:--------:|:--------:|
|             Achinese (ace)             |  0.754545 | 0.873684 | 0.809756 |
|             Afrikaans (afr)            |  0.708955 | 0.940594 | 0.808511 |
|         Alemannic German (als)         |  0.870130 | 0.752809 | 0.807229 |
|              Amharic (amh)             |  1.000000 | 0.820000 | 0.901099 |
|            Old English (ang)           |  0.966667 | 0.906250 | 0.935484 |
|              Arabic (ara)              |  0.907692 | 0.967213 | 0.936508 |
|             Aragonese (arg)            |  0.921569 | 0.959184 | 0.940000 |
|          Egyptian Arabic (arz)         |  0.964286 | 0.843750 | 0.900000 |
|             Assamese (asm)             |  0.964286 | 0.870968 | 0.915254 |
|             Asturian (ast)             |  0.880000 | 0.795181 | 0.835443 |
|               Avar (ava)               |  0.864198 | 0.843373 | 0.853659 |
|              Aymara (aym)              |  1.000000 | 0.901961 | 0.948454 |
|         South Azerbaijani (azb)        |  0.979381 | 0.989583 | 0.984456 |
|            Azerbaijani (aze)           |  0.989899 | 0.960784 | 0.975124 |
|              Bashkir (bak)             |  0.837209 | 0.857143 | 0.847059 |
|             Bavarian (bar)             |  0.741935 | 0.766667 | 0.754098 |
|           Central Bikol (bcl)          |  0.962963 | 0.928571 | 0.945455 |
| Belarusian (Taraschkewiza) (be-tarask) |  0.857143 | 0.733333 | 0.790419 |
|            Belarusian (bel)            |  0.775510 | 0.752475 | 0.763819 |
|              Bengali (ben)             |  0.861111 | 0.911765 | 0.885714 |
|             Bhojpuri (bho)             |  0.965517 | 0.933333 | 0.949153 |
|              Banjar (bjn)              |  0.891566 | 0.880952 | 0.886228 |
|              Tibetan (bod)             |  1.000000 | 1.000000 | 1.000000 |
|              Bosnian (bos)             |  0.375000 | 0.323077 | 0.347107 |
|            Bishnupriya (bpy)           |  0.986301 | 1.000000 | 0.993103 |
|              Breton (bre)              |  0.951613 | 0.893939 | 0.921875 |
|             Bulgarian (bul)            |  0.945055 | 0.877551 | 0.910053 |
|              Buryat (bxr)              |  0.955556 | 0.843137 | 0.895833 |
|              Catalan (cat)             |  0.692308 | 0.750000 | 0.720000 |
|             Chavacano (cbk)            |  0.842857 | 0.641304 | 0.728395 |
|             Min Dong (cdo)             |  0.972973 | 1.000000 | 0.986301 |
|              Cebuano (ceb)             |  0.981308 | 0.954545 | 0.967742 |
|               Czech (ces)              |  0.944444 | 0.915385 | 0.929687 |
|              Chechen (che)             |  0.875000 | 0.700000 | 0.777778 |
|             Cherokee (chr)             |  1.000000 | 0.970588 | 0.985075 |
|              Chuvash (chv)             |  0.875000 | 0.836957 | 0.855556 |
|          Central Kurdish (ckb)         |  1.000000 | 0.983051 | 0.991453 |
|              Cornish (cor)             |  0.979592 | 0.969697 | 0.974619 |
|             Corsican (cos)             |  0.986842 | 0.925926 | 0.955414 |
|           Crimean Tatar (crh)          |  0.958333 | 0.907895 | 0.932432 |
|             Kashubian (csb)            |  0.920354 | 0.904348 | 0.912281 |
|               Welsh (cym)              |  0.971014 | 0.943662 | 0.957143 |
|              Danish (dan)              |  0.865169 | 0.777778 | 0.819149 |
|              German (deu)              |  0.721311 | 0.822430 | 0.768559 |
|               Dimli (diq)              |  0.915966 | 0.923729 | 0.919831 |
|              Dhivehi (div)             |  1.000000 | 0.991228 | 0.995595 |
|           Lower Sorbian (dsb)          |  0.898876 | 0.879121 | 0.888889 |
|              Doteli (dty)              |  0.821429 | 0.638889 | 0.718750 |
|              Emilian (egl)             |  0.988095 | 0.922222 | 0.954023 |
|           Modern Greek (ell)           |  0.988636 | 0.966667 | 0.977528 |
|              English (eng)             |  0.522727 | 0.784091 | 0.627273 |
|             Esperanto (epo)            |  0.963855 | 0.930233 | 0.946746 |
|             Estonian (est)             |  0.922222 | 0.873684 | 0.897297 |
|              Basque (eus)              |  1.000000 | 0.941176 | 0.969697 |
|           Extremaduran (ext)           |  0.925373 | 0.885714 | 0.905109 |
|              Faroese (fao)             |  0.855072 | 0.887218 | 0.870849 |
|              Persian (fas)             |  0.879630 | 0.979381 | 0.926829 |
|              Finnish (fin)             |  0.952830 | 0.943925 | 0.948357 |
|              French (fra)              |  0.676768 | 0.943662 | 0.788235 |
|              Arpitan (frp)             |  0.867925 | 0.807018 | 0.836364 |
|          Western Frisian (fry)         |  0.956989 | 0.890000 | 0.922280 |
|             Friulian (fur)             |  1.000000 | 0.857143 | 0.923077 |
|              Gagauz (gag)              |  0.939024 | 0.802083 | 0.865169 |
|          Scottish Gaelic (gla)         |  1.000000 | 0.879121 | 0.935673 |
|               Irish (gle)              |  0.989247 | 0.958333 | 0.973545 |
|             Galician (glg)             |  0.910256 | 0.922078 | 0.916129 |
|              Gilaki (glk)              |  0.964706 | 0.872340 | 0.916201 |
|               Manx (glv)               |  1.000000 | 0.965517 | 0.982456 |
|              Guarani (grn)             |  0.983333 | 1.000000 | 0.991597 |
|             Gujarati (guj)             |  1.000000 | 0.991525 | 0.995745 |
|           Hakka Chinese (hak)          |  0.955224 | 0.955224 | 0.955224 |
|          Haitian Creole (hat)          |  0.833333 | 0.666667 | 0.740741 |
|               Hausa (hau)              |  0.936709 | 0.913580 | 0.925000 |
|          Serbo-Croatian (hbs)          |  0.452830 | 0.410256 | 0.430493 |
|              Hebrew (heb)              |  0.988235 | 0.976744 | 0.982456 |
|            Fiji Hindi (hif)            |  0.936709 | 0.840909 | 0.886228 |
|               Hindi (hin)              |  0.965517 | 0.756757 | 0.848485 |
|             Croatian (hrv)             |  0.443820 | 0.537415 | 0.486154 |
|           Upper Sorbian (hsb)          |  0.951613 | 0.830986 | 0.887218 |
|             Hungarian (hun)            |  0.854701 | 0.909091 | 0.881057 |
|             Armenian (hye)             |  1.000000 | 0.816327 | 0.898876 |
|               Igbo (ibo)               |  0.974359 | 0.926829 | 0.950000 |
|                Ido (ido)               |  0.975000 | 0.987342 | 0.981132 |
|            Interlingue (ile)           |  0.880597 | 0.921875 | 0.900763 |
|               Iloko (ilo)              |  0.882353 | 0.821918 | 0.851064 |
|            Interlingua (ina)           |  0.952381 | 0.895522 | 0.923077 |
|            Indonesian (ind)            |  0.606383 | 0.695122 | 0.647727 |
|             Icelandic (isl)            |  0.978261 | 0.882353 | 0.927835 |
|              Italian (ita)             |  0.910448 | 0.910448 | 0.910448 |
|          Jamaican Patois (jam)         |  0.988764 | 0.967033 | 0.977778 |
|             Javanese (jav)             |  0.903614 | 0.862069 | 0.882353 |
|              Lojban (jbo)              |  0.943878 | 0.929648 | 0.936709 |
|             Japanese (jpn)             |  1.000000 | 0.764706 | 0.866667 |
|            Karakalpak (kaa)            |  0.940171 | 0.901639 | 0.920502 |
|              Kabyle (kab)              |  0.985294 | 0.837500 | 0.905405 |
|              Kannada (kan)             |  0.975806 | 0.975806 | 0.975806 |
|             Georgian (kat)             |  0.953704 | 0.903509 | 0.927928 |
|              Kazakh (kaz)              |  0.934579 | 0.877193 | 0.904977 |
|             Kabardian (kbd)            |  0.987952 | 0.953488 | 0.970414 |
|           Central Khmer (khm)          |  0.928571 | 0.829787 | 0.876404 |
|            Kinyarwanda (kin)           |  0.953125 | 0.938462 | 0.945736 |
|              Kirghiz (kir)             |  0.927632 | 0.881250 | 0.903846 |
|           Komi-Permyak (koi)           |  0.750000 | 0.776786 | 0.763158 |
|              Konkani (kok)             |  0.893491 | 0.872832 | 0.883041 |
|               Komi (kom)               |  0.734177 | 0.690476 | 0.711656 |
|              Korean (kor)              |  0.989899 | 0.989899 | 0.989899 |
|          Karachay-Balkar (krc)         |  0.928571 | 0.917647 | 0.923077 |
|            Ripuarisch (ksh)            |  0.915789 | 0.896907 | 0.906250 |
|              Kurdish (kur)             |  0.977528 | 0.935484 | 0.956044 |
|              Ladino (lad)              |  0.985075 | 0.904110 | 0.942857 |
|                Lao (lao)               |  0.896552 | 0.812500 | 0.852459 |
|               Latin (lat)              |  0.741935 | 0.831325 | 0.784091 |
|              Latvian (lav)             |  0.710526 | 0.878049 | 0.785455 |
|             Lezghian (lez)             |  0.975309 | 0.877778 | 0.923977 |
|             Ligurian (lij)             |  0.951807 | 0.897727 | 0.923977 |
|             Limburgan (lim)            |  0.909091 | 0.921053 | 0.915033 |
|              Lingala (lin)             |  0.942857 | 0.814815 | 0.874172 |
|            Lithuanian (lit)            |  0.892857 | 0.925926 | 0.909091 |
|              Lombard (lmo)             |  0.766234 | 0.951613 | 0.848921 |
|           Northern Luri (lrc)          |  0.972222 | 0.875000 | 0.921053 |
|             Latgalian (ltg)            |  0.895349 | 0.865169 | 0.880000 |
|           Luxembourgish (ltz)          |  0.882353 | 0.750000 | 0.810811 |
|              Luganda (lug)             |  0.946429 | 0.883333 | 0.913793 |
|         Literary Chinese (lzh)         |  1.000000 | 1.000000 | 1.000000 |
|             Maithili (mai)             |  0.893617 | 0.823529 | 0.857143 |
|             Malayalam (mal)            |  1.000000 | 0.975000 | 0.987342 |
|          Banyumasan (map-bms)          |  0.924242 | 0.772152 | 0.841379 |
|              Marathi (mar)             |  0.874126 | 0.919118 | 0.896057 |
|              Moksha (mdf)              |  0.771242 | 0.830986 | 0.800000 |
|           Eastern Mari (mhr)           |  0.820000 | 0.860140 | 0.839590 |
|            Minangkabau (min)           |  0.973684 | 0.973684 | 0.973684 |
|            Macedonian (mkd)            |  0.895652 | 0.953704 | 0.923767 |
|             Malagasy (mlg)             |  1.000000 | 0.966102 | 0.982759 |
|              Maltese (mlt)             |  0.987952 | 0.964706 | 0.976190 |
|          Min Nan Chinese (nan)         |  0.975000 | 1.000000 | 0.987342 |
|             Mongolian (mon)            |  0.954545 | 0.933333 | 0.943820 |
|               Maori (mri)              |  0.985294 | 1.000000 | 0.992593 |
|           Western Mari (mrj)           |  0.966292 | 0.914894 | 0.939891 |
|               Malay (msa)              |  0.770270 | 0.695122 | 0.730769 |
|             Mirandese (mwl)            |  0.970588 | 0.891892 | 0.929577 |
|              Burmese (mya)             |  1.000000 | 0.964286 | 0.981818 |
|               Erzya (myv)              |  0.535714 | 0.681818 | 0.600000 |
|            Mazanderani (mzn)           |  0.968750 | 0.898551 | 0.932331 |
|            Neapolitan (nap)            |  0.892308 | 0.865672 | 0.878788 |
|              Navajo (nav)              |  0.984375 | 0.984375 | 0.984375 |
|         Classical Nahuatl (nci)        |  0.901408 | 0.761905 | 0.825806 |
|            Low German (nds)            |  0.896226 | 0.913462 | 0.904762 |
|        West Low German (nds-nl)        |  0.873563 | 0.835165 | 0.853933 |
|      Nepali (macrolanguage) (nep)      |  0.704545 | 0.861111 | 0.775000 |
|              Newari (new)              |  0.920000 | 0.741935 | 0.821429 |
|               Dutch (nld)              |  0.925926 | 0.872093 | 0.898204 |
|         Norwegian Nynorsk (nno)        |  0.847059 | 0.808989 | 0.827586 |
|              Bokmål (nob)              |  0.861386 | 0.852941 | 0.857143 |
|               Narom (nrm)              |  0.966667 | 0.983051 | 0.974790 |
|          Northern Sotho (nso)          |  0.897436 | 0.921053 | 0.909091 |
|              Occitan (oci)             |  0.958333 | 0.696970 | 0.807018 |
|          Livvi-Karelian (olo)          |  0.967742 | 0.937500 | 0.952381 |
|               Oriya (ori)              |  0.933333 | 1.000000 | 0.965517 |
|               Oromo (orm)              |  0.977528 | 0.915789 | 0.945652 |
|             Ossetian (oss)             |  0.958333 | 0.841463 | 0.896104 |
|            Pangasinan (pag)            |  0.847328 | 0.909836 | 0.877470 |
|             Pampanga (pam)             |  0.969697 | 0.780488 | 0.864865 |
|              Panjabi (pan)             |  1.000000 | 1.000000 | 1.000000 |
|            Papiamento (pap)            |  0.876190 | 0.920000 | 0.897561 |
|              Picard (pcd)              |  0.707317 | 0.568627 | 0.630435 |
|        Pennsylvania German (pdc)       |  0.827273 | 0.827273 | 0.827273 |
|          Palatine German (pfl)         |  0.882353 | 0.914634 | 0.898204 |
|          Western Panjabi (pnb)         |  0.964286 | 0.931034 | 0.947368 |
|              Polish (pol)              |  0.859813 | 0.910891 | 0.884615 |
|            Portuguese (por)            |  0.535714 | 0.833333 | 0.652174 |
|              Pushto (pus)              |  0.989362 | 0.902913 | 0.944162 |
|              Quechua (que)             |  0.979167 | 0.903846 | 0.940000 |
|      Tarantino dialect (roa-tara)      |  0.964912 | 0.901639 | 0.932203 |
|              Romansh (roh)             |  0.914894 | 0.895833 | 0.905263 |
|             Romanian (ron)             |  0.880597 | 0.880597 | 0.880597 |
|               Rusyn (rue)              |  0.932584 | 0.805825 | 0.864583 |
|             Aromanian (rup)            |  0.783333 | 0.758065 | 0.770492 |
|              Russian (rus)             |  0.517986 | 0.765957 | 0.618026 |
|               Yakut (sah)              |  0.954023 | 0.922222 | 0.937853 |
|             Sanskrit (san)             |  0.866667 | 0.951220 | 0.906977 |
|             Sicilian (scn)             |  0.984375 | 0.940299 | 0.961832 |
|               Scots (sco)              |  0.851351 | 0.900000 | 0.875000 |
|            Samogitian (sgs)            |  0.977011 | 0.876289 | 0.923913 |
|              Sinhala (sin)             |  0.406154 | 0.985075 | 0.575163 |
|              Slovak (slk)              |  0.956989 | 0.872549 | 0.912821 |
|              Slovene (slv)             |  0.907216 | 0.854369 | 0.880000 |
|           Northern Sami (sme)          |  0.949367 | 0.892857 | 0.920245 |
|               Shona (sna)              |  0.936508 | 0.855072 | 0.893939 |
|              Sindhi (snd)              |  0.984962 | 0.992424 | 0.988679 |
|              Somali (som)              |  0.949153 | 0.848485 | 0.896000 |
|              Spanish (spa)             |  0.584158 | 0.746835 | 0.655556 |
|             Albanian (sqi)             |  0.988095 | 0.912088 | 0.948571 |
|             Sardinian (srd)            |  0.957746 | 0.931507 | 0.944444 |
|              Sranan (srn)              |  0.985714 | 0.945205 | 0.965035 |
|              Serbian (srp)             |  0.950980 | 0.889908 | 0.919431 |
|          Saterfriesisch (stq)          |  0.962500 | 0.875000 | 0.916667 |
|             Sundanese (sun)            |  0.778846 | 0.910112 | 0.839378 |
|      Swahili (macrolanguage) (swa)     |  0.915493 | 0.878378 | 0.896552 |
|              Swedish (swe)             |  0.989247 | 0.958333 | 0.973545 |
|             Silesian (szl)             |  0.944444 | 0.904255 | 0.923913 |
|               Tamil (tam)              |  0.990000 | 0.970588 | 0.980198 |
|               Tatar (tat)              |  0.942029 | 0.902778 | 0.921986 |
|               Tulu (tcy)               |  0.980519 | 0.967949 | 0.974194 |
|              Telugu (tel)              |  0.965986 | 0.965986 | 0.965986 |
|               Tetum (tet)              |  0.898734 | 0.855422 | 0.876543 |
|               Tajik (tgk)              |  0.974684 | 0.939024 | 0.956522 |
|              Tagalog (tgl)             |  0.965909 | 0.934066 | 0.949721 |
|               Thai (tha)               |  0.923077 | 0.882353 | 0.902256 |
|              Tongan (ton)              |  0.970149 | 0.890411 | 0.928571 |
|              Tswana (tsn)              |  0.888889 | 0.926316 | 0.907216 |
|              Turkmen (tuk)             |  0.968000 | 0.889706 | 0.927203 |
|              Turkish (tur)             |  0.871287 | 0.926316 | 0.897959 |
|               Tuvan (tyv)              |  0.948454 | 0.859813 | 0.901961 |
|              Udmurt (udm)              |  0.989362 | 0.894231 | 0.939394 |
|              Uighur (uig)              |  1.000000 | 0.953333 | 0.976109 |
|             Ukrainian (ukr)            |  0.893617 | 0.875000 | 0.884211 |
|               Urdu (urd)               |  1.000000 | 1.000000 | 1.000000 |
|               Uzbek (uzb)              |  0.636042 | 0.886700 | 0.740741 |
|             Venetian (vec)             |  1.000000 | 0.941176 | 0.969697 |
|               Veps (vep)               |  0.858586 | 0.965909 | 0.909091 |
|            Vietnamese (vie)            |  1.000000 | 0.940476 | 0.969325 |
|              Vlaams (vls)              |  0.885714 | 0.898551 | 0.892086 |
|              Volapük (vol)             |  0.975309 | 0.975309 | 0.975309 |
|               Võro (vro)               |  0.855670 | 0.864583 | 0.860104 |
|               Waray (war)              |  0.972222 | 0.909091 | 0.939597 |
|              Walloon (wln)             |  0.742138 | 0.893939 | 0.810997 |
|               Wolof (wol)              |  0.882979 | 0.954023 | 0.917127 |
|            Wu Chinese (wuu)            |  0.961538 | 0.833333 | 0.892857 |
|               Xhosa (xho)              |  0.934066 | 0.867347 | 0.899471 |
|            Mingrelian (xmf)            |  0.958333 | 0.929293 | 0.943590 |
|              Yiddish (yid)             |  0.984375 | 0.875000 | 0.926471 |
|              Yoruba (yor)              |  0.868421 | 0.857143 | 0.862745 |
|              Zeeuws (zea)              |  0.879518 | 0.793478 | 0.834286 |
|           Cantonese (zh-yue)           |  0.896552 | 0.812500 | 0.852459 |
|         Standard Chinese (zho)         |  0.906250 | 0.935484 | 0.920635 |
|                accuracy                |  0.881051 | 0.881051 | 0.881051 |
|                macro avg               |  0.903245 | 0.880618 | 0.888996 |
|              weighted avg              |  0.894174 | 0.881051 | 0.884520 |

### By Token (3 to 5)

|                language                | precision |  recall  | f1-score |
|:--------------------------------------:|:---------:|:--------:|:--------:|
|             Achinese (ace)             |  0.873846 | 0.827988 | 0.850299 |
|             Afrikaans (afr)            |  0.638060 | 0.732334 | 0.681954 |
|         Alemannic German (als)         |  0.673780 | 0.547030 | 0.603825 |
|              Amharic (amh)             |  0.997743 | 0.954644 | 0.975717 |
|            Old English (ang)           |  0.840816 | 0.693603 | 0.760148 |
|              Arabic (ara)              |  0.768737 | 0.840749 | 0.803132 |
|             Aragonese (arg)            |  0.493671 | 0.505181 | 0.499360 |
|          Egyptian Arabic (arz)         |  0.823529 | 0.741935 | 0.780606 |
|             Assamese (asm)             |  0.948454 | 0.893204 | 0.920000 |
|             Asturian (ast)             |  0.490000 | 0.508299 | 0.498982 |
|               Avar (ava)               |  0.813636 | 0.655678 | 0.726166 |
|              Aymara (aym)              |  0.795833 | 0.779592 | 0.787629 |
|         South Azerbaijani (azb)        |  0.832836 | 0.863777 | 0.848024 |
|            Azerbaijani (aze)           |  0.867470 | 0.800000 | 0.832370 |
|              Bashkir (bak)             |  0.851852 | 0.750000 | 0.797688 |
|             Bavarian (bar)             |  0.560897 | 0.522388 | 0.540958 |
|           Central Bikol (bcl)          |  0.708229 | 0.668235 | 0.687651 |
| Belarusian (Taraschkewiza) (be-tarask) |  0.615635 | 0.526462 | 0.567568 |
|            Belarusian (bel)            |  0.539952 | 0.597855 | 0.567430 |
|              Bengali (ben)             |  0.830275 | 0.885086 | 0.856805 |
|             Bhojpuri (bho)             |  0.723118 | 0.691517 | 0.706965 |
|              Banjar (bjn)              |  0.619586 | 0.726269 | 0.668699 |
|              Tibetan (bod)             |  0.999537 | 0.991728 | 0.995617 |
|              Bosnian (bos)             |  0.330849 | 0.403636 | 0.363636 |
|            Bishnupriya (bpy)           |  0.941634 | 0.949020 | 0.945312 |
|              Breton (bre)              |  0.772222 | 0.745308 | 0.758527 |
|             Bulgarian (bul)            |  0.771505 | 0.706897 | 0.737789 |
|              Buryat (bxr)              |  0.741935 | 0.753149 | 0.747500 |
|              Catalan (cat)             |  0.528716 | 0.610136 | 0.566516 |
|             Chavacano (cbk)            |  0.409449 | 0.312625 | 0.354545 |
|             Min Dong (cdo)             |  0.951264 | 0.936057 | 0.943599 |
|              Cebuano (ceb)             |  0.888298 | 0.876640 | 0.882431 |
|               Czech (ces)              |  0.806045 | 0.758294 | 0.781441 |
|              Chechen (che)             |  0.857143 | 0.600000 | 0.705882 |
|             Cherokee (chr)             |  0.997840 | 0.952577 | 0.974684 |
|              Chuvash (chv)             |  0.874346 | 0.776744 | 0.822660 |
|          Central Kurdish (ckb)         |  0.984848 | 0.953545 | 0.968944 |
|              Cornish (cor)             |  0.747596 | 0.807792 | 0.776529 |
|             Corsican (cos)             |  0.673913 | 0.708571 | 0.690808 |
|           Crimean Tatar (crh)          |  0.498801 | 0.700337 | 0.582633 |
|             Kashubian (csb)            |  0.797059 | 0.794721 | 0.795888 |
|               Welsh (cym)              |  0.829609 | 0.841360 | 0.835443 |
|              Danish (dan)              |  0.649789 | 0.622222 | 0.635707 |
|              German (deu)              |  0.559406 | 0.763514 | 0.645714 |
|               Dimli (diq)              |  0.835580 | 0.763547 | 0.797941 |
|              Dhivehi (div)             |  1.000000 | 0.980645 | 0.990228 |
|           Lower Sorbian (dsb)          |  0.740484 | 0.694805 | 0.716918 |
|              Doteli (dty)              |  0.616314 | 0.527132 | 0.568245 |
|              Emilian (egl)             |  0.822993 | 0.769625 | 0.795414 |
|           Modern Greek (ell)           |  0.972043 | 0.963753 | 0.967880 |
|              English (eng)             |  0.260492 | 0.724346 | 0.383183 |
|             Esperanto (epo)            |  0.766764 | 0.716621 | 0.740845 |
|             Estonian (est)             |  0.698885 | 0.673835 | 0.686131 |
|              Basque (eus)              |  0.882716 | 0.841176 | 0.861446 |
|           Extremaduran (ext)           |  0.570605 | 0.511628 | 0.539510 |
|              Faroese (fao)             |  0.773987 | 0.784017 | 0.778970 |
|              Persian (fas)             |  0.709836 | 0.809346 | 0.756332 |
|              Finnish (fin)             |  0.866261 | 0.796089 | 0.829694 |
|              French (fra)              |  0.496263 | 0.700422 | 0.580927 |
|              Arpitan (frp)             |  0.663366 | 0.584302 | 0.621329 |
|          Western Frisian (fry)         |  0.750000 | 0.756148 | 0.753061 |
|             Friulian (fur)             |  0.713555 | 0.675545 | 0.694030 |
|              Gagauz (gag)              |  0.728125 | 0.677326 | 0.701807 |
|          Scottish Gaelic (gla)         |  0.831601 | 0.817996 | 0.824742 |
|               Irish (gle)              |  0.868852 | 0.801296 | 0.833708 |
|             Galician (glg)             |  0.469816 | 0.454315 | 0.461935 |
|              Gilaki (glk)              |  0.703883 | 0.687204 | 0.695444 |
|               Manx (glv)               |  0.873047 | 0.886905 | 0.879921 |
|              Guarani (grn)             |  0.848580 | 0.793510 | 0.820122 |
|             Gujarati (guj)             |  0.995643 | 0.926978 | 0.960084 |
|           Hakka Chinese (hak)          |  0.898403 | 0.904971 | 0.901675 |
|          Haitian Creole (hat)          |  0.719298 | 0.518987 | 0.602941 |
|               Hausa (hau)              |  0.815353 | 0.829114 | 0.822176 |
|          Serbo-Croatian (hbs)          |  0.343465 | 0.244589 | 0.285714 |
|              Hebrew (heb)              |  0.891304 | 0.933941 | 0.912125 |
|            Fiji Hindi (hif)            |  0.662577 | 0.664615 | 0.663594 |
|               Hindi (hin)              |  0.782301 | 0.778169 | 0.780229 |
|             Croatian (hrv)             |  0.360308 | 0.374000 | 0.367026 |
|           Upper Sorbian (hsb)          |  0.745763 | 0.611111 | 0.671756 |
|             Hungarian (hun)            |  0.876812 | 0.846154 | 0.861210 |
|             Armenian (hye)             |  0.988201 | 0.917808 | 0.951705 |
|               Igbo (ibo)               |  0.825397 | 0.696429 | 0.755448 |
|                Ido (ido)               |  0.760479 | 0.814103 | 0.786378 |
|            Interlingue (ile)           |  0.701299 | 0.580645 | 0.635294 |
|               Iloko (ilo)              |  0.688356 | 0.844538 | 0.758491 |
|            Interlingua (ina)           |  0.577889 | 0.588235 | 0.583016 |
|            Indonesian (ind)            |  0.415879 | 0.514019 | 0.459770 |
|             Icelandic (isl)            |  0.855263 | 0.790754 | 0.821745 |
|              Italian (ita)             |  0.474576 | 0.561247 | 0.514286 |
|          Jamaican Patois (jam)         |  0.826087 | 0.791667 | 0.808511 |
|             Javanese (jav)             |  0.670130 | 0.658163 | 0.664093 |
|              Lojban (jbo)              |  0.896861 | 0.917431 | 0.907029 |
|             Japanese (jpn)             |  0.931373 | 0.848214 | 0.887850 |
|            Karakalpak (kaa)            |  0.790393 | 0.827744 | 0.808637 |
|              Kabyle (kab)              |  0.828571 | 0.759162 | 0.792350 |
|              Kannada (kan)             |  0.879357 | 0.847545 | 0.863158 |
|             Georgian (kat)             |  0.916399 | 0.907643 | 0.912000 |
|              Kazakh (kaz)              |  0.900901 | 0.819672 | 0.858369 |
|             Kabardian (kbd)            |  0.923345 | 0.892256 | 0.907534 |
|           Central Khmer (khm)          |  0.976667 | 0.816156 | 0.889226 |
|            Kinyarwanda (kin)           |  0.824324 | 0.726190 | 0.772152 |
|              Kirghiz (kir)             |  0.674766 | 0.779698 | 0.723447 |
|           Komi-Permyak (koi)           |  0.652830 | 0.633700 | 0.643123 |
|              Konkani (kok)             |  0.778865 | 0.728938 | 0.753075 |
|               Komi (kom)               |  0.737374 | 0.572549 | 0.644592 |
|              Korean (kor)              |  0.984615 | 0.967603 | 0.976035 |
|          Karachay-Balkar (krc)         |  0.869416 | 0.857627 | 0.863481 |
|            Ripuarisch (ksh)            |  0.709859 | 0.649485 | 0.678331 |
|              Kurdish (kur)             |  0.883777 | 0.862884 | 0.873206 |
|              Ladino (lad)              |  0.660920 | 0.576441 | 0.615797 |
|                Lao (lao)               |  0.986175 | 0.918455 | 0.951111 |
|               Latin (lat)              |  0.581250 | 0.636986 | 0.607843 |
|              Latvian (lav)             |  0.824513 | 0.797844 | 0.810959 |
|             Lezghian (lez)             |  0.898955 | 0.793846 | 0.843137 |
|             Ligurian (lij)             |  0.662903 | 0.677100 | 0.669927 |
|             Limburgan (lim)            |  0.615385 | 0.581818 | 0.598131 |
|              Lingala (lin)             |  0.836207 | 0.763780 | 0.798354 |
|            Lithuanian (lit)            |  0.756329 | 0.804714 | 0.779772 |
|              Lombard (lmo)             |  0.556818 | 0.536986 | 0.546722 |
|           Northern Luri (lrc)          |  0.838574 | 0.753296 | 0.793651 |
|             Latgalian (ltg)            |  0.759531 | 0.755102 | 0.757310 |
|           Luxembourgish (ltz)          |  0.645062 | 0.614706 | 0.629518 |
|              Luganda (lug)             |  0.787535 | 0.805797 | 0.796562 |
|         Literary Chinese (lzh)         |  0.921951 | 0.949749 | 0.935644 |
|             Maithili (mai)             |  0.777778 | 0.761658 | 0.769634 |
|             Malayalam (mal)            |  0.993377 | 0.949367 | 0.970874 |
|          Banyumasan (map-bms)          |  0.531429 | 0.453659 | 0.489474 |
|              Marathi (mar)             |  0.748744 | 0.818681 | 0.782152 |
|              Moksha (mdf)              |  0.728745 | 0.800000 | 0.762712 |
|           Eastern Mari (mhr)           |  0.790323 | 0.760870 | 0.775316 |
|            Minangkabau (min)           |  0.953271 | 0.886957 | 0.918919 |
|            Macedonian (mkd)            |  0.816399 | 0.849722 | 0.832727 |
|             Malagasy (mlg)             |  0.925187 | 0.918317 | 0.921739 |
|              Maltese (mlt)             |  0.869421 | 0.890017 | 0.879599 |
|          Min Nan Chinese (nan)         |  0.743707 | 0.820707 | 0.780312 |
|             Mongolian (mon)            |  0.852194 | 0.838636 | 0.845361 |
|               Maori (mri)              |  0.934726 | 0.937173 | 0.935948 |
|           Western Mari (mrj)           |  0.818792 | 0.827119 | 0.822934 |
|               Malay (msa)              |  0.508065 | 0.376119 | 0.432247 |
|             Mirandese (mwl)            |  0.650407 | 0.685225 | 0.667362 |
|              Burmese (mya)             |  0.995968 | 0.972441 | 0.984064 |
|               Erzya (myv)              |  0.475783 | 0.503012 | 0.489019 |
|            Mazanderani (mzn)           |  0.775362 | 0.701639 | 0.736661 |
|            Neapolitan (nap)            |  0.628993 | 0.595349 | 0.611708 |
|              Navajo (nav)              |  0.955882 | 0.937500 | 0.946602 |
|         Classical Nahuatl (nci)        |  0.679758 | 0.589005 | 0.631136 |
|            Low German (nds)            |  0.669789 | 0.690821 | 0.680143 |
|        West Low German (nds-nl)        |  0.513889 | 0.504545 | 0.509174 |
|      Nepali (macrolanguage) (nep)      |  0.640476 | 0.649758 | 0.645084 |
|              Newari (new)              |  0.928571 | 0.745902 | 0.827273 |
|               Dutch (nld)              |  0.553763 | 0.553763 | 0.553763 |
|         Norwegian Nynorsk (nno)        |  0.569277 | 0.519231 | 0.543103 |
|              Bokmål (nob)              |  0.519856 | 0.562500 | 0.540338 |
|               Narom (nrm)              |  0.691275 | 0.605882 | 0.645768 |
|          Northern Sotho (nso)          |  0.950276 | 0.815166 | 0.877551 |
|              Occitan (oci)             |  0.483444 | 0.366834 | 0.417143 |
|          Livvi-Karelian (olo)          |  0.816850 | 0.790780 | 0.803604 |
|               Oriya (ori)              |  0.981481 | 0.963636 | 0.972477 |
|               Oromo (orm)              |  0.885714 | 0.829218 | 0.856536 |
|             Ossetian (oss)             |  0.822006 | 0.855219 | 0.838284 |
|            Pangasinan (pag)            |  0.842105 | 0.715655 | 0.773748 |
|             Pampanga (pam)             |  0.770000 | 0.435028 | 0.555957 |
|              Panjabi (pan)             |  0.996154 | 0.984791 | 0.990440 |
|            Papiamento (pap)            |  0.674672 | 0.661670 | 0.668108 |
|              Picard (pcd)              |  0.407895 | 0.356322 | 0.380368 |
|        Pennsylvania German (pdc)       |  0.487047 | 0.509485 | 0.498013 |
|          Palatine German (pfl)         |  0.614173 | 0.570732 | 0.591656 |
|          Western Panjabi (pnb)         |  0.926267 | 0.887417 | 0.906426 |
|              Polish (pol)              |  0.797059 | 0.734417 | 0.764457 |
|            Portuguese (por)            |  0.500914 | 0.586724 | 0.540434 |
|              Pushto (pus)              |  0.941489 | 0.898477 | 0.919481 |
|              Quechua (que)             |  0.854167 | 0.797665 | 0.824950 |
|      Tarantino dialect (roa-tara)      |  0.669794 | 0.724138 | 0.695906 |
|              Romansh (roh)             |  0.745527 | 0.760649 | 0.753012 |
|             Romanian (ron)             |  0.805486 | 0.769048 | 0.786845 |
|               Rusyn (rue)              |  0.718543 | 0.645833 | 0.680251 |
|             Aromanian (rup)            |  0.288482 | 0.730245 | 0.413580 |
|              Russian (rus)             |  0.530120 | 0.690583 | 0.599805 |
|               Yakut (sah)              |  0.853521 | 0.865714 | 0.859574 |
|             Sanskrit (san)             |  0.931343 | 0.896552 | 0.913616 |
|             Sicilian (scn)             |  0.734139 | 0.618321 | 0.671271 |
|               Scots (sco)              |  0.571429 | 0.540816 | 0.555701 |
|            Samogitian (sgs)            |  0.829167 | 0.748120 | 0.786561 |
|              Sinhala (sin)             |  0.909474 | 0.935065 | 0.922092 |
|              Slovak (slk)              |  0.738235 | 0.665782 | 0.700139 |
|              Slovene (slv)             |  0.671123 | 0.662269 | 0.666667 |
|           Northern Sami (sme)          |  0.800676 | 0.825784 | 0.813036 |
|               Shona (sna)              |  0.761702 | 0.724696 | 0.742739 |
|              Sindhi (snd)              |  0.950172 | 0.946918 | 0.948542 |
|              Somali (som)              |  0.849462 | 0.802030 | 0.825065 |
|              Spanish (spa)             |  0.325234 | 0.413302 | 0.364017 |
|             Albanian (sqi)             |  0.875899 | 0.832479 | 0.853637 |
|             Sardinian (srd)            |  0.750000 | 0.711061 | 0.730012 |
|              Sranan (srn)              |  0.888889 | 0.771084 | 0.825806 |
|              Serbian (srp)             |  0.824561 | 0.814356 | 0.819427 |
|          Saterfriesisch (stq)          |  0.790087 | 0.734417 | 0.761236 |
|             Sundanese (sun)            |  0.764192 | 0.631769 | 0.691700 |
|      Swahili (macrolanguage) (swa)     |  0.763496 | 0.796247 | 0.779528 |
|              Swedish (swe)             |  0.838284 | 0.723647 | 0.776758 |
|             Silesian (szl)             |  0.819788 | 0.750809 | 0.783784 |
|               Tamil (tam)              |  0.985765 | 0.955172 | 0.970228 |
|               Tatar (tat)              |  0.469780 | 0.795349 | 0.590674 |
|               Tulu (tcy)               |  0.893300 | 0.873786 | 0.883436 |
|              Telugu (tel)              |  1.000000 | 0.913690 | 0.954899 |
|               Tetum (tet)              |  0.765116 | 0.744344 | 0.754587 |
|               Tajik (tgk)              |  0.828418 | 0.813158 | 0.820717 |
|              Tagalog (tgl)             |  0.751468 | 0.757396 | 0.754420 |
|               Thai (tha)               |  0.933884 | 0.807143 | 0.865900 |
|              Tongan (ton)              |  0.920245 | 0.923077 | 0.921659 |
|              Tswana (tsn)              |  0.873397 | 0.889070 | 0.881164 |
|              Turkmen (tuk)             |  0.898438 | 0.837887 | 0.867107 |
|              Turkish (tur)             |  0.666667 | 0.716981 | 0.690909 |
|               Tuvan (tyv)              |  0.857143 | 0.805063 | 0.830287 |
|              Udmurt (udm)              |  0.865517 | 0.756024 | 0.807074 |
|              Uighur (uig)              |  0.991597 | 0.967213 | 0.979253 |
|             Ukrainian (ukr)            |  0.771341 | 0.702778 | 0.735465 |
|               Urdu (urd)               |  0.877647 | 0.855505 | 0.866434 |
|               Uzbek (uzb)              |  0.655652 | 0.797040 | 0.719466 |
|             Venetian (vec)             |  0.611111 | 0.527233 | 0.566082 |
|               Veps (vep)               |  0.672862 | 0.688213 | 0.680451 |
|            Vietnamese (vie)            |  0.932406 | 0.914230 | 0.923228 |
|              Vlaams (vls)              |  0.594427 | 0.501305 | 0.543909 |
|              Volapük (vol)             |  0.765625 | 0.942308 | 0.844828 |
|               Võro (vro)               |  0.797203 | 0.740260 | 0.767677 |
|               Waray (war)              |  0.930876 | 0.930876 | 0.930876 |
|              Walloon (wln)             |  0.636804 | 0.693931 | 0.664141 |
|               Wolof (wol)              |  0.864220 | 0.845601 | 0.854809 |
|            Wu Chinese (wuu)            |  0.848921 | 0.830986 | 0.839858 |
|               Xhosa (xho)              |  0.837398 | 0.759214 | 0.796392 |
|            Mingrelian (xmf)            |  0.943396 | 0.874126 | 0.907441 |
|              Yiddish (yid)             |  0.955729 | 0.897311 | 0.925599 |
|              Yoruba (yor)              |  0.812010 | 0.719907 | 0.763190 |
|              Zeeuws (zea)              |  0.617737 | 0.550409 | 0.582133 |
|           Cantonese (zh-yue)           |  0.859649 | 0.649007 | 0.739623 |
|         Standard Chinese (zho)         |  0.845528 | 0.781955 | 0.812500 |
|                accuracy                |  0.749527 | 0.749527 | 0.749527 |
|                macro avg               |  0.762866 | 0.742101 | 0.749261 |
|              weighted avg              |  0.762006 | 0.749527 | 0.752910 |

## Questions?
Post a Github issue from [HERE](https://github.com/m3hrdadfi/zabanshenas/issues).


---
language: fa
datasets:
- common_voice
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
widget:
- example_title: Common Voice sample 1
  src: https://huggingface.co/m3hrdadfi/wav2vec2-large-xlsr-persian-v3/resolve/main/sample1.flac
- example_title: Common Voice sample 2978
  src: https://huggingface.co/m3hrdadfi/wav2vec2-large-xlsr-persian-v3/resolve/main/sample2978.flac
- example_title: Common Voice sample 5168
  src: https://huggingface.co/m3hrdadfi/wav2vec2-large-xlsr-persian-v3/resolve/main/sample5168.flac
model-index:
- name: XLSR Wav2Vec2 Persian (Farsi) V3 by Mehrdad Farahani
  results:
  - task:
      name: Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Common Voice fa
      type: common_voice
      args: fa
    metrics:
       - name: Test WER
         type: wer
         value: 10.36
---

# Wav2Vec2-Large-XLSR-53-Persian V3

## Usage
Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) in Persian (Farsi) using [Common Voice](https://huggingface.co/datasets/common_voice). When using this model, make sure that your speech input is sampled at 16kHz.

**Requirements**
```bash
# requirement packages
!pip install git+https://github.com/huggingface/datasets.git
!pip install git+https://github.com/huggingface/transformers.git
!pip install torchaudio
!pip install librosa
!pip install jiwer
!pip install parsivar
!pip install num2fawords
```

**Normalizer**
```bash
# Normalizer
!wget -O normalizer.py https://huggingface.co/m3hrdadfi/"wav2vec2-large-xlsr-persian-v3/raw/main/dictionary.py
!wget -O normalizer.py https://huggingface.co/m3hrdadfi/"wav2vec2-large-xlsr-persian-v3/raw/main/normalizer.py
```

**Downloading data**
```bash
wget https://voice-prod-bundler-ee1969a6ce8178826482b88e843c335139bd3fb4.s3.amazonaws.com/cv-corpus-6.1-2020-12-11/fa.tar.gz

tar -xzf fa.tar.gz
rm -rf fa.tar.gz
```

**Cleaning**
```python
from normalizer import normalizer

def cleaning(text):
    if not isinstance(text, str):
        return None

    return normalizer({"sentence": text}, return_dict=False)

data_dir = "/content/cv-corpus-6.1-2020-12-11/fa"

test = pd.read_csv(f"{data_dir}/test.tsv", sep="	")
test["path"] = data_dir + "/clips/" + test["path"]
print(f"Step 0: {len(test)}")

test["status"] = test["path"].apply(lambda path: True if os.path.exists(path) else None)
test = test.dropna(subset=["path"])
test = test.drop("status", 1)
print(f"Step 1: {len(test)}")

test["sentence"] = test["sentence"].apply(lambda t: cleaning(t))
test = test.dropna(subset=["sentence"])
print(f"Step 2: {len(test)}")

test = test.reset_index(drop=True)
print(test.head())

test = test[["path", "sentence"]]
test.to_csv("/content/test.csv", sep="	", encoding="utf-8", index=False)
```

**Prediction**
```python
import numpy as np
import pandas as pd

import librosa
import torch
import torchaudio
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
from datasets import load_dataset, load_metric

import IPython.display as ipd

model_name_or_path = "m3hrdadfi/wav2vec2-large-xlsr-persian-v3"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(model_name_or_path, device)

processor = Wav2Vec2Processor.from_pretrained(model_name_or_path)
model = Wav2Vec2ForCTC.from_pretrained(model_name_or_path).to(device)

def speech_file_to_array_fn(batch):
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    speech_array = speech_array.squeeze().numpy()
    speech_array = librosa.resample(np.asarray(speech_array), sampling_rate, processor.feature_extractor.sampling_rate)

    batch["speech"] = speech_array
    return batch

def predict(batch):
    features = processor(
        batch["speech"],
        sampling_rate=processor.feature_extractor.sampling_rate,
        return_tensors="pt",
        padding=True
    )

    input_values = features.input_values.to(device)
    attention_mask = features.attention_mask.to(device)

    with torch.no_grad():
        logits = model(input_values, attention_mask=attention_mask).logits

    pred_ids = torch.argmax(logits, dim=-1)

    batch["predicted"] = processor.batch_decode(pred_ids)
    return batch

dataset = load_dataset("csv", data_files={"test": "/content/test.csv"}, delimiter="	")["test"]
dataset = dataset.map(speech_file_to_array_fn)
result = dataset.map(predict, batched=True, batch_size=4)
```

**WER Score**
```python
wer = load_metric("wer")
print("WER: {:.2f}".format(100 * wer.compute(predictions=result["predicted"], references=result["sentence"])))
```

**Output**
```python
max_items = np.random.randint(0, len(result), 20).tolist()
for i in max_items:
    reference, predicted =  result["sentence"][i], result["predicted"][i]
    print("reference:", reference)
    print("predicted:", predicted)
    print('---')
```

```text
reference: ماجرا رو براش تعریف کردم اون گفت مریم اگه میدونی پسر خوبیه خب چه اشکالی داره با‌هاش بیش‌تر اشنا بشو
predicted: ماجرا رو براش تعریف کردم اون گفت مریم اگه میدونی پسر خوبیه خب چه اشکالی داره با‌هاش بیش‌تر اشنا بشو
---
reference: بیا پایین تو اجازه نداری بری اون بالا
predicted: بیا پایین تو اجازه نداری بری اون بالا
---
reference: هر روز یک دو مداد کش می رفتتم تااین که تا پایان ترم از تمامی دوستانم مداد برداشته بودم
predicted: هر روز یک دو مداد کش می رفتم تااین که تا پایین ترم از تمامی دوستان و مداد برداشته بودم
---
reference: فکر میکنی آروم میشینه
predicted: فکر میکنی آروم میشینه
---
reference: هرکسی با گوشی هوشمند خود میتواند با کایلا متصل گردد در یک محدوده مکانی
predicted: هرکسی با گوشی هوشمند خود میتواند با کایلا متصل گردد در یک محدوده مکانی
---
reference: برو از مهرداد بپرس
predicted: برو از مهرداد بپرس
---
reference: می خواهم شما را با این قدم‌ها آشنا کنم
predicted: می خواهم شما را با این قدم‌ها آشنا کنم
---
reference: میدونم یه روز دوباره می تونم تو رو ببینم
predicted: میدونم یه روز دوباره می تونم تو رو ببینم
---
reference: بسیار خوب خواهد بود دعوت او را بپذیری
predicted: بسیار خوب خواهد بود دعوت او را بپذیری
---
reference: بهت بگن آشغالی خوبه
predicted: بهت بگن آشغالی خوبه
---
reference: چرا معاشرت با هم ایمانان ما را محفوظ نگه میدارد
predicted: چرا معاشرت با هم ایمانان آ را م حفوظ نگه میدارد
---
reference: بولیوی پس از گویان فقیر‌ترین کشور آمریکای جنوبی است
predicted: بولیوی پس از گویان فقیر‌ترین کشور آمریکای جنوبی است
---
reference: بعد از مدتی اینکار برایم عادی شد
predicted: بعد از مدتی اینکار برایم عادو شد
---
reference: به نظر اون هم همینطوره
predicted: به نظر اون هم همینطوره
---
reference: هیچ مایونز ی دارید
predicted: هیچ مایونز ی دارید
---
reference: هیچ یک از انان کاری به سنگ نداشتند
predicted: هیچ شک از انان کاری به سنگ نداشتند
---
reference: می خواهم کمی کتاب شعر ببینم
predicted: می خواهم کتاب شعر ببینم
---
reference: همین شوهر فهیمه مگه نمی گفتی فرمانده بوده کو
predicted: همین شوهر فهیمه بینامی گفتی فهمانده بود کو
---
reference: اون جا‌ها کسی رو نمیبینی که تو دستش کتاب نباشه
predicted: اون جا‌ها کسی رو نمیبینی که تو دستش کتاب نباشه
---
reference: زندان رفتن من در این سال‌های اخیر برام شانس بزرگی بود که معما و مشکل چندین سال‌هام را حل کرد
predicted: زندان رفتن من در این سال‌ها اخی براب شانس بزرگی بود که معما و مشکل چندین سال‌هام را حل کرد
---
```

## Evaluation

**Test Result:**
- WER: 10.36%


---
language: fa
tags:
- albert-persian
- persian-lm
license: apache-2.0
---

# ALBERT-Persian

A Lite BERT for Self-supervised Learning of Language Representations for the Persian Language
> میتونی بهش بگی برت_کوچولو

## Introduction

ALBERT-Persian trained on a massive amount of public corpora ([Persian Wikidumps](https://dumps.wikimedia.org/fawiki/), [MirasText](https://github.com/miras-tech/MirasText)) and six other manually crawled text data from a various type of websites ([BigBang Page](https://bigbangpage.com/) `scientific`, [Chetor](https://www.chetor.com/) `lifestyle`, [Eligasht](https://www.eligasht.com/Blog/) `itinerary`,  [Digikala](https://www.digikala.com/mag/) `digital magazine`, [Ted Talks](https://www.ted.com/talks) `general conversational`, Books `novels, storybooks, short stories from old to the contemporary era`).

Please follow the [ALBERT-Persian](https://github.com/m3hrdadfi/albert-persian) repo for the latest information about previous and current models.

## Intended uses & limitations

You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to
be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?search=albert-fa) to look for
fine-tuned versions on a task that interests you.

### How to use

- for using any type of Albert you have to install sentencepiece
- run this in your notebook ``` !pip install -q sentencepiece ```

#### TensorFlow 2.0

```python
from transformers import AutoConfig, AutoTokenizer, TFAutoModel

config = AutoConfig.from_pretrained("m3hrdadfi/albert-fa-base-v2")
tokenizer = AutoTokenizer.from_pretrained("m3hrdadfi/albert-fa-base-v2")
model = TFAutoModel.from_pretrained("m3hrdadfi/albert-fa-base-v2")

text = "ما در هوشواره معتقدیم با انتقال صحیح دانش و آگاهی، همه افراد می‌توانند از ابزارهای هوشمند استفاده کنند. شعار ما هوش مصنوعی برای همه است."
tokenizer.tokenize(text)

>>> ['▁ما', '▁در', '▁هوش', 'واره', '▁معتقد', 'یم', '▁با', '▁انتقال', '▁صحیح', '▁دانش', '▁و', '▁اگاه', 'ی', '،', '▁همه', '▁افراد', '▁می', '▁توانند', '▁از', '▁ابزارهای', '▁هوشمند', '▁استفاده', '▁کنند', '.', '▁شعار', '▁ما', '▁هوش', '▁مصنوعی', '▁برای', '▁همه', '▁است', '.']

```

#### Pytorch

```python
from transformers import AutoConfig, AutoTokenizer, AutoModel

config = AutoConfig.from_pretrained("m3hrdadfi/albert-fa-base-v2")
tokenizer = AutoTokenizer.from_pretrained("m3hrdadfi/albert-fa-base-v2")
model = AutoModel.from_pretrained("m3hrdadfi/albert-fa-base-v2")
```

## Training

ALBERT-Persian is the first attempt on ALBERT for the Persian Language. The model was trained based on Google's ALBERT BASE Version 2.0 over various writing styles from numerous subjects (e.g., scientific, novels, news) with more than `3.9M` documents, `73M` sentences, and `1.3B` words, like the way we did for [ParsBERT](https://github.com/hooshvare/parsbert).

## Goals
Objective goals during training are as below (after 140K steps).

``` bash
***** Eval results *****
global_step = 140000
loss = 2.0080082
masked_lm_accuracy = 0.6141017
masked_lm_loss = 1.9963315
sentence_order_accuracy = 0.985
sentence_order_loss = 0.06908702
```

## Derivative models

### Base Config

#### Albert Model
- [m3hrdadfi/albert-face-base-v2](https://huggingface.co/m3hrdadfi/albert-fa-base-v2)

#### Albert Sentiment Analysis
- [m3hrdadfi/albert-fa-base-v2-sentiment-digikala](https://huggingface.co/m3hrdadfi/albert-fa-base-v2-sentiment-digikala)
- [m3hrdadfi/albert-fa-base-v2-sentiment-snappfood](https://huggingface.co/m3hrdadfi/albert-fa-base-v2-sentiment-snappfood)
- [m3hrdadfi/albert-fa-base-v2-sentiment-deepsentipers-binary](https://huggingface.co/m3hrdadfi/albert-fa-base-v2-sentiment-deepsentipers-binary)
- [m3hrdadfi/albert-fa-base-v2-sentiment-deepsentipers-multi](https://huggingface.co/m3hrdadfi/albert-fa-base-v2-sentiment-deepsentipers-multi)
- [m3hrdadfi/albert-fa-base-v2-sentiment-binary](https://huggingface.co/m3hrdadfi/albert-fa-base-v2-sentiment-binary)
- [m3hrdadfi/albert-fa-base-v2-sentiment-multi](https://huggingface.co/m3hrdadfi/albert-fa-base-v2-sentiment-multi)
- [m3hrdadfi/albert-fa-base-v2-sentiment-multi](https://huggingface.co/m3hrdadfi/albert-fa-base-v2-sentiment-multi)

#### Albert Text Classification
- [m3hrdadfi/albert-fa-base-v2-clf-digimag](https://huggingface.co/m3hrdadfi/albert-fa-base-v2-clf-digimag)
- [m3hrdadfi/albert-fa-base-v2-clf-persiannews](https://huggingface.co/m3hrdadfi/albert-fa-base-v2-clf-persiannews)

#### Albert NER
- [m3hrdadfi/albert-fa-base-v2-ner](https://huggingface.co/m3hrdadfi/albert-fa-base-v2-ner)
- [m3hrdadfi/albert-fa-base-v2-ner-arman](https://huggingface.co/m3hrdadfi/albert-fa-base-v2-ner-arman)
- [m3hrdadfi/albert-fa-base-v2-ner-arman](https://huggingface.co/m3hrdadfi/albert-fa-base-v2-ner-arman)

## Eval results

The following tables summarize the F1 scores obtained by ALBERT-Persian as compared to other models and architectures.

### Sentiment Analysis (SA) Task

|          Dataset         | ALBERT-fa-base-v2 | ParsBERT-v1 | mBERT | DeepSentiPers |
|:------------------------:|:-----------------:|:-----------:|:-----:|:-------------:|
|  Digikala User Comments  |       81.12       |    81.74    | 80.74 |       -       |
|  SnappFood User Comments |       85.79       |    88.12    | 87.87 |       -       |
|  SentiPers (Multi Class) |       66.12       |    71.11    |   -   |     69.33     |
| SentiPers (Binary Class) |       91.09       |    92.13    |   -   |     91.98     |

### Text Classification (TC) Task

|      Dataset      | ALBERT-fa-base-v2 | ParsBERT-v1 | mBERT |
|:-----------------:|:-----------------:|:-----------:|:-----:|
| Digikala Magazine |       92.33       |    93.59    | 90.72 |
|    Persian News   |       97.01       |    97.19    | 95.79 |

### Named Entity Recognition (NER) Task

| Dataset | ALBERT-fa-base-v2 | ParsBERT-v1 | mBERT | MorphoBERT | Beheshti-NER | LSTM-CRF | Rule-Based CRF | BiLSTM-CRF |
|:-------:|:-----------------:|:-----------:|:-----:|:----------:|:------------:|:--------:|:--------------:|:----------:|
|  PEYMA  |       88.99       |    93.10    | 86.64 |      -     |     90.59    |     -    |      84.00     |      -     |
|  ARMAN  |       97.43       |    98.79    | 95.89 |    89.9    |     84.03    |   86.55  |        -       |    77.45   |

### BibTeX entry and citation info

Please cite in publications as the following:

```bibtex
@misc{ALBERT-Persian,
  author = {Mehrdad Farahani},
  title = {ALBERT-Persian: A Lite BERT for Self-supervised Learning of Language Representations for the Persian Language},
  year = {2020},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/m3hrdadfi/albert-persian}},
}

@article{ParsBERT,
    title={ParsBERT: Transformer-based Model for Persian Language Understanding},
    author={Mehrdad Farahani, Mohammad Gharachorloo, Marzieh Farahani, Mohammad Manthouri},
    journal={ArXiv},
    year={2020},
    volume={abs/2005.12515}
}
```

## Questions?
Post a Github issue on the [ALBERT-Persian](https://github.com/m3hrdadfi/albert-persian) repo.


---
language: fa
datasets:
- ShEMO
tags:
- audio
- speech
- speech-emotion-recognition
license: apache-2.0
---

# Emotion Recognition in Persian (fa) Speech using HuBERT

## How to use

### Requirements

```bash
# requirement packages
!pip install git+https://github.com/huggingface/datasets.git
!pip install git+https://github.com/huggingface/transformers.git
!pip install torchaudio
!pip install librosa
```

```bash
!git clone https://github.com/m3hrdadfi/soxan.git .
```

### Prediction

```python
import torch
import torch.nn as nn
import torch.nn.functional as F
import torchaudio
from transformers import AutoConfig, Wav2Vec2FeatureExtractor
from src.models import Wav2Vec2ForSpeechClassification, HubertForSpeechClassification

import librosa
import IPython.display as ipd
import numpy as np
import pandas as pd
```

```python
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model_name_or_path = "m3hrdadfi/hubert-base-persian-speech-emotion-recognition"
config = AutoConfig.from_pretrained(model_name_or_path)
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(model_name_or_path)
sampling_rate = feature_extractor.sampling_rate
model = HubertForSpeechClassification.from_pretrained(model_name_or_path).to(device)
```

```python
def speech_file_to_array_fn(path, sampling_rate):
    speech_array, _sampling_rate = torchaudio.load(path)
    resampler = torchaudio.transforms.Resample(_sampling_rate)
    speech = resampler(speech_array).squeeze().numpy()
    return speech

def predict(path, sampling_rate):
    speech = speech_file_to_array_fn(path, sampling_rate)
    inputs = feature_extractor(speech, sampling_rate=sampling_rate, return_tensors="pt", padding=True)
    inputs = {key: inputs[key].to(device) for key in inputs}

    with torch.no_grad():
        logits = model(**inputs).logits

    scores = F.softmax(logits, dim=1).detach().cpu().numpy()[0]
    outputs = [{"Label": config.id2label[i], "Score": f"{round(score * 100, 3):.1f}%"} for i, score in enumerate(scores)]
    return outputs
```

```python
path = "/path/to/sadness.wav"
outputs = predict(path, sampling_rate)
```

```bash
[
{'Label': 'Anger', 'Score': '0.0%'},
{'Label': 'Fear', 'Score': '0.0%'},
{'Label': 'Happiness', 'Score': '0.0%'},
{'Label': 'Neutral', 'Score': '0.0%'},
{'Label': 'Sadness', 'Score': '99.9%'},
{'Label': 'Surprise', 'Score': '0.0%'}
]
```

## Evaluation
The following tables summarize the scores obtained by model overall and per each class.

|  Emotions | precision | recall | f1-score | accuracy |
|:---------:|:---------:|:------:|:--------:|:--------:|
|   Anger   |    0.96   |  0.96  |   0.96   |          |
|    Fear   |    1.00   |  0.50  |   0.67   |          |
| Happiness |    0.79   |  0.87  |   0.83   |          |
|  Neutral  |    0.93   |  0.94  |   0.93   |          |
|  Sadness  |    0.87   |  0.94  |   0.91   |          |
|  Surprise |    0.97   |  0.75  |   0.85   |          |
|           |           |        |  Overal  |   0.92   |

## Questions?
Post a Github issue from [HERE](https://github.com/m3hrdadfi/soxan/issues).