---
language:
- multilingual
- ar
- as
- br
- ca
- cnh
- cs
- cv
- cy
- de
- dv
- el
- en
- eo
- es
- et
- eu
- fa
- fi
- fr
- hi
- hsb
- hu
- ia
- id
- ja
- ka
- ky
- lg
- lt
- ly
- mn
- mt
- nl
- or
- pl
- pt
- ro
- ru
- sah
- sl
- ta
- th
- tr
- tt
- uk
- vi
license: apache-2.0
tags:
- audio
- automatic-speech-recognition
- hf-asr-leaderboard
- robust-speech-event
- speech
- xlsr-fine-tuning-week
datasets:
- common_voice
language_bcp47:
- fy-NL
- ga-IE
- pa-IN
- rm-sursilv
- rm-vallader
- sy-SE
- zh-CN
- zh-HK
- zh-TW
model-index:
- name: XLSR Wav2Vec2 for 56 language by Voidful
results:
- task:
type: automatic-speech-recognition
name: Speech Recognition
dataset:
name: Common Voice
type: common_voice
metrics:
- type: cer
value: 23.21
name: Test CER
---
# Model Card for wav2vec2-xlsr-multilingual-56
# Model Details
## Model Description
- **Developed by:** voidful
- **Shared by [Optional]:** Hugging Face
- **Model type:** automatic-speech-recognition
- **Language(s) (NLP):** multilingual (*56 language, 1 model Multilingual ASR*)
- **License:** Apache-2.0
- **Related Models:**
- **Parent Model:** wav2vec
- **Resources for more information:**
- [GitHub Repo](https://github.com/voidful/wav2vec2-xlsr-multilingual-56)
- [Model Space](https://huggingface.co/spaces/Kamtera/Persian_Automatic_Speech_Recognition_and-more)
# Uses
## Direct Use
This model can be used for the task of automatic-speech-recognition
## Downstream Use [Optional]
More information needed
## Out-of-Scope Use
The model should not be used to intentionally create hostile or alienating environments for people.
# Bias, Risks, and Limitations
Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups.
## Recommendations
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
# Training Details
## Training Data
See the [common_voice dataset card](https://huggingface.co/datasets/common_voice)
Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on 56 language using the [Common Voice](https://huggingface.co/datasets/common_voice).
## Training Procedure
### Preprocessing
More information needed
### Speeds, Sizes, Times
When using this model, make sure that your speech input is sampled at 16kHz.
# Evaluation
## Testing Data, Factors & Metrics
### Testing Data
More information needed
### Factors
### Metrics
More information needed
## Results
Click to expand
| Common Voice Languages | Num. of data | Hour | WER | CER |
|------------------------|--------------|--------|--------|-------|
| ar | 21744 | 81.5 | 75.29 | 31.23 |
| as | 394 | 1.1 | 95.37 | 46.05 |
| br | 4777 | 7.4 | 93.79 | 41.16 |
| ca | 301308 | 692.8 | 24.80 | 10.39 |
| cnh | 1563 | 2.4 | 68.11 | 23.10 |
| cs | 9773 | 39.5 | 67.86 | 12.57 |
| cv | 1749 | 5.9 | 95.43 | 34.03 |
| cy | 11615 | 106.7 | 67.03 | 23.97 |
| de | 262113 | 822.8 | 27.03 | 6.50 |
| dv | 4757 | 18.6 | 92.16 | 30.15 |
| el | 3717 | 11.1 | 94.48 | 58.67 |
| en | 580501 | 1763.6 | 34.87 | 14.84 |
| eo | 28574 | 162.3 | 37.77 | 6.23 |
| es | 176902 | 337.7 | 19.63 | 5.41 |
| et | 5473 | 35.9 | 86.87 | 20.79 |
| eu | 12677 | 90.2 | 44.80 | 7.32 |
| fa | 12806 | 290.6 | 53.81 | 15.09 |
| fi | 875 | 2.6 | 93.78 | 27.57 |
| fr | 314745 | 664.1 | 33.16 | 13.94 |
| fy-NL | 6717 | 27.2 | 72.54 | 26.58 |
| ga-IE | 1038 | 3.5 | 92.57 | 51.02 |
| hi | 292 | 2.0 | 90.95 | 57.43 |
| hsb | 980 | 2.3 | 89.44 | 27.19 |
| hu | 4782 | 9.3 | 97.15 | 36.75 |
| ia | 5078 | 10.4 | 52.00 | 11.35 |
| id | 3965 | 9.9 | 82.50 | 22.82 |
| it | 70943 | 178.0 | 39.09 | 8.72 |
| ja | 1308 | 8.2 | 99.21 | 62.06 |
| ka | 1585 | 4.0 | 90.53 | 18.57 |
| ky | 3466 | 12.2 | 76.53 | 19.80 |
| lg | 1634 | 17.1 | 98.95 | 43.84 |
| lt | 1175 | 3.9 | 92.61 | 26.81 |
| lv | 4554 | 6.3 | 90.34 | 30.81 |
| mn | 4020 | 11.6 | 82.68 | 30.14 |
| mt | 3552 | 7.8 | 84.18 | 22.96 |
| nl | 14398 | 71.8 | 57.18 | 19.01 |
| or | 517 | 0.9 | 90.93 | 27.34 |
| pa-IN | 255 | 0.8 | 87.95 | 42.03 |
| pl | 12621 | 112.0 | 56.14 | 12.06 |
| pt | 11106 | 61.3 | 53.24 | 16.32 |
| rm-sursilv | 2589 | 5.9 | 78.17 | 23.31 |
| rm-vallader | 931 | 2.3 | 73.67 | 21.76 |
| ro | 4257 | 8.7 | 83.84 | 21.95 |
| ru | 23444 | 119.1 | 61.83 | 15.18 |
| sah | 1847 | 4.4 | 94.38 | 38.46 |
| sl | 2594 | 6.7 | 84.21 | 20.54 |
| sv-SE | 4350 | 20.8 | 83.68 | 30.79 |
| ta | 3788 | 18.4 | 84.19 | 21.60 |
| th | 4839 | 11.7 | 141.87 | 37.16 |
| tr | 3478 | 22.3 | 66.77 | 15.55 |
| tt | 13338 | 26.7 | 86.80 | 33.57 |
| uk | 7271 | 39.4 | 70.23 | 14.34 |
| vi | 421 | 1.7 | 96.06 | 66.25 |
| zh-CN | 27284 | 58.7 | 89.67 | 23.96 |
| zh-HK | 12678 | 92.1 | 81.77 | 18.82 |
| zh-TW | 6402 | 56.6 | 85.08 | 29.07 |
# Model Examination
More information needed
# Environmental Impact
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
- **Hardware Type:** More information needed
- **Hours used:** More information needed
- **Cloud Provider:** More information needed
- **Compute Region:** More information needed
- **Carbon Emitted:** More information needed
# Technical Specifications [optional]
## Model Architecture and Objective
More information needed
## Compute Infrastructure
More information needed
### Hardware
More information needed
### Software
More information needed
# Citation
**BibTeX:**
```
More information needed
```
**APA:**
```
More information needed
```
# Glossary [optional]
More information needed
# More Information [optional]
More information needed
# Model Card Authors [optional]
voidful in collaboration with Ezi Ozoani and the Hugging Face team
# Model Card Contact
More information needed
# How to Get Started with the Model
Use the code below to get started with the model.
Click to expand
## Env setup:
```
!pip install torchaudio
!pip install datasets transformers
!pip install asrp
!wget -O lang_ids.pk https://huggingface.co/voidful/wav2vec2-xlsr-multilingual-56/raw/main/lang_ids.pk
```
## Usage
```
import torchaudio
from datasets import load_dataset, load_metric
from transformers import (
Wav2Vec2ForCTC,
Wav2Vec2Processor,
AutoTokenizer,
AutoModelWithLMHead
)
import torch
import re
import sys
import soundfile as sf
model_name = "voidful/wav2vec2-xlsr-multilingual-56"
device = "cuda"
processor_name = "voidful/wav2vec2-xlsr-multilingual-56"
import pickle
with open("lang_ids.pk", 'rb') as output:
lang_ids = pickle.load(output)
model = Wav2Vec2ForCTC.from_pretrained(model_name).to(device)
processor = Wav2Vec2Processor.from_pretrained(processor_name)
model.eval()
def load_file_to_data(file,sampling_rate=16_000):
batch = {}
speech, _ = torchaudio.load(file)
if sampling_rate != '16_000' or sampling_rate != '16000':
resampler = torchaudio.transforms.Resample(orig_freq=sampling_rate, new_freq=16_000)
batch["speech"] = resampler.forward(speech.squeeze(0)).numpy()
batch["sampling_rate"] = resampler.new_freq
else:
batch["speech"] = speech.squeeze(0).numpy()
batch["sampling_rate"] = '16000'
return batch
def predict(data):
features = processor(data["speech"], sampling_rate=data["sampling_rate"], padding=True, return_tensors="pt")
input_values = features.input_values.to(device)
attention_mask = features.attention_mask.to(device)
with torch.no_grad():
logits = model(input_values, attention_mask=attention_mask).logits
decoded_results = []
for logit in logits:
pred_ids = torch.argmax(logit, dim=-1)
mask = pred_ids.ge(1).unsqueeze(-1).expand(logit.size())
vocab_size = logit.size()[-1]
voice_prob = torch.nn.functional.softmax((torch.masked_select(logit, mask).view(-1,vocab_size)),dim=-1)
comb_pred_ids = torch.argmax(voice_prob, dim=-1)
decoded_results.append(processor.decode(comb_pred_ids))
return decoded_results
def predict_lang_specific(data,lang_code):
features = processor(data["speech"], sampling_rate=data["sampling_rate"], padding=True, return_tensors="pt")
input_values = features.input_values.to(device)
attention_mask = features.attention_mask.to(device)
with torch.no_grad():
logits = model(input_values, attention_mask=attention_mask).logits
decoded_results = []
for logit in logits:
pred_ids = torch.argmax(logit, dim=-1)
mask = ~pred_ids.eq(processor.tokenizer.pad_token_id).unsqueeze(-1).expand(logit.size())
vocab_size = logit.size()[-1]
voice_prob = torch.nn.functional.softmax((torch.masked_select(logit, mask).view(-1,vocab_size)),dim=-1)
filtered_input = pred_ids[pred_ids!=processor.tokenizer.pad_token_id].view(1,-1).to(device)
if len(filtered_input[0]) == 0:
decoded_results.append("")
else:
lang_mask = torch.empty(voice_prob.shape[-1]).fill_(0)
lang_index = torch.tensor(sorted(lang_ids[lang_code]))
lang_mask.index_fill_(0, lang_index, 1)
lang_mask = lang_mask.to(device)
comb_pred_ids = torch.argmax(lang_mask*voice_prob, dim=-1)
decoded_results.append(processor.decode(comb_pred_ids))
return decoded_results
predict(load_file_to_data('audio file path',sampling_rate=16_000)) # beware of the audio file sampling rate
predict_lang_specific(load_file_to_data('audio file path',sampling_rate=16_000),'en') # beware of the audio file sampling rate
```
```python
{{ get_started_code | default("More information needed", true)}}
```