tags: pytorch, jax, wav2vec2, automatic-speech-recognition, hsb, dataset:common_voice, transformers, audio, speech, xlsr-fine-tuning-week, license:apache-2.0, model-index


language:


- mr
license: apache-2.0
tags:
- generated_from_trainer
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_8_0
metrics:
- wer
model-index:
- name: wav2vec2-large-xls-r-300m-mr
results:
  - task:
type: automatic-speech-recognition
name: Speech Recognition
dataset:
type: mozilla-foundation/common_voice_8_0
name: Common Voice 8
args: mr
metrics:
    - type: wer    # Required. Example: wer
value: 32.811  # Required. Example: 20.90
name: Test WER    # Optional. Example: Test WER
    - name: Test CER
type: cer
value: 7.692


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# wav2vec2-large-xls-r-300m-mr


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the common_voice dataset.
It achieves the following results on the evaluation set:


- Loss: 0.5479
- Wer: 0.5740


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0001
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 200


### Training results


| Training Loss | Epoch  | Step | Validation Loss | Wer    |
|:-------------:|:------:|:----:|:---------------:|:------:|
| 3.7378        | 18.18  | 400  | 3.5047          | 1.0    |
| 3.1707        | 36.36  | 800  | 2.6166          | 0.9912 |
| 1.4942        | 54.55  | 1200 | 0.5778          | 0.6927 |
| 1.2058        | 72.73  | 1600 | 0.5168          | 0.6362 |
| 1.0558        | 90.91  | 2000 | 0.5105          | 0.6069 |
| 0.9488        | 109.09 | 2400 | 0.5151          | 0.6089 |
| 0.8588        | 127.27 | 2800 | 0.5157          | 0.5989 |
| 0.7991        | 145.45 | 3200 | 0.5179          | 0.5740 |
| 0.7545        | 163.64 | 3600 | 0.5348          | 0.5740 |
| 0.7144        | 181.82 | 4000 | 0.5518          | 0.5724 |
| 0.7041        | 200.0  | 4400 | 0.5479          | 0.5740 |


### Framework versions


- Transformers 4.16.0
- Pytorch 1.10.0+cu111
- Datasets 1.18.1
- Tokenizers 0.11.0


#### Evaluation Commands


1. To evaluate on `mozilla-foundation/common_voice_8_0` with split `test`


```bash
python eval.py --model_id anuragshas/wav2vec2-large-xls-r-300m-mr --dataset mozilla-foundation/common_voice_8_0 --config mr --split test
```


### Inference With LM


```python
import torch
from datasets import load_dataset
from transformers import AutoModelForCTC, AutoProcessor
import torchaudio.functional as F
model_id = "anuragshas/wav2vec2-large-xls-r-300m-mr"
sample_iter = iter(load_dataset("mozilla-foundation/common_voice_8_0", "mr", split="test", streaming=True, use_auth_token=True))
sample = next(sample_iter)
resampled_audio = F.resample(torch.tensor(sample["audio"]["array"]), 48_000, 16_000).numpy()
model = AutoModelForCTC.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id)
input_values = processor(resampled_audio, return_tensors="pt").input_values
with torch.no_grad():
    logits = model(input_values).logits
transcription = processor.batch_decode(logits.numpy()).text
# => "या पानास लेखाचे स्वरूप यायला हावे"
```


### Eval results on Common Voice 8 "test" (WER):


| Without LM | With LM (run `./eval.py`) |
|---|---|
| 49.177 | 32.811 |


model name: wav2vec2-large-xls-r-300m-mr


model author: anuragshas


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, mr, dataset:mozilla-foundation/common_voice_8_0, transformers, license:apache-2.0, generated_from_trainer, robust-speech-event, model-index


language: or
datasets:


- common_voice
metrics:
- wer
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
model-index:
- name: Anurag Singh XLSR Wav2Vec2 Large 53 Odia
results:
  - task:
name: Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice or
type: common_voice
args: or
metrics:
    - name: Test WER
type: wer
value: 57.10


# Wav2Vec2-Large-XLSR-53-Odia


Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on Odia using the [Common Voice](https://huggingface.co/datasets/common_voice).
When using this model, make sure that your speech input is sampled at 16kHz.


## Usage


The model can be used directly (without a language model) as follows:


```python
import torch
import torchaudio
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
test_dataset = load_dataset("common_voice", "or", split="test[:2%]")
processor = Wav2Vec2Processor.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-odia")
model = Wav2Vec2ForCTC.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-odia")
resampler = torchaudio.transforms.Resample(48_000, 16_000)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch
test_dataset = test_dataset.map(speech_file_to_array_fn)
inputs = processor(test_dataset["speech"][:2], sampling_rate=16_000, return_tensors="pt", padding=True)
with torch.no_grad():
    logits = model(inputs.input_values, attention_mask=inputs.attention_mask).logits
predicted_ids = torch.argmax(logits, dim=-1)
print("Prediction:", processor.batch_decode(predicted_ids))
print("Reference:", test_dataset["sentence"][:2])
```


## Evaluation


The model can be evaluated as follows on the Odia test data of Common Voice.


```python
import torch
import torchaudio
from datasets import load_dataset, load_metric
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import re
test_dataset = load_dataset("common_voice", "or", split="test")
wer = load_metric("wer")
processor = Wav2Vec2Processor.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-odia")
model = Wav2Vec2ForCTC.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-odia")
model.to("cuda")
chars_to_ignore_regex = '[\,\?\.\!\-\;\:\"\“]'
resampler = torchaudio.transforms.Resample(48_000, 16_000)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch
test_dataset = test_dataset.map(speech_file_to_array_fn)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def evaluate(batch):
    inputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)
    with torch.no_grad():
        logits = model(inputs.input_values.to("cuda"), attention_mask=inputs.attention_mask.to("cuda")).logits
    pred_ids = torch.argmax(logits, dim=-1)
    batch["pred_strings"] = processor.batch_decode(pred_ids)
    return batch
result = test_dataset.map(evaluate, batched=True, batch_size=8)
print("WER: {:2f}".format(100 * wer.compute(predictions=result["pred_strings"], references=result["sentence"])))
```


**Test Result**: 57.10 %


## Training


The Common Voice `train` and `validation` datasets were used for training.


model name: wav2vec2-large-xlsr-53-odia


model author: anuragshas


task: automatic-speech-recognition


tags: pytorch, jax, wav2vec2, automatic-speech-recognition, or, dataset:common_voice, transformers, audio, speech, xlsr-fine-tuning-week, license:apache-2.0, model-index


language:


- mr
license: apache-2.0
tags:
- automatic-speech-recognition
- mozilla-foundation/common_voice_8_0
- generated_from_trainer
datasets:
- common_voice
model-index:
- name: ''
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the MOZILLA-FOUNDATION/COMMON_VOICE_8_0 - MR dataset.
It achieves the following results on the evaluation set:


- Loss: 0.6693
- Wer: 0.5921


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 7.5e-05
- train_batch_size: 32
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 500.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch  | Step  | Validation Loss | Wer    |
|:-------------:|:------:|:-----:|:---------------:|:------:|
| 4.9504        | 18.18  | 400   | 4.6730          | 1.0    |
| 3.3766        | 36.36  | 800   | 3.3464          | 1.0    |
| 3.1128        | 54.55  | 1200  | 3.0177          | 0.9980 |
| 1.7966        | 72.73  | 1600  | 0.8733          | 0.8039 |
| 1.4085        | 90.91  | 2000  | 0.5555          | 0.6458 |
| 1.1731        | 109.09 | 2400  | 0.4930          | 0.6438 |
| 1.0271        | 127.27 | 2800  | 0.4780          | 0.6093 |
| 0.9045        | 145.45 | 3200  | 0.4647          | 0.6578 |
| 0.807         | 163.64 | 3600  | 0.4505          | 0.5925 |
| 0.741         | 181.82 | 4000  | 0.4746          | 0.6025 |
| 0.6706        | 200.0  | 4400  | 0.5004          | 0.5844 |
| 0.6186        | 218.18 | 4800  | 0.4984          | 0.5997 |
| 0.5508        | 236.36 | 5200  | 0.5298          | 0.5636 |
| 0.5123        | 254.55 | 5600  | 0.5410          | 0.5110 |
| 0.4623        | 272.73 | 6000  | 0.5591          | 0.5383 |
| 0.4281        | 290.91 | 6400  | 0.5775          | 0.5600 |
| 0.4045        | 309.09 | 6800  | 0.5924          | 0.5580 |
| 0.3651        | 327.27 | 7200  | 0.5671          | 0.5684 |
| 0.343         | 345.45 | 7600  | 0.6083          | 0.5945 |
| 0.3085        | 363.64 | 8000  | 0.6243          | 0.5728 |
| 0.2941        | 381.82 | 8400  | 0.6245          | 0.5580 |
| 0.2735        | 400.0  | 8800  | 0.6458          | 0.5804 |
| 0.262         | 418.18 | 9200  | 0.6566          | 0.5824 |
| 0.2578        | 436.36 | 9600  | 0.6558          | 0.5965 |
| 0.2388        | 454.55 | 10000 | 0.6598          | 0.5993 |
| 0.2328        | 472.73 | 10400 | 0.6700          | 0.6041 |
| 0.2286        | 490.91 | 10800 | 0.6684          | 0.5957 |


### Framework versions


- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.4.dev0
- Tokenizers 0.11.0


model name: wav2vec2-xls-r-300m-mr-cv8-with-lm


model author: anuragshas


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, mr, dataset:common_voice, transformers, license:apache-2.0, mozilla-foundation/common_voice_8_0, generated_from_trainer, model-index


language:


- ha
license: apache-2.0
tags:
- generated_from_trainer
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_8_0
metrics:
- wer
model-index:
- name: XLS-R-300M - Hausa
results:
  - task:
type: automatic-speech-recognition
name: Speech Recognition
dataset:
type: mozilla-foundation/common_voice_8_0
name: Common Voice 8
args: ha
metrics:
    - type: wer    # Required. Example: wer
value: 36.295  # Required. Example: 20.90
name: Test WER    # Optional. Example: Test WER
    - name: Test CER
type: cer
value: 11.073


# XLS-R-300M - Hausa


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the common_voice dataset.
It achieves the following results on the evaluation set:


- Loss: 0.6094
- Wer: 0.5234


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 0.0001
- train_batch_size: 16
- eval_batch_size: 8
- seed: 13
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine_with_restarts
- lr_scheduler_warmup_steps: 1000
- num_epochs: 100


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 2.9599        | 6.56  | 400  | 2.8650          | 1.0    |
| 2.7357        | 13.11 | 800  | 2.7377          | 0.9951 |
| 1.3012        | 19.67 | 1200 | 0.6686          | 0.7111 |
| 1.0454        | 26.23 | 1600 | 0.5686          | 0.6137 |
| 0.9069        | 32.79 | 2000 | 0.5576          | 0.5815 |
| 0.82          | 39.34 | 2400 | 0.5502          | 0.5591 |
| 0.7413        | 45.9  | 2800 | 0.5970          | 0.5586 |
| 0.6872        | 52.46 | 3200 | 0.5817          | 0.5428 |
| 0.634         | 59.02 | 3600 | 0.5636          | 0.5314 |
| 0.6022        | 65.57 | 4000 | 0.5780          | 0.5229 |
| 0.5705        | 72.13 | 4400 | 0.6036          | 0.5323 |
| 0.5408        | 78.69 | 4800 | 0.6119          | 0.5336 |
| 0.5225        | 85.25 | 5200 | 0.6105          | 0.5270 |
| 0.5265        | 91.8  | 5600 | 0.6034          | 0.5231 |
| 0.5154        | 98.36 | 6000 | 0.6094          | 0.5234 |


### Framework versions


- Transformers 4.16.1
- Pytorch 1.10.0+cu111
- Datasets 1.18.2
- Tokenizers 0.11.0


#### Evaluation Commands


1. To evaluate on `mozilla-foundation/common_voice_8_0` with split `test`


```bash
python eval.py --model_id anuragshas/wav2vec2-large-xls-r-300m-ha-cv8 --dataset mozilla-foundation/common_voice_8_0 --config ha --split test
```


### Inference With LM


```python
import torch
from datasets import load_dataset
from transformers import AutoModelForCTC, AutoProcessor
import torchaudio.functional as F
model_id = "anuragshas/wav2vec2-large-xls-r-300m-ha-cv8"
sample_iter = iter(load_dataset("mozilla-foundation/common_voice_8_0", "ha", split="test", streaming=True, use_auth_token=True))
sample = next(sample_iter)
resampled_audio = F.resample(torch.tensor(sample["audio"]["array"]), 48_000, 16_000).numpy()
model = AutoModelForCTC.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id)
input_values = processor(resampled_audio, return_tensors="pt").input_values
with torch.no_grad():
    logits = model(input_values).logits
transcription = processor.batch_decode(logits.numpy()).text
# => "kakin hade ya ke da kyautar"
```


### Eval results on Common Voice 8 "test" (WER):


| Without LM | With LM (run `./eval.py`) |
|---|---|
| 47.821 | 36.295 |


model name: wav2vec2-large-xls-r-300m-ha-cv8


model author: anuragshas


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, ha, dataset:mozilla-foundation/common_voice_8_0, transformers, license:apache-2.0, generated_from_trainer, robust-speech-event, model-index


language: rm-sursilv
datasets:


- common_voice
metrics:
- wer
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
model-index:
- name: Anurag Singh XLSR Wav2Vec2 Large 53 Romansh Sursilv
results:
  - task:
name: Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice rm-sursilv
type: common_voice
args: rm-sursilv
metrics:
    - name: Test WER
type: wer
value: 25.78


# Wav2Vec2-Large-XLSR-53-Romansh Sursilv


Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on Romansh Sursilv using the [Common Voice](https://huggingface.co/datasets/common_voice).
When using this model, make sure that your speech input is sampled at 16kHz.


## Usage


The model can be used directly (without a language model) as follows:


```python
import torch
import torchaudio
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
test_dataset = load_dataset("common_voice", "rm-sursilv", split="test[:2%]")
processor = Wav2Vec2Processor.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-rm-sursilv")
model = Wav2Vec2ForCTC.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-rm-sursilv")
resampler = torchaudio.transforms.Resample(48_000, 16_000)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch
test_dataset = test_dataset.map(speech_file_to_array_fn)
inputs = processor(test_dataset["speech"][:2], sampling_rate=16_000, return_tensors="pt", padding=True)
with torch.no_grad():
    logits = model(inputs.input_values, attention_mask=inputs.attention_mask).logits
predicted_ids = torch.argmax(logits, dim=-1)
print("Prediction:", processor.batch_decode(predicted_ids))
print("Reference:", test_dataset["sentence"][:2])
```


## Evaluation


The model can be evaluated as follows on the Romansh Sursilv test data of Common Voice.


```python
import torch
import torchaudio
from datasets import load_dataset, load_metric
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import re
test_dataset = load_dataset("common_voice", "rm-sursilv", split="test")
wer = load_metric("wer")
processor = Wav2Vec2Processor.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-rm-sursilv")
model = Wav2Vec2ForCTC.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-rm-sursilv")
model.to("cuda")
chars_to_ignore_regex = '[\,\?\.\!\-\;\:\"\“\%\”\„\–\…\«\»]'
resampler = torchaudio.transforms.Resample(48_000, 16_000)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch
test_dataset = test_dataset.map(speech_file_to_array_fn)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def evaluate(batch):
    inputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)
    with torch.no_grad():
        logits = model(inputs.input_values.to("cuda"), attention_mask=inputs.attention_mask.to("cuda")).logits
    pred_ids = torch.argmax(logits, dim=-1)
    batch["pred_strings"] = processor.batch_decode(pred_ids)
    return batch
result = test_dataset.map(evaluate, batched=True, batch_size=8)
print("WER: {:2f}".format(100 * wer.compute(predictions=result["pred_strings"], references=result["sentence"])))
```


**Test Result**: 25.78 %


## Training


The Common Voice `train` and `validation` datasets were used for training.


model name: wav2vec2-large-xlsr-53-rm-sursilv


model author: anuragshas


task: automatic-speech-recognition


tags: pytorch, jax, wav2vec2, automatic-speech-recognition, rm-sursilv, dataset:common_voice, transformers, audio, speech, xlsr-fine-tuning-week, license:apache-2.0, model-index


language: te
datasets:


- openslr
metrics:
- wer
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
model-index:
- name: Anurag Singh XLSR Wav2Vec2 Large 53 Telugu
results:
  - task:
name: Speech Recognition
type: automatic-speech-recognition
dataset:
name: OpenSLR te
type: openslr
args: te
metrics:
    - name: Test WER
type: wer
value: 44.98


# Wav2Vec2-Large-XLSR-53-Telugu


Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on Telugu using the [OpenSLR SLR66](http://openslr.org/66/) dataset.
When using this model, make sure that your speech input is sampled at 16kHz.


## Usage


```python
import torch
import torchaudio
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import pandas as pd
# Evaluation notebook contains the procedure to download the data
df = pd.read_csv("/content/te/test.tsv", sep="\t")
df["path"] = "/content/te/clips/" + df["path"]
test_dataset = Dataset.from_pandas(df)
processor = Wav2Vec2Processor.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-telugu")
model = Wav2Vec2ForCTC.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-telugu")
resampler = torchaudio.transforms.Resample(48_000, 16_000)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch
test_dataset = test_dataset.map(speech_file_to_array_fn)
inputs = processor(test_dataset["speech"][:2], sampling_rate=16_000, return_tensors="pt", padding=True)
with torch.no_grad():
    logits = model(inputs.input_values, attention_mask=inputs.attention_mask).logits
predicted_ids = torch.argmax(logits, dim=-1)
print("Prediction:", processor.batch_decode(predicted_ids))
print("Reference:", test_dataset["sentence"][:2])
```


```python
import torch
import torchaudio
from datasets import Dataset, load_metric
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import re
from sklearn.model_selection import train_test_split
import pandas as pd
# Evaluation notebook contains the procedure to download the data
df = pd.read_csv("/content/te/test.tsv", sep="\t")
df["path"] = "/content/te/clips/" + df["path"]
test_dataset = Dataset.from_pandas(df)
wer = load_metric("wer")
processor = Wav2Vec2Processor.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-telugu")
model = Wav2Vec2ForCTC.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-telugu")
model.to("cuda")
chars_to_ignore_regex = '[\,\?\.\!\-\_\;\:\"\“\%\‘\”\।\’\'\&]'
resampler = torchaudio.transforms.Resample(48_000, 16_000)
def normalizer(text):
    # Use your custom normalizer
    text = text.replace("\\n","\n")
    text = ' '.join(text.split())
    text = re.sub(r'''([a-z]+)''','',text,flags=re.IGNORECASE)
    text = re.sub(r'''%'''," శాతం ", text)
    text = re.sub(r'''(/|-|_)'''," ", text)
    text = re.sub("ై","ై", text)
    text = text.strip()
    return text
def speech_file_to_array_fn(batch):
    batch["sentence"] = normalizer(batch["sentence"])
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()+ " "
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch
test_dataset = test_dataset.map(speech_file_to_array_fn)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def evaluate(batch):
    inputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)
    with torch.no_grad():
        logits = model(inputs.input_values.to("cuda"), attention_mask=inputs.attention_mask.to("cuda")).logits
    pred_ids = torch.argmax(logits, dim=-1)
    batch["pred_strings"] = processor.batch_decode(pred_ids)
    return batch
result = test_dataset.map(evaluate, batched=True, batch_size=8)
print("WER: {:2f}".format(100 * wer.compute(predictions=result["pred_strings"], references=result["sentence"])))
```


**Test Result**: 44.98%


70% of the OpenSLR Telugu dataset was used for training.


Train Split of annotations is [here](https://www.dropbox.com/s/xqc0wtour7f9h4c/train.tsv)


Test Split of annotations is [here](https://www.dropbox.com/s/qw1uy63oj4qdiu4/test.tsv)


Training Data Preparation notebook can be found [here](https://colab.research.google.com/drive/1_VR1QtY9qoiabyXBdJcOI29-xIKGdIzU?usp=sharing)


Training notebook can be found[here](https://colab.research.google.com/drive/14N-j4m0Ng_oktPEBN5wiUhDDbyrKYt8I?usp=sharing)


Evaluation notebook is [here](https://colab.research.google.com/drive/1SLEvbTWBwecIRTNqpQ0fFTqmr1-7MnSI?usp=sharing)


model name: wav2vec2-large-xlsr-53-telugu


model author: anuragshas


task: automatic-speech-recognition


tags: pytorch, jax, wav2vec2, automatic-speech-recognition, te, dataset:openslr, transformers, audio, speech, xlsr-fine-tuning-week, license:apache-2.0, model-index


language:


- pa
- pa-IN
license: apache-2.0
tags:
- generated_from_trainer
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_7_0
metrics:
- wer
model-index:
- name: XLS-R-300M - Punjabi
results:
  - task:
type: automatic-speech-recognition
name: Speech Recognition
dataset:
type: mozilla-foundation/common_voice_7_0
name: Common Voice 7
args: pa-IN
metrics:
    - type: wer    # Required. Example: wer
value: 45.611  # Required. Example: 20.90
name: Test WER    # Optional. Example: Test WER
    - name: Test CER
type: cer
value: 15.584


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# XLS-R-300M - Punjabi


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the common_voice dataset.
It achieves the following results on the evaluation set:


- Loss: 1.2548
- Wer: 0.5677


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0003
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.12
- num_epochs: 120
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch  | Step | Validation Loss | Wer    |
|:-------------:|:------:|:----:|:---------------:|:------:|
| 6.4804        | 16.65  | 400  | 1.8461          | 1.0    |
| 0.474         | 33.33  | 800  | 1.1018          | 0.6624 |
| 0.1389        | 49.98  | 1200 | 1.1918          | 0.6103 |
| 0.0919        | 66.65  | 1600 | 1.1889          | 0.6058 |
| 0.0657        | 83.33  | 2000 | 1.2266          | 0.5931 |
| 0.0479        | 99.98  | 2400 | 1.2512          | 0.5902 |
| 0.0355        | 116.65 | 2800 | 1.2548          | 0.5677 |


### Framework versions


- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.18.0
- Tokenizers 0.10.3


#### Evaluation Commands


1. To evaluate on `mozilla-foundation/common_voice_7_0` with split `test`


```bash
python eval.py --model_id anuragshas/wav2vec2-large-xls-r-300m-pa-in --dataset mozilla-foundation/common_voice_7_0 --config pa-IN --split test
```


### Inference With LM


```python
import torch
from datasets import load_dataset
from transformers import AutoModelForCTC, AutoProcessor
import torchaudio.functional as F
model_id = "anuragshas/wav2vec2-large-xls-r-300m-pa-in"
sample_iter = iter(load_dataset("mozilla-foundation/common_voice_7_0", "pa-IN", split="test", streaming=True, use_auth_token=True))
sample = next(sample_iter)
resampled_audio = F.resample(torch.tensor(sample["audio"]["array"]), 48_000, 16_000).numpy()
model = AutoModelForCTC.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id)
input_values = processor(resampled_audio, return_tensors="pt").input_values
with torch.no_grad():
    logits = model(input_values).logits
transcription = processor.batch_decode(logits.numpy()).text
# => "ਉਨ੍ਹਾਂ ਨੇ ਸਾਰੇ ਤੇਅਰਵੇ ਵੱਖਰੀ ਕਿਸਮ ਦੇ ਕੀਤੇ ਹਨ"
```


### Eval results on Common Voice 7 "test" (WER):


| Without LM | With LM (run `./eval.py`) |
|---|---|
| 51.968 | 45.611 |


model name: wav2vec2-large-xls-r-300m-pa-in


model author: anuragshas


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, pa, pa-IN, dataset:mozilla-foundation/common_voice_7_0, transformers, license:apache-2.0, generated_from_trainer, robust-speech-event, model-index


language:


- sl
license: apache-2.0
tags:
- automatic-speech-recognition
- mozilla-foundation/common_voice_8_0
- generated_from_trainer
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_8_0
model-index:
- name: XLS-R-300M - Slovenian
results:
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice 8
type: mozilla-foundation/common_voice_8_0
args: sl
metrics:
    - name: Test WER
type: wer
value: 12.736
    - name: Test CER
type: cer
value: 3.605
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Robust Speech Event - Dev Data
type: speech-recognition-community-v2/dev_data
args: sl
metrics:
    - name: Test WER
type: wer
value: 45.587
    - name: Test CER
type: cer
value: 20.886


# XLS-R-300M - Slovenian


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the MOZILLA-FOUNDATION/COMMON_VOICE_8_0 - SL dataset.
It achieves the following results on the evaluation set:


- Loss: 0.2578
- Wer: 0.2273


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 7.5e-05
- train_batch_size: 32
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 60.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 3.1829        | 4.88  | 400  | 3.1228          | 1.0    |
| 2.8675        | 9.76  | 800  | 2.8616          | 0.9993 |
| 1.583         | 14.63 | 1200 | 0.6392          | 0.6239 |
| 1.1959        | 19.51 | 1600 | 0.3602          | 0.3651 |
| 1.0276        | 24.39 | 2000 | 0.3021          | 0.2981 |
| 0.9671        | 29.27 | 2400 | 0.2872          | 0.2739 |
| 0.873         | 34.15 | 2800 | 0.2593          | 0.2459 |
| 0.8513        | 39.02 | 3200 | 0.2617          | 0.2473 |
| 0.8132        | 43.9  | 3600 | 0.2548          | 0.2426 |
| 0.7935        | 48.78 | 4000 | 0.2637          | 0.2353 |
| 0.7565        | 53.66 | 4400 | 0.2629          | 0.2322 |
| 0.7359        | 58.54 | 4800 | 0.2579          | 0.2253 |


### Framework versions


- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.11.0


#### Evaluation Commands


1. To evaluate on `mozilla-foundation/common_voice_8_0` with split `test`


```bash
python eval.py --model_id anuragshas/wav2vec2-xls-r-300m-sl-cv8-with-lm --dataset mozilla-foundation/common_voice_8_0 --config sl --split test
```


2. To evaluate on `speech-recognition-community-v2/dev_data`


```bash
python eval.py --model_id anuragshas/wav2vec2-xls-r-300m-sl-cv8-with-lm --dataset speech-recognition-community-v2/dev_data --config sl --split validation --chunk_length_s 5.0 --stride_length_s 1.0
```


### Inference With LM


```python
import torch
from datasets import load_dataset
from transformers import AutoModelForCTC, AutoProcessor
import torchaudio.functional as F
model_id = "anuragshas/wav2vec2-xls-r-300m-sl-cv8-with-lm"
sample_iter = iter(load_dataset("mozilla-foundation/common_voice_8_0", "sl", split="test", streaming=True, use_auth_token=True))
sample = next(sample_iter)
resampled_audio = F.resample(torch.tensor(sample["audio"]["array"]), 48_000, 16_000).numpy()
model = AutoModelForCTC.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id)
input_values = processor(resampled_audio, return_tensors="pt").input_values
with torch.no_grad():
    logits = model(input_values).logits
transcription = processor.batch_decode(logits.numpy()).text
# => "zmago je divje od letel s helikopterjem visoko vzrak"
```


### Eval results on Common Voice 8 "test" (WER):


| Without LM | With LM (run `./eval.py`) |
|---|---|
| 19.938 | 12.736 |


model name: wav2vec2-xls-r-300m-sl-cv8-with-lm


model author: anuragshas


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, sl, dataset:mozilla-foundation/common_voice_8_0, transformers, license:apache-2.0, mozilla-foundation/common_voice_8_0, generated_from_trainer, robust-speech-event, model-index


language: dv
datasets:


- common_voice
metrics:
- wer
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
model-index:
- name: Anurag Singh XLSR Wav2Vec2 Large 53 Dhivehi
results:
  - task:
name: Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice dv
type: common_voice
args: dv
metrics:
    - name: Test WER
type: wer
value: 55.68


# Wav2Vec2-Large-XLSR-53-Dhivehi


Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on Dhivehi using the [Common Voice](https://huggingface.co/datasets/common_voice).
When using this model, make sure that your speech input is sampled at 16kHz.


## Usage


The model can be used directly (without a language model) as follows:


```python
import torch
import torchaudio
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
test_dataset = load_dataset("common_voice", "dv", split="test[:2%]")
processor = Wav2Vec2Processor.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-dv")
model = Wav2Vec2ForCTC.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-dv")
resampler = torchaudio.transforms.Resample(48_000, 16_000)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch
test_dataset = test_dataset.map(speech_file_to_array_fn)
inputs = processor(test_dataset["speech"][:2], sampling_rate=16_000, return_tensors="pt", padding=True)
with torch.no_grad():
    logits = model(inputs.input_values, attention_mask=inputs.attention_mask).logits
predicted_ids = torch.argmax(logits, dim=-1)
print("Prediction:", processor.batch_decode(predicted_ids))
print("Reference:", test_dataset["sentence"][:2])
```


## Evaluation


The model can be evaluated as follows on the Dhivehi test data of Common Voice.


```python
import torch
import torchaudio
from datasets import load_dataset, load_metric
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import re
test_dataset = load_dataset("common_voice", "dv", split="test")
wer = load_metric("wer")
processor = Wav2Vec2Processor.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-dv")
model = Wav2Vec2ForCTC.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-dv")
model.to("cuda")
chars_to_ignore_regex = '[\,\?\.\!\-\;\:\"\“\%\‘\”\،\.\؟\–\'\’]'
resampler = torchaudio.transforms.Resample(48_000, 16_000)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch
test_dataset = test_dataset.map(speech_file_to_array_fn)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def evaluate(batch):
    inputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)
    with torch.no_grad():
        logits = model(inputs.input_values.to("cuda"), attention_mask=inputs.attention_mask.to("cuda")).logits
    pred_ids = torch.argmax(logits, dim=-1)
    batch["pred_strings"] = processor.batch_decode(pred_ids)
    return batch
result = test_dataset.map(evaluate, batched=True, batch_size=8)
print("WER: {:2f}".format(100 * wer.compute(predictions=result["pred_strings"], references=result["sentence"])))
```


**Test Result**: 55.68 %


## Training


The Common Voice `train` and `validation` datasets were used for training.


model name: wav2vec2-large-xlsr-53-dv


model author: anuragshas


task: automatic-speech-recognition


tags: pytorch, jax, wav2vec2, automatic-speech-recognition, dv, dataset:common_voice, transformers, audio, speech, xlsr-fine-tuning-week, license:apache-2.0, model-index


language:


- pa-IN
license: apache-2.0
tags:
- automatic-speech-recognition
- mozilla-foundation/common_voice_8_0
- generated_from_trainer
datasets:
- common_voice
model-index:
- name: ''
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the MOZILLA-FOUNDATION/COMMON_VOICE_8_0 - PA-IN dataset.
It achieves the following results on the evaluation set:


- Loss: 0.6864
- Wer: 0.6707


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 7.5e-05
- train_batch_size: 32
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 200.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch  | Step | Validation Loss | Wer    |
|:-------------:|:------:|:----:|:---------------:|:------:|
| 4.3322        | 14.81  | 400  | 3.7450          | 1.0    |
| 3.2662        | 29.63  | 800  | 3.2571          | 0.9996 |
| 1.6408        | 44.44  | 1200 | 0.9098          | 0.8162 |
| 1.2289        | 59.26  | 1600 | 0.6757          | 0.7099 |
| 1.0551        | 74.07  | 2000 | 0.6417          | 0.7044 |
| 0.966         | 88.89  | 2400 | 0.6365          | 0.6789 |
| 0.8713        | 103.7  | 2800 | 0.6617          | 0.6954 |
| 0.8055        | 118.52 | 3200 | 0.6371          | 0.6762 |
| 0.7489        | 133.33 | 3600 | 0.6798          | 0.6911 |
| 0.7073        | 148.15 | 4000 | 0.6567          | 0.6731 |
| 0.6609        | 162.96 | 4400 | 0.6742          | 0.6840 |
| 0.6435        | 177.78 | 4800 | 0.6862          | 0.6633 |
| 0.6282        | 192.59 | 5200 | 0.6865          | 0.6731 |


### Framework versions


- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.4.dev0
- Tokenizers 0.11.0


model name: wav2vec2-xls-r-300m-pa-IN-cv8-with-lm


model author: anuragshas


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, pa-IN, dataset:common_voice, transformers, license:apache-2.0, mozilla-foundation/common_voice_8_0, generated_from_trainer, model-index


language:


- hi
license: apache-2.0
tags:
- automatic-speech-recognition
- mozilla-foundation/common_voice_8_0
- generated_from_trainer
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_8_0
metrics:
- wer
model-index:
- name: XLS-R-1B - Hindi
results:
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice 8
type: mozilla-foundation/common_voice_8_0
args: hi
metrics:
    - name: Test WER
type: wer
value: 15.899
    - name: Test CER
type: cer
value: 5.830


# XLS-R-1B - Hindi


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-1b](https://huggingface.co/facebook/wav2vec2-xls-r-1b) on the MOZILLA-FOUNDATION/COMMON_VOICE_8_0 - HI dataset.
It achieves the following results on the evaluation set:


- Loss: 0.6921
- Wer: 0.3547


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 16
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1500
- num_epochs: 50.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 2.0674        | 2.07  | 400  | 1.3411          | 0.8835 |
| 1.324         | 4.15  | 800  | 0.9311          | 0.7142 |
| 1.2023        | 6.22  | 1200 | 0.8060          | 0.6170 |
| 1.1573        | 8.29  | 1600 | 0.7415          | 0.4972 |
| 1.1117        | 10.36 | 2000 | 0.7248          | 0.4588 |
| 1.0672        | 12.44 | 2400 | 0.6729          | 0.4350 |
| 1.0336        | 14.51 | 2800 | 0.7117          | 0.4346 |
| 1.0025        | 16.58 | 3200 | 0.7019          | 0.4272 |
| 0.9578        | 18.65 | 3600 | 0.6792          | 0.4118 |
| 0.9272        | 20.73 | 4000 | 0.6863          | 0.4156 |
| 0.9321        | 22.8  | 4400 | 0.6535          | 0.3972 |
| 0.8802        | 24.87 | 4800 | 0.6766          | 0.3906 |
| 0.844         | 26.94 | 5200 | 0.6782          | 0.3949 |
| 0.8387        | 29.02 | 5600 | 0.6916          | 0.3921 |
| 0.8042        | 31.09 | 6000 | 0.6806          | 0.3797 |
| 0.793         | 33.16 | 6400 | 0.7120          | 0.3831 |
| 0.7567        | 35.23 | 6800 | 0.6862          | 0.3808 |
| 0.7463        | 37.31 | 7200 | 0.6893          | 0.3709 |
| 0.7053        | 39.38 | 7600 | 0.7096          | 0.3701 |
| 0.6906        | 41.45 | 8000 | 0.6921          | 0.3676 |
| 0.6891        | 43.52 | 8400 | 0.7167          | 0.3663 |
| 0.658         | 45.6  | 8800 | 0.6833          | 0.3580 |
| 0.6576        | 47.67 | 9200 | 0.6914          | 0.3569 |
| 0.6358        | 49.74 | 9600 | 0.6922          | 0.3551 |


### Framework versions


- Transformers 4.16.0.dev0
- Pytorch 1.10.1+cu102
- Datasets 1.17.1.dev0
- Tokenizers 0.11.0


#### Evaluation Commands


1. To evaluate on `mozilla-foundation/common_voice_8_0` with split `test`


```bash
python eval.py --model_id anuragshas/wav2vec2-xls-r-1b-hi-with-lm --dataset mozilla-foundation/common_voice_8_0 --config hi --split test
```


### Inference With LM


```python
import torch
from datasets import load_dataset
from transformers import AutoModelForCTC, AutoProcessor
import torchaudio.functional as F
model_id = "anuragshas/wav2vec2-xls-r-1b-hi-with-lm"
sample_iter = iter(load_dataset("mozilla-foundation/common_voice_8_0", "hi", split="test", streaming=True, use_auth_token=True))
sample = next(sample_iter)
resampled_audio = F.resample(torch.tensor(sample["audio"]["array"]), 48_000, 16_000).numpy()
model = AutoModelForCTC.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id)
input_values = processor(resampled_audio, return_tensors="pt").input_values
with torch.no_grad():
    logits = model(input_values).logits
transcription = processor.batch_decode(logits.numpy()).text
# => "तुम्हारे पास तीन महीने बचे हैं"
```


### Eval results on Common Voice 8 "test" (WER):


| Without LM | With LM (run `./eval.py`) |
|---|---|
| 26.209 | 15.899 |


model name: wav2vec2-xls-r-1b-hi-with-lm


model author: anuragshas


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, hi, dataset:mozilla-foundation/common_voice_8_0, transformers, license:apache-2.0, mozilla-foundation/common_voice_8_0, generated_from_trainer, robust-speech-event, model-index


language: as
datasets:


- common_voice
metrics:
- wer
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
model-index:
- name: Anurag Singh XLSR Wav2Vec2 Large 53 Assamese
results:
  - task:
name: Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice as
type: common_voice
args: as
metrics:
    - name: Test WER
type: wer
value: 69.63


# Wav2Vec2-Large-XLSR-53-Assamese


Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on Assamese using the [Common Voice](https://huggingface.co/datasets/common_voice).
When using this model, make sure that your speech input is sampled at 16kHz.


## Usage


The model can be used directly (without a language model) as follows:


```python
import torch
import torchaudio
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
test_dataset = load_dataset("common_voice", "as", split="test[:2%]")
processor = Wav2Vec2Processor.from_pretrained("anuragshas/wav2vec2-large-xlsr-as")
model = Wav2Vec2ForCTC.from_pretrained("anuragshas/wav2vec2-large-xlsr-as")
resampler = torchaudio.transforms.Resample(48_000, 16_000)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch
test_dataset = test_dataset.map(speech_file_to_array_fn)
inputs = processor(test_dataset["speech"][:2], sampling_rate=16_000, return_tensors="pt", padding=True)
with torch.no_grad():
    logits = model(inputs.input_values, attention_mask=inputs.attention_mask).logits
predicted_ids = torch.argmax(logits, dim=-1)
print("Prediction:", processor.batch_decode(predicted_ids))
print("Reference:", test_dataset["sentence"][:2])
```


## Evaluation


The model can be evaluated as follows on the Assamese test data of Common Voice.


```python
import torch
import torchaudio
from datasets import load_dataset, load_metric
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import re
test_dataset = load_dataset("common_voice", "as", split="test")
wer = load_metric("wer")
processor = Wav2Vec2Processor.from_pretrained("anuragshas/wav2vec2-large-xlsr-as")
model = Wav2Vec2ForCTC.from_pretrained("anuragshas/wav2vec2-large-xlsr-as")
model.to("cuda")
chars_to_ignore_regex = '[\\,\\?\\.\\!\\-\\;\\:\\"\\“\\%\\”\\়\\।]'
resampler = torchaudio.transforms.Resample(48_000, 16_000)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    batch["sentence"] = re.sub('’ ',' ',batch["sentence"])
    batch["sentence"] = re.sub(' ‘',' ',batch["sentence"])
    batch["sentence"] = re.sub('’|‘','\'',batch["sentence"])
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch
test_dataset = test_dataset.map(speech_file_to_array_fn)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def evaluate(batch):
    inputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)
    with torch.no_grad():
        logits = model(inputs.input_values.to("cuda"), attention_mask=inputs.attention_mask.to("cuda")).logits
    pred_ids = torch.argmax(logits, dim=-1)
    batch["pred_strings"] = processor.batch_decode(pred_ids)
    return batch
result = test_dataset.map(evaluate, batched=True, batch_size=8)
print("WER: {:2f}".format(100 * wer.compute(predictions=result["pred_strings"], references=result["sentence"])))
```


**Test Result**: 69.63 %


## Training


The Common Voice `train` and `validation` datasets were used for training.


model name: wav2vec2-large-xlsr-as


model author: anuragshas


task: automatic-speech-recognition


tags: pytorch, jax, wav2vec2, automatic-speech-recognition, as, dataset:common_voice, transformers, audio, speech, xlsr-fine-tuning-week, license:apache-2.0, model-index


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- common_voice
model-index:
- name: wav2vec2-large-xls-r-300m-hi
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# wav2vec2-large-xls-r-300m-hi


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the common_voice dataset.
It achieves the following results on the evaluation set:


- Loss: 2.4156
- Wer: 0.7181


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0003
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 30


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 5.7703        | 2.72  | 400  | 2.2274          | 0.9259 |
| 0.6515        | 5.44  | 800  | 1.5812          | 0.7581 |
| 0.339         | 8.16  | 1200 | 2.0590          | 0.7825 |
| 0.2262        | 10.88 | 1600 | 2.0324          | 0.7603 |
| 0.1665        | 13.6  | 2000 | 2.1396          | 0.7481 |
| 0.1311        | 16.33 | 2400 | 2.2090          | 0.7379 |
| 0.1079        | 19.05 | 2800 | 2.3907          | 0.7612 |
| 0.0927        | 21.77 | 3200 | 2.5294          | 0.7478 |
| 0.0748        | 24.49 | 3600 | 2.5024          | 0.7452 |
| 0.0644        | 27.21 | 4000 | 2.4715          | 0.7307 |
| 0.0569        | 29.93 | 4400 | 2.4156          | 0.7181 |


### Framework versions


- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


model name: wav2vec2-large-xls-r-300m-hi


model author: anuragshas


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, dataset:common_voice, transformers, license:apache-2.0, generated_from_trainer, model-index


language:


- ur
license: apache-2.0
tags:
- generated_from_trainer
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_8_0
metrics:
- wer
model-index:
- name: wav2vec2-large-xls-r-300m-ur-cv8
results:
  - task:
type: automatic-speech-recognition
name: Speech Recognition
dataset:
type: mozilla-foundation/common_voice_8_0
name: Common Voice 8
args: ur
metrics:
    - type: wer    # Required. Example: wer
value: 42.376  # Required. Example: 20.90
name: Test WER    # Optional. Example: Test WER
    - name: Test CER
type: cer
value: 18.180


# wav2vec2-large-xls-r-300m-ur-cv8


- Loss: 1.1443
- Wer: 0.5677


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 0.0001
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 200


### Training results


| Training Loss | Epoch  | Step | Validation Loss | Wer    |
|:-------------:|:------:|:----:|:---------------:|:------:|
| 3.6269        | 15.98  | 400  | 3.3246          | 1.0    |
| 3.0546        | 31.98  | 800  | 2.8148          | 0.9963 |
| 1.4589        | 47.98  | 1200 | 1.0237          | 0.6584 |
| 1.0911        | 63.98  | 1600 | 0.9524          | 0.5966 |
| 0.8879        | 79.98  | 2000 | 0.9827          | 0.5822 |
| 0.7467        | 95.98  | 2400 | 0.9923          | 0.5840 |
| 0.6427        | 111.98 | 2800 | 0.9988          | 0.5714 |
| 0.5685        | 127.98 | 3200 | 1.0872          | 0.5807 |
| 0.5068        | 143.98 | 3600 | 1.1194          | 0.5822 |
| 0.463         | 159.98 | 4000 | 1.1138          | 0.5692 |
| 0.4212        | 175.98 | 4400 | 1.1232          | 0.5714 |
| 0.4056        | 191.98 | 4800 | 1.1443          | 0.5677 |


### Framework versions


- Transformers 4.16.0
- Pytorch 1.10.0+cu111
- Datasets 1.18.1
- Tokenizers 0.11.0


#### Evaluation Commands


1. To evaluate on `mozilla-foundation/common_voice_8_0` with split `test`


```bash
python eval.py --model_id anuragshas/wav2vec2-large-xls-r-300m-ur-cv8 --dataset mozilla-foundation/common_voice_8_0 --config ur --split test
```


### Inference With LM


```python
import torch
from datasets import load_dataset
from transformers import AutoModelForCTC, AutoProcessor
import torchaudio.functional as F
model_id = "anuragshas/wav2vec2-large-xls-r-300m-ur-cv8"
sample_iter = iter(load_dataset("mozilla-foundation/common_voice_8_0", "ur", split="test", streaming=True, use_auth_token=True))
sample = next(sample_iter)
resampled_audio = F.resample(torch.tensor(sample["audio"]["array"]), 48_000, 16_000).numpy()
model = AutoModelForCTC.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id)
input_values = processor(resampled_audio, return_tensors="pt").input_values
with torch.no_grad():
    logits = model(input_values).logits
transcription = processor.batch_decode(logits.numpy()).text
# => "اب نے ٹ پیس ان لیتے ہیں"
```


### Eval results on Common Voice 8 "test" (WER):


| Without LM | With LM (run `./eval.py`) |
|---|---|
| 52.146 | 42.376 |


model name: wav2vec2-large-xls-r-300m-ur-cv8


model author: anuragshas


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, ur, dataset:mozilla-foundation/common_voice_8_0, transformers, license:apache-2.0, generated_from_trainer, robust-speech-event, model-index


language:


- sk
license: apache-2.0
tags:
- automatic-speech-recognition
- mozilla-foundation/common_voice_8_0
- generated_from_trainer
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_8_0
model-index:
- name: XLS-R-300M - Slovak
results:
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice 8
type: mozilla-foundation/common_voice_8_0
args: sk
metrics:
    - name: Test WER
type: wer
value: 18.609
    - name: Test CER
type: cer
value: 5.488
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Robust Speech Event - Dev Data
type: speech-recognition-community-v2/dev_data
args: sk
metrics:
    - name: Test WER
type: wer
value: 40.548
    - name: Test CER
type: cer
value: 17.733


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# XLS-R-300M - Slovak


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the MOZILLA-FOUNDATION/COMMON_VOICE_8_0 - SK dataset.
It achieves the following results on the evaluation set:


- Loss: 0.3067
- Wer: 0.2678


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 7.5e-05
- train_batch_size: 32
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1500
- num_epochs: 60.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 5.175         | 2.41  | 400  | 4.6909          | 1.0    |
| 3.3785        | 4.82  | 800  | 3.3080          | 1.0    |
| 2.6964        | 7.23  | 1200 | 2.0651          | 1.1055 |
| 1.3008        | 9.64  | 1600 | 0.5845          | 0.6207 |
| 1.1185        | 12.05 | 2000 | 0.4195          | 0.4193 |
| 1.0252        | 14.46 | 2400 | 0.3824          | 0.3570 |
| 0.935         | 16.87 | 2800 | 0.3693          | 0.3462 |
| 0.8818        | 19.28 | 3200 | 0.3587          | 0.3318 |
| 0.8534        | 21.69 | 3600 | 0.3420          | 0.3180 |
| 0.8137        | 24.1  | 4000 | 0.3426          | 0.3130 |
| 0.7968        | 26.51 | 4400 | 0.3349          | 0.3102 |
| 0.7558        | 28.92 | 4800 | 0.3216          | 0.3019 |
| 0.7313        | 31.33 | 5200 | 0.3451          | 0.3060 |
| 0.7358        | 33.73 | 5600 | 0.3272          | 0.2967 |
| 0.718         | 36.14 | 6000 | 0.3315          | 0.2882 |
| 0.6991        | 38.55 | 6400 | 0.3299          | 0.2830 |
| 0.6529        | 40.96 | 6800 | 0.3140          | 0.2836 |
| 0.6225        | 43.37 | 7200 | 0.3128          | 0.2751 |
| 0.633         | 45.78 | 7600 | 0.3211          | 0.2774 |
| 0.5876        | 48.19 | 8000 | 0.3162          | 0.2764 |
| 0.588         | 50.6  | 8400 | 0.3082          | 0.2722 |
| 0.5915        | 53.01 | 8800 | 0.3120          | 0.2681 |
| 0.5798        | 55.42 | 9200 | 0.3133          | 0.2709 |
| 0.5736        | 57.83 | 9600 | 0.3086          | 0.2676 |


### Framework versions


- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.4.dev0
- Tokenizers 0.11.0


#### Evaluation Commands


```bash
python eval.py --model_id anuragshas/wav2vec2-xls-r-300m-sk-cv8-with-lm --dataset mozilla-foundation/common_voice_8_0 --config sk --split test
```


2. To evaluate on `speech-recognition-community-v2/dev_data`


```bash
python eval.py --model_id anuragshas/wav2vec2-xls-r-300m-sk-cv8-with-lm --dataset speech-recognition-community-v2/dev_data --config sk --split validation --chunk_length_s 5.0 --stride_length_s 1.0
```


### Inference With LM


```python
import torch
from datasets import load_dataset
from transformers import AutoModelForCTC, AutoProcessor
import torchaudio.functional as F
model_id = "anuragshas/wav2vec2-xls-r-300m-sk-cv8-with-lm"
sample_iter = iter(load_dataset("mozilla-foundation/common_voice_8_0", "sk", split="test", streaming=True, use_auth_token=True))
sample = next(sample_iter)
resampled_audio = F.resample(torch.tensor(sample["audio"]["array"]), 48_000, 16_000).numpy()
model = AutoModelForCTC.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id)
input_values = processor(resampled_audio, return_tensors="pt").input_values
with torch.no_grad():
    logits = model(input_values).logits
transcription = processor.batch_decode(logits.numpy()).text
# => ""
```


| Without LM | With LM (run `./eval.py`) |
|---|---|
| 26.707 | 18.609 |


model name: wav2vec2-xls-r-300m-sk-cv8-with-lm


model author: anuragshas


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, sk, dataset:mozilla-foundation/common_voice_8_0, transformers, license:apache-2.0, mozilla-foundation/common_voice_8_0, generated_from_trainer, robust-speech-event, model-index


language: pa-IN
datasets:


- common_voice
metrics:
- wer
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
model-index:
- name: Anurag Singh XLSR Wav2Vec2 Large 53 Punjabi
results:
  - task:
name: Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice pa-IN
type: common_voice
args: pa-IN
metrics:
    - name: Test WER
type: wer
value: 58.05


# Wav2Vec2-Large-XLSR-53-Punjabi


Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on Punjabi using the [Common Voice](https://huggingface.co/datasets/common_voice).
When using this model, make sure that your speech input is sampled at 16kHz.


## Usage


The model can be used directly (without a language model) as follows:


```python
import torch
import torchaudio
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
test_dataset = load_dataset("common_voice", "pa-IN", split="test[:2%]")
processor = Wav2Vec2Processor.from_pretrained("anuragshas/wav2vec2-xlsr-53-pa-in")
model = Wav2Vec2ForCTC.from_pretrained("anuragshas/wav2vec2-xlsr-53-pa-in")
resampler = torchaudio.transforms.Resample(48_000, 16_000)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch
test_dataset = test_dataset.map(speech_file_to_array_fn)
inputs = processor(test_dataset["speech"][:2], sampling_rate=16_000, return_tensors="pt", padding=True)
with torch.no_grad():
    logits = model(inputs.input_values, attention_mask=inputs.attention_mask).logits
predicted_ids = torch.argmax(logits, dim=-1)
print("Prediction:", processor.batch_decode(predicted_ids))
print("Reference:", test_dataset["sentence"][:2])
```


## Evaluation


The model can be evaluated as follows on the Punjabi test data of Common Voice.


```python
import torch
import torchaudio
from datasets import load_dataset, load_metric
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import re
test_dataset = load_dataset("common_voice", "pa-IN", split="test")
wer = load_metric("wer")
processor = Wav2Vec2Processor.from_pretrained("anuragshas/wav2vec2-xlsr-53-pa-in")
model = Wav2Vec2ForCTC.from_pretrained("anuragshas/wav2vec2-xlsr-53-pa-in")
model.to("cuda")
chars_to_ignore_regex = '[\,\?\.\!\-\;\:\"\“\%\‘\”\।\’\'\…]'
resampler = torchaudio.transforms.Resample(48_000, 16_000)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch
test_dataset = test_dataset.map(speech_file_to_array_fn)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def evaluate(batch):
    inputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)
    with torch.no_grad():
        logits = model(inputs.input_values.to("cuda"), attention_mask=inputs.attention_mask.to("cuda")).logits
    pred_ids = torch.argmax(logits, dim=-1)
    batch["pred_strings"] = processor.batch_decode(pred_ids)
    return batch
result = test_dataset.map(evaluate, batched=True, batch_size=8)
print("WER: {:2f}".format(100 * wer.compute(predictions=result["pred_strings"], references=result["sentence"])))
```


**Test Result**: 58.05 %


## Training


The Common Voice `train` and `validation` datasets were used for training.


model name: wav2vec2-xlsr-53-pa-in


model author: anuragshas


task: automatic-speech-recognition


tags: pytorch, jax, wav2vec2, automatic-speech-recognition, pa-IN, dataset:common_voice, transformers, audio, speech, xlsr-fine-tuning-week, license:apache-2.0, model-index


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- common_voice
model-index:
- name: wav2vec2-large-xls-r-300m-ur
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# wav2vec2-large-xls-r-300m-ur


This model is a fine-tuned version of [anuragshas/wav2vec2-large-xls-r-300m-ur](https://huggingface.co/anuragshas/wav2vec2-large-xls-r-300m-ur) on the common_voice dataset.
It achieves the following results on the evaluation set:


- Loss: 2.0508
- Wer: 0.7328


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 7.5e-05
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.12
- num_epochs: 240


### Training results


| Training Loss | Epoch  | Step | Validation Loss | Wer    |
|:-------------:|:------:|:----:|:---------------:|:------:|
| 0.0719        | 66.67  | 400  | 1.8510          | 0.7432 |
| 0.0284        | 133.33 | 800  | 2.0088          | 0.7415 |
| 0.014         | 200.0  | 1200 | 2.0508          | 0.7328 |


### Framework versions


- Transformers 4.15.0
- Pytorch 1.10.0+cu111
- Datasets 1.17.0
- Tokenizers 0.10.3


model name: wav2vec2-large-xls-r-300m-ur


model author: anuragshas


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, dataset:common_voice, transformers, license:apache-2.0, generated_from_trainer, model-index


language: ia
datasets:


- common_voice
metrics:
- wer
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
model-index:
- name: Anurag Singh XLSR Wav2Vec2 Large 53 Interlingua
results:
  - task:
name: Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice ia
type: common_voice
args: ia
metrics:
    - name: Test WER
type: wer
value: 22.08


# Wav2Vec2-Large-XLSR-53-Interlingua


Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on Interlingua using the [Common Voice](https://huggingface.co/datasets/common_voice).
When using this model, make sure that your speech input is sampled at 16kHz.


## Usage


The model can be used directly (without a language model) as follows:


```python
import torch
import torchaudio
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
test_dataset = load_dataset("common_voice", "ia", split="test[:2%]")
processor = Wav2Vec2Processor.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-ia")
model = Wav2Vec2ForCTC.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-ia")
resampler = torchaudio.transforms.Resample(48_000, 16_000)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch
test_dataset = test_dataset.map(speech_file_to_array_fn)
inputs = processor(test_dataset["speech"][:2], sampling_rate=16_000, return_tensors="pt", padding=True)
with torch.no_grad():
    logits = model(inputs.input_values, attention_mask=inputs.attention_mask).logits
predicted_ids = torch.argmax(logits, dim=-1)
print("Prediction:", processor.batch_decode(predicted_ids))
print("Reference:", test_dataset["sentence"][:2])
```


## Evaluation


The model can be evaluated as follows on the Interlingua test data of Common Voice.


```python
import torch
import torchaudio
from datasets import load_dataset, load_metric
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import re
test_dataset = load_dataset("common_voice", "ia", split="test")
wer = load_metric("wer")
processor = Wav2Vec2Processor.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-ia")
model = Wav2Vec2ForCTC.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-ia")
model.to("cuda")
chars_to_ignore_regex = '[\.\,\!\?\-\"\:\;\'\“\”]'
resampler = torchaudio.transforms.Resample(48_000, 16_000)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch
test_dataset = test_dataset.map(speech_file_to_array_fn)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def evaluate(batch):
    inputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)
    with torch.no_grad():
        logits = model(inputs.input_values.to("cuda"), attention_mask=inputs.attention_mask.to("cuda")).logits
    pred_ids = torch.argmax(logits, dim=-1)
    batch["pred_strings"] = processor.batch_decode(pred_ids)
    return batch
result = test_dataset.map(evaluate, batched=True, batch_size=8)
print("WER: {:2f}".format(100 * wer.compute(predictions=result["pred_strings"], references=result["sentence"])))
```


**Test Result**: 22.08 %


## Training


The Common Voice `train` and `validation` datasets were used for training.


model name: wav2vec2-large-xlsr-53-ia


model author: anuragshas


task: automatic-speech-recognition


tags: pytorch, jax, wav2vec2, automatic-speech-recognition, ia, dataset:common_voice, transformers, audio, speech, xlsr-fine-tuning-week, license:apache-2.0, model-index


language: vi
datasets:


- common_voice
metrics:
- wer
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
model-index:
- name: Anurag Singh XLSR Wav2Vec2 Large 53 Vietnamese
results:
  - task:
name: Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice vi
type: common_voice
args: vi
metrics:
    - name: Test WER
type: wer
value: 66.78


# Wav2Vec2-Large-XLSR-53-Vietnamese


Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on Vietnamese using the [Common Voice](https://huggingface.co/datasets/common_voice).
When using this model, make sure that your speech input is sampled at 16kHz.


## Usage


```python
import torch
import torchaudio
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
test_dataset = load_dataset("common_voice", "vi", split="test[:2%]")
processor = Wav2Vec2Processor.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-vietnamese")
model = Wav2Vec2ForCTC.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-vietnamese")
resampler = torchaudio.transforms.Resample(48_000, 16_000)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch
test_dataset = test_dataset.map(speech_file_to_array_fn)
inputs = processor(test_dataset["speech"][:2], sampling_rate=16_000, return_tensors="pt", padding=True)
with torch.no_grad():
    logits = model(inputs.input_values, attention_mask=inputs.attention_mask).logits
predicted_ids = torch.argmax(logits, dim=-1)
print("Prediction:", processor.batch_decode(predicted_ids))
print("Reference:", test_dataset["sentence"][:2])
```


The model can be evaluated as follows on the Vietnamese test data of Common Voice.


```python
import torch
import torchaudio
from datasets import load_dataset, load_metric
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import re
test_dataset = load_dataset("common_voice", "vi", split="test")
wer = load_metric("wer")
processor = Wav2Vec2Processor.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-vietnamese")
model = Wav2Vec2ForCTC.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-vietnamese")
model.to("cuda")
chars_to_ignore_regex = '[\,\?\.\!\-\;\:\"\“]'
resampler = torchaudio.transforms.Resample(48_000, 16_000)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch
test_dataset = test_dataset.map(speech_file_to_array_fn)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def evaluate(batch):
    inputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)
    with torch.no_grad():
        logits = model(inputs.input_values.to("cuda"), attention_mask=inputs.attention_mask.to("cuda")).logits
    pred_ids = torch.argmax(logits, dim=-1)
    batch["pred_strings"] = processor.batch_decode(pred_ids)
    return batch
result = test_dataset.map(evaluate, batched=True, batch_size=8)
print("WER: {:2f}".format(100 * wer.compute(predictions=result["pred_strings"], references=result["sentence"])))
```


**Test Result**: 66.78 %


model name: wav2vec2-large-xlsr-53-vietnamese


tags: pytorch, jax, wav2vec2, automatic-speech-recognition, vi, dataset:common_voice, transformers, audio, speech, xlsr-fine-tuning-week, license:apache-2.0, model-index


language: rm-vallader
datasets:


- common_voice
metrics:
- wer
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
model-index:
- name: Anurag Singh XLSR Wav2Vec2 Large 53 Romansh Vallader
results:
  - task:
name: Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice rm-vallader
type: common_voice
args: rm-vallader
metrics:
    - name: Test WER
type: wer
value: 32.89


# Wav2Vec2-Large-XLSR-53-Romansh Vallader


Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on Romansh Vallader using the [Common Voice](https://huggingface.co/datasets/common_voice).
When using this model, make sure that your speech input is sampled at 16kHz.


```python
import torch
import torchaudio
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
test_dataset = load_dataset("common_voice", "rm-vallader", split="test[:2%]")
processor = Wav2Vec2Processor.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-rm-vallader")
model = Wav2Vec2ForCTC.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-rm-vallader")
resampler = torchaudio.transforms.Resample(48_000, 16_000)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch
test_dataset = test_dataset.map(speech_file_to_array_fn)
inputs = processor(test_dataset["speech"][:2], sampling_rate=16_000, return_tensors="pt", padding=True)
with torch.no_grad():
    logits = model(inputs.input_values, attention_mask=inputs.attention_mask).logits
predicted_ids = torch.argmax(logits, dim=-1)
print("Prediction:", processor.batch_decode(predicted_ids))
print("Reference:", test_dataset["sentence"][:2])
```


## Evaluation


The model can be evaluated as follows on the Romansh Vallader test data of Common Voice.


```python
import torch
import torchaudio
from datasets import load_dataset, load_metric
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import re
test_dataset = load_dataset("common_voice", "rm-vallader", split="test")
wer = load_metric("wer")
processor = Wav2Vec2Processor.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-rm-vallader")
model = Wav2Vec2ForCTC.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-rm-vallader")
model.to("cuda")
chars_to_ignore_regex = '[\,\?\.\!\-\;\:\"\“\%\”\„\–\…\«\»]'
resampler = torchaudio.transforms.Resample(48_000, 16_000)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    batch["sentence"] = re.sub('’ ',' ',batch["sentence"])
    batch["sentence"] = re.sub(' ‘',' ',batch["sentence"])
    batch["sentence"] = re.sub('’|‘','\'',batch["sentence"])
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch
test_dataset = test_dataset.map(speech_file_to_array_fn)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def evaluate(batch):
    inputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)
    with torch.no_grad():
        logits = model(inputs.input_values.to("cuda"), attention_mask=inputs.attention_mask.to("cuda")).logits
    pred_ids = torch.argmax(logits, dim=-1)
    batch["pred_strings"] = processor.batch_decode(pred_ids)
    return batch
result = test_dataset.map(evaluate, batched=True, batch_size=8)
print("WER: {:2f}".format(100 * wer.compute(predictions=result["pred_strings"], references=result["sentence"])))
```


**Test Result**: 32.89 %


## Training


model name: wav2vec2-large-xlsr-53-rm-vallader


model author: anuragshas


task: automatic-speech-recognition


tags: pytorch, jax, wav2vec2, automatic-speech-recognition, rm-vallader, dataset:common_voice, transformers, audio, speech, xlsr-fine-tuning-week, license:apache-2.0, model-index


language:


- lv
license: apache-2.0
tags:
- automatic-speech-recognition
- mozilla-foundation/common_voice_8_0
- generated_from_trainer
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_8_0
model-index:
- name: XLS-R-300M - Latvian
results:
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice 8
type: mozilla-foundation/common_voice_8_0
args: lv
metrics:
    - name: Test WER
type: wer
value: 9.633
    - name: Test CER
type: cer
value: 2.614
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Robust Speech Event - Dev Data
type: speech-recognition-community-v2/dev_data
args: lv
metrics:
    - name: Test WER
type: wer
value: 36.110
    - name: Test CER
type: cer
value: 14.244


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# XLS-R-300M - Latvian


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the MOZILLA-FOUNDATION/COMMON_VOICE_8_0 - LV dataset.
It achieves the following results on the evaluation set:


- Loss: 0.1660
- Wer: 0.1705


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 7.5e-05
- train_batch_size: 32
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 50.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 3.489         | 2.56  | 400  | 3.3590          | 1.0    |
| 2.9903        | 5.13  | 800  | 2.9704          | 1.0001 |
| 1.6712        | 7.69  | 1200 | 0.6179          | 0.6566 |
| 1.2635        | 10.26 | 1600 | 0.3176          | 0.4531 |
| 1.0819        | 12.82 | 2000 | 0.2517          | 0.3508 |
| 1.0136        | 15.38 | 2400 | 0.2257          | 0.3124 |
| 0.9625        | 17.95 | 2800 | 0.1975          | 0.2311 |
| 0.901         | 20.51 | 3200 | 0.1986          | 0.2097 |
| 0.8842        | 23.08 | 3600 | 0.1904          | 0.2039 |
| 0.8542        | 25.64 | 4000 | 0.1847          | 0.1981 |
| 0.8244        | 28.21 | 4400 | 0.1805          | 0.1847 |
| 0.7689        | 30.77 | 4800 | 0.1736          | 0.1832 |
| 0.7825        | 33.33 | 5200 | 0.1698          | 0.1821 |
| 0.7817        | 35.9  | 5600 | 0.1758          | 0.1803 |
| 0.7488        | 38.46 | 6000 | 0.1663          | 0.1760 |
| 0.7171        | 41.03 | 6400 | 0.1636          | 0.1721 |
| 0.7222        | 43.59 | 6800 | 0.1663          | 0.1729 |
| 0.7156        | 46.15 | 7200 | 0.1633          | 0.1715 |
| 0.7121        | 48.72 | 7600 | 0.1666          | 0.1718 |


### Framework versions


- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.11.0


#### Evaluation Commands


1. To evaluate on `mozilla-foundation/common_voice_8_0` with split `test`


```bash
python eval.py --model_id anuragshas/wav2vec2-xls-r-300m-lv-cv8-with-lm --dataset mozilla-foundation/common_voice_8_0 --config lv --split test
```


2. To evaluate on `speech-recognition-community-v2/dev_data`


```bash
python eval.py --model_id anuragshas/wav2vec2-xls-r-300m-lv-cv8-with-lm --dataset speech-recognition-community-v2/dev_data --config lv --split validation --chunk_length_s 5.0 --stride_length_s 1.0
```


### Inference With LM


```python
import torch
from datasets import load_dataset
from transformers import AutoModelForCTC, AutoProcessor
import torchaudio.functional as F
model_id = "anuragshas/wav2vec2-xls-r-300m-lv-cv8-with-lm"
sample_iter = iter(load_dataset("mozilla-foundation/common_voice_8_0", "lv", split="test", streaming=True, use_auth_token=True))
sample = next(sample_iter)
resampled_audio = F.resample(torch.tensor(sample["audio"]["array"]), 48_000, 16_000).numpy()
model = AutoModelForCTC.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id)
input_values = processor(resampled_audio, return_tensors="pt").input_values
with torch.no_grad():
    logits = model(input_values).logits
transcription = processor.batch_decode(logits.numpy()).text
# => "domāju ka viņam viss labi"
```


### Eval results on Common Voice 8 "test" (WER):


| Without LM | With LM (run `./eval.py`) |
|---|---|
| 16.997 | 9.633 |


model name: wav2vec2-xls-r-300m-lv-cv8-with-lm


model author: anuragshas


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, lv, dataset:mozilla-foundation/common_voice_8_0, transformers, license:apache-2.0, mozilla-foundation/common_voice_8_0, generated_from_trainer, robust-speech-event, model-index


language: sah
datasets:


- common_voice
metrics:
- wer
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0
model-index:
- name: Anurag Singh XLSR Wav2Vec2 Large 53 Sakha
results:
  - task:
name: Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice sah
type: common_voice
args: sah
metrics:
    - name: Test WER
type: wer
value: 38.04


# Wav2Vec2-Large-XLSR-53-Sakha


Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on Sakha using the [Common Voice](https://huggingface.co/datasets/common_voice).
When using this model, make sure that your speech input is sampled at 16kHz.


## Usage


The model can be used directly (without a language model) as follows:


```python
import torch
import torchaudio
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
test_dataset = load_dataset("common_voice", "sah", split="test[:2%]")
processor = Wav2Vec2Processor.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-sah")
model = Wav2Vec2ForCTC.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-sah")
resampler = torchaudio.transforms.Resample(48_000, 16_000)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch
test_dataset = test_dataset.map(speech_file_to_array_fn)
inputs = processor(test_dataset["speech"][:2], sampling_rate=16_000, return_tensors="pt", padding=True)
with torch.no_grad():
    logits = model(inputs.input_values, attention_mask=inputs.attention_mask).logits
predicted_ids = torch.argmax(logits, dim=-1)
print("Prediction:", processor.batch_decode(predicted_ids))
print("Reference:", test_dataset["sentence"][:2])
```


## Evaluation


The model can be evaluated as follows on the Sakha test data of Common Voice.


```python
import torch
import torchaudio
from datasets import load_dataset, load_metric
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import re
test_dataset = load_dataset("common_voice", "sah", split="test")
wer = load_metric("wer")
processor = Wav2Vec2Processor.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-sah")
model = Wav2Vec2ForCTC.from_pretrained("anuragshas/wav2vec2-large-xlsr-53-sah")
model.to("cuda")
chars_to_ignore_regex = '[\,\?\.\!\-\;\:\"\“\%\”\„\–\…\«\»]'
resampler = torchaudio.transforms.Resample(48_000, 16_000)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
    batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()
    speech_array, sampling_rate = torchaudio.load(batch["path"])
    batch["speech"] = resampler(speech_array).squeeze().numpy()
    return batch
test_dataset = test_dataset.map(speech_file_to_array_fn)
# Preprocessing the datasets.
# We need to read the aduio files as arrays
def evaluate(batch):
    inputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)
    with torch.no_grad():
        logits = model(inputs.input_values.to("cuda"), attention_mask=inputs.attention_mask.to("cuda")).logits
    pred_ids = torch.argmax(logits, dim=-1)
    batch["pred_strings"] = processor.batch_decode(pred_ids)
    return batch
result = test_dataset.map(evaluate, batched=True, batch_size=8)
print("WER: {:2f}".format(100 * wer.compute(predictions=result["pred_strings"], references=result["sentence"])))
```


**Test Result**: 38.04 %


## Training


The Common Voice `train` and `validation` datasets were used for training.


model name: wav2vec2-large-xlsr-53-sah


model author: anuragshas


task: automatic-speech-recognition


tags: pytorch, jax, wav2vec2, automatic-speech-recognition, sah, dataset:common_voice, transformers, audio, speech, xlsr-fine-tuning-week, license:apache-2.0, model-index


language:


- hi
license: apache-2.0
tags:
- automatic-speech-recognition
- mozilla-foundation/common_voice_7_0
- generated_from_trainer
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_7_0
metrics:
- wer
model-index:
- name: wav2vec2-xls-r-1b-hi-cv7
results:
  - task:
type: automatic-speech-recognition
name: Speech Recognition
dataset:
type: mozilla-foundation/common_voice_7_0
name: Common Voice 7
args: hi
metrics:
    - type: wer    # Required. Example: wer
value: 18.504  # Required. Example: 20.90
name: Test WER    # Optional. Example: Test WER
    - name: Test CER
type: cer
value: 6.655


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# wav2vec2-xls-r-1b-hi-cv7


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-1b](https://huggingface.co/facebook/wav2vec2-xls-r-1b) on the MOZILLA-FOUNDATION/COMMON_VOICE_7_0 - HI dataset.
It achieves the following results on the evaluation set:


- Loss: 0.5878
- Wer: 0.3419


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 7.5e-05
- train_batch_size: 8
- eval_batch_size: 16
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2000
- num_epochs: 100.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step  | Validation Loss | Wer    |
|:-------------:|:-----:|:-----:|:---------------:|:------:|
| 1.9859        | 2.72  | 400   | 1.1663          | 0.7948 |
| 1.2969        | 5.44  | 800   | 0.7725          | 0.6562 |
| 1.1954        | 8.16  | 1200  | 0.5940          | 0.4904 |
| 1.164         | 10.88 | 1600  | 0.5338          | 0.4316 |
| 1.1464        | 13.6  | 2000  | 0.5432          | 0.4226 |
| 1.1553        | 16.33 | 2400  | 0.5471          | 0.4260 |
| 1.0985        | 19.05 | 2800  | 0.5290          | 0.4076 |
| 1.0421        | 21.77 | 3200  | 0.5672          | 0.4181 |
| 0.9831        | 24.49 | 3600  | 0.5741          | 0.4141 |
| 0.9827        | 27.21 | 4000  | 0.5754          | 0.4179 |
| 0.9669        | 29.93 | 4400  | 0.5310          | 0.3889 |
| 0.9496        | 32.65 | 4800  | 0.5649          | 0.4062 |
| 0.9112        | 35.37 | 5200  | 0.5738          | 0.3926 |
| 0.8838        | 38.1  | 5600  | 0.5232          | 0.3768 |
| 0.8666        | 40.81 | 6000  | 0.5510          | 0.3852 |
| 0.8366        | 43.54 | 6400  | 0.5436          | 0.3837 |
| 0.7957        | 46.26 | 6800  | 0.5337          | 0.3775 |
| 0.7834        | 48.98 | 7200  | 0.5611          | 0.3844 |
| 0.7685        | 51.7  | 7600  | 0.5710          | 0.4008 |
| 0.7431        | 54.42 | 8000  | 0.5636          | 0.3726 |
| 0.7353        | 57.14 | 8400  | 0.5937          | 0.3836 |
| 0.7001        | 59.86 | 8800  | 0.5815          | 0.3858 |
| 0.6799        | 62.58 | 9200  | 0.5862          | 0.3696 |
| 0.6459        | 65.31 | 9600  | 0.6181          | 0.3762 |
| 0.6121        | 68.03 | 10000 | 0.5637          | 0.3590 |
| 0.5942        | 70.75 | 10400 | 0.6374          | 0.3882 |
| 0.5769        | 73.47 | 10800 | 0.6015          | 0.3640 |
| 0.5689        | 76.19 | 11200 | 0.5669          | 0.3508 |
| 0.5461        | 78.91 | 11600 | 0.5967          | 0.3621 |
| 0.5286        | 81.63 | 12000 | 0.5840          | 0.3605 |
| 0.5057        | 84.35 | 12400 | 0.5848          | 0.3489 |
| 0.482         | 87.07 | 12800 | 0.5860          | 0.3488 |
| 0.4655        | 89.79 | 13200 | 0.5780          | 0.3453 |
| 0.4523        | 92.52 | 13600 | 0.6150          | 0.3532 |
| 0.4422        | 95.24 | 14000 | 0.5930          | 0.3452 |
| 0.4436        | 97.96 | 14400 | 0.5867          | 0.3428 |


### Framework versions


- Transformers 4.16.0.dev0
- Pytorch 1.10.1+cu102
- Datasets 1.17.1.dev0
- Tokenizers 0.11.0


#### Evaluation Commands


1. To evaluate on `mozilla-foundation/common_voice_7_0` with split `test`


```bash
python eval.py --model_id anuragshas/wav2vec2-xls-r-1b-hi --dataset mozilla-foundation/common_voice_7_0 --config hi --split test
```


### Inference With LM


```python
import torch
from datasets import load_dataset
from transformers import AutoModelForCTC, AutoProcessor
import torchaudio.functional as F
model_id = "anuragshas/wav2vec2-xls-r-1b-hi"
sample_iter = iter(load_dataset("mozilla-foundation/common_voice_7_0", "hi", split="test", streaming=True, use_auth_token=True))
sample = next(sample_iter)
resampled_audio = F.resample(torch.tensor(sample["audio"]["array"]), 48_000, 16_000).numpy()
model = AutoModelForCTC.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id)
input_values = processor(resampled_audio, return_tensors="pt").input_values
with torch.no_grad():
    logits = model(input_values).logits
transcription = processor.batch_decode(logits.numpy()).text
# => "तुम्हारे पास तीन महीने बचे हैं"
```


### Eval results on Common Voice 7 "test" (WER):


| Without LM | With LM (run `./eval.py`) |
|---|---|
| 28.942 | 18.504 |


model name: wav2vec2-xls-r-1b-hi


model author: anuragshas


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, hi, dataset:mozilla-foundation/common_voice_7_0, transformers, license:apache-2.0, mozilla-foundation/common_voice_7_0, generated_from_trainer, robust-speech-event, model-index


language:


- bg
license: apache-2.0
tags:
- automatic-speech-recognition
- mozilla-foundation/common_voice_8_0
- generated_from_trainer
- robust-speech-event
datasets:
- mozilla-foundation/common_voice_8_0
model-index:
- name: XLS-R-300M - Bulgarian
results:
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Common Voice 8
type: mozilla-foundation/common_voice_8_0
args: bg
metrics:
    - name: Test WER
type: wer
value: 21.195
    - name: Test CER
type: cer
value: 4.786
  - task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: Robust Speech Event - Dev Data
type: speech-recognition-community-v2/dev_data
args: bg
metrics:
    - name: Test WER
type: wer
value: 32.667
    - name: Test CER
type: cer
value: 12.452


# XLS-R-300M - Bulgarian


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the MOZILLA-FOUNDATION/COMMON_VOICE_8_0 - BG dataset.
It achieves the following results on the evaluation set:


- Loss: 0.2473
- Wer: 0.3002


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 7.5e-05
- train_batch_size: 32
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 50.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 3.1589        | 3.48  | 400  | 3.0830          | 1.0    |
| 2.8921        | 6.96  | 800  | 2.6605          | 0.9982 |
| 1.3049        | 10.43 | 1200 | 0.5069          | 0.5707 |
| 1.1349        | 13.91 | 1600 | 0.4159          | 0.5041 |
| 1.0686        | 17.39 | 2000 | 0.3815          | 0.4746 |
| 0.999         | 20.87 | 2400 | 0.3541          | 0.4343 |
| 0.945         | 24.35 | 2800 | 0.3266          | 0.4132 |
| 0.9058        | 27.83 | 3200 | 0.2969          | 0.3771 |
| 0.8672        | 31.3  | 3600 | 0.2802          | 0.3553 |
| 0.8313        | 34.78 | 4000 | 0.2662          | 0.3380 |
| 0.8068        | 38.26 | 4400 | 0.2528          | 0.3181 |
| 0.7796        | 41.74 | 4800 | 0.2537          | 0.3073 |
| 0.7621        | 45.22 | 5200 | 0.2503          | 0.3036 |
| 0.7611        | 48.7  | 5600 | 0.2477          | 0.2991 |


### Framework versions


- Transformers 4.17.0.dev0
- Pytorch 1.10.2+cu102
- Datasets 1.18.2.dev0
- Tokenizers 0.11.0


#### Evaluation Commands


1. To evaluate on `mozilla-foundation/common_voice_8_0` with split `test`


```bash
python eval.py --model_id anuragshas/wav2vec2-large-xls-r-300m-bg --dataset mozilla-foundation/common_voice_8_0 --config bg --split test
```


2. To evaluate on `speech-recognition-community-v2/dev_data`


```bash
python eval.py --model_id anuragshas/wav2vec2-large-xls-r-300m-bg --dataset speech-recognition-community-v2/dev_data --config bg --split validation --chunk_length_s 5.0 --stride_length_s 1.0
```


### Inference With LM


```python
import torch
from datasets import load_dataset
from transformers import AutoModelForCTC, AutoProcessor
import torchaudio.functional as F
model_id = "anuragshas/wav2vec2-large-xls-r-300m-bg"
sample_iter = iter(load_dataset("mozilla-foundation/common_voice_8_0", "bg", split="test", streaming=True, use_auth_token=True))
sample = next(sample_iter)
resampled_audio = F.resample(torch.tensor(sample["audio"]["array"]), 48_000, 16_000).numpy()
model = AutoModelForCTC.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id)
input_values = processor(resampled_audio, return_tensors="pt").input_values
with torch.no_grad():
    logits = model(input_values).logits
transcription = processor.batch_decode(logits.numpy()).text
# => "и надутият му ката блоонкурем взе да се събира"
```


### Eval results on Common Voice 8 "test" (WER):


| Without LM | With LM (run `./eval.py`) |
|---|---|
| 30.07 | 21.195 |


model name: wav2vec2-large-xls-r-300m-bg


model author: anuragshas


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, bg, dataset:mozilla-foundation/common_voice_8_0, transformers, license:apache-2.0, mozilla-foundation/common_voice_8_0, generated_from_trainer, robust-speech-event, model-index


pipeline_tag: sentence-similarity
tags:


- sentence-transformers
- feature-extraction
- sentence-similarity
- transformers


# {MODEL_NAME}


This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search.


<!--- Describe your model here -->


## Usage (Sentence-Transformers)


Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:


```
pip install -U sentence-transformers
```


Then you can use the model like this:


```python
from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]

model = SentenceTransformer('{MODEL_NAME}')
embeddings = model.encode(sentences)
print(embeddings)
```


## Usage (HuggingFace Transformers)


Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.


```python
from transformers import AutoTokenizer, AutoModel
import torch

#Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

# Sentences we want sentence embeddings for
sentences = ['This is an example sentence', 'Each sentence is converted']

# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('{MODEL_NAME}')
model = AutoModel.from_pretrained('{MODEL_NAME}')

# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')

# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

# Perform pooling. In this case, max pooling.
sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])

print("Sentence embeddings:")
print(sentence_embeddings)
```


## Evaluation Results


<!--- Describe how your model was evaluated -->


For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name={MODEL_NAME})


## Training


The model was trained with the parameters:


**DataLoader**:


`torch.utils.data.dataloader.DataLoader` of length 365 with parameters:


```
{'batch_size': 32, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}
```


**Loss**:


`sentence_transformers.losses.CosineSimilarityLoss.CosineSimilarityLoss`


Parameters of the fit()-Method:


```
{
    "callback": null,
    "epochs": 4,
    "evaluation_steps": 1000,
    "evaluator": "sentence_transformers.evaluation.EmbeddingSimilarityEvaluator.EmbeddingSimilarityEvaluator",
    "max_grad_norm": 1,
    "optimizer_class": "<class 'transformers.optimization.AdamW'>",
    "optimizer_params": {
        "lr": 2e-05
    },
    "scheduler": "WarmupLinear",
    "steps_per_epoch": null,
    "warmup_steps": 146,
    "weight_decay": 0.01
}
```


## Full Model Architecture


```
SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: RobertaModel
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
)
```


## Citing & Authors


<!--- Describe where people can find more information -->


model name: KLUE-STS-roberta-base


model author: ParkMyungkyu


task: sentence-similarity


tags: pytorch, roberta, feature-extraction, sentence-transformers, sentence-similarity, transformers, infinity_compatible


model name: try


model author: Prim9000


tags:


- conversational


model name: sheldonbot


model author: For


task: conversational