README.md

metadata

language: vi
datasets:
  - vivos
  - common_voice
metrics:
  - wer
pipeline_tag: automatic-speech-recognition
tags:
  - audio
  - speech
  - speechbrain
  - Transformer
license: cc-by-nc-4.0
widget:
  - example_title: Example 1
    src: >-
      https://huggingface.co/dragonSwing/wav2vec2-base-vn-270h/raw/main/example.mp3
  - example_title: Example 2
    src: >-
      https://huggingface.co/dragonSwing/wav2vec2-base-vn-270h/raw/main/example2.mp3
model-index:
  - name: Wav2vec2 Base Vietnamese 270h
    results:
      - task:
          name: Speech Recognition
          type: automatic-speech-recognition
        dataset:
          name: Common Voice vi
          type: common_voice
          args: vi
        metrics:
          - name: Test WER
            type: wer
            value: 9.66
      - task:
          name: Speech Recognition
          type: automatic-speech-recognition
        dataset:
          name: Common Voice 7.0
          type: mozilla-foundation/common_voice_7_0
          args: vi
        metrics:
          - name: Test WER
            type: wer
            value: 5.57
      - task:
          name: Speech Recognition
          type: automatic-speech-recognition
        dataset:
          name: Common Voice 8.0
          type: mozilla-foundation/common_voice_8_0
          args: vi
        metrics:
          - name: Test WER
            type: wer
            value: 5.76
      - task:
          name: Speech Recognition
          type: automatic-speech-recognition
        dataset:
          name: VIVOS
          type: vivos
          args: vi
        metrics:
          - name: Test WER
            type: wer
            value: 3.7

Wav2Vec2-Base-Vietnamese-270h

Fine-tuned Wav2Vec2 model on Vietnamese Speech Recognition task using about 270h labelled data combined from multiple datasets including Common Voice, VIVOS, VLSP2020. The model was fine-tuned using SpeechBrain toolkit with a custom tokenizer. For a better experience, we encourage you to learn more about SpeechBrain.
When using this model, make sure that your speech input is sampled at 16kHz.
Please refer to huggingface blog or speechbrain on how to fine-tune Wav2Vec2 model on a specific language.

Benchmark WER result:

	VIVOS	COMMON VOICE 7.0	COMMON VOICE 8.0
without LM	8.23	12.15	12.15
with 4-grams LM	3.70	5.57	5.76

The language model was trained using OSCAR dataset on about 32GB of crawled text.

Install SpeechBrain

To use this model, you should install speechbrain > 0.5.10

Usage

The model can be used directly (without a language model) as follows:

from speechbrain.pretrained import EncoderASR

model = EncoderASR.from_hparams(source="dragonSwing/wav2vec2-base-vn-270h", savedir="pretrained_models/asr-wav2vec2-vi")
model.transcribe_file('dragonSwing/wav2vec2-base-vn-270h/example.mp3')
# Output: được hồ chí minh coi là một động lực lớn của sự phát triển đất nước

Inference on GPU

To perform inference on the GPU, add run_opts={"device":"cuda"} when calling the from_hparams method.

Evaluation

The model can be evaluated as follows on the Vietnamese test data of Common Voice 8.0.

import torch
import torchaudio
from datasets import load_dataset, load_metric, Audio
from transformers import Wav2Vec2FeatureExtractor
from speechbrain.pretrained import EncoderASR
import re
test_dataset = load_dataset("mozilla-foundation/common_voice_8_0", "vi", split="test", use_auth_token=True)
test_dataset = test_dataset.cast_column("audio", Audio(sampling_rate=16_000))
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
wer = load_metric("wer")
extractor = Wav2Vec2FeatureExtractor.from_pretrained("dragonSwing/wav2vec2-base-vn-270h")
model = EncoderASR.from_hparams(source="dragonSwing/wav2vec2-base-vn-270h", savedir="pretrained_models/asr-wav2vec2-vi", run_opts={'device': device})
chars_to_ignore_regex = r'[,?.!\-;:"“%\'�]'
# Preprocessing the datasets.
# We need to read the audio files as arrays
def speech_file_to_array_fn(batch):
  audio = batch["audio"]
  batch["target_text"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()
  batch['speech'] = audio['array']
  return batch
test_dataset = test_dataset.map(speech_file_to_array_fn)

def evaluate(batch):
  # For padding inputs only
  inputs = extractor(
    batch['speech'], 
    sampling_rate=16000, 
    return_tensors="pt", 
    padding=True, 
    do_normalize=False
  ).input_values
  input_lens = torch.ones(inputs.shape[0])
  pred_str, pred_tokens = model.transcribe_batch(inputs, input_lens)
  batch["pred_strings"] = pred_str
  
  return batch
result = test_dataset.map(evaluate, batched=True, batch_size=1)
print("WER: {:2f}".format(100 * wer.compute(predictions=result["pred_strings"], references=result["target_text"])))

Test Result: 12.155553%

Citation

@misc{SB2021,
    author = {Ravanelli, Mirco and Parcollet, Titouan and Rouhe, Aku and Plantinga, Peter and Rastorgueva, Elena and Lugosch, Loren and Dawalatabad, Nauman and Ju-Chieh, Chou and Heba, Abdel and Grondin, Francois and Aris, William and Liao, Chien-Feng and Cornell, Samuele and Yeh, Sung-Lin and Na, Hwidong and Gao, Yan and Fu, Szu-Wei and Subakan, Cem and De Mori, Renato and Bengio, Yoshua },
    title = {SpeechBrain},
    year = {2021},
    publisher = {GitHub},
    journal = {GitHub repository},
    howpublished = {\\\\url{https://github.com/speechbrain/speechbrain}},
  }

About SpeechBrain

SpeechBrain is an open-source and all-in-one speech toolkit. It is designed to be simple, extremely flexible, and user-friendly. Competitive or state-of-the-art performance is obtained in various domains.
Website: https://speechbrain.github.io
GitHub: https://github.com/speechbrain/speechbrain