|
--- |
|
language: vi |
|
datasets: |
|
- vlsp |
|
- vivos |
|
tags: |
|
- audio |
|
- automatic-speech-recognition |
|
license: cc-by-nc-4.0 |
|
widget: |
|
- example_title: VLSP ASR 2020 test T1 |
|
src: https://huggingface.co/nguyenvulebinh/wav2vec2-base-vietnamese-250h/raw/main/audio-test/t1_0001-00010.wav |
|
- example_title: VLSP ASR 2020 test T1 |
|
src: https://huggingface.co/nguyenvulebinh/wav2vec2-base-vietnamese-250h/raw/main/audio-test/t1_utt000000042.wav |
|
- example_title: VLSP ASR 2020 test T2 |
|
src: https://huggingface.co/nguyenvulebinh/wav2vec2-base-vietnamese-250h/raw/main/audio-test/t2_0000006682.wav |
|
model-index: |
|
- name: Vietnamese end-to-end speech recognition using wav2vec 2.0 by VietAI |
|
results: |
|
- task: |
|
name: Speech Recognition |
|
type: automatic-speech-recognition |
|
dataset: |
|
name: Common Voice vi |
|
type: common_voice |
|
args: vi |
|
metrics: |
|
- name: Test WER |
|
type: wer |
|
value: 11.52 |
|
- task: |
|
name: Speech Recognition |
|
type: automatic-speech-recognition |
|
dataset: |
|
name: VIVOS |
|
type: vivos |
|
args: vi |
|
metrics: |
|
- name: Test WER |
|
type: wer |
|
value: 6.15 |
|
--- |
|
|
|
# Vietnamese end-to-end speech recognition using wav2vec 2.0 |
|
|
|
[](https://paperswithcode.com/sota/speech-recognition-on-common-voice-vi?p=vietnamese-end-to-end-speech-recognition) |
|
|
|
[](https://paperswithcode.com/sota/speech-recognition-on-vivos?p=vietnamese-end-to-end-speech-recognition) |
|
|
|
|
|
[Facebook's Wav2Vec2](https://ai.facebook.com/blog/wav2vec-20-learning-the-structure-of-speech-from-raw-audio/) |
|
|
|
### Model description |
|
|
|
[Our models](https://huggingface.co/nguyenvulebinh/wav2vec2-base-vietnamese-250h) are pre-trained on 13k hours of Vietnamese youtube audio (un-label data) and fine-tuned on 250 hours labeled of [VLSP ASR dataset](https://vlsp.org.vn/vlsp2020/eval/asr) on 16kHz sampled speech audio. |
|
|
|
We use [wav2vec2 architecture](https://ai.facebook.com/blog/wav2vec-20-learning-the-structure-of-speech-from-raw-audio/) for the pre-trained model. Follow wav2vec2 paper: |
|
|
|
>For the first time that learning powerful representations from speech audio alone followed by fine-tuning on transcribed speech can outperform the best semi-supervised methods while being conceptually simpler. |
|
|
|
For fine-tuning phase, wav2vec2 is fine-tuned using Connectionist Temporal Classification (CTC), which is an algorithm that is used to train neural networks for sequence-to-sequence problems and mainly in Automatic Speech Recognition and handwriting recognition. |
|
|
|
| Model | #params | Pre-training data | Fine-tune data | |
|
|---|---|---|---| |
|
| [base]((https://huggingface.co/nguyenvulebinh/wav2vec2-base-vietnamese-250h)) | 95M | 13k hours | 250 hours | |
|
|
|
In a formal ASR system, two components are required: acoustic model and language model. Here ctc-wav2vec fine-tuned model works as an acoustic model. For the language model, we provide a [4-grams model](https://huggingface.co/nguyenvulebinh/wav2vec2-base-vietnamese-250h/blob/main/vi_lm_4grams.bin.zip) trained on 2GB of spoken text. |
|
|
|
Detail of training and fine-tuning process, the audience can follow [fairseq github](https://github.com/pytorch/fairseq/tree/master/examples/wav2vec) and [huggingface blog](https://huggingface.co/blog/fine-tune-wav2vec2-english). |
|
|
|
### Benchmark WER result: |
|
|
|
| | [VIVOS](https://ailab.hcmus.edu.vn/vivos) | [COMMON VOICE VI](https://paperswithcode.com/dataset/common-voice) | [VLSP-T1](https://vlsp.org.vn/vlsp2020/eval/asr) | [VLSP-T2](https://vlsp.org.vn/vlsp2020/eval/asr) | |
|
|---|---|---|---|---| |
|
|without LM| 10.77 | 18.34 | 13.33 | 51.45 | |
|
|with 4-grams LM| 6.15 | 11.52 | 9.11 | 40.81 | |
|
|
|
|
|
### Example usage |
|
|
|
When using the model make sure that your speech input is sampled at 16Khz. Audio length should be shorter than 10s. Following the Colab link below to use a combination of CTC-wav2vec and 4-grams LM. |
|
|
|
[](https://colab.research.google.com/drive/1pVBY46gSoWer2vDf0XmZ6uNV3d8lrMxx?usp=sharing) |
|
|
|
|
|
```python |
|
from transformers import Wav2Vec2Processor, Wav2Vec2ForCTC |
|
from datasets import load_dataset |
|
import soundfile as sf |
|
import torch |
|
|
|
# load model and tokenizer |
|
processor = Wav2Vec2Processor.from_pretrained("nguyenvulebinh/wav2vec2-base-vietnamese-250h") |
|
model = Wav2Vec2ForCTC.from_pretrained("nguyenvulebinh/wav2vec2-base-vietnamese-250h") |
|
|
|
# define function to read in sound file |
|
def map_to_array(batch): |
|
speech, _ = sf.read(batch["file"]) |
|
batch["speech"] = speech |
|
return batch |
|
|
|
# load dummy dataset and read soundfiles |
|
ds = map_to_array({ |
|
"file": 'audio-test/t1_0001-00010.wav' |
|
}) |
|
|
|
# tokenize |
|
input_values = processor(ds["speech"], return_tensors="pt", padding="longest").input_values # Batch size 1 |
|
|
|
# retrieve logits |
|
logits = model(input_values).logits |
|
|
|
# take argmax and decode |
|
predicted_ids = torch.argmax(logits, dim=-1) |
|
transcription = processor.batch_decode(predicted_ids) |
|
``` |
|
|
|
### Model Parameters License |
|
|
|
The ASR model parameters are made available for non-commercial use only, under the terms of the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license. You can find details at: https://creativecommons.org/licenses/by-nc/4.0/legalcode |
|
|
|
### Citation |
|
|
|
[](https://github.com/vietai/ASR) |
|
|
|
```text |
|
@misc{Thai_Binh_Nguyen_wav2vec2_vi_2021, |
|
author = {Thai Binh Nguyen}, |
|
doi = {10.5281/zenodo.5356039}, |
|
month = {09}, |
|
title = {{Vietnamese end-to-end speech recognition using wav2vec 2.0}}, |
|
url = {https://github.com/vietai/ASR}, |
|
year = {2021} |
|
} |
|
``` |
|
**Please CITE** our repo when it is used to help produce published results or is incorporated into other software. |
|
|
|
# Contact |
|
|
|
nguyenvulebinh@gmail.com / binh@vietai.org |
|
|
|
[](https://twitter.com/intent/follow?screen_name=nguyenvulebinh) |
