|
--- |
|
language: |
|
- "zh" |
|
thumbnail: "Mandarin-wav2vec2.0 fine-tuned on AISHELL-1 dataset" |
|
tags: |
|
- automatic-speech-recognition |
|
- speech |
|
- wav2vec2.0 |
|
- audio |
|
datasets: |
|
- AISHELL-1 |
|
metrics: |
|
- cer |
|
|
|
--- |
|
|
|
The Mandarin-wav2vec2.0 model is pre-trained on 1000 hours of AISHELL-2 dataset. The pre-training detail can be found at https://github.com/kehanlu/mandarin-wav2vec2. This model is fine-tuned on 178 hours of AISHELL-1 dataset and is the baseline model in the paper "A context-aware knowledge transferring strategy for CTC-based ASR |
|
"([preprint](https://arxiv.org/abs/2210.06244)). |
|
|
|
## Results on AISHELL-1 |
|
|CER|dev|test| |
|
| - | - | - | |
|
|vanilla w2v2-CTC | 4.85 | 5.13| |
|
|
|
## Usage |
|
**Note:** the model is fine-tuned using ESPNET toolkit, then converted to huggingface model for simple usage. |
|
|
|
```python |
|
import torch |
|
import soundfile as sf |
|
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor |
|
|
|
class ExtendedWav2Vec2ForCTC(Wav2Vec2ForCTC): |
|
""" |
|
In ESPNET there is a LayerNorm layer between encoder output and CTC classification head. |
|
""" |
|
def __init__(self, config): |
|
super().__init__(config) |
|
self.lm_head = torch.nn.Sequential( |
|
torch.nn.LayerNorm(config.hidden_size), |
|
self.lm_head |
|
) |
|
|
|
model = ExtendedWav2Vec2ForCTC.from_pretrained("kehanlu/mandarin-wav2vec2-aishell1") |
|
processor = Wav2Vec2Processor.from_pretrained("kehanlu/mandarin-wav2vec2-aishell1") |
|
|
|
audio_input, sample_rate = sf.read("/path/to/data_aishell/wav/dev/S0724/BAC009S0724W0121.wav") |
|
inputs = processor(audio_input, sampling_rate=sample_rate, return_tensors="pt") |
|
|
|
with torch.no_grad(): |
|
model.eval() |
|
logits = model(**inputs).logits |
|
predicted_ids = torch.argmax(logits, dim=-1) |
|
transcription = processor.batch_decode(predicted_ids) |
|
print(transcription[0]) |
|
|
|
# 广州市房地产中介协会分析 |
|
``` |
|
|
|
|
|
## Licence |
|
|
|
The pre-trained corpus, AISHELL-2, is supported by AISHELL fundation. The outcome model also follow the licence of AISHELL-2. It is free to use for academic purpose and should not be used on any commercial purpose without the permission from AISHELL fundation. (https://www.aishelltech.com/aishell_2) |
|
|
|
``` |
|
@ARTICLE{aishell2, |
|
author = {{Du}, J. and {Na}, X. and {Liu}, X. and {Bu}, H.}, |
|
title = "{AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scale}", |
|
journal = {ArXiv}, |
|
eprint = {1808.10583}, |
|
primaryClass = "cs.CL", |
|
year = 2018, |
|
month = Aug, |
|
} |
|
``` |
|
|
|
if you find this useful, please cite |
|
``` |
|
@article{lu2022context, |
|
title={A context-aware knowledge transferring strategy for CTC-based ASR}, |
|
author={Lu, Ke-Han and Chen, Kuan-Yu}, |
|
journal={arXiv preprint arXiv:2210.06244}, |
|
year={2022} |
|
} |
|
``` |
