|
--- |
|
language: "cs" |
|
tags: |
|
- Czech |
|
- KKY |
|
- FAV |
|
license: "cc-by-nc-sa-4.0" |
|
--- |
|
|
|
# wav2vec2-base-cs-80k-ClTRUS |
|
**C**zech **l**anguage **TR**ransformer from **U**nlabeled **S**peech (ClTRUS) is a monolingual Czech Wav2Vec 2.0 base model pre-trained from 80 thousand hours of Czech speech. |
|
|
|
This model does not have a tokenizer as it was pretrained on audio alone. In order to use this model for speech recognition, a tokenizer should be created and the model should be fine-tuned on labeled text data. |
|
|
|
**Note:** This is a checkpoint of the model after 4 epochs over the whole dataset. If you want some earlier or later checkpoints, please feel free to contact the author (jlehecka(at)kky.zcu.cz). |
|
|
|
## Pretraining data |
|
More than 80 thousand hours of unlabeled Czech speech: |
|
- recordings from radio (22k hours), |
|
- unlabeled data from VoxPopuli dataset (18.7k hours), |
|
- TV shows (15k hours), |
|
- shadow speakers (12k hours), |
|
- sports (5k hours), |
|
- telephone data (2k hours), |
|
- and a smaller amount of data from several other domains including the public CommonVoice dataset. |
|
|
|
## Usage |
|
Inputs must be 16kHz mono audio files. |
|
|
|
This model can be used e.g. to extract per-frame contextual embeddings from audio: |
|
```python |
|
from transformers import Wav2Vec2Model, Wav2Vec2FeatureExtractor |
|
import torchaudio |
|
|
|
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("fav-kky/wav2vec2-base-cs-80k-ClTRUS") |
|
model = Wav2Vec2Model.from_pretrained("fav-kky/wav2vec2-base-cs-80k-ClTRUS") |
|
|
|
speech_array, sampling_rate = torchaudio.load("/path/to/audio/file.wav") |
|
inputs = feature_extractor( |
|
speech_array, |
|
sampling_rate=16_000, |
|
return_tensors="pt" |
|
)["input_values"][0] |
|
|
|
output = model(inputs) |
|
embeddings = output.last_hidden_state.detach().numpy()[0] |
|
``` |
|
|
|
## Speech recognition results |
|
After fine-tuning, the model scored the following results on public datasets: |
|
- Czech portion of CommonVoice v7.0: **WER = 3.8%** |
|
- Czech portion of VoxPopuli: **WER = 8.8%** |
|
|
|
See our paper for details. |
|
|
|
## Paper |
|
The preprint of our paper (accepted to INTERSPEECH 2022) is available at http://arxiv.org/abs/2206.07627 |
|
|
|
## Citation |
|
If you find this model useful, please cite our paper: |
|
``` |
|
@inproceedings{wav2vec2-base-cs-80k-ClTRUS, |
|
title = {{Exploring Capabilities of Monolingual Audio Transformers using Large Datasets in Automatic Speech Recognition of Czech}}, |
|
author = { |
|
Jan Lehe\v{c}ka and |
|
Jan \v{S}vec and |
|
Ale\v{s} Pra\v{z}\'ak and |
|
Josef V. Psutka |
|
}, |
|
booktitle={Proc. Interspeech 2022}, |
|
pages={1831--1835}, |
|
year = {2022}, |
|
doi={10.21437/Interspeech.2022-10439} |
|
} |
|
``` |
|
|
|
## Related works |
|
- [Transformer-based Automatic Speech Recognition of Formal and Colloquial Czech in MALACH Project](https://arxiv.org/abs/2206.07666) |
|
- [Yehor/wav2vec2-xls-r-base-uk-with-small-lm](https://huggingface.co/Yehor/wav2vec2-xls-r-base-uk-with-small-lm) |
|
|
