metadata

language: cs
tags:
  - Czech
  - KKY
  - FAV
license: cc-by-nc-sa-4.0

wav2vec2-base-cs-50k

This is a monolingual Czech Wav2Vec 2.0 base model pre-trained from 50 thousand hours of Czech speech.

This model does not have a tokenizer as it was pretrained on audio alone. In order to use this model for speech recognition, a tokenizer should be created and the model should be fine-tuned on labeled text data.

Speech recognition results

After fine-tuning, the model scored the following results on public datasets:

Czech portion of CommonVoice v16.0: WER = 11.36%

See our paper for details.

Paper

The preprint of our paper (accepted to INTERSPEECH 2024) is available at [tbd]

All models released within the paper

https://huggingface.co/fav-kky/wav2vec2-base-cs-50k (monolingual Czech)
https://huggingface.co/fav-kky/wav2vec2-base-de-50k (monolingual German)
https://huggingface.co/fav-kky/wav2vec2-base-cs-en-100k (bilingual Czech+English)
https://huggingface.co/fav-kky/wav2vec2-base-cs-de-100k (bilingual Czech+German)
https://huggingface.co/fav-kky/wav2vec2-base-en-de-100k (bilingual English+German)
https://huggingface.co/fav-kky/wav2vec2-base-cs-en-de-150k (trilingual Czech+English+German)

Citation

If you find this model useful, please cite our paper:

tbd

Usage

Inputs must be 16kHz mono audio files.

This model can be used e.g. to extract per-frame contextual embeddings from audio:

from transformers import Wav2Vec2Model, Wav2Vec2FeatureExtractor
import torchaudio

feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("fav-kky/wav2vec2-base-cs-50k")
model = Wav2Vec2Model.from_pretrained("fav-kky/wav2vec2-base-cs-50k")

speech_array, sampling_rate = torchaudio.load("/path/to/audio/file.wav")
inputs = feature_extractor(
    speech_array, 
    sampling_rate=16_000, 
    return_tensors="pt"
)["input_values"][0]

output = model(inputs)
embeddings = output.last_hidden_state.detach().numpy()[0]

fav-kky
/

wav2vec2-base-cs-50k