|
--- |
|
language: |
|
- en |
|
- de |
|
datasets: |
|
- covost2 |
|
- librispeech_asr |
|
tags: |
|
- audio |
|
- speech-translation |
|
- automatic-speech-recognition |
|
- speech2text2 |
|
license: MIT |
|
pipeline_tag: automatic-speech-recognition |
|
widget: |
|
- label: Librispeech sample 1 |
|
src: https://cdn-media.huggingface.co/speech_samples/sample1.flac |
|
- label: Librispeech sample 2 |
|
src: https://cdn-media.huggingface.co/speech_samples/sample2.flac |
|
--- |
|
|
|
|
|
# S2T2-Wav2Vec2-CoVoST2-EN-DE-ST |
|
|
|
`s2t-wav2vec2-large-en-de` is a Speech to Text Transformer (S2T) model trained for end-to-end Speech Translation (ST). |
|
The S2T model was proposed in [Large-Scale Self- and Semi-Supervised Learning for Speech Translation](https://arxiv.org/pdf/2104.06678.pdf) and officially released in |
|
[Fairseq](https://github.com/pytorch/fairseq/blob/6f847c8654d56b4d1b1fbacec027f47419426ddb/fairseq/models/wav2vec/wav2vec2_asr.py#L266). |
|
|
|
|
|
## Model description |
|
|
|
S2T2 is a transformer-based seq2seq (speech encoder-decoder) model designed for end-to-end Automatic Speech Recognition (ASR) and Speech |
|
Translation (ST). It uses a pretrained [Wav2Vec2](https://huggingface.co/transformers/model_doc/wav2vec2.html) as the encoder and a transformer-based decoder. The model is trained with standard autoregressive cross-entropy loss and generates the translations autoregressively. |
|
|
|
## Intended uses & limitations |
|
|
|
This model can be used for end-to-end English speech to German text translation. |
|
See the [model hub](https://huggingface.co/models?filter=speech2text2) to look for other S2T2 checkpoints. |
|
|
|
|
|
### How to use |
|
|
|
As this a standard sequence to sequence transformer model, you can use the `generate` method to generate the |
|
transcripts by passing the speech features to the model. |
|
|
|
You can use the model directly via the ASR pipeline |
|
|
|
```python |
|
from datasets import load_dataset |
|
from transformers import pipeline |
|
|
|
librispeech_en = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean", split="validation") |
|
asr = pipeline("automatic-speech-recognition", model="facebook/s2t-wav2vec2-large-en-de", feature_extractor="facebook/s2t-wav2vec2-large-en-de") |
|
|
|
translation_de = asr(librispeech_en[0]["file"]) |
|
``` |
|
|
|
or step-by-step as follows: |
|
|
|
```python |
|
import torch |
|
from transformers import Speech2Text2Processor, SpeechEncoderDecoder |
|
from datasets import load_dataset |
|
|
|
import soundfile as sf |
|
model = SpeechEncoderDecoder.from_pretrained("facebook/s2t-wav2vec2-large-en-de") |
|
processor = Speech2Text2Processor.from_pretrained("facebook/s2t-wav2vec2-large-en-de") |
|
|
|
def map_to_array(batch): |
|
speech, _ = sf.read(batch["file"]) |
|
batch["speech"] = speech |
|
return batch |
|
|
|
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean", split="validation") |
|
ds = ds.map(map_to_array) |
|
|
|
inputs = processor(ds["speech"][0], sampling_rate=16_000, return_tensors="pt") |
|
generated_ids = model.generate(input_ids=inputs["input_features"], attention_mask=inputs["attention_mask"]) |
|
transcription = processor.batch_decode(generated_ids) |
|
``` |
|
|
|
## Evaluation results |
|
|
|
CoVoST-V2 test results for en-de (BLEU score): **27.2** |
|
|
|
For more information, please have a look at the [official paper](https://arxiv.org/pdf/2104.06678.pdf) |
|
|
|
### BibTeX entry and citation info |
|
|
|
```bibtex |
|
@article{DBLP:journals/corr/abs-2104-06678, |
|
author = {Changhan Wang and |
|
Anne Wu and |
|
Juan Miguel Pino and |
|
Alexei Baevski and |
|
Michael Auli and |
|
Alexis Conneau}, |
|
title = {Large-Scale Self- and Semi-Supervised Learning for Speech Translation}, |
|
journal = {CoRR}, |
|
volume = {abs/2104.06678}, |
|
year = {2021}, |
|
url = {https://arxiv.org/abs/2104.06678}, |
|
archivePrefix = {arXiv}, |
|
eprint = {2104.06678}, |
|
timestamp = {Thu, 12 Aug 2021 15:37:06 +0200}, |
|
biburl = {https://dblp.org/rec/journals/corr/abs-2104-06678.bib}, |
|
bibsource = {dblp computer science bibliography, https://dblp.org} |
|
} |
|
|
|
``` |
|
|
