tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, dataset:timit_asr, transformers, timit_asr, generated_from_trainer, model-index


# Shared Roberta2Roberta Summarization with 🤗 EncoderDecoder Framework


This model is a shared Roberta2Roberta model, meaning that the encoder and decoder weights are tied, fine-tuned on summarization.


Roberta2Roberta is a `EncoderDecoderModel`, meaning that both the encoder and the decoder are `roberta-base`
RoBERTa models. In this setup the encoder and decoder weights are tied. Leveraging the [EncoderDecoderFramework](https://huggingface.co/transformers/model_doc/encoderdecoder.html#encoder-decoder-models), the
two pretrained models can simply be loaded into the framework via:


```python
roberta2roberta = EncoderDecoderModel.from_encoder_decoder_pretrained("roberta-base", "roberta-base", tie_encoder_decoder=True)
```


The decoder of an `EncoderDecoder` model needs cross-attention layers and usually makes use of causal
masking for auto-regressiv generation.
Thus, `roberta2roberta` is consequently fined-tuned on the `CNN/Daily Mail`dataset and the resulting model
`roberta2roberta-share-cnn_dailymail-fp16` is uploaded here.


## Example


The model is by no means a state-of-the-art model, but nevertheless
produces reasonable summarization results. It was mainly fine-tuned
as a proof-of-concept for the 🤗 EncoderDecoder Framework.


The model can be used as follows:


```python
from transformers import RobertaTokenizer, EncoderDecoderModel

model = EncoderDecoderModel.from_pretrained("patrickvonplaten/roberta2roberta-share-cnn_dailymail-fp16")
tokenizer = RobertaTokenizer.from_pretrained("roberta-base")

article = """(CNN)Sigma Alpha Epsilon is under fire for a video showing party-bound fraternity members singing a racist chant. SAE's national chapter suspended the students, but University of Oklahoma President David B
oren took it a step further, saying the university's affiliation with the fraternity is permanently done. The news is shocking, but it's not the first time SAE has faced controversy. SAE was founded March 9, 185
6, at the University of Alabama, five years before the American Civil War, according to the fraternity website. When the war began, the group had fewer than 400 members, of which "369 went to war for the Confede
rate States and seven for the Union Army," the website says. The fraternity now boasts more than 200,000 living alumni, along with about 15,000 undergraduates populating 219 chapters and 20 "colonies" seeking fu
ll membership at universities. SAE has had to work hard to change recently after a string of member deaths, many blamed on the hazing of new recruits, SAE national President Bradley Cohen wrote in a message on t
he fraternity's website. The fraternity's website lists more than 130 chapters cited or suspended for "health and safety incidents" since 2010. At least 30 of the incidents involved hazing, and dozens more invol
ved alcohol. However, the list is missing numerous incidents from recent months. Among them, according to various media outlets: Yale University banned the SAEs from campus activities last month after members al
legedly tried to interfere with a sexual misconduct investigation connected to an initiation rite. Stanford University in December suspended SAE housing privileges after finding sorority members attending a frat
ernity function were subjected to graphic sexual content. And Johns Hopkins University in November suspended the fraternity for underage drinking. "The media has labeled us as the 'nation's deadliest fraternity,
' " Cohen said. In 2011, for example, a student died while being coerced into excessive alcohol consumption, according to a lawsuit. SAE's previous insurer dumped the fraternity. "As a result, we are paying Lloy
d's of London the highest insurance rates in the Greek-letter world," Cohen said. Universities have turned down SAE's attempts to open new chapters, and the fraternity had to close 12 in 18 months over hazing in
cidents."""

input_ids = tokenizer(article, return_tensors="pt").input_ids
output_ids = model.generate(input_ids)

print(tokenizer.decode(output_ids[0], skip_special_tokens=True))
# should produce
# SAE's national chapter suspended after video shows party-bound fraternity members singing racist chant. University of Oklahoma president says university's affiliation with fraternity is permanently done.
# SAE has had to close 12 chapters since 2010 after members were killed in hazing. The fraternity has had more than 130 chapters in 18 months.
```


## Training script:


**IMPORTANT**: In order for this code to work, make sure you checkout to the branch
[more_general_trainer_metric](https://github.com/huggingface/transformers/tree/more_general_trainer_metric), which slightly adapts
the `Trainer` for `EncoderDecoderModels` according to this PR: https://github.com/huggingface/transformers/pull/5840.


The following code shows the complete training script that was used to fine-tune `roberta2roberta-cnn_dailymail-fp16 ` for reproducability. The training last ~9h on a standard GPU.


```python
#!/usr/bin/env python3
import nlp
import logging
from transformers import RobertaTokenizer, EncoderDecoderModel, Trainer, TrainingArguments

logging.basicConfig(level=logging.INFO)

model = EncoderDecoderModel.from_encoder_decoder_pretrained("roberta-base", "roberta-base", tie_encoder_decoder=True)
tokenizer =  RobertaTokenizer.from_pretrained("roberta-base")

# load train and validation data
train_dataset = nlp.load_dataset("cnn_dailymail", "3.0.0", split="train")
val_dataset = nlp.load_dataset("cnn_dailymail", "3.0.0", split="validation[:5%]")

# load rouge for validation
rouge = nlp.load_metric("rouge", experiment_id=0)

# set decoding params
model.config.decoder_start_token_id = tokenizer.bos_token_id
model.config.eos_token_id = tokenizer.eos_token_id
model.config.max_length = 142
model.config.min_length = 56
model.config.no_repeat_ngram_size = 3
model.early_stopping = True
model.length_penalty = 2.0
model.num_beams = 4

encoder_length = 512
decoder_length = 128
batch_size = 16

# map data correctly
def map_to_encoder_decoder_inputs(batch):
    # Tokenizer will automatically set [BOS] <text> [EOS]
    # cut off at Longformer at 2048
    inputs = tokenizer(batch["article"], padding="max_length", truncation=True, max_length=encoder_length)
    # force summarization <= 256
    outputs = tokenizer(batch["highlights"], padding="max_length", truncation=True, max_length=decoder_length)

    batch["input_ids"] = inputs.input_ids
    batch["attention_mask"] = inputs.attention_mask
    batch["decoder_input_ids"] = outputs.input_ids
    batch["labels"] = outputs.input_ids.copy()
    # mask loss for padding
    batch["labels"] = [
        [-100 if token == tokenizer.pad_token_id else token for token in labels] for labels in batch["labels"]
    ]
    batch["decoder_attention_mask"] = outputs.attention_mask

    assert all([len(x) == encoder_length for x in inputs.input_ids])
    assert all([len(x) == decoder_length for x in outputs.input_ids])

    return batch

def compute_metrics(pred):
    labels_ids = pred.label_ids
    pred_ids = pred.predictions

    # all unnecessary tokens are removed
    pred_str = tokenizer.batch_decode(pred_ids, skip_special_tokens=True)
    labels_ids[labels_ids == -100] = tokenizer.eos_token_id
    label_str = tokenizer.batch_decode(labels_ids, skip_special_tokens=True)

    rouge_output = rouge.compute(predictions=pred_str, references=label_str, rouge_types=["rouge2"])["rouge2"].mid

    return {
        "rouge2_precision": round(rouge_output.precision, 4),
        "rouge2_recall": round(rouge_output.recall, 4),
        "rouge2_fmeasure": round(rouge_output.fmeasure, 4),
    }

# make train dataset ready
train_dataset = train_dataset.map(
    map_to_encoder_decoder_inputs, batched=True, batch_size=batch_size, remove_columns=["article", "highlights"],
)
train_dataset.set_format(
    type="torch", columns=["input_ids", "attention_mask", "decoder_attention_mask", "decoder_input_ids", "labels"],
)

# same for validation dataset
val_dataset = val_dataset.map(
    map_to_encoder_decoder_inputs, batched=True, batch_size=batch_size, remove_columns=["article", "highlights"],
)
val_dataset.set_format(
    type="torch", columns=["input_ids", "decoder_attention_mask", "attention_mask", "decoder_input_ids", "labels"],
)

# set training arguments - these params are not really tuned, feel free to change
training_args = TrainingArguments(
    output_dir="./",
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    predict_from_generate=True,
    evaluate_during_training=True,
    do_train=True,
    do_eval=True,
    logging_steps=1000,
    save_steps=1000,
    eval_steps=1000,
    overwrite_output_dir=True,
    warmup_steps=2000,
    save_total_limit=3,
    fp16=True,
)

# instantiate trainer
trainer = Trainer(
    model=model,
    args=training_args,
    compute_metrics=compute_metrics,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
)

# start training
trainer.train()
```


## Evaluation


The following script evaluates the model on the test set of
CNN/Daily Mail.


```python
#!/usr/bin/env python3
import nlp
from transformers import RobertaTokenizer, EncoderDecoderModel

tokenizer = RobertaTokenizer.from_pretrained("roberta-base")
model = EncoderDecoderModel.from_pretrained("patrickvonplaten/roberta2roberta-share-cnn_dailymail-fp16")
model.to("cuda")

test_dataset = nlp.load_dataset("cnn_dailymail", "3.0.0", split="test")
batch_size = 128

# map data correctly
def generate_summary(batch):
    # Tokenizer will automatically set [BOS] <text> [EOS]
    # cut off at BERT max length 512
    inputs = tokenizer(batch["article"], padding="max_length", truncation=True, max_length=512, return_tensors="pt")
    input_ids = inputs.input_ids.to("cuda")
    attention_mask = inputs.attention_mask.to("cuda")

    outputs = model.generate(input_ids, attention_mask=attention_mask)

    # all special tokens including will be removed
    output_str = tokenizer.batch_decode(outputs, skip_special_tokens=True)

    batch["pred"] = output_str

    return batch

results = test_dataset.map(generate_summary, batched=True, batch_size=batch_size, remove_columns=["article"])

# load rouge for validation
rouge = nlp.load_metric("rouge")

pred_str = results["pred"]
label_str = results["highlights"]

rouge_output = rouge.compute(predictions=pred_str, references=label_str, rouge_types=["rouge2"])["rouge2"].mid

print(rouge_output)
```


The obtained results should be:


| -   |      Rouge2 - mid -precision      |  Rouge2 - mid - recall | Rouge2 - mid - fmeasure |
|----------|:-------------:|:------:|:------:|
| **CNN/Daily Mail** |  15.6 | 18.79 | **16.59** |


model name: roberta2roberta-share-cnn_dailymail-fp16


model author: patrickvonplaten


task: text2text-generation


tags: pytorch, encoder_decoder, text2text-generation, transformers, autonlp_compatible


license: apache-2.0
tags:


- automatic-speech-recognition
- multilingual_librispeech
- generated_from_trainer
datasets:
- multilingual_librispeech
model-index:
- name: wav2vec2-xlsr-53-300m-mls-german-ft
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# wav2vec2-xlsr-53-300m-mls-german-ft


This model is a fine-tuned version of [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on the MULTILINGUAL_LIBRISPEECH - GERMAN 10h dataset.
It achieves the following results on the evaluation set:


- Loss: 0.2219
- Wer: 0.1288


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0001
- train_batch_size: 32
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 200.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch  | Step  | Validation Loss | Wer    |
|:-------------:|:------:|:-----:|:---------------:|:------:|
| 2.9888        | 7.25   | 500   | 2.9192          | 1.0    |
| 2.9313        | 14.49  | 1000  | 2.8698          | 1.0    |
| 1.068         | 21.74  | 1500  | 0.2647          | 0.2565 |
| 0.8151        | 28.99  | 2000  | 0.2067          | 0.1719 |
| 0.764         | 36.23  | 2500  | 0.1975          | 0.1568 |
| 0.7332        | 43.48  | 3000  | 0.1812          | 0.1463 |
| 0.5952        | 50.72  | 3500  | 0.1923          | 0.1428 |
| 0.6655        | 57.97  | 4000  | 0.1900          | 0.1404 |
| 0.574         | 65.22  | 4500  | 0.1822          | 0.1370 |
| 0.6211        | 72.46  | 5000  | 0.1937          | 0.1355 |
| 0.5883        | 79.71  | 5500  | 0.1872          | 0.1335 |
| 0.5666        | 86.96  | 6000  | 0.1874          | 0.1324 |
| 0.5526        | 94.2   | 6500  | 0.1998          | 0.1368 |
| 0.5671        | 101.45 | 7000  | 0.2054          | 0.1365 |
| 0.5514        | 108.7  | 7500  | 0.1987          | 0.1340 |
| 0.5382        | 115.94 | 8000  | 0.2104          | 0.1344 |
| 0.5819        | 123.19 | 8500  | 0.2125          | 0.1334 |
| 0.5277        | 130.43 | 9000  | 0.2063          | 0.1330 |
| 0.4626        | 137.68 | 9500  | 0.2105          | 0.1310 |
| 0.5842        | 144.93 | 10000 | 0.2087          | 0.1307 |
| 0.535         | 152.17 | 10500 | 0.2137          | 0.1309 |
| 0.5081        | 159.42 | 11000 | 0.2215          | 0.1302 |
| 0.6033        | 166.67 | 11500 | 0.2162          | 0.1302 |
| 0.5549        | 173.91 | 12000 | 0.2198          | 0.1286 |
| 0.5389        | 181.16 | 12500 | 0.2241          | 0.1293 |
| 0.4912        | 188.41 | 13000 | 0.2190          | 0.1290 |
| 0.4671        | 195.65 | 13500 | 0.2218          | 0.1290 |


### Framework versions


- Transformers 4.13.0.dev0
- Pytorch 1.10.0
- Datasets 1.15.2.dev0
- Tokenizers 0.10.3


model name: wav2vec2-xlsr-53-300m-mls-german-ft


model author: patrickvonplaten


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, dataset:multilingual_librispeech, transformers, license:apache-2.0, multilingual_librispeech, generated_from_trainer, model-index


model name: wav2vec2-large-repro-960h-libri-120k-steps


tags: pytorch, wav2vec2, pretraining, transformers


language:


- tr
tags:
- automatic-speech-recognition
- common_voice
- generated_from_trainer
datasets:
- common_voice
model-index:
- name: wav2vec2-xlarge-...-common_voice-tr-demo
results: []


# wav2vec2-xlarge-...-common_voice-tr-demo


This model is a fine-tuned version of [facebook/wav2vec2-xlarge-xlsr-...](https://huggingface.co/facebook/wav2vec2-xlarge-xlsr-...) on the COMMON_VOICE - TR dataset.
It achieves the following results on the evaluation set:


- Loss: 0.2701
- Wer: 0.2309
- Cer: 0.0527


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 0.00005
- train_batch_size: 2
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- num_devices: 2
- gradient_accumulation_steps: 8
- total_train_batch_size: 32
- total_eval_batch_size: 16
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 30.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 0.4388        | 3.7   | 400  | 1.366          | 0.9701 |
| 0.3766        | 7.4   | 800  | 0.4914          | 0.5374 |
| 0.2295        | 11.11 | 1200 | 0.3934          | 0.4125 |
| 0.1121        | 14.81 | 1600 | 0.3264          | 0.2904 |
| 0.1473        | 18.51 | 2000 | 0.3103          | 0.2671 |
| 0.1013        | 22.22 | 2400 | 0.2589          | 0.2324 |
| 0.0704        | 25.92 | 2800 | 0.2826          | 0.2339 |
| 0.0537        | 29.63 | 3200 | 0.2704          | 0.2309 |


### Framework versions


- Transformers 4.12.0.dev0
- Pytorch 1.8.1
- Datasets 1.14.1.dev0
- Tokenizers 0.10.3


model name: wav2vec2-xlarge-dotdotdot-common_voice-tr-demo


model author: patrickvonplaten


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, tr, dataset:common_voice, transformers, common_voice, generated_from_trainer, model-index


license: apache-2.0
tags:


- automatic-speech-recognition
- multilingual_librispeech
- generated_from_trainer
datasets:
- multilingual_librispeech
model-index:
- name: wav2vec2-100m-mls-german-ft-2
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# wav2vec2-100m-mls-german-ft-2


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-100m](https://huggingface.co/facebook/wav2vec2-xls-r-100m) on the MULTILINGUAL_LIBRISPEECH - GERMAN dataset.
It achieves the following results on the evaluation set:


- Loss: 2.9304
- Wer: 1.0


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0005
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- total_train_batch_size: 64
- total_eval_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 100.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer |
|:-------------:|:-----:|:----:|:---------------:|:---:|
| 2.9545        | 14.29 | 500  | 2.9354          | 1.0 |
| 2.9537        | 28.57 | 1000 | 2.9359          | 1.0 |
| 2.9602        | 42.86 | 1500 | 2.9302          | 1.0 |
| 2.9586        | 57.14 | 2000 | 2.9298          | 1.0 |
| 2.9331        | 71.43 | 2500 | 2.9314          | 1.0 |
| 2.9321        | 85.71 | 3000 | 2.9304          | 1.0 |
| 2.9652        | 100.0 | 3500 | 2.9304          | 1.0 |


### Framework versions


- Transformers 4.13.0.dev0
- Pytorch 1.9.0+cu111
- Datasets 1.15.2.dev0
- Tokenizers 0.10.3


model name: wav2vec2-100m-mls-german-ft-2


model author: patrickvonplaten


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, dataset:multilingual_librispeech, transformers, license:apache-2.0, multilingual_librispeech, generated_from_trainer, model-index


This is an example repo for a possible integration with TensorSpeech, see [here](https://github.com/TensorSpeech/TensorFlowTTS/pull/555).


model name: tf_tts_fast_speech_2


tags:


- automatic-speech-recognition
- librispeech_asr
- generated_from_trainer
- wavlm_libri_finetune
model-index:
- name: wavlm-librispeech-clean-100h-dist
results: []


# wavlm-libri-clean-100h-large


This model is a fine-tuned version of [microsoft/wavlm-large](https://huggingface.co/microsoft/wavlm-large) on the LIBRISPEECH_ASR - CLEAN dataset.
It achieves the following results on the evaluation set:


- Loss: 0.0601
- Wer: 0.0491


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 0.0003
- train_batch_size: 2
- eval_batch_size: 2
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- total_eval_batch_size: 16
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 3.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 0.8069        | 0.34  | 300  | 0.7510          | 0.5809 |
| 0.2483        | 0.67  | 600  | 0.2023          | 0.1929 |
| 0.1033        | 1.01  | 900  | 0.1123          | 0.1028 |
| 0.0742        | 1.35  | 1200 | 0.0858          | 0.0771 |
| 0.057         | 1.68  | 1500 | 0.0722          | 0.0663 |
| 0.0421        | 2.02  | 1800 | 0.0682          | 0.0582 |
| 0.0839        | 2.35  | 2100 | 0.0630          | 0.0534 |
| 0.0307        | 2.69  | 2400 | 0.0603          | 0.0508 |


### Framework versions


- Transformers 4.15.0.dev0
- Pytorch 1.9.0+cu111
- Datasets 1.16.2.dev0
- Tokenizers 0.10.3


model name: wavlm-libri-clean-100h-large


model author: patrickvonplaten


task: automatic-speech-recognition


tags: pytorch, tensorboard, wavlm, automatic-speech-recognition, transformers, librispeech_asr, generated_from_trainer, wavlm_libri_finetune, model-index


language: en
datasets:


- librispeech_asr
tags:
- speech
license: apache-2.0


# Wav2Vec2-Base


[Facebook's Wav2Vec2](https://ai.facebook.com/blog/wav2vec-20-learning-the-structure-of-speech-from-raw-audio/)


The base model pretrained on 16kHz sampled speech audio. When using the model make sure that your speech input is also sampled at 16Khz. Note that this model should be fine-tuned on a downstream task, like Automatic Speech Recognition. Check out [this blog](https://huggingface.co/blog/fine-tune-wav2vec2-english) for more information.


Authors: Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, Michael Auli


**Abstract**
We show for the first time that learning powerful representations from speech audio alone followed by fine-tuning on transcribed speech can outperform the best semi-supervised methods while being conceptually simpler. wav2vec 2.0 masks the speech input in the latent space and solves a contrastive task defined over a quantization of the latent representations which are jointly learned. Experiments using all labeled data of Librispeech achieve 1.8/3.3 WER on the clean/other test sets. When lowering the amount of labeled data to one hour, wav2vec 2.0 outperforms the previous state of the art on the 100 hour subset while using 100 times less labeled data. Using just ten minutes of labeled data and pre-training on 53k hours of unlabeled data still achieves 4.8/8.2 WER. This demonstrates the feasibility of speech recognition with limited amounts of labeled data.
The original model can be found under https://github.com/pytorch/fairseq/tree/master/examples/wav2vec#wav2vec-20.


# Usage


See [this notebook](https://colab.research.google.com/drive/1FjTsqbYKphl9kL-eILgUc-bl4zVThL8F?usp=sharing) for more information on how to fine-tune the model.


model name: wav2vec2-base


tags: pytorch, wav2vec2, pretraining, en, dataset:librispeech_asr, arxiv:2006.11477, transformers, speech, license:apache-2.0


tags:


- automatic-speech-recognition
- timit_asr
- generated_from_trainer
datasets:
- timit_asr
model-index:
- name: unispeech-sat-large-timit-ft
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# unispeech-sat-large-timit-ft


This model is a fine-tuned version of [microsoft/unispeech-sat-large](https://huggingface.co/microsoft/unispeech-sat-large) on the TIMIT_ASR - NA dataset.
It achieves the following results on the evaluation set:


- Loss: 0.6074
- Wer: 0.3880


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0001
- train_batch_size: 32
- eval_batch_size: 1
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 20.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 6.2516        | 0.69  | 100  | 5.8638          | 1.0    |
| 2.9596        | 1.38  | 200  | 2.9550          | 1.0    |
| 2.8831        | 2.07  | 300  | 2.8547          | 1.0    |
| 2.3223        | 2.76  | 400  | 2.2044          | 1.0063 |
| 1.2104        | 3.45  | 500  | 1.0845          | 0.7706 |
| 0.6779        | 4.14  | 600  | 0.7342          | 0.5663 |
| 0.6319        | 4.83  | 700  | 0.6054          | 0.4881 |
| 0.664         | 5.52  | 800  | 0.5808          | 0.4913 |
| 0.402         | 6.21  | 900  | 0.5647          | 0.4611 |
| 0.3176        | 6.9   | 1000 | 0.5211          | 0.4440 |
| 0.3392        | 7.59  | 1100 | 0.5187          | 0.4359 |
| 0.3888        | 8.28  | 1200 | 0.5501          | 0.4391 |
| 0.2874        | 8.97  | 1300 | 0.5249          | 0.4148 |
| 0.208         | 9.66  | 1400 | 0.5407          | 0.4152 |
| 0.1457        | 10.34 | 1500 | 0.5722          | 0.4155 |
| 0.2375        | 11.03 | 1600 | 0.5780          | 0.4059 |
| 0.2111        | 11.72 | 1700 | 0.5823          | 0.4094 |
| 0.1422        | 12.41 | 1800 | 0.5754          | 0.3977 |
| 0.125         | 13.1  | 1900 | 0.5784          | 0.4031 |
| 0.1996        | 13.79 | 2000 | 0.5630          | 0.3956 |
| 0.1747        | 14.48 | 2100 | 0.5880          | 0.3964 |
| 0.1263        | 15.17 | 2200 | 0.5987          | 0.3951 |
| 0.11          | 15.86 | 2300 | 0.5688          | 0.3964 |
| 0.1411        | 16.55 | 2400 | 0.6223          | 0.3906 |
| 0.1647        | 17.24 | 2500 | 0.6135          | 0.3960 |
| 0.1162        | 17.93 | 2600 | 0.6224          | 0.3960 |
| 0.098         | 18.62 | 2700 | 0.6017          | 0.3907 |
| 0.1183        | 19.31 | 2800 | 0.6121          | 0.3885 |
| 0.1717        | 20.0  | 2900 | 0.6074          | 0.3880 |


### Framework versions


- Transformers 4.12.0.dev0
- Pytorch 1.8.1
- Datasets 1.14.1.dev0
- Tokenizers 0.10.3


model name: unispeech-sat-large-timit-ft


model author: patrickvonplaten


task: automatic-speech-recognition


tags: pytorch, tensorboard, unispeech-sat, automatic-speech-recognition, dataset:timit_asr, transformers, timit_asr, generated_from_trainer, model-index


language:


- tr
license: apache-2.0
tags:
- speech-recognition
- common_voice
- generated_from_trainer
datasets:
- common_voice
model-index:
- name: wav2vec2-common_voice-tr-demo
results: []


# wav2vec2-common_voice-tr-demo


This model is a fine-tuned version of [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on the COMMON_VOICE - TR dataset.
It achieves the following results on the evaluation set:


- Loss: 0.3856
- Wer: 0.3556


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 0.0003
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 15.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 3.7391        | 0.92  | 100  | 3.5760          | 1.0    |
| 2.927         | 1.83  | 200  | 3.0796          | 0.9999 |
| 0.9009        | 2.75  | 300  | 0.9278          | 0.8226 |
| 0.6529        | 3.67  | 400  | 0.5926          | 0.6367 |
| 0.3623        | 4.59  | 500  | 0.5372          | 0.5692 |
| 0.2888        | 5.5   | 600  | 0.4407          | 0.4838 |
| 0.285         | 6.42  | 700  | 0.4341          | 0.4694 |
| 0.0842        | 7.34  | 800  | 0.4153          | 0.4302 |
| 0.1415        | 8.26  | 900  | 0.4317          | 0.4136 |
| 0.1552        | 9.17  | 1000 | 0.4145          | 0.4013 |
| 0.1184        | 10.09 | 1100 | 0.4115          | 0.3844 |
| 0.0556        | 11.01 | 1200 | 0.4182          | 0.3862 |
| 0.0851        | 11.93 | 1300 | 0.3985          | 0.3688 |
| 0.0961        | 12.84 | 1400 | 0.4030          | 0.3665 |
| 0.0596        | 13.76 | 1500 | 0.3880          | 0.3631 |
| 0.0917        | 14.68 | 1600 | 0.3878          | 0.3582 |


### Framework versions


- Transformers 4.11.0.dev0
- Pytorch 1.9.0+cu111
- Datasets 1.12.1
- Tokenizers 0.10.3


model name: wav2vec2-common_voice-tr-demo


model author: patrickvonplaten


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, tr, dataset:common_voice, transformers, license:apache-2.0, speech-recognition, common_voice, generated_from_trainer, model-index


language: en
datasets:


- scientific_papers


## license: apache-2.0


## Introduction


[Allenai's Longformer Encoder-Decoder (LED)](https://github.com/allenai/longformer#longformer).


This is an unofficial *led-large-16384* checkpoint that is fine-tuned on the [pubmed dataset](https://huggingface.co/datasets/scientific_papers).


The model was fine-tuned and evaluated as detailed in [this notebook](https://colab.research.google.com/drive/12LjJazBl7Gam0XBPy_y0CTOJZeZ34c2v?usp=sharing)


## Results


The model achieves a **Rouge-2** score of 19.33 on Pubmed which is competitive to state-of-the-art models.


## Usage


The model can be used as follows. The input is taken from the test data of the [pubmed dataset](https://huggingface.co/datasets/scientific_papers).


```python
LONG_ARTICLE = """"anxiety affects quality of life in those living
with parkinson 's disease ( pd ) more so than
overall cognitive status , motor deficits , apathy
, and depression [ 13 ] . although anxiety and
depression are often related and coexist in pd
patients , recent research suggests that anxiety
rather than depression is the most prominent and
prevalent mood disorder in pd [ 5 , 6 ] . yet ,
our current understanding of anxiety and its
impact on cognition in pd , as well as its neural
basis and best treatment practices , remains
meager and lags far behind that of depression .
overall , neuropsychiatric symptoms in pd have
been shown to be negatively associated with
cognitive performance . for example , higher
depression scores have been correlated with lower
scores on the mini - mental state exam ( mmse ) [
8 , 9 ] as well as tests of memory and executive
functions ( e.g. , attention ) [ 1014 ] . likewise
, apathy and anhedonia in pd patients have been
associated with executive dysfunction [ 10 , 1523
] . however , few studies have specifically
investigated the relationship between anxiety and
cognition in pd . one study showed a strong
negative relationship between anxiety ( both state
and trait ) and overall cognitive performance (
measured by the total of the repeatable battery
for the assessment of neuropsychological status
index ) within a sample of 27 pd patients .
furthermore , trait anxiety was negatively
associated with each of the cognitive domains
assessed by the rbans ( i.e. , immediate memory ,
visuospatial construction , language , attention ,
and delayed memory ) . two further studies have
examined whether anxiety differentially affects
cognition in patients with left - sided dominant
pd ( lpd ) versus right - sided dominant pd ( rpd
) ; however , their findings were inconsistent .
the first study found that working memory
performance was worse in lpd patients with anxiety
compared to rpd patients with anxiety , whereas
the second study reported that , in lpd , apathy
but not anxiety was associated with performance on
nonverbally mediated executive functions and
visuospatial tasks ( e.g. , tmt - b , wms - iii
spatial span ) , while in rpd , anxiety but not
apathy significantly correlated with performance
on verbally mediated tasks ( e.g. , clock reading
test and boston naming test ) . furthermore ,
anxiety was significantly correlated with
neuropsychological measures of attention and
executive and visuospatial functions . taken
together , it is evident that there are limited
and inconsistent findings describing the
relationship between anxiety and cognition in pd
and more specifically how anxiety might influence
particular domains of cognition such as attention
and memory and executive functioning . it is also
striking that , to date , no study has examined
the influence of anxiety on cognition in pd by
directly comparing groups of pd patients with and
without anxiety while excluding depression . given
that research on healthy young adults suggests
that anxiety reduces processing capacity and
impairs processing efficiency , especially in the
central executive and attentional systems of
working memory [ 26 , 27 ] , we hypothesized that
pd patients with anxiety would show impairments in
attentional set - shifting and working memory
compared to pd patients without anxiety .
furthermore , since previous work , albeit limited
, has focused on the influence of symptom
laterality on anxiety and cognition , we also
explored this relationship . seventeen pd patients
with anxiety and thirty - three pd patients
without anxiety were included in this study ( see
table 1 ) . the cross - sectional data from these
participants was taken from a patient database
that has been compiled over the past 8 years (
since 2008 ) at the parkinson 's disease research
clinic at the brain and mind centre , university
of sydney . inclusion criteria involved a
diagnosis of idiopathic pd according to the united
kingdom parkinson 's disease society brain bank
criteria   and were confirmed by a neurologist (
sjgl ) . patients also had to have an adequate
proficiency in english and have completed a full
neuropsychological assessment . ten patients in
this study ( 5 pd with anxiety ; 5 pd without
anxiety ) were taking psychotropic drugs ( i.e. ,
benzodiazepine or selective serotonin reuptake
inhibitor ) . patients were also excluded if they
had other neurological disorders , psychiatric
disorders other than affective disorders ( such as
anxiety ) , or if they reported a score greater
than six on the depression subscale of the
hospital anxiety and depression scale ( hads ) .
thus , all participants who scored within a
depressed  ( hads - d > 6 ) range were excluded
from this study , in attempt to examine a refined
sample of pd patients with and without anxiety in
order to determine the independent effect of
anxiety on cognition . this research was approved
by the human research ethics committee of the
university of sydney , and written informed
consent was obtained from all participants . self
- reported hads was used to assess anxiety in pd
and has been previously shown to be a useful
measure of clinical anxiety in pd . a cut - off
score of > 8 on the anxiety subscale of the hads (
hads - a ) was used to identify pd cases with
anxiety ( pda+ ) , while a cut - off score of < 6
on the hads - a was used to identify pd cases
without anxiety ( pda ) . this criterion was more
stringent than usual ( > 7 cut - off score ) , in
effort to create distinct patient groups . the
neurological evaluation rated participants
according to hoehn and yahr ( h&y ) stages   and
assessed their motor symptoms using part iii of
the revised mds task force unified parkinson 's
disease rating scale ( updrs ) . in a similar way
this was determined by calculating a total left
and right score from rigidity items 3035 ,
voluntary movement items 3643 , and tremor items
5057 from the mds - updrs part iii ( see table 1 )
. processing speed was assessed using the trail
making test , part a ( tmt - a , z - score ) .
attentional set - shifting was measured using the
trail making test , part b ( tmt - b , z - score )
. working memory was assessed using the digit span
forward and backward subtest of the wechsler
memory scale - iii ( raw scores ) . language was
assessed with semantic and phonemic verbal fluency
via the controlled oral word associated test (
cowat animals and letters , z - score ) . the
ability to retain learned verbal memory was
assessed using the logical memory subtest from the
wechsler memory scale - iii ( lm - i z - score ,
lm - ii z - score , % lm retention z - score ) .
the mini - mental state examination ( mmse )
demographic , clinical , and neuropsychological
variables were compared between the two groups
with the independent t - test or mann  whitney u
test , depending on whether the variable met
parametric assumptions . chi - square tests were
used to examine gender and symptom laterality
differences between groups . all analyses employed
an alpha level of p < 0.05 and were two - tailed .
spearman correlations were performed separately in
each group to examine associations between anxiety
and/or depression ratings and cognitive functions
. as expected , the pda+ group reported
significant greater levels of anxiety on the hads
- a ( u = 0 , p < 0.001 ) and higher total score
on the hads ( u = 1 , p < 0.001 ) compared to the
pda group ( table 1 ) . groups were matched in age
( t(48 ) = 1.31 , p = 0.20 ) , disease duration (
u = 259 , p = 0.66 ) , updrs - iii score ( u =
250.5 , p = 0.65 ) , h&y ( u = 245 , p = 0.43 ) ,
ledd ( u = 159.5 , p = 0.80 ) , and depression (
hads - d ) ( u = 190.5 , p = 0.06 ) . additionally
, all groups were matched in the distribution of
gender (  = 0.098 , p = 0.75 ) and side - affected
(  = 0.765 , p = 0.38 ) . there were no group
differences for tmt - a performance ( u = 256 , p
= 0.62 ) ( table 2 ) ; however , the pda+ group
had worse performance on the trail making test
part b ( t(46 ) = 2.03 , p = 0.048 ) compared to
the pda group ( figure 1 ) . the pda+ group also
demonstrated significantly worse performance on
the digit span forward subtest ( t(48 ) = 2.22 , p
= 0.031 ) and backward subtest ( u = 190.5 , p =
0.016 ) compared to the pda group ( figures 2(a )
and 2(b ) ) . neither semantic verbal fluency (
t(47 ) = 0.70 , p = 0.49 ) nor phonemic verbal
fluency ( t(47 ) = 0.39 , p = 0.70 ) differed
between groups . logical memory i immediate recall
test ( u = 176 , p = 0.059 ) showed a trend that
the pda+ group had worse new verbal learning and
immediate recall abilities than the pda group .
however , logical memory ii test performance ( u =
219 , p = 0.204 ) and logical memory % retention (
u = 242.5 , p = 0.434 ) did not differ between
groups . there were also no differences between
groups in global cognition ( mmse ) ( u = 222.5 ,
p = 0.23 ) . participants were split into lpd and
rpd , and then further group differences were
examined between pda+ and pda. importantly , the
groups remained matched in age , disease duration
, updrs - iii , dde , h&y stage , and depression
but remained significantly different on self -
reported anxiety . lpda+ demonstrated worse
performance on the digit span forward test ( t(19
) = 2.29 , p = 0.033 ) compared to lpda , whereas
rpda+ demonstrated worse performance on the digit
span backward test ( u = 36.5 , p = 0.006 ) , lm -
i immediate recall ( u = 37.5 , p = 0.008 ) , and
lm - ii ( u = 45.0 , p = 0.021 ) but not lm %
retention ( u = 75.5 , p = 0.39 ) compared to
rpda. this study is the first to directly compare
cognition between pd patients with and without
anxiety . the findings confirmed our hypothesis
that anxiety negatively influences attentional set
- shifting and working memory in pd . more
specifically , we found that pd patients with
anxiety were more impaired on the trail making
test part b which assessed attentional set -
shifting , on both digit span tests which assessed
working memory and attention , and to a lesser
extent on the logical memory test which assessed
memory and new verbal learning compared to pd
patients without anxiety . taken together , these
findings suggest that anxiety in pd may reduce
processing capacity and impair processing
efficiency , especially in the central executive
and attentional systems of working memory in a
similar way as seen in young healthy adults [ 26 ,
27 ] . although the neurobiology of anxiety in pd
remains unknown , many researchers have postulated
that anxiety disorders are related to
neurochemical changes that occur during the early
, premotor stages of pd - related degeneration [
37 , 38 ] such as nigrostriatal dopamine depletion
, as well as cell loss within serotonergic and
noradrenergic brainstem nuclei ( i.e. , raphe
nuclei and locus coeruleus , resp . , which
provide massive inputs to corticolimbic regions )
. over time , chronic dysregulation of
adrenocortical and catecholamine functions can
lead to hippocampal damage as well as
dysfunctional prefrontal neural circuitries [ 39 ,
40 ] , which play a key role in memory and
attention . recent functional neuroimaging work
has suggested that enhanced hippocampal activation
during executive functioning and working memory
tasks may represent compensatory processes for
impaired frontostriatal functions in pd patients
compared to controls . therefore , chronic stress
from anxiety , for example , may disrupt
compensatory processes in pd patients and explain
the cognitive impairments specifically in working
memory and attention seen in pd patients with
anxiety . it has also been suggested that
hyperactivation within the putamen may reflect a
compensatory striatal mechanism to maintain normal
working memory performance in pd patients ;
however , losing this compensatory activation has
been shown to contribute to poor working memory
performance . anxiety in mild pd has been linked
to reduced putamen dopamine uptake which becomes
more extensive as the disease progresses . this
further supports the notion that anxiety may
disrupt compensatory striatal mechanisms as well ,
providing another possible explanation for the
cognitive impairments observed in pd patients with
anxiety in this study . noradrenergic and
serotonergic systems should also be considered
when trying to explain the mechanisms by which
anxiety may influence cognition in pd . although
these neurotransmitter systems are relatively
understudied in pd cognition , treating the
noradrenergic and serotonergic systems has shown
beneficial effects on cognition in pd . selective
serotonin reuptake inhibitor , citalopram , was
shown to improve response inhibition deficits in
pd , while noradrenaline reuptake blocker ,
atomoxetine , has been recently reported to have
promising effects on cognition in pd [ 45 , 46 ] .
overall , very few neuroimaging studies have been
conducted in pd in order to understand the neural
correlates of pd anxiety and its underlying neural
pathology . future research should focus on
relating anatomical changes and neurochemical
changes to neural activation in order to gain a
clearer understanding on how these pathologies
affect anxiety in pd . to further understand how
anxiety and cognitive dysfunction are related ,
future research should focus on using advanced
structural and function imaging techniques to
explain both cognitive and neural breakdowns that
are associated with anxiety in pd patients .
research has indicated that those with amnestic
mild cognitive impairment who have more
neuropsychiatric symptoms have a greater risk of
developing dementia compared to those with fewer
neuropsychiatric symptoms . future studies should
also examine whether treating neuropsychiatric
symptoms might impact the progression of cognitive
decline and improve cognitive impairments in pd
patients . previous studies have used pd symptom
laterality as a window to infer asymmetrical
dysfunction of neural circuits . for example , lpd
patients have greater inferred right hemisphere
pathology , whereas rpd patients have greater
inferred left hemisphere pathology . thus ,
cognitive domains predominantly subserved by the
left hemisphere ( e.g. , verbally mediated tasks
of executive function and verbal memory ) might be
hypothesized to be more affected in rpd than lpd ;
however , this remains controversial . it has also
been suggested that since anxiety is a common
feature of left hemisphere involvement [ 48 , 49 ]
, cognitive domains subserved by the left
hemisphere may also be more strongly related to
anxiety . results from this study showed selective
verbal memory deficits in rpd patients with
anxiety compared to rpd without anxiety , whereas
lpd patients with anxiety had greater attentional
/ working memory deficits compared to lpd without
anxiety . although these results align with
previous research , interpretations of these
findings should be made with caution due to the
small sample size in the lpd comparison
specifically . recent work has suggested that the
hads questionnaire may underestimate the burden of
anxiety related symptomology and therefore be a
less sensitive measure of anxiety in pd [ 30 , 50
] . in addition , our small sample size also
limited the statistical power for detecting
significant findings . based on these limitations
, our findings are likely conservative and
underrepresent the true impact anxiety has on
cognition in pd . additionally , the current study
employed a very brief neuropsychological
assessment including one or two tests for each
cognitive domain . future studies are encouraged
to collect a more complex and comprehensive
battery from a larger sample of pd participants in
order to better understand the role anxiety plays
on cognition in pd . another limitation of this
study was the absence of diagnostic interviews to
characterize participants ' psychiatric symptoms
and specify the type of anxiety disorders included
in this study . future studies should perform
diagnostic interviews with participants ( e.g. ,
using dsm - v criteria ) rather than relying on
self - reported measures to group participants ,
in order to better understand whether the type of
anxiety disorder ( e.g. , social anxiety , phobias
, panic disorders , and generalized anxiety )
influences cognitive performance differently in pd
. one advantage the hads questionnaire provided
over other anxiety scales was that it assessed
both anxiety and depression simultaneously and
allowed us to control for coexisting depression .
although there was a trend that the pda+ group
self - reported higher levels of depression than
the pda group , all participants included in the
study scored < 6 on the depression subscale of the
hads . controlling for depression while assessing
anxiety has been identified as a key shortcoming
in the majority of recent work . considering many
previous studies have investigated the influence
of depression on cognition in pd without
accounting for the presence of anxiety and the
inconsistent findings reported to date , we
recommend that future research should try to
disentangle the influence of anxiety versus
depression on cognitive impairments in pd .
considering the growing number of clinical trials
for treating depression , there are few if any for
the treatment of anxiety in pd . anxiety is a key
contributor to decreased quality of life in pd and
greatly requires better treatment options .
moreover , anxiety has been suggested to play a
key role in freezing of gait ( fog ) , which is
also related to attentional set - shifting [ 52 ,
53 ] . future research should examine the link
between anxiety , set - shifting , and fog , in
order to determine whether treating anxiety might
be a potential therapy for improving fog ."""

from transformers import LEDForConditionalGeneration, LEDTokenizer
import torch

tokenizer = LEDTokenizer.from_pretrained("patrickvonplaten/led-large-16384-pubmed")

input_ids = tokenizer(LONG_ARTICLE, return_tensors="pt").input_ids.to("cuda")
global_attention_mask = torch.zeros_like(input_ids)
# set global_attention_mask on first token
global_attention_mask[:, 0] = 1

model = LEDForConditionalGeneration.from_pretrained("patrickvonplaten/led-large-16384-pubmed", return_dict_in_generate=True).to("cuda")

sequences = model.generate(input_ids, global_attention_mask=global_attention_mask).sequences

summary = tokenizer.batch_decode(sequences)
```


model name: led-large-16384-pubmed


task: text2text-generation


tags: pytorch, tf, led, text2text-generation, en, dataset:scientific_papers, transformers, license:apache-2.0, autonlp_compatible


This repository is used to debug models, functionalities in transformers, etc...


# 1. Generation


# 2. Flax Wav2Vec2 Pretraining


Go into the `flax_wav2vec2` folder.


1. **Check PT loss works correctly**


`./run_pt_fsq_comp.sh` shows that HF PyTorch and Fairseq PT yield equivalent loss. Make sure to use the correct library versions as defined `branches_to_use.txt`.


2. **Check Flax loss works correctly**


`./run_flax_fsq_comp.sh` shows that HF PyTorch and HF Flax yield equivalent loss. Make sure to use the correct library versions as defined `branches_to_use.txt`.


model name: debug_repo


model author: patrickvonplaten


tags:


language:


- sv
tags:
- automatic-speech-recognition
- common_voice
- generated_from_trainer
datasets:
- common_voice
model-index:
- name: wav2vec2-xls-r-phoneme-300m-sv
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# Wav2vec2-xls-r-phoneme-300m-sv


**Note**: The tokenizer was created from the official Swedish phoneme vocabulary as defined here: https://github.com/microsoft/UniSpeech/blob/main/UniSpeech/examples/unispeech/data/sv/phonesMatches_reduced.json


One can simply download the file, rename it to `vocab.json` and load a `Wav2Vec2PhonemeCTCTokenizer.from_pretrained("./directory/with/vocab.json/")`.


This model is a fine-tuned version of [wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the COMMON_VOICE - SV-SE dataset.


It achieves the following results on the evaluation set:


- Loss: 0.9707
- PER: 0.2215


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0005
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- num_devices: 2
- total_train_batch_size: 32
- total_eval_batch_size: 16
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 20.0
- mixed_precision_training: Native AMP


### Training results


See Tensorboard traces


### Framework versions


- Transformers 4.13.0.dev0
- Pytorch 1.8.1
- Datasets 1.16.2.dev0
- Tokenizers 0.10.3


model name: wav2vec2-xls-r-phoneme-300m-sv


model author: patrickvonplaten


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, sv, dataset:common_voice, transformers, common_voice, generated_from_trainer, model-index


tags:


- automatic-speech-recognition
- librispeech_asr
- generated_from_trainer
- asr_seq2esq
model-index:
- name: wav2vec2-2-bart-base
results: []
widget:
- example_title: Librispeech sample 1
src: https://cdn-media.huggingface.co/speech_samples/sample1.flac
- example_title: Librispeech sample 2
src: https://cdn-media.huggingface.co/speech_samples/sample2.flac
- example_title: Common Voice sample
src: https://cdn-media.huggingface.co/speech_samples/common_voice_en_18301577.mp3


To rerun this experiment, please clone this directory and run:


```bash
python create_model.py
```


followed by


```bash
./run_librispeech.sh
```


# wav2vec2-2-bart-base


This model is a fine-tuned version of [facebook/wav2vec2-base](https://huggingface.co/facebook/wav2vec2-base) and [bart-base](https://huggingface.co/facebook/bart-base) on the librispeech_asr - clean dataset.


- Loss: 0.405
- Wer: 0.0728


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 0.0003
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- total_train_batch_size: 64
- total_eval_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 400
- num_epochs: 5
- mixed_precision_training: Native AMP


### Training results


See Training Metrics Tab.


### Framework versions


- Transformers 4.15.0.dev0
- Pytorch 1.9.0+cu111
- Datasets 1.16.2.dev0
- Tokenizers 0.10.3


model name: wav2vec2-2-bart-base


model author: patrickvonplaten


task: automatic-speech-recognition


tags: pytorch, tensorboard, speech-encoder-decoder, automatic-speech-recognition, transformers, librispeech_asr, generated_from_trainer, asr_seq2esq, model-index


tags:


- automatic-speech-recognition
- mozilla-foundation/common_voice_3_0
- generated_from_trainer
model-index:
- name: xls-r-300m-tr-phoneme
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# xls-r-300m-tr-phoneme


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the mozilla-foundation/common_voice_3_0 - TR dataset.
It achieves the following results on the evaluation set:


- Loss: 0.4378
- Wer: 0.09936


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 0.000075
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- total_train_batch_size: 32
- total_eval_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 150
- mixed_precision_training: Native AMP


### Training results


See Training Metrics Tab.


### Framework versions


model name: xls-r-300m-tr-phoneme


model author: patrickvonplaten


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, transformers, mozilla-foundation/common_voice_3_0, generated_from_trainer, model-index


license: apache-2.0
tags:


- automatic-speech-recognition
- timit_asr
- generated_from_trainer
datasets:
- timit_asr
model-index:
- name: sew-d-small-100k-timit
results: []


# sew-d-small-100k-timit


This model is a fine-tuned version of [asapp/sew-d-small-100k](https://huggingface.co/asapp/sew-d-small-100k) on the TIMIT_ASR - NA dataset.
It achieves the following results on the evaluation set:


- Loss: 1.7541
- Wer: 0.8061


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0001
- train_batch_size: 32
- eval_batch_size: 1
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 20.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 4.2068        | 0.69  | 100  | 4.0802          | 1.0    |
| 2.9805        | 1.38  | 200  | 2.9792          | 1.0    |
| 2.9781        | 2.07  | 300  | 2.9408          | 1.0    |
| 2.9655        | 2.76  | 400  | 2.9143          | 1.0    |
| 2.8953        | 3.45  | 500  | 2.8775          | 1.0    |
| 2.7718        | 4.14  | 600  | 2.7787          | 1.0    |
| 2.6711        | 4.83  | 700  | 2.6401          | 0.9786 |
| 2.6403        | 5.52  | 800  | 2.5435          | 1.0392 |
| 2.4052        | 6.21  | 900  | 2.4580          | 1.0706 |
| 2.1708        | 6.9   | 1000 | 2.2800          | 1.0090 |
| 2.2555        | 7.59  | 1100 | 2.1493          | 0.9579 |
| 2.3673        | 8.28  | 1200 | 2.0709          | 0.9051 |
| 2.091         | 8.97  | 1300 | 2.0258          | 0.8926 |
| 1.8433        | 9.66  | 1400 | 1.9645          | 0.8243 |
| 1.6824        | 10.34 | 1500 | 1.9211          | 0.8707 |
| 2.2282        | 11.03 | 1600 | 1.8914          | 0.8695 |
| 1.9027        | 11.72 | 1700 | 1.8718          | 0.8343 |
| 1.6303        | 12.41 | 1800 | 1.8646          | 0.8232 |
| 1.648         | 13.1  | 1900 | 1.8297          | 0.8177 |
| 2.0429        | 13.79 | 2000 | 1.8127          | 0.8642 |
| 1.8833        | 14.48 | 2100 | 1.8005          | 0.8307 |
| 1.5996        | 15.17 | 2200 | 1.7926          | 0.8467 |
| 1.4876        | 15.86 | 2300 | 1.7795          | 0.8341 |
| 1.8925        | 16.55 | 2400 | 1.7716          | 0.8199 |
| 1.814         | 17.24 | 2500 | 1.7846          | 0.8086 |
| 1.536         | 17.93 | 2600 | 1.7655          | 0.8019 |
| 1.4476        | 18.62 | 2700 | 1.7599          | 0.8070 |
| 1.7629        | 19.31 | 2800 | 1.7589          | 0.8119 |
| 1.7646        | 20.0  | 2900 | 1.7541          | 0.8061 |


### Framework versions


- Transformers 4.12.0.dev0
- Pytorch 1.8.1
- Datasets 1.14.1.dev0
- Tokenizers 0.10.3


model name: sew-d-small-100k-timit


model author: patrickvonplaten


task: automatic-speech-recognition


tags: pytorch, tensorboard, sew-d, automatic-speech-recognition, dataset:timit_asr, transformers, license:apache-2.0, timit_asr, generated_from_trainer, model-index


language:


- tr
license: apache-2.0
tags:
- automatic-speech-recognition
- common_voice
- generated_from_trainer
- xls_r_repro_common_voice_tr
datasets:
- common_voice
model-index:
- name: wav2vec2-large-xlsr-53-common_voice-tr-ft
results: []


# wav2vec2-large-xlsr-53-common_voice-tr-ft


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on the COMMON_VOICE - TR dataset.
It achieves the following results on the evaluation set:


- Loss: 0.4231
- Wer: 0.3104
- Cer: 0.0737


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 0.0005
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- total_train_batch_size: 64
- total_eval_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 100.0
- mixed_precision_training: Native AMP


### Training results


see under *Training Metrics* Tab.


### Framework versions


- Transformers 4.13.0.dev0
- Pytorch 1.9.0+cu111
- Datasets 1.15.2.dev0
- Tokenizers 0.10.3


model name: wav2vec2-large-xlsr-53-common_voice-tr-ft


model author: patrickvonplaten


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, tr, dataset:common_voice, transformers, license:apache-2.0, common_voice, generated_from_trainer, xls_r_repro_common_voice_tr, model-index


model name: wav2vec2-base-repro-960h-libri-85k-steps


tags: pytorch, wav2vec2, pretraining, transformers


license: apache-2.0
tags:


- automatic-speech-recognition
- librispeech_asr
- generated_from_trainer
model-index:
- name: sew-d-mid-400k-librispeech-clean-100h-ft
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# sew-d-mid-400k-librispeech-clean-100h-ft


This model is a fine-tuned version of [asapp/sew-d-mid-400k](https://huggingface.co/asapp/sew-d-mid-400k) on the LIBRISPEECH_ASR - CLEAN dataset.
It achieves the following results on the evaluation set:


- Loss: 2.3540
- Wer: 1.0536


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 3e-05
- train_batch_size: 4
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- total_train_batch_size: 32
- total_eval_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 3.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 7.319         | 0.11  | 100  | 11.0572         | 1.0    |
| 3.6726        | 0.22  | 200  | 4.2003          | 1.0    |
| 2.981         | 0.34  | 300  | 3.5742          | 0.9919 |
| 2.9411        | 0.45  | 400  | 3.2599          | 1.0    |
| 2.903         | 0.56  | 500  | 2.9350          | 1.0    |
| 2.8597        | 0.67  | 600  | 2.9514          | 1.0    |
| 2.7771        | 0.78  | 700  | 2.8521          | 1.0    |
| 2.7926        | 0.9   | 800  | 2.7821          | 1.0120 |
| 2.6623        | 1.01  | 900  | 2.7027          | 0.9924 |
| 2.5893        | 1.12  | 1000 | 2.6667          | 1.0240 |
| 2.5733        | 1.23  | 1100 | 2.6341          | 1.0368 |
| 2.5455        | 1.35  | 1200 | 2.5928          | 1.0411 |
| 2.4919        | 1.46  | 1300 | 2.5695          | 1.0817 |
| 2.5182        | 1.57  | 1400 | 2.5559          | 1.1072 |
| 2.4766        | 1.68  | 1500 | 2.5229          | 1.1257 |
| 2.4267        | 1.79  | 1600 | 2.4991          | 1.1151 |
| 2.3919        | 1.91  | 1700 | 2.4768          | 1.1139 |
| 2.3883        | 2.02  | 1800 | 2.4452          | 1.0636 |
| 2.3737        | 2.13  | 1900 | 2.4304          | 1.0594 |
| 2.3569        | 2.24  | 2000 | 2.4095          | 1.0539 |
| 2.3641        | 2.35  | 2100 | 2.3997          | 1.0511 |
| 2.3281        | 2.47  | 2200 | 2.3856          | 1.0414 |
| 2.2912        | 2.58  | 2300 | 2.3750          | 1.0696 |
| 2.3028        | 2.69  | 2400 | 2.3684          | 1.0436 |
| 2.2906        | 2.8   | 2500 | 2.3613          | 1.0538 |
| 2.2822        | 2.91  | 2600 | 2.3558          | 1.0506 |


### Framework versions


- Transformers 4.12.0.dev0
- Pytorch 1.9.0+cu111
- Datasets 1.13.4.dev0
- Tokenizers 0.10.3


model name: sew-d-mid-400k-librispeech-clean-100h-ft


model author: patrickvonplaten


task: automatic-speech-recognition


tags: pytorch, tensorboard, sew-d, automatic-speech-recognition, transformers, license:apache-2.0, librispeech_asr, generated_from_trainer, model-index


language:


- tr
tags:
- automatic-speech-recognition
- common_voice
- generated_from_trainer
datasets:
- common_voice
model-index:
- name: wav2vec2-xls-r-phoneme-300m-tr
results: []


# Wav2vec2-xls-r-phoneme-300m-tr


This model is a fine-tuned version of [wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the COMMON_VOICE - TR dataset.
It achieves the following results on the evaluation set:


- Loss: 0.6380
- PER: 0.1664


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 0.0005
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- num_devices: 2
- total_train_batch_size: 32
- total_eval_batch_size: 16
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 20.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | PER    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 13.6687       | 0.92  | 100  | 12.4567         | 1.0    |
| 3.4219        | 1.83  | 200  | 3.4704          | 1.0    |
| 3.1846        | 2.75  | 300  | 3.2281          | 0.9935 |
| 2.0076        | 3.67  | 400  | 1.7415          | 0.5222 |
| 1.0244        | 4.59  | 500  | 1.0290          | 0.3323 |
| 0.7095        | 5.5   | 600  | 0.8424          | 0.2859 |
| 0.619         | 6.42  | 700  | 0.7389          | 0.2232 |
| 0.3541        | 7.34  | 800  | 0.7049          | 0.2043 |
| 0.2946        | 8.26  | 900  | 0.7065          | 0.2153 |
| 0.2868        | 9.17  | 1000 | 0.6840          | 0.2115 |
| 0.2245        | 10.09 | 1100 | 0.6714          | 0.1952 |
| 0.1394        | 11.01 | 1200 | 0.6864          | 0.1954 |
| 0.1288        | 11.93 | 1300 | 0.6696          | 0.2017 |
| 0.1568        | 12.84 | 1400 | 0.6468          | 0.1843 |
| 0.1269        | 13.76 | 1500 | 0.6736          | 0.1965 |
| 0.1101        | 14.68 | 1600 | 0.6689          | 0.1915 |
| 0.1388        | 15.6  | 1700 | 0.6690          | 0.1782 |
| 0.0739        | 16.51 | 1800 | 0.6364          | 0.1734 |
| 0.0897        | 17.43 | 1900 | 0.6480          | 0.1748 |
| 0.0795        | 18.35 | 2000 | 0.6356          | 0.1695 |
| 0.0823        | 19.27 | 2100 | 0.6382          | 0.1685 |


### Framework versions


- Transformers 4.13.0.dev0
- Pytorch 1.8.1
- Datasets 1.16.2.dev0
- Tokenizers 0.10.3


model name: wav2vec2-xls-r-phoneme-300m-tr


model author: patrickvonplaten


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, tr, dataset:common_voice, transformers, common_voice, generated_from_trainer, model-index


language: en
license: apache-2.0
datasets:


- xsum
tags:
- summarization


Shared RoBERTa2RoBERTa Summarization with 🤗EncoderDecoder Framework
This model is a warm-started *RoBERTaShared* model fine-tuned on the *BBC XSum* summarization dataset.


The model achieves a **16.89** ROUGE-2 score on *BBC XSUM*'s test dataset.


For more details on how the model was fine-tuned, please refer to
[this](https://colab.research.google.com/drive/1Ekd5pUeCX7VOrMx94_czTkwNtLN32Uyu?usp=sharing) notebook.


model name: roberta_shared_bbc_xsum


task: summarization


tags: pytorch, encoder-decoder, text2text-generation, en, dataset:xsum, transformers, license:apache-2.0, summarization, autonlp_compatible


language:


- ta
license: apache-2.0
tags:
- automatic-speech-recognition
- common_voice
- generated_from_trainer
datasets:
- common_voice
model-index:
- name: wav2vec2-common_voice-tamil
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# wav2vec2-common_voice-tamil


This model is a fine-tuned version of [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on the COMMON_VOICE - TA dataset.
It achieves the following results on the evaluation set:


- Loss: 1.1172
- Wer: 1.0070


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0003
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 3.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| No log        | 0.84  | 100  | 4.0148          | 1.0    |
| No log        | 1.69  | 200  | 3.1738          | 1.0    |
| No log        | 2.54  | 300  | 2.5980          | 1.0236 |


### Framework versions


- Transformers 4.17.0.dev0
- Pytorch 1.10.1+cu113
- Datasets 1.18.1.dev0
- Tokenizers 0.10.3


model name: wav2vec2-common_voice-tamil


model author: patrickvonplaten


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, ta, dataset:common_voice, transformers, license:apache-2.0, common_voice, generated_from_trainer, model-index


license: apache-2.0
tags:


- automatic-speech-recognition
- timit_asr
- generated_from_trainer
datasets:
- timit_asr
model-index:
- name: sew-d-small-100k-ft-timit
results: []


# sew-d-small-100k-ft-timit


This model is a fine-tuned version of [asapp/sew-d-small-100k](https://huggingface.co/asapp/sew-d-small-100k) on the TIMIT_ASR - NA dataset.
It achieves the following results on the evaluation set:


- Loss: 1.7482
- Wer: 0.7987


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 0.0001
- train_batch_size: 32
- eval_batch_size: 1
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 20.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 4.2068        | 0.69  | 100  | 4.0802          | 1.0    |
| 2.9805        | 1.38  | 200  | 2.9792          | 1.0    |
| 2.9781        | 2.07  | 300  | 2.9408          | 1.0    |
| 2.9655        | 2.76  | 400  | 2.9143          | 1.0    |
| 2.8953        | 3.45  | 500  | 2.8775          | 1.0    |
| 2.7719        | 4.14  | 600  | 2.7815          | 0.9999 |
| 2.6531        | 4.83  | 700  | 2.6375          | 1.0065 |
| 2.6425        | 5.52  | 800  | 2.5602          | 1.0210 |
| 2.3963        | 6.21  | 900  | 2.4665          | 1.0591 |
| 2.1447        | 6.9   | 1000 | 2.2792          | 0.9848 |
| 2.2719        | 7.59  | 1100 | 2.2237          | 0.9465 |
| 2.3629        | 8.28  | 1200 | 2.1058          | 0.8907 |
| 2.0913        | 8.97  | 1300 | 2.0113          | 0.9070 |
| 1.8334        | 9.66  | 1400 | 1.9466          | 0.8177 |
| 1.6608        | 10.34 | 1500 | 1.9217          | 0.8698 |
| 2.2194        | 11.03 | 1600 | 1.9091          | 0.8727 |
| 1.9002        | 11.72 | 1700 | 1.8746          | 0.8332 |
| 1.6268        | 12.41 | 1800 | 1.8782          | 0.7951 |
| 1.6455        | 13.1  | 1900 | 1.8230          | 0.8225 |
| 2.0308        | 13.79 | 2000 | 1.8067          | 0.8560 |
| 1.855         | 14.48 | 2100 | 1.8129          | 0.8177 |
| 1.5901        | 15.17 | 2200 | 1.7891          | 0.8367 |
| 1.4848        | 15.86 | 2300 | 1.7821          | 0.8201 |
| 1.8754        | 16.55 | 2400 | 1.7700          | 0.8137 |
| 1.7975        | 17.24 | 2500 | 1.7795          | 0.8171 |
| 1.5194        | 17.93 | 2600 | 1.7605          | 0.7977 |
| 1.4374        | 18.62 | 2700 | 1.7529          | 0.7978 |
| 1.7498        | 19.31 | 2800 | 1.7522          | 0.8023 |
| 1.7452        | 20.0  | 2900 | 1.7482          | 0.7987 |


### Framework versions


- Transformers 4.12.0.dev0
- Pytorch 1.8.1
- Datasets 1.14.1.dev0
- Tokenizers 0.10.3


model name: sew-d-small-100k-ft-timit


model author: patrickvonplaten


task: automatic-speech-recognition


tags: pytorch, tensorboard, sew-d, automatic-speech-recognition, dataset:timit_asr, transformers, license:apache-2.0, timit_asr, generated_from_trainer, model-index


tags:


- automatic-speech-recognition
- librispeech_asr
- generated_from_trainer
- wavlm_libri_finetune
model-index:
- name: wavlm-libri-clean-100h-base-plus
results: []


# wavlm-libri-clean-100h-base-plus


This model is a fine-tuned version of [microsoft/wavlm-base-plus](https://huggingface.co/microsoft/wavlm-base-plus) on the LIBRISPEECH_ASR - CLEAN dataset.
It achieves the following results on the evaluation set:


- Loss: 0.0819
- Wer: 0.0683


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 0.0003
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- total_train_batch_size: 32
- total_eval_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 3.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 2.8877        | 0.34  | 300  | 2.8649          | 1.0    |
| 0.2852        | 0.67  | 600  | 0.2196          | 0.1830 |
| 0.1198        | 1.01  | 900  | 0.1438          | 0.1273 |
| 0.0906        | 1.35  | 1200 | 0.1145          | 0.1035 |
| 0.0729        | 1.68  | 1500 | 0.1055          | 0.0955 |
| 0.0605        | 2.02  | 1800 | 0.0936          | 0.0859 |
| 0.0402        | 2.35  | 2100 | 0.0885          | 0.0746 |
| 0.0421        | 2.69  | 2400 | 0.0848          | 0.0700 |


### Framework versions


- Transformers 4.15.0.dev0
- Pytorch 1.9.0+cu111
- Datasets 1.16.2.dev0
- Tokenizers 0.10.3


model name: wavlm-libri-clean-100h-base-plus


model author: patrickvonplaten


task: automatic-speech-recognition


tags: pytorch, tensorboard, wavlm, automatic-speech-recognition, transformers, librispeech_asr, generated_from_trainer, wavlm_libri_finetune, model-index


## Roberta-Base


This repo trains [roberta-base](https://huggingface.co/roberta-base) from scratch on the [Norwegian training subset of Oscar](https://oscar-corpus.com/) containing roughly 4.7 GB of data according to [this](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling) example.


Training is done on a TPUv3-8 in Flax. More statistics on the training run can be found under [tf.hub](https://tensorboard.dev/experiment/GdYmdak2TWeVz0DDRYOrrg).


model name: norwegian-roberta-base


task: fill-mask


tags: pytorch, jax, roberta, fill-mask, transformers, autonlp_compatible, infinity_compatible


license: apache-2.0
tags:


- automatic-speech-recognition
- timit_asr
- generated_from_trainer
datasets:
- timit_asr
model-index:
- name: sew-small-100k-timit
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# sew-small-100k-timit


This model is a fine-tuned version of [asapp/sew-small-100k](https://huggingface.co/asapp/sew-small-100k) on the TIMIT_ASR - NA dataset.
It achieves the following results on the evaluation set:


- Loss: 0.4926
- Wer: 0.2988


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0001
- train_batch_size: 32
- eval_batch_size: 1
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 20.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 3.071         | 0.69  | 100  | 3.0262          | 1.0    |
| 2.9304        | 1.38  | 200  | 2.9297          | 1.0    |
| 2.8823        | 2.07  | 300  | 2.8367          | 1.0    |
| 1.5668        | 2.76  | 400  | 1.2310          | 0.8807 |
| 0.7422        | 3.45  | 500  | 0.7080          | 0.5957 |
| 0.4121        | 4.14  | 600  | 0.5829          | 0.5073 |
| 0.3981        | 4.83  | 700  | 0.5153          | 0.4461 |
| 0.5038        | 5.52  | 800  | 0.4908          | 0.4151 |
| 0.2899        | 6.21  | 900  | 0.5122          | 0.4111 |
| 0.2198        | 6.9   | 1000 | 0.4908          | 0.3803 |
| 0.2129        | 7.59  | 1100 | 0.4668          | 0.3789 |
| 0.3007        | 8.28  | 1200 | 0.4788          | 0.3562 |
| 0.2264        | 8.97  | 1300 | 0.5113          | 0.3635 |
| 0.1536        | 9.66  | 1400 | 0.4950          | 0.3441 |
| 0.1206        | 10.34 | 1500 | 0.5062          | 0.3421 |
| 0.2021        | 11.03 | 1600 | 0.4900          | 0.3283 |
| 0.1458        | 11.72 | 1700 | 0.5019          | 0.3307 |
| 0.1151        | 12.41 | 1800 | 0.4989          | 0.3270 |
| 0.0985        | 13.1  | 1900 | 0.4925          | 0.3173 |
| 0.1412        | 13.79 | 2000 | 0.4868          | 0.3125 |
| 0.1579        | 14.48 | 2100 | 0.4983          | 0.3147 |
| 0.1043        | 15.17 | 2200 | 0.4914          | 0.3091 |
| 0.0773        | 15.86 | 2300 | 0.4858          | 0.3102 |
| 0.1327        | 16.55 | 2400 | 0.5084          | 0.3064 |
| 0.1281        | 17.24 | 2500 | 0.5017          | 0.3025 |
| 0.0845        | 17.93 | 2600 | 0.5001          | 0.3012 |
| 0.0717        | 18.62 | 2700 | 0.4894          | 0.3004 |
| 0.0835        | 19.31 | 2800 | 0.4963          | 0.2998 |
| 0.1181        | 20.0  | 2900 | 0.4926          | 0.2988 |


### Framework versions


- Transformers 4.12.0.dev0
- Pytorch 1.8.1
- Datasets 1.14.1.dev0
- Tokenizers 0.10.3


model name: sew-small-100k-timit


model author: patrickvonplaten


task: automatic-speech-recognition


tags: pytorch, tensorboard, sew, automatic-speech-recognition, dataset:timit_asr, transformers, license:apache-2.0, timit_asr, generated_from_trainer, model-index


# Librispeech dev-clean dataset


This is the Libirspeech "dev-clean" dataset that can also be obtained by running:


```
curl https://www.openslr.org/resources/12/dev-clean.tar.gz --output dev-clean.tar.gz
tar xf dev-clean.tar.gz
```


model name: LibriSpeechTest


tags:


# Roberta2Roberta Summarization with 🤗 EncoderDecoder Framework


This model is a Roberta2Roberta model fine-tuned on summarization.


Roberta2Roberta is a `EncoderDecoderModel`, meaning that both the encoder and the decoder are `roberta-base`
RoBERTa models. Leveraging the [EncoderDecoderFramework](https://huggingface.co/transformers/model_doc/encoderdecoder.html#encoder-decoder-models), the
two pretrained models can simply be loaded into the framework via:


```python
roberta2roberta = EncoderDecoderModel.from_encoder_decoder_pretrained("roberta-base", "roberta-base")
```


The decoder of an `EncoderDecoder` model needs cross-attention layers and usually makes use of causal
masking for auto-regressiv generation.
Thus, `roberta2roberta` is consequently fined-tuned on the `CNN/Daily Mail`dataset and the resulting model
`roberta2roberta-cnn_dailymail-fp16` is uploaded here.


## Example


The model is by no means a state-of-the-art model, but nevertheless
produces reasonable summarization results. It was mainly fine-tuned
as a proof-of-concept for the 🤗 EncoderDecoder Framework.


The model can be used as follows:


```python
from transformers import BertTokenizer, EncoderDecoderModel

model = EncoderDecoderModel.from_pretrained("patrickvonplaten/roberta2roberta-cnn_dailymail-fp16")
tokenizer = RobertaTokenizer.from_pretrained("roberta-base")

article = """(CNN)Sigma Alpha Epsilon is under fire for a video showing party-bound fraternity members singing a racist chant. SAE's national chapter suspended the students, but University of Oklahoma President David B
oren took it a step further, saying the university's affiliation with the fraternity is permanently done. The news is shocking, but it's not the first time SAE has faced controversy. SAE was founded March 9, 185
6, at the University of Alabama, five years before the American Civil War, according to the fraternity website. When the war began, the group had fewer than 400 members, of which "369 went to war for the Confede
rate States and seven for the Union Army," the website says. The fraternity now boasts more than 200,000 living alumni, along with about 15,000 undergraduates populating 219 chapters and 20 "colonies" seeking fu
ll membership at universities. SAE has had to work hard to change recently after a string of member deaths, many blamed on the hazing of new recruits, SAE national President Bradley Cohen wrote in a message on t
he fraternity's website. The fraternity's website lists more than 130 chapters cited or suspended for "health and safety incidents" since 2010. At least 30 of the incidents involved hazing, and dozens more invol
ved alcohol. However, the list is missing numerous incidents from recent months. Among them, according to various media outlets: Yale University banned the SAEs from campus activities last month after members al
legedly tried to interfere with a sexual misconduct investigation connected to an initiation rite. Stanford University in December suspended SAE housing privileges after finding sorority members attending a frat
ernity function were subjected to graphic sexual content. And Johns Hopkins University in November suspended the fraternity for underage drinking. "The media has labeled us as the 'nation's deadliest fraternity,
' " Cohen said. In 2011, for example, a student died while being coerced into excessive alcohol consumption, according to a lawsuit. SAE's previous insurer dumped the fraternity. "As a result, we are paying Lloy
d's of London the highest insurance rates in the Greek-letter world," Cohen said. Universities have turned down SAE's attempts to open new chapters, and the fraternity had to close 12 in 18 months over hazing in
cidents."""

input_ids = tokenizer(article, return_tensors="pt").input_ids
output_ids = model.generate(input_ids)

print(tokenizer.decode(output_ids[0], skip_special_tokens=True))
# should produce
# Sigma Alpha Epsilon is under fire for a video showing party-bound fraternity members singing racist chants. The fraternity's national chapter has had to close 12 in 18 months over hazing.
# Sigma has had more than 130 chapters in 18 states. University of Oklahoma president says fraternity has been "deteriorated".
```


## Training script:


**IMPORTANT**: In order for this code to work, make sure you checkout to the branch
[more_general_trainer_metric](https://github.com/huggingface/transformers/tree/more_general_trainer_metric), which slightly adapts
the `Trainer` for `EncoderDecoderModels` according to this PR: https://github.com/huggingface/transformers/pull/5840.


The following code shows the complete training script that was used to fine-tune `roberta2roberta-cnn_dailymail-fp16 ` for reproducability. The training last ~9h on a standard GPU.


```python
#!/usr/bin/env python3
import nlp
import logging
from transformers import RobertaTokenizer, EncoderDecoderModel, Trainer, TrainingArguments

logging.basicConfig(level=logging.INFO)

model = EncoderDecoderModel.from_encoder_decoder_pretrained("roberta-base", "roberta-base")
tokenizer =  RobertaTokenizer.from_pretrained("roberta-base")

# load train and validation data
train_dataset = nlp.load_dataset("cnn_dailymail", "3.0.0", split="train")
val_dataset = nlp.load_dataset("cnn_dailymail", "3.0.0", split="validation[:5%]")

# load rouge for validation
rouge = nlp.load_metric("rouge", experiment_id=0)

# set decoding params
model.config.decoder_start_token_id = tokenizer.bos_token_id
model.config.eos_token_id = tokenizer.eos_token_id
model.config.max_length = 142
model.config.min_length = 56
model.config.no_repeat_ngram_size = 3
model.early_stopping = True
model.length_penalty = 2.0
model.num_beams = 4

encoder_length = 512
decoder_length = 128
batch_size = 16

# map data correctly
def map_to_encoder_decoder_inputs(batch):
    # Tokenizer will automatically set [BOS] <text> [EOS]
    # cut off at Longformer at 2048
    inputs = tokenizer(batch["article"], padding="max_length", truncation=True, max_length=encoder_length)
    # force summarization <= 256
    outputs = tokenizer(batch["highlights"], padding="max_length", truncation=True, max_length=decoder_length)

    batch["input_ids"] = inputs.input_ids
    batch["attention_mask"] = inputs.attention_mask
    batch["decoder_input_ids"] = outputs.input_ids
    batch["labels"] = outputs.input_ids.copy()
    # mask loss for padding
    batch["labels"] = [
        [-100 if token == tokenizer.pad_token_id else token for token in labels] for labels in batch["labels"]
    ]
    batch["decoder_attention_mask"] = outputs.attention_mask

    assert all([len(x) == encoder_length for x in inputs.input_ids])
    assert all([len(x) == decoder_length for x in outputs.input_ids])

    return batch

def compute_metrics(pred):
    labels_ids = pred.label_ids
    pred_ids = pred.predictions

    # all unnecessary tokens are removed
    pred_str = tokenizer.batch_decode(pred_ids, skip_special_tokens=True)
    labels_ids[labels_ids == -100] = tokenizer.eos_token_id
    label_str = tokenizer.batch_decode(labels_ids, skip_special_tokens=True)

    rouge_output = rouge.compute(predictions=pred_str, references=label_str, rouge_types=["rouge2"])["rouge2"].mid

    return {
        "rouge2_precision": round(rouge_output.precision, 4),
        "rouge2_recall": round(rouge_output.recall, 4),
        "rouge2_fmeasure": round(rouge_output.fmeasure, 4),
    }

# make train dataset ready
train_dataset = train_dataset.map(
    map_to_encoder_decoder_inputs, batched=True, batch_size=batch_size, remove_columns=["article", "highlights"],
)
train_dataset.set_format(
    type="torch", columns=["input_ids", "attention_mask", "decoder_attention_mask", "decoder_input_ids", "labels"],
)

# same for validation dataset
val_dataset = val_dataset.map(
    map_to_encoder_decoder_inputs, batched=True, batch_size=batch_size, remove_columns=["article", "highlights"],
)
val_dataset.set_format(
    type="torch", columns=["input_ids", "decoder_attention_mask", "attention_mask", "decoder_input_ids", "labels"],
)

# set training arguments - these params are not really tuned, feel free to change
training_args = TrainingArguments(
    output_dir="./",
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    predict_from_generate=True,
    evaluate_during_training=True,
    do_train=True,
    do_eval=True,
    logging_steps=1000,
    save_steps=1000,
    eval_steps=1000,
    overwrite_output_dir=True,
    warmup_steps=2000,
    save_total_limit=3,
    fp16=True,
)

# instantiate trainer
trainer = Trainer(
    model=model,
    args=training_args,
    compute_metrics=compute_metrics,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
)

# start training
trainer.train()
```


## Evaluation


The following script evaluates the model on the test set of
CNN/Daily Mail.


```python
#!/usr/bin/env python3
import nlp
from transformers import RobertaTokenizer, EncoderDecoderModel

tokenizer = RobertaTokenizer.from_pretrained("roberta-base")
model = EncoderDecoderModel.from_pretrained("patrickvonplaten/roberta2roberta-cnn_dailymail-fp16")
model.to("cuda")

test_dataset = nlp.load_dataset("cnn_dailymail", "3.0.0", split="test")
batch_size = 128

# map data correctly
def generate_summary(batch):
    # Tokenizer will automatically set [BOS] <text> [EOS]
    # cut off at BERT max length 512
    inputs = tokenizer(batch["article"], padding="max_length", truncation=True, max_length=512, return_tensors="pt")
    input_ids = inputs.input_ids.to("cuda")
    attention_mask = inputs.attention_mask.to("cuda")

    outputs = model.generate(input_ids, attention_mask=attention_mask)

    # all special tokens including will be removed
    output_str = tokenizer.batch_decode(outputs, skip_special_tokens=True)

    batch["pred"] = output_str

    return batch

results = test_dataset.map(generate_summary, batched=True, batch_size=batch_size, remove_columns=["article"])

# load rouge for validation
rouge = nlp.load_metric("rouge")

pred_str = results["pred"]
label_str = results["highlights"]

rouge_output = rouge.compute(predictions=pred_str, references=label_str, rouge_types=["rouge2"])["rouge2"].mid

print(rouge_output)
```


The obtained results should be:


| -   |      Rouge2 - mid -precision      |  Rouge2 - mid - recall | Rouge2 - mid - fmeasure |
|----------|:-------------:|:------:|:------:|
| **CNN/Daily Mail** |  15.79 | 19.05 | **16.79** |


model name: roberta2roberta-cnn_dailymail-fp16


model author: patrickvonplaten


task: text2text-generation


tags: pytorch, encoder_decoder, text2text-generation, transformers, autonlp_compatible


license: apache-2.0
tags:


- generated_from_trainer
model-index:
- name: wav2vec2-base-timit-demo-colab
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# wav2vec2-base-timit-demo-colab


This model is a fine-tuned version of [facebook/wav2vec2-base](https://huggingface.co/facebook/wav2vec2-base) on the None dataset.
It achieves the following results on the evaluation set:


- Loss: 0.4888
- Wer: 0.3392


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training:


- learning_rate: 0.0001
- train_batch_size: 32
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 30
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 0.1134        | 4.0   | 500  | 0.4250          | 0.3626 |
| 0.1035        | 8.0   | 1000 | 0.4980          | 0.3650 |
| 0.0801        | 12.0  | 1500 | 0.5563          | 0.3632 |
| 0.0592        | 16.0  | 2000 | 0.6222          | 0.3607 |
| 0.0563        | 20.0  | 2500 | 0.4763          | 0.3457 |
| 0.0611        | 24.0  | 3000 | 0.4938          | 0.3489 |
| 0.0475        | 28.0  | 3500 | 0.4888          | 0.3392 |


### Framework versions


- Transformers 4.11.3
- Pytorch 1.9.0+cu111
- Datasets 1.13.3
- Tokenizers 0.10.3


model name: wav2vec2-base-timit-demo-colab


model author: patrickvonplaten


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, transformers, license:apache-2.0, generated_from_trainer, model-index


language:


- tr
tags:
- automatic-speech-recognition
- common_voice
- generated_from_trainer
datasets:
- common_voice
model-index:
- name: hello_2b
results: []


# hello_2b


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-2b](https://huggingface.co/facebook/wav2vec2-xls-r-2b) on the COMMON_VOICE - TR dataset.
It achieves the following results on the evaluation set:


- Loss: 1.2725
- Wer: 0.9531


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 1e-05
- train_batch_size: 2
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- num_devices: 2
- gradient_accumulation_steps: 8
- total_train_batch_size: 32
- total_eval_batch_size: 16
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 30.0
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 3.1646        | 0.92  | 100  | 3.2106          | 1.0    |
| 0.368         | 1.85  | 200  | 2.9963          | 1.0    |
| 0.2252        | 2.77  | 300  | 2.8078          | 0.9999 |
| 0.1546        | 3.7   | 400  | 2.3458          | 0.9996 |
| 0.1468        | 4.63  | 500  | 2.0086          | 0.9986 |
| 0.1261        | 5.55  | 600  | 1.8269          | 0.9985 |
| 0.1206        | 6.48  | 700  | 1.7347          | 0.9956 |
| 0.1959        | 7.4   | 800  | 1.6819          | 0.9955 |
| 0.0502        | 8.33  | 900  | 1.6809          | 0.9965 |
| 0.0811        | 9.26  | 1000 | 1.6674          | 0.9916 |
| 0.0534        | 10.18 | 1100 | 1.5719          | 0.9898 |
| 0.0402        | 11.11 | 1200 | 1.4620          | 0.9821 |
| 0.057         | 12.04 | 1300 | 1.3015          | 0.9554 |
| 0.0385        | 12.96 | 1400 | 1.3798          | 0.9600 |
| 0.0422        | 13.88 | 1500 | 1.3538          | 0.9699 |
| 0.014         | 14.81 | 1600 | 1.2507          | 0.9443 |
| 0.0232        | 15.74 | 1700 | 1.3318          | 0.9465 |
| 0.0554        | 16.66 | 1800 | 1.2784          | 0.9462 |
| 0.0316        | 17.59 | 1900 | 1.2503          | 0.9481 |
| 0.0524        | 18.51 | 2000 | 1.3920          | 0.9604 |
| 0.0142        | 19.44 | 2100 | 1.4224          | 0.9698 |
| 0.0288        | 20.37 | 2200 | 1.3475          | 0.9635 |
| 0.0106        | 21.29 | 2300 | 1.2232          | 0.9264 |
| 0.0396        | 22.22 | 2400 | 1.3323          | 0.9615 |
| 0.0349        | 23.15 | 2500 | 1.2741          | 0.9587 |
| 0.0121        | 24.07 | 2600 | 1.2671          | 0.9586 |
| 0.0224        | 24.99 | 2700 | 1.3001          | 0.9611 |
| 0.0449        | 25.92 | 2800 | 1.2777          | 0.9572 |
| 0.0186        | 26.85 | 2900 | 1.2766          | 0.9607 |
| 0.0365        | 27.77 | 3000 | 1.2935          | 0.9598 |
| 0.0105        | 28.7  | 3100 | 1.2761          | 0.9588 |
| 0.021         | 29.63 | 3200 | 1.2686          | 0.9528 |


### Framework versions


- Transformers 4.13.0.dev0
- Pytorch 1.10.0
- Datasets 1.15.2.dev0
- Tokenizers 0.10.3


model name: hello_2b


model author: patrickvonplaten


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, tr, dataset:common_voice, transformers, common_voice, generated_from_trainer, model-index


license: apache-2.0
tags:


- generated_from_trainer
datasets:
- common_voice
model-index:
- name: wav2vec2-large-xlsr-turkish-demo-colab
results: []


# wav2vec2-large-xlsr-turkish-demo-colab


This model is a fine-tuned version of [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on the common_voice dataset.
It achieves the following results on the evaluation set:


- Loss: 0.4055
- Wer: 0.4800


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


- learning_rate: 0.0003
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 30
- mixed_precision_training: Native AMP


### Training results


| Training Loss | Epoch | Step | Validation Loss | Wer    |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 5.0179        | 4.21  | 400  | 1.4935          | 1.0249 |
| 0.7075        | 8.42  | 800  | 0.4546          | 0.6071 |
| 0.3072        | 12.63 | 1200 | 0.3947          | 0.5401 |
| 0.2145        | 16.84 | 1600 | 0.4049          | 0.5194 |
| 0.1647        | 21.05 | 2000 | 0.4199          | 0.5003 |
| 0.1338        | 25.26 | 2400 | 0.4144          | 0.4859 |
| 0.116         | 29.47 | 2800 | 0.4055          | 0.4800 |


### Framework versions


- Transformers 4.11.3
- Pytorch 1.9.1+cu102
- Datasets 1.13.3
- Tokenizers 0.10.3


model name: wav2vec2-large-xlsr-turkish-demo-colab


model author: patrickvonplaten


task: automatic-speech-recognition


tags: pytorch, tensorboard, wav2vec2, automatic-speech-recognition, dataset:common_voice, transformers, license:apache-2.0, generated_from_trainer, model-index


language: es
datasets:


- common_voice
metrics:
- wer
- cer
tags:
- audio
- automatic-speech-recognition
- speech
- xlsr-fine-tuning-week
license: apache-2.0


# Wav2Vec2-Large-XLSR-53-Spanish-With-LM


This is a model copy of [Wav2Vec2-Large-XLSR-53-Spanish](https://huggingface.co/jonatasgrosman/wav2vec2-large-xlsr-53-spanish)
that has language model support.


This model card can be seen as a demo for the [pyctcdecode](https://github.com/kensho-technologies/pyctcdecode) integration
with Transformers led by [this PR](https://github.com/huggingface/transformers/pull/14339). The PR explains in-detail how the
integration works.


In a nutshell: This PR adds a new Wav2Vec2WithLMProcessor class as drop-in replacement for Wav2Vec2Processor.


The only change from the existing ASR pipeline will be:


```diff
import torch
from datasets import load_dataset
from transformers import AutoModelForCTC, AutoProcessor
import torchaudio.functional as F

model_id = "patrickvonplaten/wav2vec2-xlsr-53-es-kenlm"

sample = next(iter(load_dataset("common_voice", "es", split="test", streaming=True)))
resampled_audio = F.resample(torch.tensor(sample["audio"]["array"]), 48_000, 16_000).numpy()

model = AutoModelForCTC.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id)

input_values = processor(resampled_audio, return_tensors="pt").input_values

with torch.no_grad():
    logits = model(input_values).logits

-prediction_ids = torch.argmax(logits, dim=-1)
-transcription = processor.batch_decode(prediction_ids)
+transcription = processor.batch_decode(logits.numpy()).text
# => 'bien y qué regalo vas a abrir primero'
```


**Improvement**


This model has been compared on 512 speech samples from the Spanish Common Voice Test set and
gives a nice *20 %* performance boost:


The results can be reproduced by running *from this model repository*:


| Model | WER | CER |
| ------------- | ------------- | ------------- |
| patrickvonplaten/wav2vec2-xlsr-53-es-kenlm | **8.44%** | **2.93%** |
| jonatasgrosman/wav2vec2-large-xlsr-53-spanish | **10.20%** | **3.24%** |


```
bash run_ngram_wav2vec2.py 1 512
```


```
bash run_ngram_wav2vec2.py 0 512
```


with `run_ngram_wav2vec2.py` being
https://huggingface.co/patrickvonplaten/wav2vec2-large-xlsr-53-spanish-with-lm/blob/main/run_ngram_wav2vec2.py


model name: wav2vec2-xlsr-53-es-kenlm


model author: patrickvonplaten


task: automatic-speech-recognition


tags: pytorch, wav2vec2, automatic-speech-recognition, es, dataset:common_voice, transformers, audio, speech, xlsr-fine-tuning-week, license:apache-2.0


language: en
license: apache-2.0
datasets:


- cnn_dailymail
tags:
- summarization


Bert2Bert Summarization with 🤗EncoderDecoder Framework
This model is a warm-started *BERT2BERT* model fine-tuned on the *CNN/Dailymail* summarization dataset.


The model achieves a **18.22** ROUGE-2 score on *CNN/Dailymail*'s test dataset.


For more details on how the model was fine-tuned, please refer to
[this](https://colab.research.google.com/drive/1Ekd5pUeCX7VOrMx94_czTkwNtLN32Uyu?usp=sharing) notebook.


model name: bert2bert_cnn_daily_mail


task: summarization


tags: pytorch, encoder-decoder, text2text-generation, en, dataset:cnn_dailymail, transformers, license:apache-2.0, summarization, autonlp_compatible


language:


- tr
tags:
- automatic-speech-recognition
- common_voice
- generated_from_trainer
datasets:
- common_voice
model-index:
- name: hello_2b_3
results: []


<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->


# hello_2b_3


This model is a fine-tuned version of [facebook/wav2vec2-xls-r-2b](https://huggingface.co/facebook/wav2vec2-xls-r-2b) on the COMMON_VOICE - TR dataset.
It achieves the following results on the evaluation set:


- Loss: 1.5615
- Wer: 0.9808


## Model description


More information needed


## Intended uses & limitations


## Training and evaluation data


## Training procedure


### Training hyperparameters


The following hyperparameters were used during training: