Edit model card

libri-alpha-0.75-Temp-1-attention-3-layers-distil-with-6-layers-mse-take-3

This model is a fine-tuned version of rohitp1/libri-alpha-0.75-Temp-1-attention-3-layers-distil-with-6-layers-mse on the None dataset. It achieves the following results on the evaluation set:

  • Loss: 28.9263
  • Wer: 0.3301

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0005
  • train_batch_size: 4
  • eval_batch_size: 1
  • seed: 42
  • gradient_accumulation_steps: 2
  • total_train_batch_size: 8
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_ratio: 0.2
  • num_epochs: 40
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss Wer
291.1088 0.22 400 28.4207 0.3362
284.1968 0.45 800 28.1458 0.3314
288.1414 0.67 1200 28.1397 0.3326
290.0272 0.9 1600 28.4186 0.3323
287.3224 1.12 2000 28.3548 0.3283
279.1482 1.35 2400 28.5373 0.3309
285.8217 1.57 2800 28.4447 0.3301
282.9265 1.79 3200 28.5379 0.3365
292.6254 2.02 3600 28.2632 0.3299
279.215 2.24 4000 28.9263 0.3301

Framework versions

  • Transformers 4.24.0
  • Pytorch 1.12.1
  • Datasets 2.7.1
  • Tokenizers 0.11.0
Downloads last month
7