libri-finetune-6-layers-transformer

This model is a fine-tuned version of on the None dataset. It achieves the following results on the evaluation set:

• Loss: 83.8115
• Wer: 0.7450

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.2
• num_epochs: 100
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer
15064.3113	2.99	400	5870.5229	0.9999
7699.3169	5.97	800	4746.7100	0.9999
6725.2825	8.96	1200	2085.7102	1.0483
2585.5492	11.94	1600	627.4510	0.8690
1256.1203	14.93	2000	305.7397	0.8220
829.551	17.91	2400	195.8744	0.8150
562.8857	20.9	2800	134.9562	0.8043
383.4462	23.88	3200	103.3252	0.7821
251.6083	26.87	3600	83.2899	0.7851
176.5321	29.85	4000	72.0544	0.7668
107.0474	32.84	4400	81.2677	0.7692
47.8587	35.82	4800	61.3930	0.7735
23.0181	38.81	5200	77.2446	0.7795
-41.5376	41.79	5600	55.3390	0.7637
-82.3179	44.78	6000	60.2555	0.7582
-89.828	47.76	6400	68.0762	0.7566
-132.0839	50.75	6800	78.1840	0.7478
-149.8906	53.73	7200	87.0740	0.7535
-181.8106	56.72	7600	83.8115	0.7450

Framework versions

• Transformers 4.23.1
• Pytorch 1.12.1
• Datasets 2.6.1
• Tokenizers 0.13.1