wav2vec2-pretraining-demo

This model is a fine-tuned version of on the librispeech_asr dataset. It achieves the following results on the evaluation set:

• Loss: nan
• Constrast Loss: 708.2256
• Div Loss: nan

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 10
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Constrast Loss	Div Loss
343.7458	0.3745	100	nan	706.8953	nan
360.8719	0.7491	200	nan	709.1356	nan
561.7222	1.1236	300	nan	725.8750	nan
450.0023	1.4981	400	nan	704.0518	nan
412.6041	1.8727	500	nan	702.4941	nan
955.4866	2.2472	600	nan	717.8548	nan
947.9007	2.6217	700	775.8956	763.6932	122.0238
487.9696	2.9963	800	nan	706.2196	nan
359.6353	3.3708	900	nan	727.8684	nan
299.9773	3.7453	1000	nan	689.3754	nan
557.0656	4.1199	1100	nan	741.6811	nan
625.427	4.4944	1200	nan	669.9859	nan
486.9457	4.8689	1300	nan	725.6080	nan
1067.3776	5.2434	1400	nan	703.2175	nan
958.6181	5.6180	1500	nan	742.9335	nan
482.8976	5.9925	1600	nan	724.2250	nan
350.0361	6.3670	1700	nan	715.3557	nan
350.6836	6.7416	1800	nan	711.0261	nan
522.1924	7.1161	1900	nan	702.5721	nan
483.4517	7.4906	2000	nan	741.1333	nan
590.2781	7.8652	2100	nan	728.1261	nan
1039.0322	8.2397	2200	nan	720.8858	nan
1146.0051	8.6142	2300	nan	717.9496	nan
626.2248	8.9888	2400	nan	712.2637	nan
371.0907	9.3633	2500	nan	727.3933	nan
362.5648	9.7378	2600	nan	708.2256	nan

Framework versions

• Transformers 4.49.0.dev0
• Pytorch 2.6.0+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0