b20-wav2vec2-large-xls-r-romansh-colab

This model is a fine-tuned version of facebook/wav2vec2-xls-r-300m on the common_voice_13_0 dataset. It achieves the following results on the evaluation set:

• Loss: 0.3738
• Wer: 0.3181

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 100
• num_epochs: 30
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer
9.6653	0.76	100	3.2423	1.0
3.0224	1.52	200	3.0321	1.0
2.969	2.29	300	3.0174	1.0
2.964	3.05	400	2.9531	1.0
2.9488	3.81	500	2.9441	1.0
2.962	4.58	600	2.9383	1.0
2.9646	5.34	700	2.9377	1.0
2.9411	6.11	800	2.9303	1.0
2.9313	6.87	900	2.9264	1.0
2.9327	7.63	1000	2.9211	1.0
2.9574	8.4	1100	2.9145	1.0
2.9227	9.16	1200	2.9034	1.0
2.8916	9.92	1300	2.8764	1.0
2.8311	10.68	1400	2.5611	0.9995
2.0497	11.45	1500	1.1256	0.8784
1.2359	12.21	1600	0.7668	0.7143
0.9607	12.97	1700	0.6340	0.6388
0.804	13.74	1800	0.5658	0.5806
0.693	14.5	1900	0.5147	0.5389
0.6403	15.27	2000	0.4711	0.4797
0.5716	16.03	2100	0.4298	0.4520
0.5124	16.79	2200	0.4353	0.4313
0.5104	17.56	2300	0.3991	0.3952
0.4416	18.32	2400	0.4012	0.3933
0.4419	19.08	2500	0.3945	0.3687
0.406	19.84	2600	0.4003	0.3675
0.3946	20.61	2700	0.3901	0.3579
0.379	21.37	2800	0.3963	0.3537
0.3663	22.14	2900	0.3826	0.3435
0.3425	22.9	3000	0.3850	0.3435
0.3396	23.66	3100	0.3852	0.3405
0.3041	24.43	3200	0.3771	0.3265
0.3194	25.19	3300	0.3796	0.3265
0.312	25.95	3400	0.3734	0.3228
0.313	26.71	3500	0.3864	0.3270
0.3039	27.48	3600	0.3734	0.3149
0.2929	28.24	3700	0.3785	0.3223
0.2884	29.01	3800	0.3734	0.3160
0.2812	29.77	3900	0.3738	0.3181

Framework versions

• Transformers 4.26.0
• Pytorch 2.0.1+cu118
• Datasets 2.14.4
• Tokenizers 0.13.3