wav2vec2-large-xlrs-korean-v5

This model is a fine-tuned version of facebook/wav2vec2-xls-r-300m on the zeroth_korean dataset. It achieves the following results on the evaluation set:

• Loss: 0.1300
• Wer: 0.2433

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 32
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 1000
• num_epochs: 30
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer
5.1453	1.4368	500	3.1530	1.0
2.4287	2.8736	1000	0.6084	0.8317
0.5556	4.3103	1500	0.3414	0.6165
0.3929	5.7471	2000	0.2729	0.5386
0.3211	7.1839	2500	0.2294	0.4794
0.281	8.6207	3000	0.2052	0.4298
0.2483	10.0575	3500	0.1911	0.4061
0.2243	11.4943	4000	0.1685	0.3873
0.2023	12.9310	4500	0.1627	0.3524
0.188	14.3678	5000	0.1572	0.3272
0.1784	15.8046	5500	0.1495	0.3131
0.1677	17.2414	6000	0.1424	0.2881
0.1533	18.6782	6500	0.1418	0.2709
0.1501	20.1149	7000	0.1387	0.2822
0.1402	21.5517	7500	0.1401	0.2697
0.1353	22.9885	8000	0.1367	0.2643
0.133	24.4253	8500	0.1337	0.2578
0.1254	25.8621	9000	0.1355	0.2560
0.1262	27.2989	9500	0.1339	0.2474
0.121	28.7356	10000	0.1300	0.2433

Framework versions

• Transformers 4.41.0
• Pytorch 2.3.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1