fpadovani/fr_wiki_mlm_30_new

fr_wiki_mlm_30

This model is a fine-tuned version of on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.0899

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 16
• eval_batch_size: 16
• seed: 30
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 40000
• training_steps: 100000

Training results

Training Loss	Epoch	Step	Validation Loss
No log	1.7190	2000	7.6699
7.6889	3.4379	4000	6.6554
7.6889	5.1569	6000	6.5451
6.5434	6.8758	8000	6.4810
6.5434	8.5948	10000	6.4034
6.3985	10.3137	12000	6.3405
6.3985	12.0327	14000	6.3003
6.2938	13.7516	16000	6.2494
6.2938	15.4706	18000	6.1585
6.1683	17.1895	20000	6.0396
6.1683	18.9085	22000	5.6432
5.7111	20.6274	24000	5.0941
5.7111	22.3464	26000	4.5845
4.7685	24.0653	28000	4.1427
4.7685	25.7843	30000	3.8185
3.9833	27.5032	32000	3.5973
3.9833	29.2222	34000	3.3900
3.5291	30.9411	36000	3.2362
3.5291	32.6601	38000	3.1203
3.2203	34.3790	40000	2.9705
3.2203	36.0980	42000	2.8947
2.9765	37.8169	44000	2.8023
2.9765	39.5359	46000	2.7170
2.7888	41.2548	48000	2.6235
2.7888	42.9738	50000	2.5756
2.6487	44.6927	52000	2.5370
2.6487	46.4117	54000	2.5089
2.5403	48.1306	56000	2.4711
2.5403	49.8496	58000	2.4114
2.4503	51.5685	60000	2.4126
2.4503	53.2875	62000	2.3348
2.3773	55.0064	64000	2.3154
2.3773	56.7254	66000	2.3191
2.3137	58.4443	68000	2.2695
2.3137	60.1633	70000	2.2463
2.2639	61.8823	72000	2.2415
2.2639	63.6012	74000	2.2218
2.2121	65.3202	76000	2.1900
2.2121	67.0391	78000	2.1908
2.1731	68.7581	80000	2.1536
2.1731	70.4770	82000	2.1406
2.1447	72.1960	84000	2.1389
2.1447	73.9149	86000	2.1326
2.1134	75.6339	88000	2.1130
2.1134	77.3528	90000	2.1237
2.0866	79.0718	92000	2.0951
2.0866	80.7907	94000	2.1022
2.0645	82.5097	96000	2.0837
2.0645	84.2286	98000	2.0891
2.0527	85.9476	100000	2.0899

Framework versions

• Transformers 4.45.2
• Pytorch 2.5.1+cu124
• Datasets 3.0.1
• Tokenizers 0.20.1