BERT_WordLevel_phoneme_wikitext

This model is a fine-tuned version of on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.2557

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 256
• eval_batch_size: 256
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 1024
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-06
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 10000
• training_steps: 106000
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
3.0796	0.9394	2000	3.0683
2.1293	1.8788	4000	1.8796
1.0751	2.8182	6000	0.9697
0.8026	3.7576	8000	0.7028
0.6745	4.6970	10000	0.5962
0.5973	5.6364	12000	0.5183
0.5651	6.5759	14000	0.4832
0.5151	7.5153	16000	0.4465
0.488	8.4547	18000	0.4234
0.4706	9.3941	20000	0.4077
0.4528	10.3335	22000	0.3910
0.4375	11.2729	24000	0.3734
0.4273	12.2123	26000	0.3705
0.4171	13.1517	28000	0.3640
0.4078	14.0911	30000	0.3485
0.4007	15.0305	32000	0.3467
0.39	15.9699	34000	0.3367
0.3824	16.9093	36000	0.3300
0.3765	17.8488	38000	0.3277
0.372	18.7882	40000	0.3230
0.3682	19.7276	42000	0.3186
0.3638	20.6670	44000	0.3160
0.357	21.6064	46000	0.3101
0.3544	22.5458	48000	0.3007
0.3495	23.4852	50000	0.3037
0.3456	24.4246	52000	0.2982
0.3416	25.3640	54000	0.2902
0.3382	26.3034	56000	0.2908
0.3337	27.2428	58000	0.2866
0.3323	28.1822	60000	0.2920
0.3284	29.1217	62000	0.2820
0.3265	30.0611	64000	0.2772
0.3249	31.0005	66000	0.2791
0.3214	31.9399	68000	0.2804
0.3192	32.8793	70000	0.2742
0.3151	33.8187	72000	0.2707
0.3154	34.7581	74000	0.2684
0.3126	35.6975	76000	0.2699
0.3116	36.6369	78000	0.2736
0.3087	37.5763	80000	0.2693
0.3074	38.5157	82000	0.2652
0.3044	39.4551	84000	0.2548
0.3044	40.3946	86000	0.2649
0.3035	41.3340	88000	0.2650
0.2994	42.2734	90000	0.2519
0.3014	43.2128	92000	0.2564
0.2971	44.1522	94000	0.2602
0.2983	45.0916	96000	0.2552
0.297	46.0310	98000	0.2556
0.296	46.9704	100000	0.2553
0.2936	47.9098	102000	0.2568
0.2929	48.8492	104000	0.2480
0.2935	49.7886	106000	0.2557

Framework versions

• Transformers 4.45.1
• Pytorch 2.4.1+cu121
• Datasets 3.0.1
• Tokenizers 0.20.0