typescriptmate-500000

This model is a fine-tuned version of gpt2 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 1.1136

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 8
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 5

Training results

Training Loss	Epoch	Step	Validation Loss
1.8011	0.0518	500	1.6532
1.678	0.1036	1000	1.5555
1.5942	0.1555	1500	1.5039
1.5568	0.2073	2000	1.4671
1.5595	0.2591	2500	1.4372
1.5126	0.3109	3000	1.4117
1.4806	0.3628	3500	1.3926
1.4451	0.4146	4000	1.3764
1.4274	0.4664	4500	1.3599
1.433	0.5182	5000	1.3466
1.4115	0.5701	5500	1.3372
1.4328	0.6219	6000	1.3246
1.3804	0.6737	6500	1.3123
1.3596	0.7255	7000	1.3050
1.393	0.7774	7500	1.2951
1.3176	0.8292	8000	1.2860
1.353	0.8810	8500	1.2762
1.3695	0.9328	9000	1.2723
1.3406	0.9847	9500	1.2651
1.3488	1.0365	10000	1.2578
1.3148	1.0883	10500	1.2539
1.3104	1.1401	11000	1.2470
1.2822	1.1920	11500	1.2443
1.2954	1.2438	12000	1.2400
1.2589	1.2956	12500	1.2317
1.2742	1.3474	13000	1.2277
1.2625	1.3993	13500	1.2251
1.2297	1.4511	14000	1.2192
1.2458	1.5029	14500	1.2148
1.2429	1.5547	15000	1.2118
1.2663	1.6066	15500	1.2079
1.2459	1.6584	16000	1.2036
1.28	1.7102	16500	1.2020
1.2599	1.7620	17000	1.1995
1.2102	1.8138	17500	1.1938
1.2385	1.8657	18000	1.1904
1.2258	1.9175	18500	1.1882
1.2303	1.9693	19000	1.1858
1.2119	2.0211	19500	1.1837
1.2305	2.0730	20000	1.1791
1.2013	2.1248	20500	1.1779
1.1969	2.1766	21000	1.1743
1.2009	2.2284	21500	1.1737
1.2005	2.2803	22000	1.1710
1.1802	2.3321	22500	1.1694
1.2228	2.3839	23000	1.1672
1.201	2.4357	23500	1.1637
1.1836	2.4876	24000	1.1611
1.1918	2.5394	24500	1.1594
1.1931	2.5912	25000	1.1581
1.1989	2.6430	25500	1.1550
1.1829	2.6949	26000	1.1533
1.1703	2.7467	26500	1.1531
1.1834	2.7985	27000	1.1501
1.1486	2.8503	27500	1.1483
1.1481	2.9022	28000	1.1457
1.1711	2.9540	28500	1.1457
1.1466	3.0058	29000	1.1430
1.1409	3.0576	29500	1.1413
1.1535	3.1095	30000	1.1407
1.1309	3.1613	30500	1.1397
1.1552	3.2131	31000	1.1392
1.1316	3.2649	31500	1.1364
1.1655	3.3167	32000	1.1353
1.1463	3.3686	32500	1.1358
1.1475	3.4204	33000	1.1331
1.155	3.4722	33500	1.1319
1.1437	3.5240	34000	1.1307
1.1152	3.5759	34500	1.1310
1.1237	3.6277	35000	1.1298
1.1617	3.6795	35500	1.1276
1.1202	3.7313	36000	1.1266
1.1404	3.7832	36500	1.1262
1.1331	3.8350	37000	1.1244
1.1366	3.8868	37500	1.1238
1.1154	3.9386	38000	1.1235
1.1435	3.9905	38500	1.1224
1.1084	4.0423	39000	1.1209
1.115	4.0941	39500	1.1223
1.1077	4.1459	40000	1.1209
1.0867	4.1978	40500	1.1197
1.1049	4.2496	41000	1.1191
1.0976	4.3014	41500	1.1199
1.1199	4.3532	42000	1.1183
1.109	4.4051	42500	1.1172
1.1119	4.4569	43000	1.1168
1.1006	4.5087	43500	1.1165
1.1261	4.5605	44000	1.1168
1.0676	4.6124	44500	1.1165
1.1358	4.6642	45000	1.1160
1.1384	4.7160	45500	1.1151
1.0974	4.7678	46000	1.1152
1.1357	4.8197	46500	1.1143
1.1186	4.8715	47000	1.1142
1.1156	4.9233	47500	1.1140
1.1155	4.9751	48000	1.1136

Framework versions

• Transformers 4.52.4
• Pytorch 2.6.0+cu124
• Datasets 3.6.0
• Tokenizers 0.21.1