collapse_gemma-2-2b_hs2_replace_iter1_sftsd0

This model is a fine-tuned version of google/gemma-2-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.1098
• Num Input Tokens Seen: 3714531

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 1
• eval_batch_size: 1
• seed: 0
• gradient_accumulation_steps: 16
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.5574	0
1.5728	0.0064	5	1.5530	24085
1.513	0.0128	10	1.5215	47453
1.3347	0.0192	15	1.4468	70909
1.3476	0.0255	20	1.3877	95452
1.4852	0.0319	25	1.3363	119981
1.1471	0.0383	30	1.2983	144786
1.3088	0.0447	35	1.2671	166464
1.201	0.0511	40	1.2375	190744
1.124	0.0575	45	1.2105	214420
1.0562	0.0638	50	1.1885	239168
1.3103	0.0702	55	1.1785	264823
1.2666	0.0766	60	1.1738	286887
1.0661	0.0830	65	1.1713	310195
1.1574	0.0894	70	1.1683	332735
1.1514	0.0958	75	1.1670	355198
1.193	0.1021	80	1.1652	376716
1.1345	0.1085	85	1.1634	400910
1.1844	0.1149	90	1.1606	424364
1.121	0.1213	95	1.1605	448878
1.1053	0.1277	100	1.1597	473220
1.3433	0.1341	105	1.1586	494797
1.0394	0.1405	110	1.1569	516472
1.0004	0.1468	115	1.1564	542496
1.063	0.1532	120	1.1542	567455
1.0935	0.1596	125	1.1532	589914
1.1756	0.1660	130	1.1523	615559
1.1847	0.1724	135	1.1513	639427
0.9203	0.1788	140	1.1502	663647
1.2464	0.1851	145	1.1489	686562
1.2849	0.1915	150	1.1481	711202
1.0506	0.1979	155	1.1458	736818
1.0547	0.2043	160	1.1450	760835
1.2118	0.2107	165	1.1444	785946
1.0628	0.2171	170	1.1442	809541
1.2405	0.2234	175	1.1442	834219
1.0144	0.2298	180	1.1437	857046
1.0397	0.2362	185	1.1424	878565
0.97	0.2426	190	1.1426	900602
1.1232	0.2490	195	1.1426	922378
1.2127	0.2554	200	1.1421	947077
1.1946	0.2618	205	1.1419	971080
1.0858	0.2681	210	1.1409	994354
1.0305	0.2745	215	1.1398	1020247
1.2059	0.2809	220	1.1381	1044169
1.1007	0.2873	225	1.1383	1068670
1.2074	0.2937	230	1.1380	1092119
1.2036	0.3001	235	1.1381	1116057
1.1312	0.3064	240	1.1369	1139741
1.356	0.3128	245	1.1368	1161776
1.2383	0.3192	250	1.1358	1186457
0.9792	0.3256	255	1.1355	1209960
1.3327	0.3320	260	1.1350	1234993
1.224	0.3384	265	1.1345	1258158
1.1602	0.3447	270	1.1340	1281170
1.1478	0.3511	275	1.1326	1303543
1.0506	0.3575	280	1.1324	1327641
1.1367	0.3639	285	1.1322	1350399
0.9812	0.3703	290	1.1318	1375625
0.9715	0.3767	295	1.1322	1401283
1.1205	0.3831	300	1.1325	1424494
0.9563	0.3894	305	1.1317	1448682
1.2123	0.3958	310	1.1317	1473827
1.2588	0.4022	315	1.1301	1496238
1.2084	0.4086	320	1.1295	1520282
1.146	0.4150	325	1.1296	1544117
1.2708	0.4214	330	1.1281	1568119
1.063	0.4277	335	1.1276	1589956
1.1589	0.4341	340	1.1279	1613080
1.101	0.4405	345	1.1292	1636206
1.101	0.4469	350	1.1283	1658755
1.3876	0.4533	355	1.1277	1681219
1.0889	0.4597	360	1.1274	1705600
1.1535	0.4660	365	1.1271	1730155
1.2215	0.4724	370	1.1274	1752902
1.1059	0.4788	375	1.1278	1776823
1.0943	0.4852	380	1.1277	1801284
1.1276	0.4916	385	1.1273	1825215
1.4414	0.4980	390	1.1275	1849301
1.0202	0.5043	395	1.1270	1873516
0.9466	0.5107	400	1.1275	1898133
1.172	0.5171	405	1.1269	1922443
1.1545	0.5235	410	1.1259	1946674
1.2317	0.5299	415	1.1253	1969275
1.3125	0.5363	420	1.1254	1991043
1.0947	0.5427	425	1.1248	2015867
1.2208	0.5490	430	1.1251	2038041
1.1131	0.5554	435	1.1252	2060342
1.0515	0.5618	440	1.1252	2084200
1.0103	0.5682	445	1.1252	2107998
1.2081	0.5746	450	1.1242	2129784
1.0893	0.5810	455	1.1231	2155055
1.0361	0.5873	460	1.1224	2179095
1.195	0.5937	465	1.1223	2201382
1.1477	0.6001	470	1.1216	2223137
1.1631	0.6065	475	1.1221	2245234
1.1492	0.6129	480	1.1214	2269870
1.1192	0.6193	485	1.1215	2294736
1.1139	0.6256	490	1.1212	2320091
1.3045	0.6320	495	1.1208	2343005
1.1927	0.6384	500	1.1209	2368849
1.0113	0.6448	505	1.1210	2392477
0.987	0.6512	510	1.1209	2415916
0.9788	0.6576	515	1.1195	2440778
1.1421	0.6640	520	1.1191	2464786
1.3182	0.6703	525	1.1189	2487622
1.1411	0.6767	530	1.1186	2511201
1.2706	0.6831	535	1.1180	2535535
1.2678	0.6895	540	1.1179	2556715
1.0137	0.6959	545	1.1175	2580524
1.1116	0.7023	550	1.1173	2605345
1.1862	0.7086	555	1.1167	2628955
1.2881	0.7150	560	1.1164	2651931
1.0775	0.7214	565	1.1156	2676797
1.2833	0.7278	570	1.1158	2700355
1.1513	0.7342	575	1.1167	2722499
1.0315	0.7406	580	1.1165	2745577
1.0138	0.7469	585	1.1158	2770551
0.9586	0.7533	590	1.1159	2794752
1.0669	0.7597	595	1.1157	2818575
1.0782	0.7661	600	1.1149	2841338
1.0795	0.7725	605	1.1149	2865516
1.2159	0.7789	610	1.1145	2887902
0.9433	0.7853	615	1.1143	2912838
1.2748	0.7916	620	1.1153	2936217
1.285	0.7980	625	1.1148	2959937
1.1652	0.8044	630	1.1147	2984114
1.0269	0.8108	635	1.1140	3007557
1.1009	0.8172	640	1.1138	3031187
1.3179	0.8236	645	1.1135	3054281
0.9786	0.8299	650	1.1128	3081188
1.4264	0.8363	655	1.1129	3104440
1.3194	0.8427	660	1.1127	3128974
1.0863	0.8491	665	1.1133	3150918
1.3073	0.8555	670	1.1136	3172136
1.0008	0.8619	675	1.1134	3194331
1.1083	0.8682	680	1.1133	3216425
1.2404	0.8746	685	1.1130	3239182
1.0338	0.8810	690	1.1130	3263452
1.1239	0.8874	695	1.1124	3288221
0.9939	0.8938	700	1.1115	3313512
0.9771	0.9002	705	1.1105	3339071
1.1427	0.9066	710	1.1107	3363039
0.9692	0.9129	715	1.1110	3386567
1.0297	0.9193	720	1.1108	3411415
1.1646	0.9257	725	1.1103	3437441
1.1262	0.9321	730	1.1101	3459420
1.0678	0.9385	735	1.1110	3483604
1.1839	0.9449	740	1.1110	3508111
1.1017	0.9512	745	1.1108	3530644
1.1201	0.9576	750	1.1105	3556161
1.1003	0.9640	755	1.1096	3578193
1.0523	0.9704	760	1.1093	3602419
1.1821	0.9768	765	1.1093	3626838
1.0863	0.9832	770	1.1091	3651332
1.3074	0.9895	775	1.1096	3677439
1.0503	0.9959	780	1.1101	3701297

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1