UTI2_L3_1000steps_1e6rate_05beta_CSFTDPO

This model is a fine-tuned version of tsavage68/UTI_L3_1000steps_1e5rate_SFT on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.0094
• Rewards/chosen: 3.5398
• Rewards/rejected: -9.3115
• Rewards/accuracies: 0.9900
• Rewards/margins: 12.8514
• Logps/rejected: -61.8926
• Logps/chosen: -22.1453
• Logits/rejected: -1.1592
• Logits/chosen: -1.1419

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-06
• train_batch_size: 2
• eval_batch_size: 1
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 4
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 100
• training_steps: 1000

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.5446	0.3333	25	0.2409	0.7934	-0.6030	0.9800	1.3964	-44.4754	-27.6381	-1.1424	-1.1365
0.0009	0.6667	50	0.0261	2.3344	-5.4705	0.9800	7.8050	-54.2106	-24.5560	-1.1516	-1.1414
0.0001	1.0	75	0.0417	2.5058	-6.7400	0.9700	9.2458	-56.7494	-24.2133	-1.1557	-1.1437
0.0	1.3333	100	0.0150	2.4614	-7.0530	0.9900	9.5144	-57.3755	-24.3022	-1.1580	-1.1452
0.0	1.6667	125	0.0165	2.8817	-7.5738	0.9900	10.4555	-58.4170	-23.4615	-1.1561	-1.1425
0.0175	2.0	150	0.0077	2.7979	-9.1294	0.9900	11.9273	-61.5282	-23.6290	-1.1592	-1.1446
0.0	2.3333	175	0.0077	2.8004	-9.1559	0.9900	11.9563	-61.5813	-23.6240	-1.1592	-1.1445
0.0	2.6667	200	0.0098	3.5386	-9.1468	0.9900	12.6854	-61.5630	-22.1478	-1.1582	-1.1411
0.0	3.0	225	0.0098	3.5323	-9.1598	0.9900	12.6921	-61.5891	-22.1603	-1.1583	-1.1411
0.0	3.3333	250	0.0099	3.5384	-9.1504	0.9900	12.6888	-61.5704	-22.1482	-1.1580	-1.1408
0.0	3.6667	275	0.0101	3.5390	-9.1521	0.9900	12.6912	-61.5738	-22.1469	-1.1582	-1.1410
0.0173	4.0	300	0.0102	3.5300	-9.1689	0.9900	12.6988	-61.6072	-22.1650	-1.1582	-1.1410
0.0	4.3333	325	0.0095	3.5391	-9.1723	0.9900	12.7114	-61.6141	-22.1467	-1.1582	-1.1411
0.0173	4.6667	350	0.0098	3.5336	-9.1774	0.9900	12.7110	-61.6242	-22.1576	-1.1582	-1.1411
0.0	5.0	375	0.0100	3.5413	-9.1860	0.9900	12.7273	-61.6416	-22.1423	-1.1584	-1.1412
0.0173	5.3333	400	0.0097	3.5385	-9.1956	0.9900	12.7342	-61.6608	-22.1479	-1.1586	-1.1414
0.0173	5.6667	425	0.0099	3.5458	-9.1729	0.9900	12.7188	-61.6153	-22.1332	-1.1581	-1.1409
0.0	6.0	450	0.0095	3.5342	-9.2206	0.9900	12.7548	-61.7106	-22.1565	-1.1583	-1.1411
0.0	6.3333	475	0.0096	3.5378	-9.2207	0.9900	12.7585	-61.7109	-22.1492	-1.1585	-1.1413
0.0173	6.6667	500	0.0098	3.5344	-9.2288	0.9900	12.7632	-61.7271	-22.1561	-1.1588	-1.1415
0.0	7.0	525	0.0090	3.5387	-9.2492	0.9900	12.7878	-61.7678	-22.1475	-1.1587	-1.1414
0.0	7.3333	550	0.0092	3.5377	-9.2629	0.9900	12.8006	-61.7953	-22.1496	-1.1589	-1.1417
0.0173	7.6667	575	0.0093	3.5369	-9.2697	0.9900	12.8066	-61.8089	-22.1510	-1.1590	-1.1418
0.0	8.0	600	0.0094	3.5387	-9.2877	0.9900	12.8264	-61.8448	-22.1475	-1.1587	-1.1414
0.0347	8.3333	625	0.0098	3.5219	-9.2959	0.9900	12.8178	-61.8614	-22.1812	-1.1590	-1.1418
0.0	8.6667	650	0.0092	3.5332	-9.2917	0.9900	12.8249	-61.8529	-22.1584	-1.1589	-1.1416
0.0	9.0	675	0.0091	3.5324	-9.3041	0.9900	12.8365	-61.8776	-22.1600	-1.1591	-1.1418
0.0	9.3333	700	0.0096	3.5277	-9.3067	0.9900	12.8344	-61.8829	-22.1695	-1.1591	-1.1418
0.0	9.6667	725	0.0092	3.5429	-9.3040	0.9900	12.8470	-61.8776	-22.1390	-1.1591	-1.1418
0.0	10.0	750	0.0096	3.5350	-9.3114	0.9900	12.8464	-61.8923	-22.1549	-1.1588	-1.1415
0.0	10.3333	775	0.0094	3.5320	-9.3159	0.9900	12.8479	-61.9013	-22.1609	-1.1590	-1.1416
0.0	10.6667	800	0.0092	3.5430	-9.3106	0.9900	12.8535	-61.8906	-22.1389	-1.1591	-1.1418
0.0	11.0	825	0.0090	3.5293	-9.3094	0.9900	12.8387	-61.8883	-22.1663	-1.1589	-1.1416
0.0	11.3333	850	0.0093	3.5309	-9.3281	0.9900	12.8591	-61.9258	-22.1630	-1.1590	-1.1417
0.0173	11.6667	875	0.0093	3.5340	-9.3279	0.9900	12.8618	-61.9252	-22.1570	-1.1592	-1.1419
0.0	12.0	900	0.0092	3.5268	-9.3258	0.9900	12.8526	-61.9212	-22.1713	-1.1590	-1.1416
0.0	12.3333	925	0.0089	3.5337	-9.3216	0.9900	12.8553	-61.9127	-22.1576	-1.1590	-1.1417
0.0173	12.6667	950	0.0093	3.5404	-9.3113	0.9900	12.8518	-61.8922	-22.1440	-1.1591	-1.1419
0.0173	13.0	975	0.0094	3.5398	-9.3115	0.9900	12.8514	-61.8926	-22.1453	-1.1592	-1.1419
0.0	13.3333	1000	0.0094	3.5398	-9.3115	0.9900	12.8514	-61.8926	-22.1453	-1.1592	-1.1419

Framework versions

• Transformers 4.41.2
• Pytorch 2.0.0+cu117
• Datasets 2.19.2
• Tokenizers 0.19.1