UTI2_L3_1000steps_1e7rate_05beta_CSFTDPO

This model is a fine-tuned version of tsavage68/UTI_L3_1000steps_1e5rate_SFT on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.0214
• Rewards/chosen: 2.3188
• Rewards/rejected: -5.0731
• Rewards/accuracies: 0.9900
• Rewards/margins: 7.3919
• Logps/rejected: -53.4156
• Logps/chosen: -24.5872
• Logits/rejected: -1.1499
• Logits/chosen: -1.1399

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-07
• train_batch_size: 2
• eval_batch_size: 1
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 4
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 100
• training_steps: 1000

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6994	0.3333	25	0.6838	0.0135	-0.0067	0.5900	0.0202	-43.2828	-29.1979	-1.1410	-1.1363
0.6558	0.6667	50	0.6397	0.0766	-0.0351	0.9300	0.1117	-43.3396	-29.0716	-1.1411	-1.1363
0.5544	1.0	75	0.5162	0.2530	-0.1459	0.9800	0.3989	-43.5613	-28.7188	-1.1416	-1.1366
0.3409	1.3333	100	0.3357	0.6037	-0.3562	0.9700	0.9598	-43.9818	-28.0176	-1.1423	-1.1368
0.1685	1.6667	125	0.1829	0.9252	-0.8431	0.9800	1.7683	-44.9557	-27.3745	-1.1429	-1.1369
0.1015	2.0	150	0.1103	1.2176	-1.2970	0.9700	2.5146	-45.8636	-26.7898	-1.1438	-1.1373
0.0441	2.3333	175	0.0733	1.4279	-1.7766	0.9800	3.2045	-46.8227	-26.3690	-1.1445	-1.1375
0.0216	2.6667	200	0.0523	1.6140	-2.2674	0.9900	3.8814	-47.8044	-25.9969	-1.1449	-1.1374
0.0112	3.0	225	0.0428	1.7801	-2.7695	0.9800	4.5496	-48.8085	-25.6648	-1.1454	-1.1375
0.007	3.3333	250	0.0360	1.9280	-3.2399	0.9800	5.1679	-49.7493	-25.3689	-1.1464	-1.1380
0.0034	3.6667	275	0.0318	2.0520	-3.6179	0.9800	5.6698	-50.5052	-25.1210	-1.1470	-1.1382
0.0191	4.0	300	0.0304	2.1084	-3.9273	0.9800	6.0357	-51.1241	-25.0082	-1.1475	-1.1384
0.0013	4.3333	325	0.0274	2.1616	-4.1577	0.9800	6.3193	-51.5849	-24.9017	-1.1482	-1.1390
0.02	4.6667	350	0.0274	2.1909	-4.3267	0.9800	6.5176	-51.9229	-24.8431	-1.1485	-1.1392
0.0015	5.0	375	0.0255	2.2135	-4.4596	0.9800	6.6731	-52.1887	-24.7980	-1.1489	-1.1394
0.0263	5.3333	400	0.0254	2.2351	-4.5500	0.9800	6.7851	-52.3696	-24.7548	-1.1491	-1.1396
0.0179	5.6667	425	0.0250	2.2415	-4.6664	0.9800	6.9079	-52.6023	-24.7420	-1.1492	-1.1395
0.0007	6.0	450	0.0249	2.2668	-4.7384	0.9800	7.0052	-52.7462	-24.6913	-1.1494	-1.1397
0.0006	6.3333	475	0.0235	2.2771	-4.7885	0.9800	7.0656	-52.8465	-24.6708	-1.1493	-1.1395
0.0241	6.6667	500	0.0228	2.2909	-4.8459	0.9800	7.1368	-52.9612	-24.6430	-1.1496	-1.1398
0.0005	7.0	525	0.0233	2.2991	-4.9022	0.9800	7.2013	-53.0738	-24.6267	-1.1498	-1.1399
0.0009	7.3333	550	0.0234	2.3031	-4.9695	0.9800	7.2726	-53.2086	-24.6187	-1.1499	-1.1400
0.0177	7.6667	575	0.0223	2.3072	-4.9980	0.9800	7.3052	-53.2655	-24.6105	-1.1500	-1.1400
0.0003	8.0	600	0.0213	2.3161	-4.9998	0.9900	7.3158	-53.2690	-24.5928	-1.1497	-1.1397
0.035	8.3333	625	0.0223	2.3137	-5.0175	0.9800	7.3312	-53.3045	-24.5975	-1.1500	-1.1401
0.0003	8.6667	650	0.0224	2.3191	-5.0217	0.9800	7.3407	-53.3128	-24.5868	-1.1500	-1.1399
0.0063	9.0	675	0.0221	2.3281	-5.0418	0.9800	7.3699	-53.3530	-24.5686	-1.1499	-1.1400
0.0052	9.3333	700	0.0222	2.3187	-5.0539	0.9800	7.3726	-53.3773	-24.5874	-1.1500	-1.1400
0.0003	9.6667	725	0.0222	2.3309	-5.0402	0.9900	7.3711	-53.3498	-24.5630	-1.1499	-1.1399
0.0012	10.0	750	0.0223	2.3177	-5.0661	0.9800	7.3838	-53.4016	-24.5894	-1.1497	-1.1396
0.0003	10.3333	775	0.0218	2.3350	-5.0444	0.9900	7.3793	-53.3583	-24.5550	-1.1499	-1.1399
0.0003	10.6667	800	0.0221	2.3263	-5.0530	0.9800	7.3793	-53.3756	-24.5723	-1.1501	-1.1400
0.0054	11.0	825	0.0221	2.3297	-5.0668	0.9800	7.3964	-53.4030	-24.5656	-1.1498	-1.1397
0.0003	11.3333	850	0.0218	2.3202	-5.0679	0.9900	7.3881	-53.4054	-24.5845	-1.1499	-1.1399
0.0177	11.6667	875	0.0223	2.3311	-5.0581	0.9900	7.3893	-53.3858	-24.5626	-1.1500	-1.1401
0.0013	12.0	900	0.0212	2.3255	-5.0619	0.9900	7.3874	-53.3934	-24.5739	-1.1498	-1.1398
0.0003	12.3333	925	0.0209	2.3206	-5.0729	0.9900	7.3936	-53.4154	-24.5837	-1.1499	-1.1399
0.0177	12.6667	950	0.0214	2.3188	-5.0731	0.9900	7.3919	-53.4156	-24.5872	-1.1499	-1.1399
0.0178	13.0	975	0.0214	2.3188	-5.0731	0.9900	7.3919	-53.4156	-24.5872	-1.1499	-1.1399
0.0003	13.3333	1000	0.0214	2.3188	-5.0731	0.9900	7.3919	-53.4156	-24.5872	-1.1499	-1.1399

Framework versions

• Transformers 4.41.2
• Pytorch 2.0.0+cu117
• Datasets 2.19.2
• Tokenizers 0.19.1