phi-2-dpo-full-hydrox-safe

This model is a fine-tuned version of microsoft/phi-2 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.0050
• Rewards/chosen: -0.0996
• Rewards/rejected: -19.3989
• Rewards/accuracies: 0.9992
• Rewards/margins: 19.2993
• Logps/rejected: -644.0859
• Logps/chosen: -275.4780
• Logits/rejected: 0.2021
• Logits/chosen: -0.3196

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-07
• train_batch_size: 8
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• total_train_batch_size: 64
• total_eval_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6535	0.03	100	0.6404	0.0912	-0.0433	0.6936	0.1345	-450.5297	-273.5692	0.6266	0.1056
0.4293	0.07	200	0.4021	0.4957	-0.4032	0.8998	0.8989	-454.1287	-269.5248	0.6425	0.1325
0.2685	0.1	300	0.2464	0.8729	-1.0624	0.9386	1.9354	-460.7211	-265.7526	0.6700	0.1533
0.1918	0.14	400	0.1792	1.1981	-1.7756	0.9453	2.9737	-467.8528	-262.5003	0.6918	0.1719
0.1409	0.17	500	0.1423	1.4079	-2.4222	0.9503	3.8301	-474.3184	-260.4028	0.7139	0.1772
0.1437	0.2	600	0.1232	1.6054	-3.1009	0.9537	4.7063	-481.1060	-258.4279	0.7179	0.1571
0.0747	0.24	700	0.1057	1.7190	-3.9721	0.9613	5.6911	-489.8179	-257.2914	0.7179	0.1325
0.0848	0.27	800	0.0863	1.8540	-4.3289	0.9689	6.1829	-493.3858	-255.9415	0.7047	0.1024
0.0853	0.31	900	0.0744	1.8328	-5.0270	0.9722	6.8598	-500.3666	-256.1535	0.6746	0.0529
0.0561	0.34	1000	0.0639	1.7736	-5.4555	0.9747	7.2291	-504.6516	-256.7455	0.6517	0.0185
0.0657	0.37	1100	0.0546	1.7966	-5.6453	0.9840	7.4419	-506.5500	-256.5157	0.6738	0.0168
0.0445	0.41	1200	0.0499	1.7245	-6.0645	0.9823	7.7889	-510.7414	-257.2369	0.6757	0.0176
0.0498	0.44	1300	0.0459	1.6804	-6.6799	0.9848	8.3603	-516.8956	-257.6772	0.6719	0.0228
0.0554	0.48	1400	0.0549	1.5994	-6.8731	0.9848	8.4726	-518.8279	-258.4872	0.6684	0.0151
0.0368	0.51	1500	0.0365	1.3551	-7.8016	0.9865	9.1567	-528.1130	-260.9311	0.6464	-0.0223
0.0514	0.54	1600	0.0333	1.3431	-7.5785	0.9891	8.9215	-525.8817	-261.0511	0.6617	-0.0159
0.0365	0.58	1700	0.0310	1.0743	-8.8130	0.9933	9.8873	-538.2264	-263.7386	0.6354	-0.0234
0.0214	0.61	1800	0.0300	1.1310	-9.1633	0.9924	10.2942	-541.7292	-263.1721	0.6278	-0.0199
0.0321	0.65	1900	0.0267	0.9514	-9.9884	0.9924	10.9399	-549.9811	-264.9675	0.6044	-0.0268
0.0239	0.68	2000	0.0249	0.9624	-10.0608	0.9933	11.0232	-550.7047	-264.8574	0.5955	-0.0628
0.0484	0.71	2100	0.0222	0.9176	-11.0294	0.9941	11.9470	-560.3909	-265.3055	0.6043	-0.0338
0.0239	0.75	2200	0.0209	0.8247	-10.5070	0.9941	11.3317	-555.1663	-266.2345	0.5913	-0.0648
0.0219	0.78	2300	0.0221	0.7334	-10.5131	0.9924	11.2465	-555.2272	-267.1474	0.5942	-0.0808
0.0168	0.82	2400	0.0200	0.8909	-11.0818	0.9966	11.9727	-560.9146	-265.5724	0.6006	-0.0770
0.0135	0.85	2500	0.0187	0.7273	-11.3130	0.9958	12.0403	-563.2269	-267.2084	0.5948	-0.0711
0.0211	0.88	2600	0.0190	0.5390	-11.8492	0.9949	12.3883	-568.5890	-269.0913	0.5818	-0.0719
0.0209	0.92	2700	0.0170	0.4957	-12.3176	0.9949	12.8133	-573.2725	-269.5249	0.5537	-0.0912
0.0168	0.95	2800	0.0175	0.5850	-12.2737	0.9958	12.8587	-572.8340	-268.6321	0.5471	-0.0870
0.0129	0.99	2900	0.0171	0.5691	-12.7785	0.9949	13.3476	-577.8817	-268.7910	0.5441	-0.0810
0.0134	1.02	3000	0.0188	0.3713	-13.0254	0.9966	13.3966	-580.3504	-270.7691	0.5261	-0.1163
0.0127	1.05	3100	0.0155	0.1769	-14.3013	0.9958	14.4782	-593.1094	-272.7126	0.4928	-0.1129
0.0072	1.09	3200	0.0147	0.2474	-13.6499	0.9975	13.8973	-586.5952	-272.0074	0.4753	-0.1479
0.0068	1.12	3300	0.0179	0.2168	-14.6122	0.9958	14.8291	-596.2192	-272.3131	0.4722	-0.1430
0.0323	1.16	3400	0.0218	0.2536	-15.9479	0.9958	16.2015	-609.5762	-271.9459	0.4159	-0.1430
0.0232	1.19	3500	0.0171	0.4273	-14.2304	0.9958	14.6577	-592.4003	-270.2084	0.4598	-0.1410
0.0112	1.22	3600	0.0165	0.3634	-15.1069	0.9966	15.4704	-601.1661	-270.8472	0.4366	-0.1521
0.0136	1.26	3700	0.0176	0.2922	-14.4784	0.9966	14.7706	-594.8809	-271.5597	0.4553	-0.1679
0.0089	1.29	3800	0.0155	0.2349	-14.8970	0.9966	15.1319	-599.0669	-272.1331	0.4447	-0.1721
0.01	1.33	3900	0.0123	0.1309	-15.3620	0.9975	15.4930	-603.7170	-273.1724	0.4180	-0.1834
0.0055	1.36	4000	0.0132	0.0227	-16.5037	0.9975	16.5265	-615.1339	-274.2542	0.3836	-0.1899
0.004	1.39	4100	0.0122	0.2637	-15.2111	0.9975	15.4748	-602.2075	-271.8445	0.4087	-0.1870
0.0057	1.43	4200	0.0117	0.1593	-16.5411	0.9975	16.7003	-615.5074	-272.8891	0.4086	-0.1518
0.0041	1.46	4300	0.0113	-0.0815	-17.4813	0.9975	17.3998	-624.9092	-275.2963	0.3722	-0.1782
0.0039	1.5	4400	0.0115	0.2051	-17.4309	0.9983	17.6359	-624.4056	-272.4310	0.3445	-0.1975
0.0075	1.53	4500	0.0098	0.2810	-15.5168	0.9992	15.7978	-605.2650	-271.6719	0.4021	-0.1980
0.0034	1.56	4600	0.0123	0.2926	-15.4764	0.9992	15.7690	-604.8610	-271.5560	0.4225	-0.1766
0.0066	1.6	4700	0.0125	0.3192	-14.8325	0.9992	15.1517	-598.4222	-271.2900	0.4068	-0.2026
0.0079	1.63	4800	0.0124	0.1405	-15.1594	0.9992	15.2999	-601.6904	-273.0765	0.3942	-0.2089
0.0063	1.67	4900	0.0108	-0.1407	-17.4715	0.9975	17.3308	-624.8116	-275.8886	0.3294	-0.2140
0.0026	1.7	5000	0.0116	-0.0520	-17.9554	0.9992	17.9034	-629.6504	-275.0013	0.3361	-0.2062
0.0151	1.73	5100	0.0112	0.5727	-14.3495	0.9975	14.9221	-593.5914	-268.7550	0.4234	-0.1897
0.0049	1.77	5200	0.0083	0.2709	-15.7593	0.9975	16.0301	-607.6893	-271.7730	0.3788	-0.2054
0.0036	1.8	5300	0.0093	0.2336	-16.3789	0.9992	16.6125	-613.8856	-272.1456	0.3798	-0.1961
0.0076	1.84	5400	0.0086	0.2618	-16.1654	0.9992	16.4272	-611.7511	-271.8639	0.3844	-0.2090
0.0028	1.87	5500	0.0079	0.3444	-15.7817	0.9983	16.1261	-607.9135	-271.0378	0.3874	-0.2135
0.0102	1.9	5600	0.0081	0.3276	-15.7265	0.9983	16.0541	-607.3622	-271.2061	0.3713	-0.2243
0.0051	1.94	5700	0.0072	0.3979	-15.8670	0.9983	16.2649	-608.7670	-270.5030	0.3562	-0.2429
0.0068	1.97	5800	0.0069	0.2817	-16.7894	0.9992	17.0711	-617.9911	-271.6647	0.3329	-0.2424
0.0021	2.01	5900	0.0061	0.0725	-17.3567	0.9992	17.4292	-623.6634	-273.7564	0.3142	-0.2479
0.0031	2.04	6000	0.0066	-0.0179	-18.1905	0.9983	18.1726	-632.0013	-274.6606	0.2948	-0.2445
0.0032	2.07	6100	0.0064	0.0140	-17.8441	0.9992	17.8581	-628.5374	-274.3414	0.3043	-0.2424
0.0043	2.11	6200	0.0066	0.0216	-18.5824	0.9983	18.6041	-635.9210	-274.2653	0.2863	-0.2416
0.0033	2.14	6300	0.0070	0.0355	-18.3245	0.9983	18.3600	-633.3419	-274.1270	0.2783	-0.2691
0.0036	2.18	6400	0.0065	0.1473	-18.2292	0.9983	18.3765	-632.3882	-273.0083	0.2772	-0.2744
0.0023	2.21	6500	0.0064	0.0450	-17.9052	0.9992	17.9502	-629.1483	-274.0311	0.2750	-0.2756
0.0025	2.24	6600	0.0061	-0.0642	-17.4265	0.9983	17.3623	-624.3619	-275.1237	0.2773	-0.2881
0.0008	2.28	6700	0.0058	0.0951	-17.3710	0.9992	17.4661	-623.8071	-273.5309	0.2703	-0.2891
0.002	2.31	6800	0.0057	-0.0533	-18.2744	0.9992	18.2212	-632.8410	-275.0142	0.2544	-0.2823
0.0052	2.35	6900	0.0073	0.0152	-18.6276	0.9983	18.6427	-636.3723	-274.3301	0.2365	-0.2928
0.0029	2.38	7000	0.0056	0.0321	-18.4600	0.9992	18.4921	-634.6968	-274.1606	0.2366	-0.2889
0.0013	2.41	7100	0.0058	-0.1304	-19.1740	0.9983	19.0436	-641.8367	-275.7859	0.2306	-0.2898
0.0011	2.45	7200	0.0059	-0.0977	-19.2699	0.9992	19.1722	-642.7958	-275.4590	0.2227	-0.2916
0.0058	2.48	7300	0.0059	-0.0938	-18.8625	0.9992	18.7687	-638.7216	-275.4195	0.2314	-0.2945
0.004	2.52	7400	0.0067	-0.1457	-19.2264	0.9992	19.0807	-642.3610	-275.9389	0.2194	-0.2959
0.0042	2.55	7500	0.0056	-0.0100	-18.6009	0.9992	18.5909	-636.1058	-274.5812	0.2398	-0.3003
0.0033	2.58	7600	0.0057	-0.1484	-19.3114	0.9992	19.1630	-643.2104	-275.9655	0.2215	-0.2965
0.0012	2.62	7700	0.0051	-0.1574	-19.2398	0.9992	19.0824	-642.4946	-276.0552	0.2210	-0.3052
0.002	2.65	7800	0.0052	-0.1220	-19.5223	0.9992	19.4004	-645.3201	-275.7011	0.2102	-0.3165
0.0053	2.69	7900	0.0051	-0.1063	-19.6889	0.9992	19.5826	-646.9856	-275.5444	0.2106	-0.3074
0.0013	2.72	8000	0.0056	-0.1357	-19.8339	0.9992	19.6982	-648.4356	-275.8386	0.2029	-0.3129
0.002	2.75	8100	0.0050	-0.1024	-19.6476	0.9992	19.5452	-646.5726	-275.5056	0.1993	-0.3168
0.0016	2.79	8200	0.0052	-0.0378	-19.4058	0.9992	19.3680	-644.1544	-274.8596	0.2152	-0.3120
0.0015	2.82	8300	0.0049	-0.0549	-19.2544	0.9992	19.1994	-642.6404	-275.0310	0.2093	-0.3168
0.003	2.86	8400	0.0051	-0.1218	-19.2614	0.9983	19.1396	-642.7106	-275.7000	0.2057	-0.3182
0.0019	2.89	8500	0.0050	-0.1057	-19.3041	0.9992	19.1983	-643.1373	-275.5387	0.2077	-0.3151
0.0039	2.92	8600	0.0052	-0.0863	-19.2826	0.9992	19.1963	-642.9229	-275.3445	0.2057	-0.3208
0.0042	2.96	8700	0.0048	-0.1111	-19.3837	0.9992	19.2726	-643.9339	-275.5926	0.2008	-0.3215
0.0023	2.99	8800	0.0050	-0.1085	-19.4076	0.9992	19.2990	-644.1723	-275.5668	0.1985	-0.3228

Framework versions

• Transformers 4.38.0.dev0
• Pytorch 2.1.1+cu121
• Datasets 2.14.6
• Tokenizers 0.15.1