zephyr-dpop-qlora-uf-5e-7

This model is a fine-tuned version of alignment-handbook/zephyr-7b-sft-full on the HuggingFaceH4/ultrafeedback_binarized dataset. It achieves the following results on the evaluation set:

• Loss: 0.6789
• Positive Losses: 0.2624
• Dpo Losses: 0.6395
• Rewards/chosen: 0.2321
• Rewards/rejected: 0.1098
• Rewards/accuracies: 0.7180
• Rewards/margins: 0.1223
• Rewards/margins Max: 0.4485
• Rewards/margins Min: -0.1548
• Rewards/margins Std: 0.2025
• Logps/rejected: -247.6011
• Logps/chosen: -261.3811
• Logits/rejected: -2.6202
• Logits/chosen: -2.6543

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-06
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• distributed_type: multi-GPU
• num_devices: 2
• gradient_accumulation_steps: 2
• total_train_batch_size: 16
• total_eval_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Positive Losses	Dpo Losses	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Rewards/margins Max	Rewards/margins Min	Rewards/margins Std	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6911	0.03	100	0.6918	0.0097	0.6901	0.0256	0.0195	0.6670	0.0061	0.0282	-0.0136	0.0138	-256.6321	-282.0331	-2.7662	-2.8054
0.6847	0.05	200	0.6919	0.0310	0.6806	0.0844	0.0583	0.6710	0.0261	0.1132	-0.0512	0.0542	-252.7455	-276.1540	-2.7592	-2.7987
0.686	0.08	300	0.6901	0.0841	0.6693	0.1465	0.0956	0.6950	0.0509	0.2071	-0.0915	0.0989	-249.0196	-269.9467	-2.7474	-2.7859
0.6944	0.1	400	0.6911	0.1510	0.6631	0.1581	0.0931	0.7100	0.0650	0.2490	-0.1113	0.1195	-249.2730	-268.7827	-2.7115	-2.7504
0.6923	0.13	500	0.6788	0.0596	0.6647	0.1948	0.1332	0.6950	0.0617	0.2513	-0.1077	0.1190	-245.2602	-265.1090	-2.6843	-2.7243
0.663	0.16	600	0.6892	0.1483	0.6607	0.1942	0.1226	0.6770	0.0716	0.3008	-0.1286	0.1420	-246.3230	-265.1740	-2.6660	-2.7036
0.6784	0.18	700	0.6935	0.2142	0.6550	0.1892	0.1049	0.6970	0.0843	0.3275	-0.1274	0.1516	-248.0892	-265.6756	-2.6229	-2.6624
0.661	0.21	800	0.6885	0.1770	0.6538	0.1994	0.1122	0.7020	0.0872	0.3388	-0.1292	0.1549	-247.3548	-264.6508	-2.6850	-2.7245
0.6736	0.24	900	0.6827	0.1576	0.6557	0.2025	0.1192	0.6940	0.0833	0.3345	-0.1335	0.1561	-246.6593	-264.3388	-2.6814	-2.7201
0.6998	0.26	1000	0.6806	0.2131	0.6517	0.2037	0.1115	0.7070	0.0922	0.3499	-0.1335	0.1615	-247.4245	-264.2192	-2.6830	-2.7190
0.6943	0.29	1100	0.6808	0.2125	0.6503	0.2101	0.1144	0.7100	0.0957	0.3629	-0.1371	0.1674	-247.1344	-263.5789	-2.6633	-2.6979
0.6761	0.31	1200	0.6793	0.1898	0.6511	0.2157	0.1215	0.7110	0.0942	0.3704	-0.1366	0.1692	-246.4255	-263.0201	-2.6573	-2.6916
0.6976	0.34	1300	0.6730	0.1194	0.6535	0.2178	0.1297	0.7080	0.0881	0.3434	-0.1322	0.1594	-245.6055	-262.8122	-2.6282	-2.6641
0.7536	0.37	1400	0.7005	0.3143	0.6471	0.2121	0.1083	0.7030	0.1038	0.3986	-0.1530	0.1838	-247.7509	-263.3833	-2.6211	-2.6572
0.6711	0.39	1500	0.6918	0.2213	0.6489	0.2190	0.1197	0.7040	0.0994	0.3826	-0.1451	0.1760	-246.6128	-262.6917	-2.5983	-2.6356
0.7428	0.42	1600	0.6867	0.1652	0.6501	0.2193	0.1228	0.7010	0.0965	0.3730	-0.1448	0.1730	-246.2957	-262.6611	-2.5979	-2.6328
0.6593	0.44	1700	0.6785	0.2228	0.6467	0.2221	0.1173	0.7110	0.1048	0.3978	-0.1483	0.1825	-246.8526	-262.3859	-2.6262	-2.6614
0.6856	0.47	1800	0.6702	0.1343	0.6504	0.2318	0.1356	0.6980	0.0962	0.3760	-0.1454	0.1748	-245.0162	-261.4142	-2.5972	-2.6326
0.6552	0.5	1900	0.6743	0.1855	0.6484	0.2278	0.1267	0.6990	0.1011	0.3920	-0.1494	0.1816	-245.9063	-261.8096	-2.5761	-2.6118
0.6577	0.52	2000	0.6748	0.2036	0.6461	0.2310	0.1247	0.7090	0.1063	0.4016	-0.1526	0.1853	-246.1064	-261.4890	-2.5869	-2.6241
0.6695	0.55	2100	0.6841	0.2842	0.6443	0.2230	0.1124	0.7100	0.1106	0.4202	-0.1537	0.1915	-247.3420	-262.2980	-2.6033	-2.6404
0.6633	0.58	2200	0.6799	0.2580	0.6435	0.2273	0.1147	0.7140	0.1126	0.4254	-0.1549	0.1932	-247.1040	-261.8589	-2.6014	-2.6383
0.7136	0.6	2300	0.6781	0.2376	0.6443	0.2290	0.1183	0.7110	0.1107	0.4197	-0.1532	0.1914	-246.7446	-261.6907	-2.6118	-2.6471
0.6631	0.63	2400	0.6769	0.2289	0.6450	0.2285	0.1195	0.7080	0.1090	0.4134	-0.1509	0.1882	-246.6301	-261.7479	-2.6072	-2.6430
0.6884	0.65	2500	0.6854	0.3215	0.6404	0.2248	0.1047	0.7120	0.1201	0.4408	-0.1583	0.2000	-248.1103	-262.1167	-2.6064	-2.6413
0.6701	0.68	2600	0.6817	0.2661	0.6432	0.2290	0.1154	0.7240	0.1136	0.4344	-0.1554	0.1960	-247.0384	-261.6952	-2.6116	-2.6458
0.668	0.71	2700	0.6771	0.2209	0.6441	0.2330	0.1218	0.7190	0.1112	0.4213	-0.1525	0.1911	-246.4004	-261.2966	-2.6196	-2.6533
0.6851	0.73	2800	0.6777	0.2299	0.6430	0.2330	0.1192	0.7090	0.1138	0.4274	-0.1550	0.1946	-246.6621	-261.2937	-2.6278	-2.6613
0.678	0.76	2900	0.6856	0.2997	0.6402	0.2278	0.1072	0.7110	0.1207	0.4462	-0.1603	0.2028	-247.8615	-261.8085	-2.6269	-2.6602
0.6605	0.79	3000	0.6807	0.2415	0.6412	0.2316	0.1134	0.7160	0.1182	0.4380	-0.1547	0.1986	-247.2367	-261.4324	-2.6275	-2.6605
0.6874	0.81	3100	0.6753	0.2061	0.6425	0.2349	0.1199	0.7190	0.1150	0.4300	-0.1520	0.1951	-246.5852	-261.0995	-2.6151	-2.6494
0.6516	0.84	3200	0.6828	0.3006	0.6385	0.2284	0.1036	0.7160	0.1248	0.4527	-0.1586	0.2052	-248.2176	-261.7539	-2.6158	-2.6498
0.6627	0.86	3300	0.6773	0.2406	0.6403	0.2325	0.1123	0.7190	0.1203	0.4419	-0.1545	0.2003	-247.3520	-261.3398	-2.6184	-2.6526
0.6517	0.89	3400	0.6814	0.2865	0.6386	0.2300	0.1056	0.7190	0.1244	0.4519	-0.1569	0.2045	-248.0181	-261.5968	-2.6213	-2.6551
0.7267	0.92	3500	0.6810	0.2880	0.6385	0.2302	0.1056	0.7200	0.1246	0.4536	-0.1569	0.2050	-248.0208	-261.5744	-2.6222	-2.6560
0.6563	0.94	3600	0.6790	0.2627	0.6394	0.2318	0.1093	0.7220	0.1225	0.4487	-0.1550	0.2027	-247.6491	-261.4136	-2.6216	-2.6555
0.7039	0.97	3700	0.6790	0.2634	0.6396	0.2320	0.1099	0.7230	0.1222	0.4483	-0.1550	0.2025	-247.5927	-261.3918	-2.6220	-2.6559
0.6622	0.99	3800	0.6789	0.2612	0.6395	0.2320	0.1098	0.7220	0.1222	0.4482	-0.1549	0.2025	-247.6030	-261.3938	-2.6204	-2.6544

Framework versions

• PEFT 0.7.1
• Transformers 4.39.0.dev0
• Pytorch 2.1.2+cu121
• Datasets 2.14.6
• Tokenizers 0.15.2