reflection_dpo.pt
This model is a fine-tuned version of meta-llama/Llama-3.1-8B-Instruct on an unknown dataset. It achieves the following results on the evaluation set:
- Logits/chosen: -0.9495
- Logits/rejected: -0.9444
- Logps/chosen: -75.1085
- Logps/rejected: -153.4672
- Loss: 0.2019
- Rewards/accuracies: 0.9303
- Rewards/chosen: -3.3185
- Rewards/margins: 8.4758
- Rewards/rejected: -11.7943
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- training_steps: 500
Training results
| Training Loss | Epoch | Step | Logits/chosen | Logits/rejected | Logps/chosen | Logps/rejected | Validation Loss | Rewards/accuracies | Rewards/chosen | Rewards/margins | Rewards/rejected |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0.4496 | 0.0032 | 10 | -0.7050 | -0.6918 | -44.4013 | -46.5039 | 0.4352 | 0.8096 | -0.2478 | 0.8502 | -1.0980 |
| 0.3522 | 0.0064 | 20 | -0.7595 | -0.7500 | -46.6875 | -56.6043 | 0.3375 | 0.8405 | -0.4764 | 1.6316 | -2.1080 |
| 0.222 | 0.0096 | 30 | -0.7949 | -0.7888 | -46.7048 | -61.9410 | 0.2981 | 0.8520 | -0.4782 | 2.1635 | -2.6417 |
| 0.1372 | 0.0128 | 40 | -0.8280 | -0.8277 | -43.7714 | -64.4651 | 0.2727 | 0.8635 | -0.1848 | 2.7093 | -2.8941 |
| 0.3171 | 0.0160 | 50 | -0.8491 | -0.8518 | -43.7808 | -70.7219 | 0.2561 | 0.8685 | -0.1858 | 3.3340 | -3.5198 |
| 0.2395 | 0.0192 | 60 | -0.8561 | -0.8587 | -44.3195 | -74.5011 | 0.2392 | 0.8894 | -0.2396 | 3.6581 | -3.8977 |
| 0.1957 | 0.0224 | 70 | -0.8841 | -0.8821 | -46.2857 | -79.3155 | 0.2225 | 0.8958 | -0.4363 | 3.9429 | -4.3792 |
| 0.3734 | 0.0256 | 80 | -0.9419 | -0.9412 | -48.2650 | -82.6414 | 0.2188 | 0.8973 | -0.6342 | 4.0776 | -4.7118 |
| 0.1994 | 0.0288 | 90 | -0.9986 | -1.0003 | -50.5246 | -88.4025 | 0.2237 | 0.8980 | -0.8602 | 4.4277 | -5.2879 |
| 0.142 | 0.0320 | 100 | -1.0438 | -1.0429 | -60.2203 | -104.6738 | 0.2439 | 0.8901 | -1.8297 | 5.0853 | -6.9150 |
| 0.0637 | 0.0352 | 110 | -1.0523 | -1.0494 | -64.6031 | -112.8395 | 0.2623 | 0.8901 | -2.2680 | 5.4636 | -7.7316 |
| 0.3994 | 0.0384 | 120 | -1.0401 | -1.0326 | -67.1203 | -119.2853 | 0.2586 | 0.8930 | -2.5197 | 5.8564 | -8.3761 |
| 0.4963 | 0.0416 | 130 | -1.0260 | -1.0195 | -63.6645 | -114.9173 | 0.2443 | 0.8944 | -2.1741 | 5.7652 | -7.9393 |
| 0.184 | 0.0448 | 140 | -0.9956 | -0.9886 | -64.3210 | -116.1868 | 0.2440 | 0.8994 | -2.2398 | 5.8265 | -8.0663 |
| 0.4548 | 0.0480 | 150 | -0.9475 | -0.9363 | -66.2748 | -120.6504 | 0.2454 | 0.9016 | -2.4352 | 6.0775 | -8.5127 |
| 0.3672 | 0.0512 | 160 | -0.9220 | -0.9124 | -62.2740 | -113.5057 | 0.2140 | 0.9059 | -2.0351 | 5.7631 | -7.7982 |
| 0.1702 | 0.0544 | 170 | -0.9642 | -0.9637 | -53.7962 | -101.3066 | 0.1882 | 0.9167 | -1.1873 | 5.3910 | -6.5783 |
| 0.4943 | 0.0576 | 180 | -0.9846 | -0.9897 | -49.7029 | -94.7573 | 0.1848 | 0.9131 | -0.7780 | 5.1454 | -5.9233 |
| 0.4157 | 0.0608 | 190 | -0.9938 | -0.9997 | -49.2515 | -92.8510 | 0.1796 | 0.9159 | -0.7329 | 4.9999 | -5.7327 |
| 0.1773 | 0.0640 | 200 | -1.0428 | -1.0499 | -50.9003 | -96.4929 | 0.1829 | 0.9159 | -0.8977 | 5.1992 | -6.0969 |
| 0.061 | 0.0672 | 210 | -1.0744 | -1.0813 | -53.2245 | -101.3653 | 0.1894 | 0.9124 | -1.1301 | 5.4540 | -6.5841 |
| 0.2528 | 0.0704 | 220 | -1.0751 | -1.0807 | -55.2461 | -106.9378 | 0.1869 | 0.9174 | -1.3323 | 5.8091 | -7.1414 |
| 0.1233 | 0.0736 | 230 | -1.0647 | -1.0694 | -58.8487 | -115.0116 | 0.1922 | 0.9217 | -1.6926 | 6.2562 | -7.9488 |
| 0.102 | 0.0768 | 240 | -1.0603 | -1.0651 | -60.8948 | -118.7347 | 0.1928 | 0.9203 | -1.8972 | 6.4239 | -8.3211 |
| 0.3324 | 0.0800 | 250 | -1.0533 | -1.0557 | -63.6693 | -125.6155 | 0.1904 | 0.9224 | -2.1746 | 6.8345 | -9.0092 |
| 0.006 | 0.0832 | 260 | -1.0321 | -1.0304 | -72.4092 | -138.8671 | 0.2070 | 0.9195 | -3.0486 | 7.2857 | -10.3343 |
| 0.088 | 0.0864 | 270 | -1.0164 | -1.0114 | -76.2556 | -146.4741 | 0.2187 | 0.9203 | -3.4333 | 7.6618 | -11.0950 |
| 0.1346 | 0.0896 | 280 | -0.9962 | -0.9865 | -80.2022 | -149.6163 | 0.2305 | 0.9188 | -3.8279 | 7.5813 | -11.4092 |
| 0.4596 | 0.0928 | 290 | -0.9952 | -0.9845 | -79.7203 | -148.6709 | 0.2291 | 0.9210 | -3.7797 | 7.5350 | -11.3147 |
| 0.3019 | 0.0960 | 300 | -1.0053 | -0.9956 | -78.3724 | -147.5943 | 0.2222 | 0.9203 | -3.6449 | 7.5621 | -11.2071 |
| 0.0708 | 0.0992 | 310 | -1.0132 | -1.0058 | -75.9911 | -145.4509 | 0.2133 | 0.9188 | -3.4068 | 7.5859 | -10.9927 |
| 0.0371 | 0.1024 | 320 | -1.0142 | -1.0081 | -79.0266 | -152.0327 | 0.2291 | 0.9217 | -3.7104 | 7.9405 | -11.6509 |
| 0.3383 | 0.1056 | 330 | -0.9949 | -0.9877 | -83.4509 | -159.5245 | 0.2452 | 0.9167 | -4.1528 | 8.2473 | -12.4001 |
| 1.1015 | 0.1088 | 340 | -0.9688 | -0.9596 | -84.9845 | -163.4342 | 0.2508 | 0.9210 | -4.3061 | 8.4849 | -12.7910 |
| 0.2088 | 0.1120 | 350 | -0.9577 | -0.9474 | -83.8031 | -160.2732 | 0.2460 | 0.9181 | -4.1880 | 8.2869 | -12.4749 |
| 0.3555 | 0.1152 | 360 | -0.9630 | -0.9531 | -80.5315 | -156.9231 | 0.2314 | 0.9210 | -3.8608 | 8.2791 | -12.1399 |
| 0.197 | 0.1184 | 370 | -0.9738 | -0.9651 | -78.5307 | -154.6240 | 0.2215 | 0.9246 | -3.6608 | 8.2493 | -11.9100 |
| 0.5949 | 0.1216 | 380 | -0.9840 | -0.9770 | -75.5271 | -151.0730 | 0.2079 | 0.9260 | -3.3604 | 8.1945 | -11.5549 |
| 0.3272 | 0.1248 | 390 | -0.9869 | -0.9810 | -74.7702 | -150.4924 | 0.2008 | 0.9332 | -3.2847 | 8.2121 | -11.4969 |
| 0.0613 | 0.1280 | 400 | -0.9777 | -0.9725 | -75.3109 | -151.8670 | 0.1995 | 0.9325 | -3.3388 | 8.2955 | -11.6343 |
| 0.3468 | 0.1312 | 410 | -0.9809 | -0.9769 | -73.3059 | -148.0693 | 0.1949 | 0.9289 | -3.1383 | 8.1163 | -11.2545 |
| 0.4202 | 0.1344 | 420 | -0.9788 | -0.9744 | -73.1894 | -148.3277 | 0.1945 | 0.9274 | -3.1266 | 8.1537 | -11.2804 |
| 0.2138 | 0.1376 | 430 | -0.9738 | -0.9687 | -73.5879 | -149.2324 | 0.1947 | 0.9289 | -3.1665 | 8.2044 | -11.3709 |
| 0.475 | 0.1408 | 440 | -0.9682 | -0.9636 | -73.3113 | -149.3820 | 0.1959 | 0.9303 | -3.1388 | 8.2470 | -11.3858 |
| 0.0014 | 0.1440 | 450 | -0.9672 | -0.9625 | -73.3271 | -149.4055 | 0.1976 | 0.9296 | -3.1404 | 8.2478 | -11.3882 |
| 0.0216 | 0.1472 | 460 | -0.9631 | -0.9585 | -73.6283 | -150.2499 | 0.1994 | 0.9303 | -3.1705 | 8.3021 | -11.4726 |
| 0.1517 | 0.1504 | 470 | -0.9563 | -0.9517 | -74.2343 | -151.6451 | 0.2018 | 0.9303 | -3.2311 | 8.3810 | -11.6121 |
| 0.3719 | 0.1536 | 480 | -0.9516 | -0.9466 | -74.7146 | -152.7081 | 0.2016 | 0.9303 | -3.2792 | 8.4393 | -11.7184 |
| 0.1176 | 0.1567 | 490 | -0.9502 | -0.9450 | -75.0118 | -153.3125 | 0.2020 | 0.9296 | -3.3089 | 8.4700 | -11.7789 |
| 0.2333 | 0.1599 | 500 | -0.9495 | -0.9444 | -75.1085 | -153.4672 | 0.2019 | 0.9303 | -3.3185 | 8.4758 | -11.7943 |
Framework versions
- PEFT 0.12.0
- Transformers 4.44.2
- Pytorch 2.4.0+cu124
- Datasets 2.21.0
- Tokenizers 0.19.1
- Downloads last month
- 4
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for hlee131/Llama-3-OOD-Reflection-DPO
Base model
meta-llama/Llama-3.1-8B Finetuned
meta-llama/Llama-3.1-8B-Instruct