--- license: llama2 library_name: peft tags: - trl - dpo - generated_from_trainer base_model: meta-llama/Llama-2-7b-hf model-index: - name: llama_SFT_e1_DPO_e3 results: [] --- # llama_SFT_e1_DPO_e3 This model is a fine-tuned version of [meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf) on an unknown dataset. It achieves the following results on the evaluation set: - Loss: 0.0745 - Rewards/chosen: 0.4581 - Rewards/rejected: -2.2850 - Rewards/accuracies: 1.0 - Rewards/margins: 2.7431 - Logps/rejected: -231.3585 - Logps/chosen: -178.1707 - Logits/rejected: -1.0559 - Logits/chosen: -0.8886 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 7e-07 - train_batch_size: 1 - eval_batch_size: 1 - seed: 42 - gradient_accumulation_steps: 8 - total_train_batch_size: 8 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: linear - num_epochs: 3 ### Training results | Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen | |:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:| | 0.6769 | 0.1 | 25 | 0.6597 | 0.0250 | -0.0447 | 0.8267 | 0.0696 | -208.9552 | -182.5023 | -1.0564 | -0.8828 | | 0.6046 | 0.2 | 50 | 0.5848 | 0.0656 | -0.1679 | 0.9767 | 0.2335 | -210.1874 | -182.0954 | -1.0566 | -0.8835 | | 0.5278 | 0.3 | 75 | 0.4947 | 0.1260 | -0.3302 | 1.0 | 0.4561 | -211.8100 | -181.4921 | -1.0569 | -0.8842 | | 0.4449 | 0.4 | 100 | 0.4127 | 0.1764 | -0.5163 | 1.0 | 0.6927 | -213.6715 | -180.9882 | -1.0574 | -0.8860 | | 0.3778 | 0.5 | 125 | 0.3368 | 0.2273 | -0.7228 | 1.0 | 0.9502 | -215.7366 | -180.4786 | -1.0575 | -0.8860 | | 0.2947 | 0.6 | 150 | 0.2704 | 0.2755 | -0.9495 | 1.0 | 1.2250 | -218.0034 | -179.9972 | -1.0567 | -0.8893 | | 0.2451 | 0.7 | 175 | 0.2164 | 0.3191 | -1.1794 | 1.0 | 1.4985 | -220.3023 | -179.5607 | -1.0570 | -0.8899 | | 0.1913 | 0.79 | 200 | 0.1737 | 0.3571 | -1.4023 | 1.0 | 1.7594 | -222.5310 | -179.1809 | -1.0573 | -0.8904 | | 0.1553 | 0.89 | 225 | 0.1412 | 0.3875 | -1.6187 | 1.0 | 2.0062 | -224.6958 | -178.8769 | -1.0564 | -0.8903 | | 0.1265 | 0.99 | 250 | 0.1173 | 0.4068 | -1.8122 | 1.0 | 2.2190 | -226.6304 | -178.6835 | -1.0564 | -0.8882 | | 0.1174 | 1.09 | 275 | 0.1029 | 0.4201 | -1.9500 | 1.0 | 2.3701 | -228.0080 | -178.5508 | -1.0555 | -0.8881 | | 0.0915 | 1.19 | 300 | 0.0938 | 0.4300 | -2.0411 | 1.0 | 2.4711 | -228.9190 | -178.4516 | -1.0555 | -0.8891 | | 0.0819 | 1.29 | 325 | 0.0880 | 0.4386 | -2.1112 | 1.0 | 2.5497 | -229.6201 | -178.3662 | -1.0554 | -0.8884 | | 0.09 | 1.39 | 350 | 0.0838 | 0.4485 | -2.1597 | 1.0 | 2.6082 | -230.1051 | -178.2668 | -1.0556 | -0.8886 | | 0.0865 | 1.49 | 375 | 0.0803 | 0.4551 | -2.2012 | 1.0 | 2.6562 | -230.5201 | -178.2012 | -1.0556 | -0.8904 | | 0.0731 | 1.59 | 400 | 0.0787 | 0.4544 | -2.2264 | 1.0 | 2.6807 | -230.7719 | -178.2080 | -1.0555 | -0.8884 | | 0.0734 | 1.69 | 425 | 0.0769 | 0.4580 | -2.2470 | 1.0 | 2.7050 | -230.9783 | -178.1717 | -1.0552 | -0.8884 | | 0.0827 | 1.79 | 450 | 0.0763 | 0.4591 | -2.2582 | 1.0 | 2.7173 | -231.0906 | -178.1612 | -1.0555 | -0.8884 | | 0.0784 | 1.89 | 475 | 0.0756 | 0.4545 | -2.2709 | 1.0 | 2.7253 | -231.2172 | -178.2073 | -1.0556 | -0.8886 | | 0.0768 | 1.99 | 500 | 0.0751 | 0.4564 | -2.2804 | 1.0 | 2.7368 | -231.3123 | -178.1877 | -1.0556 | -0.8883 | | 0.0696 | 2.09 | 525 | 0.0746 | 0.4607 | -2.2824 | 1.0 | 2.7431 | -231.3322 | -178.1449 | -1.0554 | -0.8901 | | 0.0691 | 2.19 | 550 | 0.0743 | 0.4597 | -2.2852 | 1.0 | 2.7449 | -231.3599 | -178.1548 | -1.0557 | -0.8886 | | 0.07 | 2.29 | 575 | 0.0747 | 0.4597 | -2.2807 | 1.0 | 2.7404 | -231.3157 | -178.1549 | -1.0559 | -0.8889 | | 0.0754 | 2.38 | 600 | 0.0744 | 0.4599 | -2.2865 | 1.0 | 2.7463 | -231.3729 | -178.1530 | -1.0555 | -0.8903 | | 0.0833 | 2.48 | 625 | 0.0743 | 0.4588 | -2.2878 | 1.0 | 2.7466 | -231.3862 | -178.1637 | -1.0557 | -0.8887 | | 0.0761 | 2.58 | 650 | 0.0746 | 0.4583 | -2.2856 | 1.0 | 2.7440 | -231.3646 | -178.1684 | -1.0557 | -0.8889 | | 0.0716 | 2.68 | 675 | 0.0745 | 0.4597 | -2.2866 | 1.0 | 2.7463 | -231.3745 | -178.1546 | -1.0558 | -0.8905 | | 0.0755 | 2.78 | 700 | 0.0746 | 0.4592 | -2.2835 | 1.0 | 2.7426 | -231.3430 | -178.1601 | -1.0560 | -0.8889 | | 0.063 | 2.88 | 725 | 0.0740 | 0.4618 | -2.2897 | 1.0 | 2.7516 | -231.4058 | -178.1337 | -1.0556 | -0.8884 | | 0.0743 | 2.98 | 750 | 0.0745 | 0.4581 | -2.2850 | 1.0 | 2.7431 | -231.3585 | -178.1707 | -1.0559 | -0.8886 | ### Framework versions - PEFT 0.8.2 - Transformers 4.38.1 - Pytorch 2.2.0+cu118 - Datasets 2.17.1 - Tokenizers 0.15.2