metadata
license: llama3
base_model: meta-llama/Meta-Llama-3-8B-Instruct
tags:
- trl
- dpo
- generated_from_trainer
model-index:
- name: MedQA_L3_1000steps_1e6rate_05beat_CSFTDPO
results: []
MedQA_L3_1000steps_1e6rate_05beat_CSFTDPO
This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on an unknown dataset. It achieves the following results on the evaluation set:
- Loss: 0.5717
- Rewards/chosen: -1.8210
- Rewards/rejected: -5.7186
- Rewards/accuracies: 0.8066
- Rewards/margins: 3.8976
- Logps/rejected: -32.7538
- Logps/chosen: -21.8647
- Logits/rejected: -1.0151
- Logits/chosen: -1.0132
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 1e-06
- train_batch_size: 2
- eval_batch_size: 1
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 4
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 100
- training_steps: 1000
Training results
| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0.7075 | 0.0489 | 50 | 0.6367 | 0.2363 | 0.0705 | 0.6571 | 0.1658 | -21.1755 | -17.7501 | -0.9379 | -0.9373 |
| 0.6451 | 0.0977 | 100 | 0.6114 | -0.8886 | -1.7629 | 0.6923 | 0.8743 | -24.8423 | -19.9998 | -0.9999 | -0.9992 |
| 0.7409 | 0.1466 | 150 | 0.6018 | -1.9813 | -3.3881 | 0.7297 | 1.4068 | -28.0927 | -22.1852 | -0.9814 | -0.9805 |
| 0.4181 | 0.1954 | 200 | 0.5971 | -1.4742 | -3.0996 | 0.7341 | 1.6254 | -27.5157 | -21.1711 | -0.9791 | -0.9778 |
| 0.7476 | 0.2443 | 250 | 0.5735 | -1.5098 | -3.3523 | 0.7648 | 1.8425 | -28.0212 | -21.2423 | -0.9317 | -0.9303 |
| 0.5351 | 0.2931 | 300 | 0.7384 | -1.9600 | -4.7179 | 0.7538 | 2.7579 | -30.7524 | -22.1427 | -0.9715 | -0.9699 |
| 0.3789 | 0.3420 | 350 | 0.6165 | -2.8286 | -5.5771 | 0.7846 | 2.7485 | -32.4706 | -23.8798 | -0.9876 | -0.9860 |
| 0.6639 | 0.3908 | 400 | 0.5874 | -1.6246 | -4.5259 | 0.7912 | 2.9013 | -30.3683 | -21.4718 | -1.0086 | -1.0070 |
| 1.046 | 0.4397 | 450 | 0.5833 | -1.4867 | -4.5791 | 0.8044 | 3.0924 | -30.4748 | -21.1961 | -0.9772 | -0.9753 |
| 1.1477 | 0.4885 | 500 | 0.5726 | -1.9020 | -4.7805 | 0.8022 | 2.8785 | -30.8775 | -22.0266 | -0.9644 | -0.9628 |
| 0.2869 | 0.5374 | 550 | 0.5733 | -1.9387 | -5.0557 | 0.8000 | 3.1170 | -31.4279 | -22.1000 | -0.9901 | -0.9887 |
| 0.3924 | 0.5862 | 600 | 0.5336 | -1.1994 | -4.6601 | 0.8066 | 3.4607 | -30.6367 | -20.6214 | -0.9897 | -0.9880 |
| 0.5685 | 0.6351 | 650 | 0.5600 | -0.6431 | -4.3081 | 0.8000 | 3.6650 | -29.9327 | -19.5088 | -1.0020 | -1.0002 |
| 0.5743 | 0.6839 | 700 | 0.5739 | -1.5294 | -5.3059 | 0.8000 | 3.7764 | -31.9282 | -21.2815 | -1.0088 | -1.0069 |
| 0.5395 | 0.7328 | 750 | 0.5778 | -1.6200 | -5.4658 | 0.8088 | 3.8459 | -32.2482 | -21.4626 | -1.0136 | -1.0117 |
| 0.3395 | 0.7816 | 800 | 0.5754 | -1.8314 | -5.7044 | 0.8000 | 3.8730 | -32.7253 | -21.8854 | -1.0148 | -1.0130 |
| 0.6214 | 0.8305 | 850 | 0.5752 | -1.8114 | -5.6937 | 0.8000 | 3.8823 | -32.7039 | -21.8454 | -1.0152 | -1.0133 |
| 0.9719 | 0.8793 | 900 | 0.5707 | -1.8135 | -5.7132 | 0.8066 | 3.8997 | -32.7430 | -21.8497 | -1.0147 | -1.0128 |
| 0.3164 | 0.9282 | 950 | 0.5710 | -1.8198 | -5.7127 | 0.8000 | 3.8929 | -32.7420 | -21.8623 | -1.0148 | -1.0129 |
| 0.1257 | 0.9770 | 1000 | 0.5717 | -1.8210 | -5.7186 | 0.8066 | 3.8976 | -32.7538 | -21.8647 | -1.0151 | -1.0132 |
Framework versions
- Transformers 4.41.0
- Pytorch 2.0.0+cu117
- Datasets 2.19.1
- Tokenizers 0.19.1