zephyr-7b-dpo-qlora
This model is a fine-tuned version of alignment-handbook/zephyr-7b-sft-qlora on the HuggingFaceH4/ultrafeedback_binarized dataset. It achieves the following results on the evaluation set:
- Loss: 0.5018
- Rewards/chosen: -2.1482
- Rewards/rejected: -3.1540
- Rewards/accuracies: 0.7590
- Rewards/margins: 1.0058
- Logps/rejected: -556.6644
- Logps/chosen: -480.1277
- Logits/rejected: -1.2931
- Logits/chosen: -1.3827
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-06
- train_batch_size: 4
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- num_devices: 2
- gradient_accumulation_steps: 4
- total_train_batch_size: 32
- total_eval_batch_size: 16
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 1
Training results
| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0.6635 | 0.0523 | 100 | 0.6640 | 0.0129 | -0.0631 | 0.6830 | 0.0761 | -247.5831 | -264.0175 | -2.0469 | -2.1424 |
| 0.6119 | 0.1047 | 200 | 0.6207 | -0.5556 | -0.8212 | 0.6790 | 0.2657 | -323.3911 | -320.8676 | -1.9524 | -2.0449 |
| 0.5874 | 0.1570 | 300 | 0.5849 | -0.4240 | -0.8044 | 0.7000 | 0.3804 | -321.7128 | -307.7115 | -1.9609 | -2.0494 |
| 0.5608 | 0.2094 | 400 | 0.5607 | -1.1817 | -1.7752 | 0.7290 | 0.5935 | -418.7894 | -383.4811 | -1.6969 | -1.7823 |
| 0.5287 | 0.2617 | 500 | 0.5434 | -1.7248 | -2.4550 | 0.7250 | 0.7303 | -486.7726 | -437.7878 | -1.5394 | -1.6284 |
| 0.5504 | 0.3141 | 600 | 0.5278 | -1.3541 | -2.1302 | 0.7370 | 0.7761 | -454.2872 | -400.7156 | -1.4439 | -1.5287 |
| 0.5243 | 0.3664 | 700 | 0.5278 | -0.9934 | -1.7415 | 0.7420 | 0.7481 | -415.4179 | -364.6462 | -1.4888 | -1.5754 |
| 0.5346 | 0.4187 | 800 | 0.5285 | -1.0509 | -1.8191 | 0.7360 | 0.7681 | -423.1764 | -370.4044 | -1.4861 | -1.5718 |
| 0.5072 | 0.4711 | 900 | 0.5197 | -1.6324 | -2.5736 | 0.7300 | 0.9412 | -498.6239 | -428.5474 | -1.3651 | -1.4531 |
| 0.5023 | 0.5234 | 1000 | 0.5158 | -1.6927 | -2.6755 | 0.7460 | 0.9828 | -508.8179 | -434.5808 | -1.2853 | -1.3779 |
| 0.4954 | 0.5758 | 1100 | 0.5126 | -1.4605 | -2.3370 | 0.7480 | 0.8765 | -474.9688 | -411.3603 | -1.3921 | -1.4843 |
| 0.4983 | 0.6281 | 1200 | 0.5105 | -2.0566 | -3.0678 | 0.7450 | 1.0112 | -548.0505 | -470.9687 | -1.1942 | -1.2848 |
| 0.4774 | 0.6805 | 1300 | 0.5093 | -1.9802 | -3.0112 | 0.7510 | 1.0311 | -542.3931 | -463.3254 | -1.2574 | -1.3491 |
| 0.4516 | 0.7328 | 1400 | 0.5058 | -2.1539 | -3.2003 | 0.7530 | 1.0464 | -561.2969 | -480.7002 | -1.2592 | -1.3500 |
| 0.4758 | 0.7851 | 1500 | 0.5018 | -2.2342 | -3.2427 | 0.7550 | 1.0085 | -565.5339 | -488.7257 | -1.2803 | -1.3710 |
| 0.4967 | 0.8375 | 1600 | 0.5019 | -2.1690 | -3.1744 | 0.7590 | 1.0054 | -558.7111 | -482.2090 | -1.2939 | -1.3837 |
| 0.4769 | 0.8898 | 1700 | 0.5018 | -2.1431 | -3.1460 | 0.7600 | 1.0029 | -555.8691 | -479.6245 | -1.2936 | -1.3834 |
| 0.4843 | 0.9422 | 1800 | 0.5019 | -2.1475 | -3.1534 | 0.7580 | 1.0059 | -556.6094 | -480.0620 | -1.2932 | -1.3829 |
| 0.5048 | 0.9945 | 1900 | 0.5019 | -2.1484 | -3.1540 | 0.7590 | 1.0056 | -556.6639 | -480.1491 | -1.2933 | -1.3829 |
Framework versions
- PEFT 0.7.1
- Transformers 4.40.1
- Pytorch 2.1.2
- Datasets 2.19.0
- Tokenizers 0.19.1
- Downloads last month
- 0
Model tree for chrlu/zephyr-7b-dpo-qlora
Base model
mistralai/Mistral-7B-v0.1