zephyr-7b-dpo-full
This model is a fine-tuned version of alignment-handbook/zephyr-7b-sft-full on the HuggingFaceH4/ultrafeedback_binarized dataset.
It achieves the following results on the evaluation set:
- Loss: 0.4980
- Rewards/chosen: -2.1242
- Rewards/rejected: -3.0843
- Rewards/accuracies: 0.7380
- Rewards/margins: 0.9601
- Logps/rejected: -497.6194
- Logps/chosen: -397.4371
- Logits/rejected: -0.2690
- Logits/chosen: -0.8689
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-07
- train_batch_size: 1
- eval_batch_size: 2
- seed: 42
- distributed_type: multi-GPU
- num_devices: 4
- gradient_accumulation_steps: 8
- total_train_batch_size: 32
- total_eval_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 1
Training results
| Training Loss |
Epoch |
Step |
Validation Loss |
Rewards/chosen |
Rewards/rejected |
Rewards/accuracies |
Rewards/margins |
Logps/rejected |
Logps/chosen |
Logits/rejected |
Logits/chosen |
| 0.5294 |
0.2617 |
500 |
0.5470 |
-1.7358 |
-2.3925 |
0.6980 |
0.6567 |
-428.4361 |
-358.6011 |
-0.4724 |
-0.8639 |
| 0.5232 |
0.5234 |
1000 |
0.5099 |
-1.9184 |
-2.7566 |
0.7160 |
0.8382 |
-464.8497 |
-376.8646 |
-0.0573 |
-0.6162 |
| 0.4707 |
0.7851 |
1500 |
0.5000 |
-2.1875 |
-3.1436 |
0.7320 |
0.9561 |
-503.5489 |
-403.7713 |
-0.0548 |
-0.6702 |
Framework versions
- Transformers 4.40.1
- Pytorch 2.1.2
- Datasets 2.19.0
- Tokenizers 0.19.1
