---
license: mit
base_model: HuggingFaceH4/mistral-7b-sft-beta
tags:
- trl
- dpo
- generated_from_trainer
model-index:
- name: zephyr-7b-dpo-full
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/sanqiang/wdpo/runs/a3szju9y)
# zephyr-7b-dpo-full

This model is a fine-tuned version of [HuggingFaceH4/mistral-7b-sft-beta](https://huggingface.co/HuggingFaceH4/mistral-7b-sft-beta) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.0321
- Rewards/chosen: -1.7573
- Rewards/rejected: -2.2672
- Rewards/accuracies: 0.6959
- Rewards/margins: 0.5100
- Logps/rejected: -377.4232
- Logps/chosen: -319.9703
- Logits/rejected: -1.8383
- Logits/chosen: -1.8642
- Debug/policy Weights: 0.0530
- Debug/losses: 0.0296
- Debug/raw Losses: 0.5668

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-07
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- gradient_accumulation_steps: 2
- total_train_batch_size: 128
- total_eval_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 1

### Training results

| Training Loss | Epoch  | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen | Debug/policy Weights | Debug/losses | Debug/raw Losses |
|:-------------:|:------:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|:--------------------:|:------------:|:----------------:|
| 0.1732        | 0.0796 | 100  | 0.1632          | -0.1423        | -0.1764          | 0.5961             | 0.0340          | -168.3350      | -158.4762    | -2.7038         | -2.7119       | 0.2383               | 0.1617       | 0.6787           |
| 0.077         | 0.1592 | 200  | 0.0820          | -0.7396        | -0.9289          | 0.6297             | 0.1892          | -243.5861      | -218.2062    | -2.5352         | -2.5459       | 0.1265               | 0.0808       | 0.6430           |
| 0.0465        | 0.2388 | 300  | 0.0460          | -1.4486        | -1.8085          | 0.6670             | 0.3600          | -331.5535      | -289.1008    | -2.1315         | -2.1516       | 0.0732               | 0.0441       | 0.6075           |
| 0.0301        | 0.3183 | 400  | 0.0302          | -1.9073        | -2.2520          | 0.6604             | 0.3447          | -375.8980      | -334.9689    | -1.9972         | -2.0210       | 0.0476               | 0.0280       | 0.6068           |
| 0.0365        | 0.3979 | 500  | 0.0424          | -1.5133        | -1.9905          | 0.6838             | 0.4772          | -349.7517      | -295.5778    | -2.1610         | -2.1866       | 0.0695               | 0.0399       | 0.5868           |
| 0.0314        | 0.4775 | 600  | 0.0309          | -1.8400        | -2.2814          | 0.6772             | 0.4414          | -378.8387      | -328.2419    | -2.1045         | -2.1289       | 0.0504               | 0.0288       | 0.5833           |
| 0.0424        | 0.5571 | 700  | 0.0462          | -1.3625        | -1.8115          | 0.6912             | 0.4490          | -331.8509      | -280.4930    | -2.0674         | -2.0900       | 0.0753               | 0.0434       | 0.5756           |
| 0.0289        | 0.6367 | 800  | 0.0295          | -1.8506        | -2.3511          | 0.6978             | 0.5005          | -385.8100      | -329.3051    | -1.8270         | -1.8539       | 0.0480               | 0.0269       | 0.5769           |
| 0.0314        | 0.7163 | 900  | 0.0339          | -1.7372        | -2.2475          | 0.6894             | 0.5102          | -375.4441      | -317.9683    | -1.8414         | -1.8673       | 0.0557               | 0.0309       | 0.5706           |
| 0.0307        | 0.7959 | 1000 | 0.0306          | -1.8549        | -2.3581          | 0.6866             | 0.5033          | -386.5125      | -329.7294    | -1.8126         | -1.8391       | 0.0503               | 0.0279       | 0.5696           |
| 0.0313        | 0.8754 | 1100 | 0.0319          | -1.7477        | -2.2469          | 0.6922             | 0.4992          | -375.3870      | -319.0127    | -1.8471         | -1.8723       | 0.0527               | 0.0294       | 0.5672           |
| 0.031         | 0.9550 | 1200 | 0.0321          | -1.7573        | -2.2672          | 0.6959             | 0.5100          | -377.4232      | -319.9703    | -1.8383         | -1.8642       | 0.0530               | 0.0296       | 0.5668           |


### Framework versions

- Transformers 4.41.0.dev0
- Pytorch 2.1.2+cu121
- Datasets 2.14.6
- Tokenizers 0.19.1