Hugging Face's logo

wzhouad
/
zephyr-7b-dpo-full

Text Generation
Transformers
Safetensors
llama
trl
dpo
Generated from Trainer
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
zephyr-7b-dpo-full / README.md
wzhouad's picture
wzhouad
Model save
6538785 verified
preview code
|
raw
history blame
No virus
5.36 kB
metadata
license: mit
base_model: HuggingFaceH4/mistral-7b-sft-beta
tags:
  - trl
  - dpo
  - generated_from_trainer
model-index:
  - name: zephyr-7b-dpo-full
    results: []

Visualize in Weights & Biases

zephyr-7b-dpo-full

This model is a fine-tuned version of HuggingFaceH4/mistral-7b-sft-beta on the None dataset. It achieves the following results on the evaluation set:

  • Loss: 0.0321
  • Rewards/chosen: -1.7573
  • Rewards/rejected: -2.2672
  • Rewards/accuracies: 0.6959
  • Rewards/margins: 0.5100
  • Logps/rejected: -377.4232
  • Logps/chosen: -319.9703
  • Logits/rejected: -1.8383
  • Logits/chosen: -1.8642
  • Debug/policy Weights: 0.0530
  • Debug/losses: 0.0296
  • Debug/raw Losses: 0.5668

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 5e-07
  • train_batch_size: 8
  • eval_batch_size: 8
  • seed: 42
  • distributed_type: multi-GPU
  • num_devices: 8
  • gradient_accumulation_steps: 2
  • total_train_batch_size: 128
  • total_eval_batch_size: 64
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 1

Training results

Training Loss Epoch Step Validation Loss Rewards/chosen Rewards/rejected Rewards/accuracies Rewards/margins Logps/rejected Logps/chosen Logits/rejected Logits/chosen Debug/policy Weights Debug/losses Debug/raw Losses
0.1732 0.0796 100 0.1632 -0.1423 -0.1764 0.5961 0.0340 -168.3350 -158.4762 -2.7038 -2.7119 0.2383 0.1617 0.6787
0.077 0.1592 200 0.0820 -0.7396 -0.9289 0.6297 0.1892 -243.5861 -218.2062 -2.5352 -2.5459 0.1265 0.0808 0.6430
0.0465 0.2388 300 0.0460 -1.4486 -1.8085 0.6670 0.3600 -331.5535 -289.1008 -2.1315 -2.1516 0.0732 0.0441 0.6075
0.0301 0.3183 400 0.0302 -1.9073 -2.2520 0.6604 0.3447 -375.8980 -334.9689 -1.9972 -2.0210 0.0476 0.0280 0.6068
0.0365 0.3979 500 0.0424 -1.5133 -1.9905 0.6838 0.4772 -349.7517 -295.5778 -2.1610 -2.1866 0.0695 0.0399 0.5868
0.0314 0.4775 600 0.0309 -1.8400 -2.2814 0.6772 0.4414 -378.8387 -328.2419 -2.1045 -2.1289 0.0504 0.0288 0.5833
0.0424 0.5571 700 0.0462 -1.3625 -1.8115 0.6912 0.4490 -331.8509 -280.4930 -2.0674 -2.0900 0.0753 0.0434 0.5756
0.0289 0.6367 800 0.0295 -1.8506 -2.3511 0.6978 0.5005 -385.8100 -329.3051 -1.8270 -1.8539 0.0480 0.0269 0.5769
0.0314 0.7163 900 0.0339 -1.7372 -2.2475 0.6894 0.5102 -375.4441 -317.9683 -1.8414 -1.8673 0.0557 0.0309 0.5706
0.0307 0.7959 1000 0.0306 -1.8549 -2.3581 0.6866 0.5033 -386.5125 -329.7294 -1.8126 -1.8391 0.0503 0.0279 0.5696
0.0313 0.8754 1100 0.0319 -1.7477 -2.2469 0.6922 0.4992 -375.3870 -319.0127 -1.8471 -1.8723 0.0527 0.0294 0.5672
0.031 0.9550 1200 0.0321 -1.7573 -2.2672 0.6959 0.5100 -377.4232 -319.9703 -1.8383 -1.8642 0.0530 0.0296 0.5668

Framework versions

  • Transformers 4.41.0.dev0
  • Pytorch 2.1.2+cu121
  • Datasets 2.14.6
  • Tokenizers 0.19.1