Hugging Face's logo

weqweasdas
/
zephyr-7b-dpo-full

Text Generation
Transformers
TensorBoard
Safetensors
mistral
alignment-handbook
Generated from Trainer
trl
dpo
conversational
text-generation-inference
Inference Endpoints
Model card Files Metrics Community
weqweasdas commited on
Commit
163a796
•
1 Parent(s): 8d6ef03

Model save

Browse files
Files changed (11) hide show
  1. README.md +37 -27
  2. all_results.json +18 -18
  3. eval_results.json +13 -13
  4. generation_config.json +1 -1
  5. model-00001-of-00003.safetensors +1 -1
  6. model-00002-of-00003.safetensors +1 -1
  7. model-00003-of-00003.safetensors +1 -1
  8. runs/May02_12-22-32_temp1a22200000C/events.out.tfevents.1714652888.temp1a22200000C.2457569.0 +2 -2
  9. runs/May02_12-22-32_temp1a22200000C/events.out.tfevents.1714702156.temp1a22200000C.2457569.1 +3 -0
  10. train_results.json +6 -6
  11. trainer_state.json +0 -0
README.md CHANGED
@@ -2,15 +2,9 @@
2
  license: apache-2.0
3
  base_model: alignment-handbook/zephyr-7b-sft-full
4
  tags:
5
- - alignment-handbook
6
  - trl
7
  - dpo
8
  - generated_from_trainer
9
- - trl
10
- - dpo
11
- - generated_from_trainer
12
- datasets:
13
- - HuggingFaceH4/ultrafeedback_binarized
14
  model-index:
15
  - name: zephyr-7b-dpo-full
16
  results: []
@@ -21,17 +15,17 @@ should probably proofread and complete it, then remove this comment. -->
21
 
22
  # zephyr-7b-dpo-full
23
 
24
- This model is a fine-tuned version of [alignment-handbook/zephyr-7b-sft-full](https://huggingface.co/alignment-handbook/zephyr-7b-sft-full) on the HuggingFaceH4/ultrafeedback_binarized dataset.
25
  It achieves the following results on the evaluation set:
26
- - Loss: 0.5027
27
- - Rewards/chosen: -1.0713
28
- - Rewards/rejected: -1.8869
29
- - Rewards/accuracies: 0.7550
30
- - Rewards/margins: 0.8156
31
- - Logps/rejected: -456.8468
32
- - Logps/chosen: -397.0062
33
- - Logits/rejected: -0.9628
34
- - Logits/chosen: -1.4755
35
 
36
  ## Model description
37
 
@@ -51,31 +45,47 @@ More information needed
51
 
52
  The following hyperparameters were used during training:
53
  - learning_rate: 5e-07
54
- - train_batch_size: 2
55
- - eval_batch_size: 2
56
  - seed: 42
57
  - distributed_type: multi-GPU
58
- - num_devices: 10
59
- - gradient_accumulation_steps: 8
60
- - total_train_batch_size: 160
61
- - total_eval_batch_size: 20
62
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
63
  - lr_scheduler_type: cosine
64
  - lr_scheduler_warmup_ratio: 0.1
65
- - num_epochs: 1
66
 
67
  ### Training results
68
 
69
  | Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
70
  |:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
71
- | 0.5474 | 0.26 | 100 | 0.5625 | -0.7544 | -1.1673 | 0.6850 | 0.4129 | -384.8893 | -365.3133 | -2.2414 | -2.3089 |
72
- | 0.5291 | 0.52 | 200 | 0.5176 | -1.2026 | -1.9705 | 0.75 | 0.7679 | -465.2114 | -410.1355 | -1.1595 | -1.5009 |
73
- | 0.4891 | 0.79 | 300 | 0.5042 | -1.1309 | -1.9677 | 0.7550 | 0.8368 | -464.9229 | -402.9644 | -0.9395 | -1.4685 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
74
 
75
 
76
  ### Framework versions
77
 
78
- - Transformers 4.39.0.dev0
79
  - Pytorch 2.1.2
80
  - Datasets 2.14.6
81
  - Tokenizers 0.15.2
 
2
  license: apache-2.0
3
  base_model: alignment-handbook/zephyr-7b-sft-full
4
  tags:
 
5
  - trl
6
  - dpo
7
  - generated_from_trainer
 
 
 
 
 
8
  model-index:
9
  - name: zephyr-7b-dpo-full
10
  results: []
 
15
 
16
  # zephyr-7b-dpo-full
17
 
18
+ This model is a fine-tuned version of [alignment-handbook/zephyr-7b-sft-full](https://huggingface.co/alignment-handbook/zephyr-7b-sft-full) on the None dataset.
19
  It achieves the following results on the evaluation set:
20
+ - Loss: 0.5590
21
+ - Rewards/chosen: -0.7818
22
+ - Rewards/rejected: -2.7115
23
+ - Rewards/accuracies: 0.7857
24
+ - Rewards/margins: 1.9297
25
+ - Logps/rejected: -287.3273
26
+ - Logps/chosen: -289.7805
27
+ - Logits/rejected: -2.4561
28
+ - Logits/chosen: -2.5007
29
 
30
  ## Model description
31
 
 
45
 
46
  The following hyperparameters were used during training:
47
  - learning_rate: 5e-07
48
+ - train_batch_size: 8
49
+ - eval_batch_size: 8
50
  - seed: 42
51
  - distributed_type: multi-GPU
52
+ - num_devices: 4
53
+ - gradient_accumulation_steps: 2
54
+ - total_train_batch_size: 64
55
+ - total_eval_batch_size: 32
56
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
57
  - lr_scheduler_type: cosine
58
  - lr_scheduler_warmup_ratio: 0.1
59
+ - num_epochs: 2
60
 
61
  ### Training results
62
 
63
  | Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
64
  |:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
65
+ | 0.6075 | 0.1 | 100 | 0.5945 | 0.3241 | -0.1206 | 0.7163 | 0.4447 | -261.4175 | -278.7209 | -2.6324 | -2.6651 |
66
+ | 0.5341 | 0.21 | 200 | 0.5471 | -0.0734 | -1.0103 | 0.7639 | 0.9369 | -270.3152 | -282.6963 | -2.5394 | -2.5779 |
67
+ | 0.5315 | 0.31 | 300 | 0.5258 | 0.1435 | -0.9757 | 0.7619 | 1.1192 | -269.9694 | -280.5274 | -2.5337 | -2.5711 |
68
+ | 0.4978 | 0.42 | 400 | 0.5366 | -0.2177 | -1.2826 | 0.7579 | 1.0649 | -273.0383 | -284.1391 | -2.5667 | -2.6011 |
69
+ | 0.5134 | 0.52 | 500 | 0.5340 | -0.4713 | -1.5140 | 0.7460 | 1.0427 | -275.3516 | -286.6748 | -2.4488 | -2.4836 |
70
+ | 0.5404 | 0.63 | 600 | 0.5188 | -0.0534 | -1.2981 | 0.7480 | 1.2447 | -273.1928 | -282.4962 | -2.3631 | -2.4039 |
71
+ | 0.5256 | 0.73 | 700 | 0.5270 | -0.2533 | -1.5704 | 0.7639 | 1.3172 | -275.9163 | -284.4948 | -2.3224 | -2.3640 |
72
+ | 0.4991 | 0.84 | 800 | 0.5278 | -0.2394 | -1.5276 | 0.7639 | 1.2882 | -275.4879 | -284.3556 | -2.3730 | -2.4144 |
73
+ | 0.5084 | 0.94 | 900 | 0.5457 | 0.2664 | -0.9546 | 0.7619 | 1.2210 | -269.7581 | -279.2981 | -2.4875 | -2.5254 |
74
+ | 0.1011 | 1.05 | 1000 | 0.5361 | -0.5236 | -2.1364 | 0.7877 | 1.6129 | -281.5762 | -287.1976 | -2.4389 | -2.4774 |
75
+ | 0.0942 | 1.15 | 1100 | 0.5454 | -0.4356 | -2.2047 | 0.7897 | 1.7691 | -282.2592 | -286.3182 | -2.4515 | -2.4926 |
76
+ | 0.0817 | 1.26 | 1200 | 0.5530 | -0.7588 | -2.5855 | 0.7857 | 1.8268 | -286.0674 | -289.5495 | -2.4441 | -2.4863 |
77
+ | 0.0697 | 1.36 | 1300 | 0.5549 | -0.5919 | -2.4690 | 0.7798 | 1.8771 | -284.9021 | -287.8810 | -2.4474 | -2.4910 |
78
+ | 0.0842 | 1.47 | 1400 | 0.5575 | -0.7425 | -2.6443 | 0.7917 | 1.9018 | -286.6550 | -289.3871 | -2.4669 | -2.5100 |
79
+ | 0.075 | 1.57 | 1500 | 0.5590 | -0.5382 | -2.4532 | 0.7956 | 1.9150 | -284.7438 | -287.3436 | -2.4699 | -2.5133 |
80
+ | 0.098 | 1.67 | 1600 | 0.5583 | -0.7761 | -2.6741 | 0.7877 | 1.8980 | -286.9528 | -289.7227 | -2.4652 | -2.5092 |
81
+ | 0.0718 | 1.78 | 1700 | 0.5593 | -0.7532 | -2.6704 | 0.7877 | 1.9172 | -286.9160 | -289.4940 | -2.4592 | -2.5036 |
82
+ | 0.0828 | 1.88 | 1800 | 0.5606 | -0.7985 | -2.7306 | 0.7897 | 1.9321 | -287.5178 | -289.9467 | -2.4560 | -2.5007 |
83
+ | 0.103 | 1.99 | 1900 | 0.5601 | -0.7805 | -2.7113 | 0.7857 | 1.9309 | -287.3255 | -289.7666 | -2.4554 | -2.5002 |
84
 
85
 
86
  ### Framework versions
87
 
88
+ - Transformers 4.36.2
89
  - Pytorch 2.1.2
90
  - Datasets 2.14.6
91
  - Tokenizers 0.15.2
all_results.json CHANGED
@@ -1,21 +1,21 @@
1
  {
2
- "epoch": 1.0,
3
- "eval_logits/chosen": -1.475502371788025,
4
- "eval_logits/rejected": -0.962776243686676,
5
- "eval_logps/chosen": -397.00616455078125,
6
- "eval_logps/rejected": -456.8468017578125,
7
- "eval_loss": 0.5026616454124451,
8
- "eval_rewards/accuracies": 0.7549999952316284,
9
- "eval_rewards/chosen": -1.0713179111480713,
10
- "eval_rewards/margins": 0.815581202507019,
11
- "eval_rewards/rejected": -1.8868991136550903,
12
- "eval_runtime": 693.918,
13
  "eval_samples": 2000,
14
- "eval_samples_per_second": 2.882,
15
- "eval_steps_per_second": 0.144,
16
- "train_loss": 0.5410362510156881,
17
- "train_runtime": 45025.8645,
18
- "train_samples": 61134,
19
- "train_samples_per_second": 1.358,
20
- "train_steps_per_second": 0.008
21
  }
 
1
  {
2
+ "epoch": 2.0,
3
+ "eval_logits/chosen": -2.500744581222534,
4
+ "eval_logits/rejected": -2.4560909271240234,
5
+ "eval_logps/chosen": -289.780517578125,
6
+ "eval_logps/rejected": -287.32733154296875,
7
+ "eval_loss": 0.5590207576751709,
8
+ "eval_rewards/accuracies": 0.7857142686843872,
9
+ "eval_rewards/chosen": -0.7818440198898315,
10
+ "eval_rewards/margins": 1.9296892881393433,
11
+ "eval_rewards/rejected": -2.711533308029175,
12
+ "eval_runtime": 224.8388,
13
  "eval_samples": 2000,
14
+ "eval_samples_per_second": 8.895,
15
+ "eval_steps_per_second": 0.28,
16
+ "train_loss": 0.31584552245614417,
17
+ "train_runtime": 49042.6248,
18
+ "train_samples": 61135,
19
+ "train_samples_per_second": 2.493,
20
+ "train_steps_per_second": 0.039
21
  }
eval_results.json CHANGED
@@ -1,16 +1,16 @@
1
  {
2
- "epoch": 1.0,
3
- "eval_logits/chosen": -1.475502371788025,
4
- "eval_logits/rejected": -0.962776243686676,
5
- "eval_logps/chosen": -397.00616455078125,
6
- "eval_logps/rejected": -456.8468017578125,
7
- "eval_loss": 0.5026616454124451,
8
- "eval_rewards/accuracies": 0.7549999952316284,
9
- "eval_rewards/chosen": -1.0713179111480713,
10
- "eval_rewards/margins": 0.815581202507019,
11
- "eval_rewards/rejected": -1.8868991136550903,
12
- "eval_runtime": 693.918,
13
  "eval_samples": 2000,
14
- "eval_samples_per_second": 2.882,
15
- "eval_steps_per_second": 0.144
16
  }
 
1
  {
2
+ "epoch": 2.0,
3
+ "eval_logits/chosen": -2.500744581222534,
4
+ "eval_logits/rejected": -2.4560909271240234,
5
+ "eval_logps/chosen": -289.780517578125,
6
+ "eval_logps/rejected": -287.32733154296875,
7
+ "eval_loss": 0.5590207576751709,
8
+ "eval_rewards/accuracies": 0.7857142686843872,
9
+ "eval_rewards/chosen": -0.7818440198898315,
10
+ "eval_rewards/margins": 1.9296892881393433,
11
+ "eval_rewards/rejected": -2.711533308029175,
12
+ "eval_runtime": 224.8388,
13
  "eval_samples": 2000,
14
+ "eval_samples_per_second": 8.895,
15
+ "eval_steps_per_second": 0.28
16
  }
generation_config.json CHANGED
@@ -2,5 +2,5 @@
2
  "_from_model_config": true,
3
  "bos_token_id": 1,
4
  "eos_token_id": 2,
5
- "transformers_version": "4.39.0.dev0"
6
  }
 
2
  "_from_model_config": true,
3
  "bos_token_id": 1,
4
  "eos_token_id": 2,
5
+ "transformers_version": "4.36.2"
6
  }
model-00001-of-00003.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:35f5ea80d91818a0be671826bc85d784733485872dd5276556683bac2831a20f
3
  size 4943162336
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d570449fc3ab875c842445dbe0419c9af52105a4493d8f3c10f459938fc90f64
3
  size 4943162336
model-00002-of-00003.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:603985a400b07943db0c3c4c47b8afb5314d9648c23a1083c7f064fd5205963d
3
  size 4999819336
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:26a01d8a995bf3ec2fb02e218913c533b2a43141aaa93a3f89a0474c0be16d81
3
  size 4999819336
model-00003-of-00003.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:a31b14bd81e7160ee0da251e70ebbc7c8cc6d82e5e461a6e3a41eb506b16d7ee
3
  size 4540516344
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b30ce3a2200f7e33b0218668aa8c2abf1f9930fa287643e501dc5b8e0ee0c5e5
3
  size 4540516344
runs/May02_12-22-32_temp1a22200000C/events.out.tfevents.1714652888.temp1a22200000C.2457569.0 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:2487aca8c5bbaf154e55e23b4d7fc480b7ff128fdba276423c2645b201d19063
3
- size 140157
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f6cb2efe603da8388b36e38085c2051ee1f453146abadd234c626384e81f7c4b
3
+ size 140511
runs/May02_12-22-32_temp1a22200000C/events.out.tfevents.1714702156.temp1a22200000C.2457569.1 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:faf1c169b32a331513106cbd059c42c296387209a9a873d82f492e7d23c42492
3
+ size 828
train_results.json CHANGED
@@ -1,8 +1,8 @@
1
  {
2
- "epoch": 1.0,
3
- "train_loss": 0.5410362510156881,
4
- "train_runtime": 45025.8645,
5
- "train_samples": 61134,
6
- "train_samples_per_second": 1.358,
7
- "train_steps_per_second": 0.008
8
  }
 
1
  {
2
+ "epoch": 2.0,
3
+ "train_loss": 0.31584552245614417,
4
+ "train_runtime": 49042.6248,
5
+ "train_samples": 61135,
6
+ "train_samples_per_second": 2.493,
7
+ "train_steps_per_second": 0.039
8
  }
trainer_state.json CHANGED
The diff for this file is too large to render. See raw diff