Hugging Face's logo

statking
/
zephyr-7b-sft-full-orpo

Text Generation
Transformers
Safetensors
mistral
alignment-handbook
trl
orpo
Generated from Trainer
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
statking commited on
Commit
5dc2393
•
1 Parent(s): 590cc04

Model save

Browse files
Files changed (9) hide show
  1. README.md +24 -49
  2. all_results.json +4 -4
  3. config.json +1 -1
  4. model-00001-of-00003.safetensors +1 -1
  5. model-00002-of-00003.safetensors +1 -1
  6. model-00003-of-00003.safetensors +1 -1
  7. train_results.json +4 -4
  8. trainer_state.json +0 -0
  9. training_args.bin +1 -1
README.md CHANGED
@@ -2,16 +2,10 @@
2
  license: apache-2.0
3
  base_model: mistralai/Mistral-7B-v0.1
4
  tags:
5
- - alignment-handbook
6
- - trl
7
- - orpo
8
- - generated_from_trainer
9
  - trl
10
  - orpo
11
  - alignment-handbook
12
  - generated_from_trainer
13
- datasets:
14
- - HuggingFaceH4/ultrafeedback_binarized
15
  model-index:
16
  - name: zephyr-7b-sft-full-orpo
17
  results: []
@@ -20,23 +14,23 @@ model-index:
20
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
21
  should probably proofread and complete it, then remove this comment. -->
22
 
23
- [<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/statking/huggingface/runs/b45ab3qe)
24
  # zephyr-7b-sft-full-orpo
25
 
26
- This model is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) on the HuggingFaceH4/ultrafeedback_binarized dataset.
27
  It achieves the following results on the evaluation set:
28
- - Loss: 1.3771
29
- - Rewards/chosen: -0.1391
30
- - Rewards/rejected: -0.1930
31
  - Rewards/accuracies: 0.6528
32
- - Rewards/margins: 0.0539
33
- - Logps/rejected: -3.8602
34
- - Logps/chosen: -2.7813
35
- - Logits/rejected: -2.8670
36
- - Logits/chosen: -2.8498
37
- - Nll Loss: 1.3532
38
- - Log Odds Ratio: -1.0480
39
- - Log Odds Chosen: 1.2201
40
 
41
  ## Model description
42
 
@@ -55,7 +49,7 @@ More information needed
55
  ### Training hyperparameters
56
 
57
  The following hyperparameters were used during training:
58
- - learning_rate: 2e-05
59
  - train_batch_size: 8
60
  - eval_batch_size: 8
61
  - seed: 42
@@ -67,40 +61,21 @@ The following hyperparameters were used during training:
67
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
68
  - lr_scheduler_type: inverse_sqrt
69
  - lr_scheduler_warmup_steps: 100
70
- - num_epochs: 3
71
 
72
  ### Training results
73
 
74
  | Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen | Nll Loss | Log Odds Ratio | Log Odds Chosen |
75
  |:-------------:|:------:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|:--------:|:--------------:|:---------------:|
76
- | 0.5668 | 0.1049 | 100 | 0.5843 | -0.0456 | -0.0529 | 0.6151 | 0.0073 | -1.0580 | -0.9113 | -3.3148 | -3.3082 | 0.5516 | -0.6530 | 0.2184 |
77
- | 0.5676 | 0.2098 | 200 | 0.5726 | -0.0441 | -0.0532 | 0.625 | 0.0092 | -1.0644 | -0.8811 | -3.0026 | -2.9992 | 0.5359 | -0.6474 | 0.2850 |
78
- | 0.5819 | 0.3146 | 300 | 0.5552 | -0.0439 | -0.0531 | 0.6290 | 0.0092 | -1.0620 | -0.8770 | -3.1424 | -3.1391 | 0.5202 | -0.6464 | 0.2830 |
79
- | 0.5738 | 0.4195 | 400 | 0.5411 | -0.0422 | -0.0517 | 0.6290 | 0.0096 | -1.0346 | -0.8434 | -3.1026 | -3.1020 | 0.5047 | -0.6522 | 0.2961 |
80
- | 0.5478 | 0.5244 | 500 | 0.5319 | -0.0421 | -0.0525 | 0.6290 | 0.0105 | -1.0509 | -0.8415 | -3.0260 | -3.0286 | 0.4970 | -0.6382 | 0.3327 |
81
- | 0.5146 | 0.6293 | 600 | 0.5240 | -0.0408 | -0.0508 | 0.6230 | 0.0100 | -1.0165 | -0.8165 | -3.1325 | -3.1275 | 0.4883 | -0.6418 | 0.3121 |
82
- | 0.5298 | 0.7341 | 700 | 0.5188 | -0.0413 | -0.0541 | 0.6429 | 0.0128 | -1.0827 | -0.8267 | -3.0761 | -3.0755 | 0.4842 | -0.6219 | 0.3869 |
83
- | 0.5181 | 0.8390 | 800 | 0.5141 | -0.0410 | -0.0524 | 0.6329 | 0.0114 | -1.0475 | -0.8198 | -3.1382 | -3.1394 | 0.4803 | -0.6322 | 0.3506 |
84
- | 0.5239 | 0.9439 | 900 | 0.5086 | -0.0402 | -0.0506 | 0.6310 | 0.0104 | -1.0129 | -0.8045 | -3.1191 | -3.1171 | 0.4748 | -0.6328 | 0.3268 |
85
- | 0.2888 | 1.0488 | 1000 | 0.5400 | -0.0436 | -0.0556 | 0.6429 | 0.0120 | -1.1128 | -0.8724 | -3.0171 | -3.0190 | 0.5058 | -0.6318 | 0.3794 |
86
- | 0.29 | 1.1536 | 1100 | 0.5385 | -0.0437 | -0.0574 | 0.6468 | 0.0138 | -1.1487 | -0.8736 | -3.0027 | -3.0029 | 0.5042 | -0.6256 | 0.4247 |
87
- | 0.2826 | 1.2585 | 1200 | 0.5428 | -0.0443 | -0.0581 | 0.6429 | 0.0139 | -1.1626 | -0.8854 | -2.9620 | -2.9583 | 0.5084 | -0.6254 | 0.4215 |
88
- | 0.2796 | 1.3634 | 1300 | 0.5393 | -0.0441 | -0.0589 | 0.6468 | 0.0147 | -1.1771 | -0.8825 | -2.9256 | -2.9285 | 0.5060 | -0.6208 | 0.4508 |
89
- | 0.2784 | 1.4683 | 1400 | 0.5365 | -0.0444 | -0.0589 | 0.6528 | 0.0145 | -1.1784 | -0.8885 | -2.9583 | -2.9594 | 0.5037 | -0.6236 | 0.4410 |
90
- | 0.2873 | 1.5732 | 1500 | 0.5330 | -0.0436 | -0.0579 | 0.6448 | 0.0143 | -1.1584 | -0.8718 | -2.9664 | -2.9657 | 0.5004 | -0.6226 | 0.4364 |
91
- | 0.276 | 1.6780 | 1600 | 0.5367 | -0.0442 | -0.0594 | 0.6409 | 0.0152 | -1.1879 | -0.8833 | -2.9358 | -2.9324 | 0.5041 | -0.6160 | 0.4570 |
92
- | 0.2715 | 1.7829 | 1700 | 0.5349 | -0.0436 | -0.0580 | 0.6448 | 0.0145 | -1.1603 | -0.8710 | -3.0209 | -3.0194 | 0.5024 | -0.6272 | 0.4425 |
93
- | 0.2717 | 1.8878 | 1800 | 0.5341 | -0.0450 | -0.0616 | 0.6548 | 0.0166 | -1.2325 | -0.8997 | -2.9579 | -2.9563 | 0.5023 | -0.6184 | 0.4824 |
94
- | 0.2857 | 1.9927 | 1900 | 0.5408 | -0.0454 | -0.0620 | 0.6548 | 0.0166 | -1.2409 | -0.9088 | -3.0279 | -3.0350 | 0.5091 | -0.6193 | 0.4892 |
95
- | 0.1137 | 2.0975 | 2000 | 0.6877 | -0.0620 | -0.0838 | 0.6706 | 0.0218 | -1.6761 | -1.2408 | -2.8815 | -2.8704 | 0.6539 | -0.6273 | 0.5767 |
96
- | 0.1192 | 2.2024 | 2100 | 0.7577 | -0.0706 | -0.0981 | 0.6726 | 0.0275 | -1.9620 | -1.4122 | -2.8433 | -2.8372 | 0.7199 | -0.6210 | 0.6958 |
97
- | 0.1178 | 2.3073 | 2200 | 1.1762 | -0.1205 | -0.1717 | 0.6528 | 0.0512 | -3.4342 | -2.4108 | -2.9107 | -2.8878 | 1.1197 | -0.7778 | 1.1628 |
98
- | 0.1184 | 2.4122 | 2300 | 1.8520 | -0.1935 | -0.2541 | 0.6369 | 0.0606 | -5.0812 | -3.8696 | -2.9226 | -2.9102 | 1.7542 | -1.0562 | 1.3233 |
99
- | 0.1172 | 2.5170 | 2400 | 1.0193 | -0.1001 | -0.1434 | 0.6409 | 0.0432 | -2.8671 | -2.0024 | -2.8710 | -2.8561 | 0.9736 | -0.8145 | 1.0075 |
100
- | 0.1109 | 2.6219 | 2500 | 1.2050 | -0.1209 | -0.1677 | 0.6329 | 0.0468 | -3.3547 | -2.4183 | -2.8571 | -2.8457 | 1.1724 | -0.9768 | 1.0766 |
101
- | 0.1238 | 2.7268 | 2600 | 2.6922 | -0.3036 | -0.3822 | 0.5873 | 0.0786 | -7.6444 | -6.0725 | -2.9967 | -2.9805 | 2.6498 | -1.6934 | 1.6674 |
102
- | 0.1192 | 2.8317 | 2700 | 1.2391 | -0.1189 | -0.1634 | 0.625 | 0.0445 | -3.2671 | -2.3779 | -2.8836 | -2.8662 | 1.1910 | -0.9507 | 1.0201 |
103
- | 0.1191 | 2.9365 | 2800 | 1.0214 | -0.0976 | -0.1394 | 0.6270 | 0.0418 | -2.7882 | -1.9523 | -2.8221 | -2.8059 | 0.9673 | -0.8558 | 0.9869 |
104
 
105
 
106
  ### Framework versions
 
2
  license: apache-2.0
3
  base_model: mistralai/Mistral-7B-v0.1
4
  tags:
 
 
 
 
5
  - trl
6
  - orpo
7
  - alignment-handbook
8
  - generated_from_trainer
 
 
9
  model-index:
10
  - name: zephyr-7b-sft-full-orpo
11
  results: []
 
14
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
15
  should probably proofread and complete it, then remove this comment. -->
16
 
17
+ [<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/statking/huggingface/runs/90a8kp39)
18
  # zephyr-7b-sft-full-orpo
19
 
20
+ This model is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) on an unknown dataset.
21
  It achieves the following results on the evaluation set:
22
+ - Loss: 0.4714
23
+ - Rewards/chosen: -0.0357
24
+ - Rewards/rejected: -0.0466
25
  - Rewards/accuracies: 0.6528
26
+ - Rewards/margins: 0.0109
27
+ - Logps/rejected: -0.9324
28
+ - Logps/chosen: -0.7143
29
+ - Logits/rejected: -2.9543
30
+ - Logits/chosen: -2.9692
31
+ - Nll Loss: 0.4361
32
+ - Log Odds Ratio: -0.6245
33
+ - Log Odds Chosen: 0.3669
34
 
35
  ## Model description
36
 
 
49
  ### Training hyperparameters
50
 
51
  The following hyperparameters were used during training:
52
+ - learning_rate: 7e-06
53
  - train_batch_size: 8
54
  - eval_batch_size: 8
55
  - seed: 42
 
61
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
62
  - lr_scheduler_type: inverse_sqrt
63
  - lr_scheduler_warmup_steps: 100
64
+ - num_epochs: 1
65
 
66
  ### Training results
67
 
68
  | Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen | Nll Loss | Log Odds Ratio | Log Odds Chosen |
69
  |:-------------:|:------:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|:--------:|:--------------:|:---------------:|
70
+ | 0.5226 | 0.1049 | 100 | 0.5280 | -0.0386 | -0.0472 | 0.6329 | 0.0086 | -0.9448 | -0.7728 | -2.7583 | -2.7860 | 0.4953 | -0.6326 | 0.2873 |
71
+ | 0.5074 | 0.2098 | 200 | 0.5134 | -0.0381 | -0.0478 | 0.6409 | 0.0098 | -0.9566 | -0.7612 | -2.6736 | -2.7002 | 0.4774 | -0.6357 | 0.3190 |
72
+ | 0.5265 | 0.3146 | 300 | 0.5012 | -0.0379 | -0.0479 | 0.6329 | 0.0099 | -0.9572 | -0.7588 | -2.7317 | -2.7594 | 0.4653 | -0.6374 | 0.3278 |
73
+ | 0.5194 | 0.4195 | 400 | 0.4912 | -0.0371 | -0.0478 | 0.6429 | 0.0107 | -0.9559 | -0.7417 | -2.6640 | -2.6974 | 0.4560 | -0.6284 | 0.3607 |
74
+ | 0.5008 | 0.5244 | 500 | 0.4847 | -0.0373 | -0.0489 | 0.6508 | 0.0117 | -0.9786 | -0.7455 | -2.5957 | -2.6294 | 0.4499 | -0.6209 | 0.3873 |
75
+ | 0.4725 | 0.6293 | 600 | 0.4794 | -0.0362 | -0.0470 | 0.6349 | 0.0107 | -0.9394 | -0.7248 | -2.6147 | -2.6477 | 0.4435 | -0.6320 | 0.3567 |
76
+ | 0.4875 | 0.7341 | 700 | 0.4767 | -0.0368 | -0.0498 | 0.6409 | 0.0129 | -0.9955 | -0.7365 | -2.6910 | -2.7213 | 0.4416 | -0.6158 | 0.4180 |
77
+ | 0.4796 | 0.8390 | 800 | 0.4740 | -0.0371 | -0.0508 | 0.6508 | 0.0137 | -1.0162 | -0.7416 | -2.7913 | -2.8114 | 0.4396 | -0.6169 | 0.4363 |
78
+ | 0.4851 | 0.9439 | 900 | 0.4714 | -0.0357 | -0.0466 | 0.6528 | 0.0109 | -0.9324 | -0.7143 | -2.9543 | -2.9692 | 0.4361 | -0.6245 | 0.3669 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
79
 
80
 
81
  ### Framework versions
all_results.json CHANGED
@@ -1,5 +1,5 @@
1
  {
2
- "epoch": 2.9984268484530676,
3
  "eval_log_odds_chosen": 1.220078706741333,
4
  "eval_log_odds_ratio": -1.047989010810852,
5
  "eval_logits/chosen": -2.849764585494995,
@@ -17,9 +17,9 @@
17
  "eval_samples_per_second": 14.663,
18
  "eval_steps_per_second": 0.463,
19
  "total_flos": 0.0,
20
- "train_loss": 0.32389816019492534,
21
- "train_runtime": 62235.4926,
22
  "train_samples": 61005,
23
- "train_samples_per_second": 2.941,
24
  "train_steps_per_second": 0.046
25
  }
 
1
  {
2
+ "epoch": 0.9994756161510225,
3
  "eval_log_odds_chosen": 1.220078706741333,
4
  "eval_log_odds_ratio": -1.047989010810852,
5
  "eval_logits/chosen": -2.849764585494995,
 
17
  "eval_samples_per_second": 14.663,
18
  "eval_steps_per_second": 0.463,
19
  "total_flos": 0.0,
20
+ "train_loss": 0.5301580581685054,
21
+ "train_runtime": 20737.8205,
22
  "train_samples": 61005,
23
+ "train_samples_per_second": 2.942,
24
  "train_steps_per_second": 0.046
25
  }
config.json CHANGED
@@ -21,6 +21,6 @@
21
  "tie_word_embeddings": false,
22
  "torch_dtype": "bfloat16",
23
  "transformers_version": "4.41.0.dev0",
24
- "use_cache": true,
25
  "vocab_size": 32000
26
  }
 
21
  "tie_word_embeddings": false,
22
  "torch_dtype": "bfloat16",
23
  "transformers_version": "4.41.0.dev0",
24
+ "use_cache": false,
25
  "vocab_size": 32000
26
  }
model-00001-of-00003.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:ee10f5eceafcd14eb38770919bb08de1ed7713961ad136a927e2ab2dc2e5054d
3
  size 4943162336
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b313dc48ecc95a24426280ae5e3e66f841af88ea853a46696d2faaae1f2f129e
3
  size 4943162336
model-00002-of-00003.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:89834df397be326e4f7d0760093ebf7a8800ec22ebd09ad4276d44d01b6d2eb5
3
  size 4999819336
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cfee51622377d86839cc88b039e910fc8cb1731ae63fd5fb76be8ccc53beca43
3
  size 4999819336
model-00003-of-00003.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:2ea0193cb9fe9772cff4067a7665f851a73e9595d6b4fdbfc36a57105015522b
3
  size 4540516344
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1d332c123718dc9f9ea7414c9e2362576015310d8b853af1269dffded4a17cec
3
  size 4540516344
train_results.json CHANGED
@@ -1,9 +1,9 @@
1
  {
2
- "epoch": 2.9984268484530676,
3
  "total_flos": 0.0,
4
- "train_loss": 0.32389816019492534,
5
- "train_runtime": 62235.4926,
6
  "train_samples": 61005,
7
- "train_samples_per_second": 2.941,
8
  "train_steps_per_second": 0.046
9
  }
 
1
  {
2
+ "epoch": 0.9994756161510225,
3
  "total_flos": 0.0,
4
+ "train_loss": 0.5301580581685054,
5
+ "train_runtime": 20737.8205,
6
  "train_samples": 61005,
7
+ "train_samples_per_second": 2.942,
8
  "train_steps_per_second": 0.046
9
  }
trainer_state.json CHANGED
The diff for this file is too large to render. See raw diff
 
training_args.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:3e94fd3a4a9d2e3763c25d9900c434cf062f5e77c71cb792f16d64f681e35200
3
  size 6648
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:427c0935c426bbb8857ae50317955d9dcaa991ddbe5ea21647c088c2a9b1ccc7
3
  size 6648