Hugging Face's logo

lole25
/
zephyr-7b-dpo-qlora

PEFT
TensorBoard
Safetensors
mistral
alignment-handbook
Generated from Trainer
trl
dpo
4-bit precision
bitsandbytes
Model card Files Metrics Community
lole25 commited on
Commit
f872924
•
1 Parent(s): 3ef9491

Model save

Browse files
Files changed (8) hide show
  1. README.md +77 -0
  2. adapter_model.safetensors +1 -1
  3. all_results.json +21 -0
  4. eval_results.json +16 -0
  5. runs/Feb26_23-36-47_gpu4-119-4/events.out.tfevents.1708951118.gpu4-119-4.3608392.0 +2 -2
  6. runs/Feb26_23-36-47_gpu4-119-4/events.out.tfevents.1708957045.gpu4-119-4.3608392.1 +3 -0
  7. train_results.json +8 -0
  8. trainer_state.json +624 -0
README.md ADDED
@@ -0,0 +1,77 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: peft
4
+ tags:
5
+ - trl
6
+ - dpo
7
+ - generated_from_trainer
8
+ base_model: mistralai/Mistral-7B-v0.1
9
+ model-index:
10
+ - name: zephyr-7b-dpo-qlora
11
+ results: []
12
+ ---
13
+
14
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
15
+ should probably proofread and complete it, then remove this comment. -->
16
+
17
+ # zephyr-7b-dpo-qlora
18
+
19
+ This model is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) on the None dataset.
20
+ It achieves the following results on the evaluation set:
21
+ - Loss: 0.6813
22
+ - Rewards/chosen: -0.0009
23
+ - Rewards/rejected: -0.0252
24
+ - Rewards/accuracies: 0.2920
25
+ - Rewards/margins: 0.0243
26
+ - Logps/rejected: -71.3009
27
+ - Logps/chosen: -65.4449
28
+ - Logits/rejected: -2.4428
29
+ - Logits/chosen: -2.4444
30
+
31
+ ## Model description
32
+
33
+ More information needed
34
+
35
+ ## Intended uses & limitations
36
+
37
+ More information needed
38
+
39
+ ## Training and evaluation data
40
+
41
+ More information needed
42
+
43
+ ## Training procedure
44
+
45
+ ### Training hyperparameters
46
+
47
+ The following hyperparameters were used during training:
48
+ - learning_rate: 5e-06
49
+ - train_batch_size: 2
50
+ - eval_batch_size: 2
51
+ - seed: 42
52
+ - distributed_type: multi-GPU
53
+ - num_devices: 4
54
+ - gradient_accumulation_steps: 4
55
+ - total_train_batch_size: 32
56
+ - total_eval_batch_size: 8
57
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
58
+ - lr_scheduler_type: cosine
59
+ - lr_scheduler_warmup_ratio: 0.1
60
+ - num_epochs: 1
61
+
62
+ ### Training results
63
+
64
+ | Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
65
+ |:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
66
+ | 0.69 | 0.26 | 100 | 0.6897 | 0.0232 | 0.0168 | 0.2680 | 0.0064 | -67.1001 | -63.0342 | -2.4904 | -2.4911 |
67
+ | 0.6869 | 0.52 | 200 | 0.6849 | 0.0066 | -0.0092 | 0.3060 | 0.0159 | -69.7060 | -64.6950 | -2.4556 | -2.4573 |
68
+ | 0.681 | 0.78 | 300 | 0.6815 | -0.0026 | -0.0264 | 0.2880 | 0.0238 | -71.4280 | -65.6224 | -2.4430 | -2.4446 |
69
+
70
+
71
+ ### Framework versions
72
+
73
+ - PEFT 0.7.1
74
+ - Transformers 4.36.2
75
+ - Pytorch 2.1.2+cu118
76
+ - Datasets 2.14.6
77
+ - Tokenizers 0.15.2
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:c25823732795917f51b090ccdd64723dc4816175f7e4d02778ed5d78c7c80f5c
3
  size 83946192
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c03cf130135167da55ddc0a06753fe3bc068d8b6139fb76450e8db473f37ebca
3
  size 83946192
all_results.json ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 1.0,
3
+ "eval_logits/chosen": -2.444389581680298,
4
+ "eval_logits/rejected": -2.4428083896636963,
5
+ "eval_logps/chosen": -65.44485473632812,
6
+ "eval_logps/rejected": -71.30088806152344,
7
+ "eval_loss": 0.68132084608078,
8
+ "eval_rewards/accuracies": 0.2919999957084656,
9
+ "eval_rewards/chosen": -0.0008568476187065244,
10
+ "eval_rewards/margins": 0.02430289424955845,
11
+ "eval_rewards/rejected": -0.025159740820527077,
12
+ "eval_runtime": 376.3354,
13
+ "eval_samples": 2000,
14
+ "eval_samples_per_second": 5.314,
15
+ "eval_steps_per_second": 0.664,
16
+ "train_loss": 0.6857879155593393,
17
+ "train_runtime": 5551.1855,
18
+ "train_samples": 12227,
19
+ "train_samples_per_second": 2.203,
20
+ "train_steps_per_second": 0.069
21
+ }
eval_results.json ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 1.0,
3
+ "eval_logits/chosen": -2.444389581680298,
4
+ "eval_logits/rejected": -2.4428083896636963,
5
+ "eval_logps/chosen": -65.44485473632812,
6
+ "eval_logps/rejected": -71.30088806152344,
7
+ "eval_loss": 0.68132084608078,
8
+ "eval_rewards/accuracies": 0.2919999957084656,
9
+ "eval_rewards/chosen": -0.0008568476187065244,
10
+ "eval_rewards/margins": 0.02430289424955845,
11
+ "eval_rewards/rejected": -0.025159740820527077,
12
+ "eval_runtime": 376.3354,
13
+ "eval_samples": 2000,
14
+ "eval_samples_per_second": 5.314,
15
+ "eval_steps_per_second": 0.664
16
+ }
runs/Feb26_23-36-47_gpu4-119-4/events.out.tfevents.1708951118.gpu4-119-4.3608392.0 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:43210e1b72c13ad9ba6693e9f95a7950a7f4df7c3bae6cf4926d4dd83fcaa5fb
3
- size 26647
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:55af144e1f7ab39ab708ab4150bade1bd02b199005ad45d2356ee016af89ca8e
3
+ size 32073
runs/Feb26_23-36-47_gpu4-119-4/events.out.tfevents.1708957045.gpu4-119-4.3608392.1 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d74112aa63f65ad97db55132aa88a32524c8d1235923db9c961725aa8dc21aa4
3
+ size 828
train_results.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 1.0,
3
+ "train_loss": 0.6857879155593393,
4
+ "train_runtime": 5551.1855,
5
+ "train_samples": 12227,
6
+ "train_samples_per_second": 2.203,
7
+ "train_steps_per_second": 0.069
8
+ }
trainer_state.json ADDED
@@ -0,0 +1,624 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": null,
3
+ "best_model_checkpoint": null,
4
+ "epoch": 0.999345977763244,
5
+ "eval_steps": 100,
6
+ "global_step": 382,
7
+ "is_hyper_param_search": false,
8
+ "is_local_process_zero": true,
9
+ "is_world_process_zero": true,
10
+ "log_history": [
11
+ {
12
+ "epoch": 0.0,
13
+ "learning_rate": 1.282051282051282e-07,
14
+ "logits/chosen": -2.3458919525146484,
15
+ "logits/rejected": -2.3298444747924805,
16
+ "logps/chosen": -24.293487548828125,
17
+ "logps/rejected": -56.14665603637695,
18
+ "loss": 0.6931,
19
+ "rewards/accuracies": 0.0,
20
+ "rewards/chosen": 0.0,
21
+ "rewards/margins": 0.0,
22
+ "rewards/rejected": 0.0,
23
+ "step": 1
24
+ },
25
+ {
26
+ "epoch": 0.03,
27
+ "learning_rate": 1.282051282051282e-06,
28
+ "logits/chosen": -2.5303263664245605,
29
+ "logits/rejected": -2.469114065170288,
30
+ "logps/chosen": -87.91757202148438,
31
+ "logps/rejected": -76.600830078125,
32
+ "loss": 0.6931,
33
+ "rewards/accuracies": 0.2361111044883728,
34
+ "rewards/chosen": 5.832286842633039e-05,
35
+ "rewards/margins": -0.00032225216273218393,
36
+ "rewards/rejected": 0.00038057498750276864,
37
+ "step": 10
38
+ },
39
+ {
40
+ "epoch": 0.05,
41
+ "learning_rate": 2.564102564102564e-06,
42
+ "logits/chosen": -2.51778507232666,
43
+ "logits/rejected": -2.4767203330993652,
44
+ "logps/chosen": -82.92424011230469,
45
+ "logps/rejected": -82.9404067993164,
46
+ "loss": 0.6929,
47
+ "rewards/accuracies": 0.4000000059604645,
48
+ "rewards/chosen": 0.001368564902804792,
49
+ "rewards/margins": 0.0007934757741168141,
50
+ "rewards/rejected": 0.0005750892451032996,
51
+ "step": 20
52
+ },
53
+ {
54
+ "epoch": 0.08,
55
+ "learning_rate": 3.846153846153847e-06,
56
+ "logits/chosen": -2.524854898452759,
57
+ "logits/rejected": -2.492816925048828,
58
+ "logps/chosen": -101.72441864013672,
59
+ "logps/rejected": -104.42093658447266,
60
+ "loss": 0.6928,
61
+ "rewards/accuracies": 0.3499999940395355,
62
+ "rewards/chosen": 0.0047247810289263725,
63
+ "rewards/margins": 0.0011784282978624105,
64
+ "rewards/rejected": 0.003546352731063962,
65
+ "step": 30
66
+ },
67
+ {
68
+ "epoch": 0.1,
69
+ "learning_rate": 4.999895137916554e-06,
70
+ "logits/chosen": -2.5693459510803223,
71
+ "logits/rejected": -2.5531678199768066,
72
+ "logps/chosen": -65.36104583740234,
73
+ "logps/rejected": -71.78102111816406,
74
+ "loss": 0.6926,
75
+ "rewards/accuracies": 0.32499998807907104,
76
+ "rewards/chosen": 0.006629495415836573,
77
+ "rewards/margins": 0.0005699460161849856,
78
+ "rewards/rejected": 0.006059548817574978,
79
+ "step": 40
80
+ },
81
+ {
82
+ "epoch": 0.13,
83
+ "learning_rate": 4.987322328603843e-06,
84
+ "logits/chosen": -2.5499138832092285,
85
+ "logits/rejected": -2.4812991619110107,
86
+ "logps/chosen": -85.08619689941406,
87
+ "logps/rejected": -91.72368621826172,
88
+ "loss": 0.6921,
89
+ "rewards/accuracies": 0.3375000059604645,
90
+ "rewards/chosen": 0.011005833745002747,
91
+ "rewards/margins": 0.00240146997384727,
92
+ "rewards/rejected": 0.008604364469647408,
93
+ "step": 50
94
+ },
95
+ {
96
+ "epoch": 0.16,
97
+ "learning_rate": 4.953897892477664e-06,
98
+ "logits/chosen": -2.3451428413391113,
99
+ "logits/rejected": -2.380490303039551,
100
+ "logps/chosen": -70.91415405273438,
101
+ "logps/rejected": -81.67916870117188,
102
+ "loss": 0.6917,
103
+ "rewards/accuracies": 0.38749998807907104,
104
+ "rewards/chosen": 0.015571129508316517,
105
+ "rewards/margins": 0.0033102519810199738,
106
+ "rewards/rejected": 0.012260876595973969,
107
+ "step": 60
108
+ },
109
+ {
110
+ "epoch": 0.18,
111
+ "learning_rate": 4.899902032011388e-06,
112
+ "logits/chosen": -2.5166893005371094,
113
+ "logits/rejected": -2.5204968452453613,
114
+ "logps/chosen": -70.66954040527344,
115
+ "logps/rejected": -66.42029571533203,
116
+ "loss": 0.6907,
117
+ "rewards/accuracies": 0.4124999940395355,
118
+ "rewards/chosen": 0.01822517439723015,
119
+ "rewards/margins": 0.006782916374504566,
120
+ "rewards/rejected": 0.011442260816693306,
121
+ "step": 70
122
+ },
123
+ {
124
+ "epoch": 0.21,
125
+ "learning_rate": 4.825787403267713e-06,
126
+ "logits/chosen": -2.2986605167388916,
127
+ "logits/rejected": -2.2916884422302246,
128
+ "logps/chosen": -55.5330924987793,
129
+ "logps/rejected": -64.45872497558594,
130
+ "loss": 0.6899,
131
+ "rewards/accuracies": 0.375,
132
+ "rewards/chosen": 0.020927606150507927,
133
+ "rewards/margins": 0.007273596711456776,
134
+ "rewards/rejected": 0.013654010370373726,
135
+ "step": 80
136
+ },
137
+ {
138
+ "epoch": 0.24,
139
+ "learning_rate": 4.732175321209399e-06,
140
+ "logits/chosen": -2.451927900314331,
141
+ "logits/rejected": -2.4550557136535645,
142
+ "logps/chosen": -83.72239685058594,
143
+ "logps/rejected": -77.17060852050781,
144
+ "loss": 0.6892,
145
+ "rewards/accuracies": 0.375,
146
+ "rewards/chosen": 0.028661560267210007,
147
+ "rewards/margins": 0.011586638167500496,
148
+ "rewards/rejected": 0.01707492023706436,
149
+ "step": 90
150
+ },
151
+ {
152
+ "epoch": 0.26,
153
+ "learning_rate": 4.619850551115105e-06,
154
+ "logits/chosen": -2.462538003921509,
155
+ "logits/rejected": -2.384904146194458,
156
+ "logps/chosen": -45.45023727416992,
157
+ "logps/rejected": -49.149192810058594,
158
+ "loss": 0.69,
159
+ "rewards/accuracies": 0.2750000059604645,
160
+ "rewards/chosen": 0.016560647636651993,
161
+ "rewards/margins": 0.0038987069856375456,
162
+ "rewards/rejected": 0.012661941349506378,
163
+ "step": 100
164
+ },
165
+ {
166
+ "epoch": 0.26,
167
+ "eval_logits/chosen": -2.4911162853240967,
168
+ "eval_logits/rejected": -2.4903504848480225,
169
+ "eval_logps/chosen": -63.03419876098633,
170
+ "eval_logps/rejected": -67.10006713867188,
171
+ "eval_loss": 0.6897407174110413,
172
+ "eval_rewards/accuracies": 0.2680000066757202,
173
+ "eval_rewards/chosen": 0.023249642923474312,
174
+ "eval_rewards/margins": 0.0064011020585894585,
175
+ "eval_rewards/rejected": 0.016848541796207428,
176
+ "eval_runtime": 376.1511,
177
+ "eval_samples_per_second": 5.317,
178
+ "eval_steps_per_second": 0.665,
179
+ "step": 100
180
+ },
181
+ {
182
+ "epoch": 0.29,
183
+ "learning_rate": 4.489754729764663e-06,
184
+ "logits/chosen": -2.5341131687164307,
185
+ "logits/rejected": -2.5058047771453857,
186
+ "logps/chosen": -82.01190185546875,
187
+ "logps/rejected": -82.14210510253906,
188
+ "loss": 0.6892,
189
+ "rewards/accuracies": 0.32499998807907104,
190
+ "rewards/chosen": 0.026387155055999756,
191
+ "rewards/margins": 0.006781895644962788,
192
+ "rewards/rejected": 0.019605260342359543,
193
+ "step": 110
194
+ },
195
+ {
196
+ "epoch": 0.31,
197
+ "learning_rate": 4.342978471545118e-06,
198
+ "logits/chosen": -2.406015157699585,
199
+ "logits/rejected": -2.4332752227783203,
200
+ "logps/chosen": -59.783790588378906,
201
+ "logps/rejected": -47.791358947753906,
202
+ "loss": 0.688,
203
+ "rewards/accuracies": 0.3375000059604645,
204
+ "rewards/chosen": 0.023528261110186577,
205
+ "rewards/margins": 0.01194488350301981,
206
+ "rewards/rejected": 0.011583376675844193,
207
+ "step": 120
208
+ },
209
+ {
210
+ "epoch": 0.34,
211
+ "learning_rate": 4.1807522256532925e-06,
212
+ "logits/chosen": -2.484022617340088,
213
+ "logits/rejected": -2.4554779529571533,
214
+ "logps/chosen": -71.26405334472656,
215
+ "logps/rejected": -76.42189025878906,
216
+ "loss": 0.6887,
217
+ "rewards/accuracies": 0.32499998807907104,
218
+ "rewards/chosen": 0.027204791083931923,
219
+ "rewards/margins": 0.01390073262155056,
220
+ "rewards/rejected": 0.013304059393703938,
221
+ "step": 130
222
+ },
223
+ {
224
+ "epoch": 0.37,
225
+ "learning_rate": 4.0044359610404505e-06,
226
+ "logits/chosen": -2.4449095726013184,
227
+ "logits/rejected": -2.513423442840576,
228
+ "logps/chosen": -73.46162414550781,
229
+ "logps/rejected": -68.35511016845703,
230
+ "loss": 0.6873,
231
+ "rewards/accuracies": 0.26249998807907104,
232
+ "rewards/chosen": 0.017034200951457024,
233
+ "rewards/margins": 0.007175432052463293,
234
+ "rewards/rejected": 0.009858767502009869,
235
+ "step": 140
236
+ },
237
+ {
238
+ "epoch": 0.39,
239
+ "learning_rate": 3.815507765571832e-06,
240
+ "logits/chosen": -2.3777384757995605,
241
+ "logits/rejected": -2.3317112922668457,
242
+ "logps/chosen": -48.66705322265625,
243
+ "logps/rejected": -60.531578063964844,
244
+ "loss": 0.6871,
245
+ "rewards/accuracies": 0.3125,
246
+ "rewards/chosen": 0.013008219189941883,
247
+ "rewards/margins": 0.010229717940092087,
248
+ "rewards/rejected": 0.00277850148268044,
249
+ "step": 150
250
+ },
251
+ {
252
+ "epoch": 0.42,
253
+ "learning_rate": 3.6155514549761644e-06,
254
+ "logits/chosen": -2.4121792316436768,
255
+ "logits/rejected": -2.4337995052337646,
256
+ "logps/chosen": -61.59517288208008,
257
+ "logps/rejected": -63.75123977661133,
258
+ "loss": 0.6884,
259
+ "rewards/accuracies": 0.3125,
260
+ "rewards/chosen": 0.021260840818285942,
261
+ "rewards/margins": 0.00982903502881527,
262
+ "rewards/rejected": 0.011431803926825523,
263
+ "step": 160
264
+ },
265
+ {
266
+ "epoch": 0.44,
267
+ "learning_rate": 3.4062432954613256e-06,
268
+ "logits/chosen": -2.4510605335235596,
269
+ "logits/rejected": -2.4552080631256104,
270
+ "logps/chosen": -78.11065673828125,
271
+ "logps/rejected": -71.26216888427734,
272
+ "loss": 0.6839,
273
+ "rewards/accuracies": 0.3375000059604645,
274
+ "rewards/chosen": 0.024866515770554543,
275
+ "rewards/margins": 0.017677070572972298,
276
+ "rewards/rejected": 0.007189445197582245,
277
+ "step": 170
278
+ },
279
+ {
280
+ "epoch": 0.47,
281
+ "learning_rate": 3.189337951302639e-06,
282
+ "logits/chosen": -2.557910203933716,
283
+ "logits/rejected": -2.5174174308776855,
284
+ "logps/chosen": -63.83809280395508,
285
+ "logps/rejected": -68.5108871459961,
286
+ "loss": 0.686,
287
+ "rewards/accuracies": 0.32499998807907104,
288
+ "rewards/chosen": 0.022966545075178146,
289
+ "rewards/margins": 0.017569351941347122,
290
+ "rewards/rejected": 0.005397191736847162,
291
+ "step": 180
292
+ },
293
+ {
294
+ "epoch": 0.5,
295
+ "learning_rate": 2.9666537752074443e-06,
296
+ "logits/chosen": -2.4893686771392822,
297
+ "logits/rejected": -2.4778811931610107,
298
+ "logps/chosen": -86.08170318603516,
299
+ "logps/rejected": -79.71577453613281,
300
+ "loss": 0.6841,
301
+ "rewards/accuracies": 0.3375000059604645,
302
+ "rewards/chosen": 0.007741170935332775,
303
+ "rewards/margins": 0.013728922232985497,
304
+ "rewards/rejected": -0.005987753160297871,
305
+ "step": 190
306
+ },
307
+ {
308
+ "epoch": 0.52,
309
+ "learning_rate": 2.7400575647692046e-06,
310
+ "logits/chosen": -2.307661294937134,
311
+ "logits/rejected": -2.314356565475464,
312
+ "logps/chosen": -34.22106170654297,
313
+ "logps/rejected": -51.29539108276367,
314
+ "loss": 0.6869,
315
+ "rewards/accuracies": 0.26249998807907104,
316
+ "rewards/chosen": 0.006807768251746893,
317
+ "rewards/margins": 0.01217351108789444,
318
+ "rewards/rejected": -0.005365744233131409,
319
+ "step": 200
320
+ },
321
+ {
322
+ "epoch": 0.52,
323
+ "eval_logits/chosen": -2.457301139831543,
324
+ "eval_logits/rejected": -2.4556241035461426,
325
+ "eval_logps/chosen": -64.6949691772461,
326
+ "eval_logps/rejected": -69.70600128173828,
327
+ "eval_loss": 0.6848968267440796,
328
+ "eval_rewards/accuracies": 0.3059999942779541,
329
+ "eval_rewards/chosen": 0.00664189737290144,
330
+ "eval_rewards/margins": 0.015852706506848335,
331
+ "eval_rewards/rejected": -0.009210809133946896,
332
+ "eval_runtime": 376.2332,
333
+ "eval_samples_per_second": 5.316,
334
+ "eval_steps_per_second": 0.664,
335
+ "step": 200
336
+ },
337
+ {
338
+ "epoch": 0.55,
339
+ "learning_rate": 2.5114489128002655e-06,
340
+ "logits/chosen": -2.2540907859802246,
341
+ "logits/rejected": -2.2756266593933105,
342
+ "logps/chosen": -68.72586822509766,
343
+ "logps/rejected": -80.93492126464844,
344
+ "loss": 0.683,
345
+ "rewards/accuracies": 0.2874999940395355,
346
+ "rewards/chosen": 0.014960775151848793,
347
+ "rewards/margins": 0.020435906946659088,
348
+ "rewards/rejected": -0.005475130397826433,
349
+ "step": 210
350
+ },
351
+ {
352
+ "epoch": 0.58,
353
+ "learning_rate": 2.2827442827369776e-06,
354
+ "logits/chosen": -2.36279296875,
355
+ "logits/rejected": -2.3906493186950684,
356
+ "logps/chosen": -66.4981689453125,
357
+ "logps/rejected": -72.55201721191406,
358
+ "loss": 0.6845,
359
+ "rewards/accuracies": 0.375,
360
+ "rewards/chosen": 0.009893990121781826,
361
+ "rewards/margins": 0.030444588512182236,
362
+ "rewards/rejected": -0.020550597459077835,
363
+ "step": 220
364
+ },
365
+ {
366
+ "epoch": 0.6,
367
+ "learning_rate": 2.0558609426156346e-06,
368
+ "logits/chosen": -2.5613975524902344,
369
+ "logits/rejected": -2.5647411346435547,
370
+ "logps/chosen": -91.46321868896484,
371
+ "logps/rejected": -82.19490051269531,
372
+ "loss": 0.6826,
373
+ "rewards/accuracies": 0.4124999940395355,
374
+ "rewards/chosen": 0.009320605546236038,
375
+ "rewards/margins": 0.036185480654239655,
376
+ "rewards/rejected": -0.026864871382713318,
377
+ "step": 230
378
+ },
379
+ {
380
+ "epoch": 0.63,
381
+ "learning_rate": 1.8327008923033224e-06,
382
+ "logits/chosen": -2.4174304008483887,
383
+ "logits/rejected": -2.395275115966797,
384
+ "logps/chosen": -75.15461730957031,
385
+ "logps/rejected": -72.01311492919922,
386
+ "loss": 0.6828,
387
+ "rewards/accuracies": 0.25,
388
+ "rewards/chosen": 0.005219348706305027,
389
+ "rewards/margins": 0.016013307496905327,
390
+ "rewards/rejected": -0.010793961584568024,
391
+ "step": 240
392
+ },
393
+ {
394
+ "epoch": 0.65,
395
+ "learning_rate": 1.6151349187243066e-06,
396
+ "logits/chosen": -2.4290623664855957,
397
+ "logits/rejected": -2.4516799449920654,
398
+ "logps/chosen": -92.50470733642578,
399
+ "logps/rejected": -91.04740905761719,
400
+ "loss": 0.6824,
401
+ "rewards/accuracies": 0.3499999940395355,
402
+ "rewards/chosen": -0.011667473241686821,
403
+ "rewards/margins": 0.016532858833670616,
404
+ "rewards/rejected": -0.028200332075357437,
405
+ "step": 250
406
+ },
407
+ {
408
+ "epoch": 0.68,
409
+ "learning_rate": 1.4049869127495886e-06,
410
+ "logits/chosen": -2.419191837310791,
411
+ "logits/rejected": -2.3789072036743164,
412
+ "logps/chosen": -55.09886932373047,
413
+ "logps/rejected": -63.9772834777832,
414
+ "loss": 0.6842,
415
+ "rewards/accuracies": 0.22499999403953552,
416
+ "rewards/chosen": -0.0026586714666336775,
417
+ "rewards/margins": 0.007516457699239254,
418
+ "rewards/rejected": -0.010175129398703575,
419
+ "step": 260
420
+ },
421
+ {
422
+ "epoch": 0.71,
423
+ "learning_rate": 1.2040185792236874e-06,
424
+ "logits/chosen": -2.366826295852661,
425
+ "logits/rejected": -2.387421131134033,
426
+ "logps/chosen": -78.35263061523438,
427
+ "logps/rejected": -85.33952331542969,
428
+ "loss": 0.6812,
429
+ "rewards/accuracies": 0.44999998807907104,
430
+ "rewards/chosen": 0.0012182873906567693,
431
+ "rewards/margins": 0.039223961532115936,
432
+ "rewards/rejected": -0.038005679845809937,
433
+ "step": 270
434
+ },
435
+ {
436
+ "epoch": 0.73,
437
+ "learning_rate": 1.0139146683069728e-06,
438
+ "logits/chosen": -2.507392168045044,
439
+ "logits/rejected": -2.514265775680542,
440
+ "logps/chosen": -80.45622253417969,
441
+ "logps/rejected": -72.48971557617188,
442
+ "loss": 0.6785,
443
+ "rewards/accuracies": 0.30000001192092896,
444
+ "rewards/chosen": 0.01629328541457653,
445
+ "rewards/margins": 0.02651815116405487,
446
+ "rewards/rejected": -0.01022486761212349,
447
+ "step": 280
448
+ },
449
+ {
450
+ "epoch": 0.76,
451
+ "learning_rate": 8.362688519416084e-07,
452
+ "logits/chosen": -2.448162078857422,
453
+ "logits/rejected": -2.432882070541382,
454
+ "logps/chosen": -69.45552062988281,
455
+ "logps/rejected": -75.71241760253906,
456
+ "loss": 0.6867,
457
+ "rewards/accuracies": 0.2750000059604645,
458
+ "rewards/chosen": 0.0061899758875370026,
459
+ "rewards/margins": 0.012647658586502075,
460
+ "rewards/rejected": -0.006457681301981211,
461
+ "step": 290
462
+ },
463
+ {
464
+ "epoch": 0.78,
465
+ "learning_rate": 6.725703638409861e-07,
466
+ "logits/chosen": -2.4520912170410156,
467
+ "logits/rejected": -2.4525935649871826,
468
+ "logps/chosen": -73.40272521972656,
469
+ "logps/rejected": -72.8621826171875,
470
+ "loss": 0.681,
471
+ "rewards/accuracies": 0.375,
472
+ "rewards/chosen": 0.0016829551896080375,
473
+ "rewards/margins": 0.024337466806173325,
474
+ "rewards/rejected": -0.022654514759778976,
475
+ "step": 300
476
+ },
477
+ {
478
+ "epoch": 0.78,
479
+ "eval_logits/chosen": -2.4445886611938477,
480
+ "eval_logits/rejected": -2.4429824352264404,
481
+ "eval_logps/chosen": -65.6224136352539,
482
+ "eval_logps/rejected": -71.427978515625,
483
+ "eval_loss": 0.6814776062965393,
484
+ "eval_rewards/accuracies": 0.2879999876022339,
485
+ "eval_rewards/chosen": -0.002632532035931945,
486
+ "eval_rewards/margins": 0.02379809506237507,
487
+ "eval_rewards/rejected": -0.026430627331137657,
488
+ "eval_runtime": 376.1246,
489
+ "eval_samples_per_second": 5.317,
490
+ "eval_steps_per_second": 0.665,
491
+ "step": 300
492
+ },
493
+ {
494
+ "epoch": 0.81,
495
+ "learning_rate": 5.24191515001774e-07,
496
+ "logits/chosen": -2.3931050300598145,
497
+ "logits/rejected": -2.394754409790039,
498
+ "logps/chosen": -76.21150970458984,
499
+ "logps/rejected": -73.65968322753906,
500
+ "loss": 0.6818,
501
+ "rewards/accuracies": 0.2874999940395355,
502
+ "rewards/chosen": 0.005356580018997192,
503
+ "rewards/margins": 0.02418547496199608,
504
+ "rewards/rejected": -0.018828898668289185,
505
+ "step": 310
506
+ },
507
+ {
508
+ "epoch": 0.84,
509
+ "learning_rate": 3.9237618939808655e-07,
510
+ "logits/chosen": -2.3786847591400146,
511
+ "logits/rejected": -2.3697614669799805,
512
+ "logps/chosen": -68.56962585449219,
513
+ "logps/rejected": -69.29893493652344,
514
+ "loss": 0.6827,
515
+ "rewards/accuracies": 0.2874999940395355,
516
+ "rewards/chosen": -0.002341556828469038,
517
+ "rewards/margins": 0.014613455161452293,
518
+ "rewards/rejected": -0.01695501245558262,
519
+ "step": 320
520
+ },
521
+ {
522
+ "epoch": 0.86,
523
+ "learning_rate": 2.782294163002261e-07,
524
+ "logits/chosen": -2.3469510078430176,
525
+ "logits/rejected": -2.324633836746216,
526
+ "logps/chosen": -69.15370178222656,
527
+ "logps/rejected": -79.9259033203125,
528
+ "loss": 0.679,
529
+ "rewards/accuracies": 0.30000001192092896,
530
+ "rewards/chosen": 0.008440351113677025,
531
+ "rewards/margins": 0.024770593270659447,
532
+ "rewards/rejected": -0.01633024401962757,
533
+ "step": 330
534
+ },
535
+ {
536
+ "epoch": 0.89,
537
+ "learning_rate": 1.8270810663494591e-07,
538
+ "logits/chosen": -2.4450182914733887,
539
+ "logits/rejected": -2.427203416824341,
540
+ "logps/chosen": -67.0771484375,
541
+ "logps/rejected": -68.71158599853516,
542
+ "loss": 0.6797,
543
+ "rewards/accuracies": 0.2750000059604645,
544
+ "rewards/chosen": 0.0001767903595464304,
545
+ "rewards/margins": 0.018586840480566025,
546
+ "rewards/rejected": -0.018410051241517067,
547
+ "step": 340
548
+ },
549
+ {
550
+ "epoch": 0.92,
551
+ "learning_rate": 1.0661303104582882e-07,
552
+ "logits/chosen": -2.5075066089630127,
553
+ "logits/rejected": -2.515806198120117,
554
+ "logps/chosen": -100.72679138183594,
555
+ "logps/rejected": -111.1237564086914,
556
+ "loss": 0.683,
557
+ "rewards/accuracies": 0.4000000059604645,
558
+ "rewards/chosen": 0.004834707826375961,
559
+ "rewards/margins": 0.03868640214204788,
560
+ "rewards/rejected": -0.03385169431567192,
561
+ "step": 350
562
+ },
563
+ {
564
+ "epoch": 0.94,
565
+ "learning_rate": 5.0582106903004636e-08,
566
+ "logits/chosen": -2.3713040351867676,
567
+ "logits/rejected": -2.476989507675171,
568
+ "logps/chosen": -80.11368560791016,
569
+ "logps/rejected": -83.44255065917969,
570
+ "loss": 0.6804,
571
+ "rewards/accuracies": 0.4000000059604645,
572
+ "rewards/chosen": 0.0026325061917304993,
573
+ "rewards/margins": 0.0385906919836998,
574
+ "rewards/rejected": -0.0359581895172596,
575
+ "step": 360
576
+ },
577
+ {
578
+ "epoch": 0.97,
579
+ "learning_rate": 1.508505053829046e-08,
580
+ "logits/chosen": -2.4487078189849854,
581
+ "logits/rejected": -2.4094619750976562,
582
+ "logps/chosen": -64.8487548828125,
583
+ "logps/rejected": -94.2999496459961,
584
+ "loss": 0.6805,
585
+ "rewards/accuracies": 0.32499998807907104,
586
+ "rewards/chosen": 0.00822712853550911,
587
+ "rewards/margins": 0.03635001927614212,
588
+ "rewards/rejected": -0.02812289260327816,
589
+ "step": 370
590
+ },
591
+ {
592
+ "epoch": 0.99,
593
+ "learning_rate": 4.19439536940569e-10,
594
+ "logits/chosen": -2.4872801303863525,
595
+ "logits/rejected": -2.508479595184326,
596
+ "logps/chosen": -71.79349517822266,
597
+ "logps/rejected": -80.17054748535156,
598
+ "loss": 0.6814,
599
+ "rewards/accuracies": 0.3375000059604645,
600
+ "rewards/chosen": -0.00255067041143775,
601
+ "rewards/margins": 0.025686126202344894,
602
+ "rewards/rejected": -0.02823679707944393,
603
+ "step": 380
604
+ },
605
+ {
606
+ "epoch": 1.0,
607
+ "step": 382,
608
+ "total_flos": 0.0,
609
+ "train_loss": 0.6857879155593393,
610
+ "train_runtime": 5551.1855,
611
+ "train_samples_per_second": 2.203,
612
+ "train_steps_per_second": 0.069
613
+ }
614
+ ],
615
+ "logging_steps": 10,
616
+ "max_steps": 382,
617
+ "num_input_tokens_seen": 0,
618
+ "num_train_epochs": 1,
619
+ "save_steps": 100,
620
+ "total_flos": 0.0,
621
+ "train_batch_size": 2,
622
+ "trial_name": null,
623
+ "trial_params": null
624
+ }