Model save

Browse files

Files changed (9) hide show

README.md +3 -24
all_results.json +5 -5
model-00001-of-00004.safetensors +1 -1
model-00002-of-00004.safetensors +1 -1
model-00003-of-00004.safetensors +1 -1
model-00004-of-00004.safetensors +1 -1
train_results.json +5 -5
trainer_state.json +789 -1269
training_args.bin +1 -1

README.md CHANGED Viewed

@@ -14,16 +14,6 @@ should probably proofread and complete it, then remove this comment. -->
 # zephyr-7b-dpo-full
 This model was trained from scratch on the None dataset.
-It achieves the following results on the evaluation set:
-- Loss: 0.5261
-- Rewards/chosen: -2.4591
-- Rewards/rejected: -3.9221
-- Rewards/accuracies: 0.7773
-- Rewards/margins: 1.4631
-- Logps/rejected: -703.8400
-- Logps/chosen: -549.4910
-- Logits/rejected: 0.0289
-- Logits/chosen: 0.0663
 ## Model description
@@ -43,12 +33,12 @@ More information needed
 The following hyperparameters were used during training:
 - learning_rate: 1e-06
-- train_batch_size: 4
 - eval_batch_size: 8
-- seed: 2
 - distributed_type: multi-GPU
 - num_devices: 8
-- gradient_accumulation_steps: 4
 - total_train_batch_size: 128
 - total_eval_batch_size: 64
 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
@@ -58,17 +48,6 @@ The following hyperparameters were used during training:
 ### Training results
-| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
-|:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
-| 0.6201        | 0.21  | 100  | 0.6253          | -0.2753        | -0.6662          | 0.7031             | 0.3909          | -378.2405      | -331.1124    | 0.4172          | 0.3706        |
-| 0.5547        | 0.42  | 200  | 0.5549          | -0.6988        | -1.4726          | 0.7656             | 0.7738          | -458.8863      | -373.4661    | 0.4261          | 0.3909        |
-| 0.5343        | 0.63  | 300  | 0.5316          | -0.8044        | -1.6474          | 0.7656             | 0.8430          | -476.3628      | -384.0199    | 0.2851          | 0.2449        |
-| 0.5323        | 0.84  | 400  | 0.5211          | -0.9068        | -1.8283          | 0.7812             | 0.9216          | -494.4600      | -394.2621    | 0.2834          | 0.2514        |
-| 0.352         | 1.05  | 500  | 0.5258          | -1.9533        | -3.4166          | 0.7969             | 1.4634          | -653.2899      | -498.9117    | -0.0846         | -0.0654       |
-| 0.3342        | 1.26  | 600  | 0.5268          | -2.3123        | -3.7246          | 0.7930             | 1.4124          | -684.0857      | -534.8101    | 0.1128          | 0.1344        |
-| 0.337         | 1.47  | 700  | 0.5290          | -2.3753        | -3.8837          | 0.7773             | 1.5084          | -699.9910      | -541.1116    | 0.0099          | 0.0414        |
-| 0.3398        | 1.67  | 800  | 0.5297          | -2.5097        | -4.0133          | 0.7734             | 1.5036          | -712.9506      | -554.5546    | 0.0381          | 0.0750        |
-| 0.307         | 1.88  | 900  | 0.5261          | -2.4591        | -3.9221          | 0.7773             | 1.4631          | -703.8400      | -549.4910    | 0.0289          | 0.0663        |
 ### Framework versions

 # zephyr-7b-dpo-full
 This model was trained from scratch on the None dataset.
 ## Model description
 The following hyperparameters were used during training:
 - learning_rate: 1e-06
+- train_batch_size: 2
 - eval_batch_size: 8
+- seed: 4
 - distributed_type: multi-GPU
 - num_devices: 8
+- gradient_accumulation_steps: 8
 - total_train_batch_size: 128
 - total_eval_batch_size: 64
 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
 ### Training results
 ### Framework versions

all_results.json CHANGED Viewed

@@ -1,8 +1,8 @@
 {
     "epoch": 2.0,
-    "train_loss": 0.44779854000739333,
-    "train_runtime": 8782.9823,
-    "train_samples": 61134,
-    "train_samples_per_second": 13.921,
-    "train_steps_per_second": 0.109
 }

 {
     "epoch": 2.0,
+    "train_loss": 0.5396277803770253,
+    "train_runtime": 10358.1577,
+    "train_samples": 45548,
+    "train_samples_per_second": 8.795,
+    "train_steps_per_second": 0.069
 }

model-00001-of-00004.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ac6928598b87aa9e7ace3c1f94d7df4eaa7164f92df0fda9603ca217542e9949
 size 4976698672

 version https://git-lfs.github.com/spec/v1
+oid sha256:20c73d37fa1abc7f882494b68d610836beb09ee5e90cce2d4b4576eb264bd637
 size 4976698672

model-00002-of-00004.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1832fd3b8ecfcb01bbcfbb4425fba1dbd3c57767b6fe8fe806acc6e01654a97b
 size 4999802720

 version https://git-lfs.github.com/spec/v1
+oid sha256:afe87beb2104fe94743a7aad2eaaf684b7d6d4a1bc13e5a8e968ee40e87d1576
 size 4999802720

model-00003-of-00004.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a888c6fcef84f2cb517c57068441aa0a6ba00c8bfc4bd4c7155e79556f8133b9
 size 4915916176

 version https://git-lfs.github.com/spec/v1
+oid sha256:6f5ac4165fed09e88facc67c437f2ed004803890e8af5b9d37e433d9f3ae3649
 size 4915916176

model-00004-of-00004.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c2d194b9549d875e2494134696a27103429d2e21501246a202a7415a40b5de72
 size 1168138808

 version https://git-lfs.github.com/spec/v1
+oid sha256:76643d477c833c3cd28dbb514282eada09b9314c8d0ae879cfa39e575f941a0c
 size 1168138808

train_results.json CHANGED Viewed

@@ -1,8 +1,8 @@
 {
     "epoch": 2.0,
-    "train_loss": 0.44779854000739333,
-    "train_runtime": 8782.9823,
-    "train_samples": 61134,
-    "train_samples_per_second": 13.921,
-    "train_steps_per_second": 0.109
 }

 {
     "epoch": 2.0,
+    "train_loss": 0.5396277803770253,
+    "train_runtime": 10358.1577,
+    "train_samples": 45548,
+    "train_samples_per_second": 8.795,
+    "train_steps_per_second": 0.069
 }

trainer_state.json CHANGED Viewed

@@ -1,1501 +1,1021 @@
 {
   "best_metric": null,
   "best_model_checkpoint": null,
-  "epoch": 1.9968602825745683,
-  "eval_steps": 100,
-  "global_step": 954,
   "is_hyper_param_search": false,
   "is_local_process_zero": true,
   "is_world_process_zero": true,
   "log_history": [
     {
-      "epoch": 0.02,
-      "learning_rate": 1.0416666666666667e-07,
-      "logits/chosen": 0.29368966817855835,
-      "logits/rejected": 0.3178113102912903,
-      "logps/chosen": -295.21783447265625,
-      "logps/rejected": -290.84619140625,
-      "loss": 0.6933,
-      "rewards/accuracies": 0.4000000059604645,
-      "rewards/chosen": 0.00023447822604794055,
-      "rewards/margins": 0.00020264319027774036,
-      "rewards/rejected": 3.183506123605184e-05,
       "step": 10
     },
     {
-      "epoch": 0.04,
-      "learning_rate": 2.0833333333333333e-07,
-      "logits/chosen": 0.25306791067123413,
-      "logits/rejected": 0.3252382278442383,
-      "logps/chosen": -318.19073486328125,
-      "logps/rejected": -289.6706237792969,
-      "loss": 0.6929,
-      "rewards/accuracies": 0.53125,
-      "rewards/chosen": 0.00048495858209207654,
-      "rewards/margins": 0.0009848512709140778,
-      "rewards/rejected": -0.0004998926888220012,
       "step": 20
     },
     {
-      "epoch": 0.06,
-      "learning_rate": 3.1249999999999997e-07,
-      "logits/chosen": 0.33226653933525085,
-      "logits/rejected": 0.3872108459472656,
-      "logps/chosen": -296.3697204589844,
-      "logps/rejected": -283.0611877441406,
-      "loss": 0.692,
-      "rewards/accuracies": 0.550000011920929,
-      "rewards/chosen": -0.0008094090735539794,
-      "rewards/margins": 0.0026363185606896877,
-      "rewards/rejected": -0.003445727750658989,
       "step": 30
     },
     {
-      "epoch": 0.08,
-      "learning_rate": 4.1666666666666667e-07,
-      "logits/chosen": 0.22801101207733154,
-      "logits/rejected": 0.32900214195251465,
-      "logps/chosen": -305.9015197753906,
-      "logps/rejected": -293.1842346191406,
-      "loss": 0.6883,
-      "rewards/accuracies": 0.643750011920929,
-      "rewards/chosen": -0.0027595984283834696,
-      "rewards/margins": 0.006589935161173344,
-      "rewards/rejected": -0.00934953335672617,
       "step": 40
     },
     {
-      "epoch": 0.1,
-      "learning_rate": 5.208333333333334e-07,
-      "logits/chosen": 0.27091675996780396,
-      "logits/rejected": 0.31866759061813354,
-      "logps/chosen": -314.2833557128906,
-      "logps/rejected": -307.02532958984375,
-      "loss": 0.683,
-      "rewards/accuracies": 0.6312500238418579,
-      "rewards/chosen": -0.0060192132368683815,
-      "rewards/margins": 0.019419629126787186,
-      "rewards/rejected": -0.025438839569687843,
       "step": 50
     },
     {
-      "epoch": 0.13,
-      "learning_rate": 6.249999999999999e-07,
-      "logits/chosen": 0.31704145669937134,
-      "logits/rejected": 0.4334793984889984,
-      "logps/chosen": -294.429931640625,
-      "logps/rejected": -272.87994384765625,
-      "loss": 0.6735,
-      "rewards/accuracies": 0.6312500238418579,
-      "rewards/chosen": -0.022902976721525192,
-      "rewards/margins": 0.04408121109008789,
-      "rewards/rejected": -0.06698418408632278,
       "step": 60
     },
     {
-      "epoch": 0.15,
-      "learning_rate": 7.291666666666666e-07,
-      "logits/chosen": 0.31964099407196045,
-      "logits/rejected": 0.3377896547317505,
-      "logps/chosen": -304.6803894042969,
-      "logps/rejected": -311.32794189453125,
-      "loss": 0.6602,
-      "rewards/accuracies": 0.6312500238418579,
-      "rewards/chosen": -0.05670947954058647,
-      "rewards/margins": 0.05069820210337639,
-      "rewards/rejected": -0.10740767419338226,
       "step": 70
     },
     {
-      "epoch": 0.17,
-      "learning_rate": 8.333333333333333e-07,
-      "logits/chosen": 0.35048729181289673,
-      "logits/rejected": 0.4193252921104431,
-      "logps/chosen": -306.3404541015625,
-      "logps/rejected": -282.4783020019531,
-      "loss": 0.6474,
-      "rewards/accuracies": 0.643750011920929,
-      "rewards/chosen": -0.10853584855794907,
-      "rewards/margins": 0.13117292523384094,
-      "rewards/rejected": -0.2397087812423706,
       "step": 80
     },
     {
-      "epoch": 0.19,
-      "learning_rate": 9.374999999999999e-07,
-      "logits/chosen": 0.32813602685928345,
-      "logits/rejected": 0.4464220404624939,
-      "logps/chosen": -341.1703186035156,
-      "logps/rejected": -299.92340087890625,
-      "loss": 0.6357,
-      "rewards/accuracies": 0.6499999761581421,
-      "rewards/chosen": -0.2554694712162018,
-      "rewards/margins": 0.12284588813781738,
-      "rewards/rejected": -0.37831538915634155,
       "step": 90
     },
     {
-      "epoch": 0.21,
-      "learning_rate": 9.999463737538052e-07,
-      "logits/chosen": 0.35799938440322876,
-      "logits/rejected": 0.3899138271808624,
-      "logps/chosen": -318.7712097167969,
-      "logps/rejected": -348.5688781738281,
-      "loss": 0.6201,
-      "rewards/accuracies": 0.6312500238418579,
-      "rewards/chosen": -0.3124231994152069,
-      "rewards/margins": 0.19829413294792175,
-      "rewards/rejected": -0.5107173323631287,
       "step": 100
     },
     {
-      "epoch": 0.21,
-      "eval_logits/chosen": 0.3705582916736603,
-      "eval_logits/rejected": 0.4172414541244507,
-      "eval_logps/chosen": -331.11236572265625,
-      "eval_logps/rejected": -378.240478515625,
-      "eval_loss": 0.6252639293670654,
-      "eval_rewards/accuracies": 0.703125,
-      "eval_rewards/chosen": -0.27527713775634766,
-      "eval_rewards/margins": 0.3908771872520447,
-      "eval_rewards/rejected": -0.6661543846130371,
-      "eval_runtime": 64.993,
-      "eval_samples_per_second": 30.773,
-      "eval_steps_per_second": 0.492,
-      "step": 100
-    },
-    {
-      "epoch": 0.23,
-      "learning_rate": 9.993432105822034e-07,
-      "logits/chosen": 0.3002661168575287,
-      "logits/rejected": 0.3483879864215851,
-      "logps/chosen": -350.9095458984375,
-      "logps/rejected": -360.30963134765625,
-      "loss": 0.6093,
-      "rewards/accuracies": 0.706250011920929,
-      "rewards/chosen": -0.37449145317077637,
-      "rewards/margins": 0.3816668689250946,
-      "rewards/rejected": -0.7561584115028381,
       "step": 110
     },
     {
-      "epoch": 0.25,
-      "learning_rate": 9.980706626858607e-07,
-      "logits/chosen": 0.2088731825351715,
-      "logits/rejected": 0.2899537980556488,
-      "logps/chosen": -395.3984069824219,
-      "logps/rejected": -433.7286682128906,
-      "loss": 0.5905,
-      "rewards/accuracies": 0.6875,
-      "rewards/chosen": -0.4185329079627991,
-      "rewards/margins": 0.4517739713191986,
-      "rewards/rejected": -0.8703069686889648,
       "step": 120
     },
     {
-      "epoch": 0.27,
-      "learning_rate": 9.961304359538434e-07,
-      "logits/chosen": 0.09751267731189728,
-      "logits/rejected": 0.22797170281410217,
-      "logps/chosen": -374.6502380371094,
-      "logps/rejected": -373.14263916015625,
-      "loss": 0.5998,
-      "rewards/accuracies": 0.6312500238418579,
-      "rewards/chosen": -0.4483565390110016,
-      "rewards/margins": 0.3120550811290741,
-      "rewards/rejected": -0.7604116201400757,
       "step": 130
     },
     {
-      "epoch": 0.29,
-      "learning_rate": 9.935251313189563e-07,
-      "logits/chosen": 0.23218217492103577,
-      "logits/rejected": 0.28826406598091125,
-      "logps/chosen": -363.15338134765625,
-      "logps/rejected": -381.83795166015625,
-      "loss": 0.5847,
-      "rewards/accuracies": 0.699999988079071,
-      "rewards/chosen": -0.49422675371170044,
-      "rewards/margins": 0.38399404287338257,
-      "rewards/rejected": -0.8782208561897278,
       "step": 140
     },
     {
-      "epoch": 0.31,
-      "learning_rate": 9.902582412711118e-07,
-      "logits/chosen": 0.29499703645706177,
-      "logits/rejected": 0.49714046716690063,
-      "logps/chosen": -342.9471130371094,
-      "logps/rejected": -372.1763610839844,
-      "loss": 0.5681,
-      "rewards/accuracies": 0.699999988079071,
-      "rewards/chosen": -0.6186890602111816,
-      "rewards/margins": 0.5347500443458557,
-      "rewards/rejected": -1.1534390449523926,
       "step": 150
     },
     {
-      "epoch": 0.33,
-      "learning_rate": 9.86334145175542e-07,
-      "logits/chosen": 0.35773637890815735,
-      "logits/rejected": 0.4941268861293793,
-      "logps/chosen": -370.3203125,
-      "logps/rejected": -428.2557067871094,
-      "loss": 0.5792,
-      "rewards/accuracies": 0.7437499761581421,
-      "rewards/chosen": -0.7682583928108215,
-      "rewards/margins": 0.6092099547386169,
-      "rewards/rejected": -1.377468228340149,
       "step": 160
     },
     {
-      "epoch": 0.36,
-      "learning_rate": 9.817581034021272e-07,
-      "logits/chosen": 0.19401590526103973,
-      "logits/rejected": 0.3156794607639313,
-      "logps/chosen": -415.65313720703125,
-      "logps/rejected": -425.68963623046875,
-      "loss": 0.5747,
-      "rewards/accuracies": 0.731249988079071,
-      "rewards/chosen": -0.5680743455886841,
-      "rewards/margins": 0.4286484122276306,
-      "rewards/rejected": -0.9967228174209595,
       "step": 170
     },
     {
-      "epoch": 0.38,
-      "learning_rate": 9.765362502737097e-07,
-      "logits/chosen": 0.2572034001350403,
-      "logits/rejected": 0.27328386902809143,
-      "logps/chosen": -358.0533447265625,
-      "logps/rejected": -398.5332946777344,
-      "loss": 0.5634,
-      "rewards/accuracies": 0.699999988079071,
-      "rewards/chosen": -0.644694447517395,
-      "rewards/margins": 0.5251447558403015,
-      "rewards/rejected": -1.1698391437530518,
       "step": 180
     },
     {
-      "epoch": 0.4,
-      "learning_rate": 9.706755858428485e-07,
-      "logits/chosen": 0.3962785303592682,
-      "logits/rejected": 0.4539657235145569,
-      "logps/chosen": -347.5164794921875,
-      "logps/rejected": -378.83184814453125,
-      "loss": 0.5423,
-      "rewards/accuracies": 0.71875,
-      "rewards/chosen": -0.7963652610778809,
-      "rewards/margins": 0.5720622539520264,
-      "rewards/rejected": -1.3684275150299072,
       "step": 190
     },
     {
-      "epoch": 0.42,
-      "learning_rate": 9.641839665080363e-07,
-      "logits/chosen": 0.3198946714401245,
-      "logits/rejected": 0.4063253402709961,
-      "logps/chosen": -352.0743713378906,
-      "logps/rejected": -419.7388610839844,
-      "loss": 0.5547,
-      "rewards/accuracies": 0.7250000238418579,
-      "rewards/chosen": -0.7651049494743347,
-      "rewards/margins": 0.6531444787979126,
-      "rewards/rejected": -1.418249487876892,
-      "step": 200
-    },
-    {
-      "epoch": 0.42,
-      "eval_logits/chosen": 0.3908616304397583,
-      "eval_logits/rejected": 0.4261176884174347,
-      "eval_logps/chosen": -373.46612548828125,
-      "eval_logps/rejected": -458.88629150390625,
-      "eval_loss": 0.5549300312995911,
-      "eval_rewards/accuracies": 0.765625,
-      "eval_rewards/chosen": -0.6988146305084229,
-      "eval_rewards/margins": 0.7737974524497986,
-      "eval_rewards/rejected": -1.4726121425628662,
-      "eval_runtime": 65.2313,
-      "eval_samples_per_second": 30.66,
-      "eval_steps_per_second": 0.491,
       "step": 200
     },
     {
-      "epoch": 0.44,
-      "learning_rate": 9.570700944819582e-07,
-      "logits/chosen": 0.3505176901817322,
-      "logits/rejected": 0.42375579476356506,
-      "logps/chosen": -397.61199951171875,
-      "logps/rejected": -454.89776611328125,
-      "loss": 0.5265,
-      "rewards/accuracies": 0.762499988079071,
-      "rewards/chosen": -0.7523492574691772,
-      "rewards/margins": 0.7344074845314026,
-      "rewards/rejected": -1.486756682395935,
       "step": 210
     },
     {
-      "epoch": 0.46,
-      "learning_rate": 9.493435061259129e-07,
-      "logits/chosen": 0.2726442813873291,
-      "logits/rejected": 0.4434526860713959,
-      "logps/chosen": -410.77667236328125,
-      "logps/rejected": -431.207275390625,
-      "loss": 0.555,
-      "rewards/accuracies": 0.731249988079071,
-      "rewards/chosen": -0.8984044194221497,
-      "rewards/margins": 0.5839776992797852,
-      "rewards/rejected": -1.48238205909729,
       "step": 220
     },
     {
-      "epoch": 0.48,
-      "learning_rate": 9.4101455916603e-07,
-      "logits/chosen": 0.28837597370147705,
-      "logits/rejected": 0.35526323318481445,
-      "logps/chosen": -363.0335998535156,
-      "logps/rejected": -436.46612548828125,
-      "loss": 0.5498,
-      "rewards/accuracies": 0.7124999761581421,
-      "rewards/chosen": -0.8101651072502136,
-      "rewards/margins": 0.737695574760437,
-      "rewards/rejected": -1.5478605031967163,
       "step": 230
     },
     {
-      "epoch": 0.5,
-      "learning_rate": 9.320944188084241e-07,
-      "logits/chosen": 0.23826150596141815,
-      "logits/rejected": 0.285171240568161,
-      "logps/chosen": -432.6297912597656,
-      "logps/rejected": -497.56341552734375,
-      "loss": 0.5392,
-      "rewards/accuracies": 0.7437499761581421,
-      "rewards/chosen": -0.7939236164093018,
-      "rewards/margins": 0.7111212611198425,
-      "rewards/rejected": -1.505044937133789,
       "step": 240
     },
     {
-      "epoch": 0.52,
-      "learning_rate": 9.225950427718974e-07,
-      "logits/chosen": 0.2762988209724426,
-      "logits/rejected": 0.31130319833755493,
-      "logps/chosen": -399.399169921875,
-      "logps/rejected": -445.702880859375,
-      "loss": 0.5465,
-      "rewards/accuracies": 0.6875,
-      "rewards/chosen": -0.83611661195755,
-      "rewards/margins": 0.7073522806167603,
-      "rewards/rejected": -1.5434690713882446,
       "step": 250
     },
     {
-      "epoch": 0.54,
-      "learning_rate": 9.125291652582547e-07,
-      "logits/chosen": 0.1327328383922577,
-      "logits/rejected": 0.3085227310657501,
-      "logps/chosen": -436.08135986328125,
-      "logps/rejected": -457.634765625,
-      "loss": 0.5194,
-      "rewards/accuracies": 0.699999988079071,
-      "rewards/chosen": -0.9428914785385132,
-      "rewards/margins": 0.6950392723083496,
-      "rewards/rejected": -1.6379308700561523,
       "step": 260
     },
     {
-      "epoch": 0.57,
-      "learning_rate": 9.019102798817195e-07,
-      "logits/chosen": 0.23745720088481903,
-      "logits/rejected": 0.34172096848487854,
-      "logps/chosen": -421.8299865722656,
-      "logps/rejected": -466.4856872558594,
-      "loss": 0.5496,
-      "rewards/accuracies": 0.71875,
-      "rewards/chosen": -0.9325121641159058,
-      "rewards/margins": 0.7680062651634216,
-      "rewards/rejected": -1.7005186080932617,
       "step": 270
     },
     {
-      "epoch": 0.59,
-      "learning_rate": 8.90752621580335e-07,
-      "logits/chosen": 0.16251161694526672,
-      "logits/rejected": 0.2581509053707123,
-      "logps/chosen": -418.5828552246094,
-      "logps/rejected": -507.22412109375,
-      "loss": 0.5168,
-      "rewards/accuracies": 0.7250000238418579,
-      "rewards/chosen": -1.113261342048645,
-      "rewards/margins": 0.7757157683372498,
-      "rewards/rejected": -1.88897705078125,
       "step": 280
     },
     {
-      "epoch": 0.61,
-      "learning_rate": 8.79071147533597e-07,
-      "logits/chosen": 0.18345972895622253,
-      "logits/rejected": 0.24752414226531982,
-      "logps/chosen": -374.2388916015625,
-      "logps/rejected": -421.7548828125,
-      "loss": 0.5452,
-      "rewards/accuracies": 0.699999988079071,
-      "rewards/chosen": -0.7729798555374146,
-      "rewards/margins": 0.6754422187805176,
-      "rewards/rejected": -1.448421835899353,
       "step": 290
     },
     {
-      "epoch": 0.63,
-      "learning_rate": 8.668815171119019e-07,
-      "logits/chosen": 0.11917382478713989,
-      "logits/rejected": 0.2862890362739563,
-      "logps/chosen": -396.6165466308594,
-      "logps/rejected": -400.9344787597656,
-      "loss": 0.5343,
-      "rewards/accuracies": 0.75,
-      "rewards/chosen": -0.8496394157409668,
-      "rewards/margins": 0.5671547055244446,
-      "rewards/rejected": -1.4167941808700562,
-      "step": 300
-    },
-    {
-      "epoch": 0.63,
-      "eval_logits/chosen": 0.24486932158470154,
-      "eval_logits/rejected": 0.2851215898990631,
-      "eval_logps/chosen": -384.0199279785156,
-      "eval_logps/rejected": -476.3627624511719,
-      "eval_loss": 0.5315821766853333,
-      "eval_rewards/accuracies": 0.765625,
-      "eval_rewards/chosen": -0.8043524026870728,
-      "eval_rewards/margins": 0.8430246114730835,
-      "eval_rewards/rejected": -1.6473771333694458,
-      "eval_runtime": 64.8898,
-      "eval_samples_per_second": 30.822,
-      "eval_steps_per_second": 0.493,
       "step": 300
     },
     {
-      "epoch": 0.65,
-      "learning_rate": 8.54200070884685e-07,
-      "logits/chosen": 0.21563191711902618,
-      "logits/rejected": 0.1952591836452484,
-      "logps/chosen": -438.91552734375,
-      "logps/rejected": -499.61395263671875,
-      "loss": 0.5567,
-      "rewards/accuracies": 0.699999988079071,
-      "rewards/chosen": -1.173762559890747,
-      "rewards/margins": 0.5995947122573853,
-      "rewards/rejected": -1.7733571529388428,
       "step": 310
     },
     {
-      "epoch": 0.67,
-      "learning_rate": 8.410438087153911e-07,
-      "logits/chosen": 0.031750187277793884,
-      "logits/rejected": 0.14312420785427094,
-      "logps/chosen": -457.0936584472656,
-      "logps/rejected": -459.15411376953125,
-      "loss": 0.5366,
-      "rewards/accuracies": 0.612500011920929,
-      "rewards/chosen": -1.0222933292388916,
-      "rewards/margins": 0.4668423533439636,
-      "rewards/rejected": -1.4891356229782104,
       "step": 320
     },
     {
-      "epoch": 0.69,
-      "learning_rate": 8.274303669726426e-07,
-      "logits/chosen": 0.07745673507452011,
-      "logits/rejected": 0.07082104682922363,
-      "logps/chosen": -399.32464599609375,
-      "logps/rejected": -523.4277954101562,
-      "loss": 0.532,
-      "rewards/accuracies": 0.7749999761581421,
-      "rewards/chosen": -0.9549884796142578,
-      "rewards/margins": 0.8862001299858093,
-      "rewards/rejected": -1.8411887884140015,
       "step": 330
     },
     {
-      "epoch": 0.71,
-      "learning_rate": 8.133779948881513e-07,
-      "logits/chosen": 0.10294970124959946,
-      "logits/rejected": 0.09352216869592667,
-      "logps/chosen": -409.33770751953125,
-      "logps/rejected": -510.10089111328125,
-      "loss": 0.5217,
-      "rewards/accuracies": 0.7124999761581421,
-      "rewards/chosen": -1.1990526914596558,
-      "rewards/margins": 0.7596696615219116,
-      "rewards/rejected": -1.958722472190857,
       "step": 340
     },
     {
-      "epoch": 0.73,
-      "learning_rate": 7.989055300930704e-07,
-      "logits/chosen": 0.16737070679664612,
-      "logits/rejected": 0.21901166439056396,
-      "logps/chosen": -437.2059020996094,
-      "logps/rejected": -538.3087158203125,
-      "loss": 0.5091,
-      "rewards/accuracies": 0.7437499761581421,
-      "rewards/chosen": -1.2976617813110352,
-      "rewards/margins": 0.8197441101074219,
-      "rewards/rejected": -2.117405891418457,
       "step": 350
     },
     {
-      "epoch": 0.75,
-      "learning_rate": 7.840323733655778e-07,
-      "logits/chosen": 0.1267612874507904,
-      "logits/rejected": 0.20460394024848938,
-      "logps/chosen": -465.1019592285156,
-      "logps/rejected": -518.5805053710938,
-      "loss": 0.5185,
-      "rewards/accuracies": 0.7562500238418579,
-      "rewards/chosen": -1.174440860748291,
-      "rewards/margins": 0.9074214696884155,
-      "rewards/rejected": -2.081862211227417,
       "step": 360
     },
     {
-      "epoch": 0.77,
-      "learning_rate": 7.687784626235447e-07,
-      "logits/chosen": 0.18740633130073547,
-      "logits/rejected": 0.27840983867645264,
-      "logps/chosen": -468.21807861328125,
-      "logps/rejected": -493.06646728515625,
-      "loss": 0.5158,
-      "rewards/accuracies": 0.675000011920929,
-      "rewards/chosen": -1.1525232791900635,
-      "rewards/margins": 0.7130603790283203,
-      "rewards/rejected": -1.8655836582183838,
       "step": 370
     },
     {
-      "epoch": 0.8,
-      "learning_rate": 7.531642461971514e-07,
-      "logits/chosen": 0.15459200739860535,
-      "logits/rejected": 0.23283176124095917,
-      "logps/chosen": -453.07904052734375,
-      "logps/rejected": -535.761474609375,
-      "loss": 0.5392,
-      "rewards/accuracies": 0.7562500238418579,
-      "rewards/chosen": -1.2102587223052979,
-      "rewards/margins": 1.0239769220352173,
-      "rewards/rejected": -2.2342355251312256,
       "step": 380
     },
     {
-      "epoch": 0.82,
-      "learning_rate": 7.372106554172801e-07,
-      "logits/chosen": 0.19951777160167694,
-      "logits/rejected": 0.2895793318748474,
-      "logps/chosen": -411.4781799316406,
-      "logps/rejected": -466.41253662109375,
-      "loss": 0.5356,
-      "rewards/accuracies": 0.7124999761581421,
-      "rewards/chosen": -0.8807674646377563,
-      "rewards/margins": 0.7255697846412659,
-      "rewards/rejected": -1.606337308883667,
       "step": 390
     },
     {
-      "epoch": 0.84,
-      "learning_rate": 7.209390765564318e-07,
-      "logits/chosen": 0.1757899820804596,
-      "logits/rejected": 0.23791635036468506,
-      "logps/chosen": -390.8957214355469,
-      "logps/rejected": -478.49407958984375,
-      "loss": 0.5323,
-      "rewards/accuracies": 0.7437499761581421,
-      "rewards/chosen": -0.9380720257759094,
-      "rewards/margins": 0.7229949235916138,
-      "rewards/rejected": -1.661067008972168,
-      "step": 400
-    },
-    {
-      "epoch": 0.84,
-      "eval_logits/chosen": 0.25136542320251465,
-      "eval_logits/rejected": 0.28342366218566895,
-      "eval_logps/chosen": -394.2620849609375,
-      "eval_logps/rejected": -494.46002197265625,
-      "eval_loss": 0.521114706993103,
-      "eval_rewards/accuracies": 0.78125,
-      "eval_rewards/chosen": -0.90677410364151,
-      "eval_rewards/margins": 0.9215754270553589,
-      "eval_rewards/rejected": -1.8283497095108032,
-      "eval_runtime": 65.2477,
-      "eval_samples_per_second": 30.652,
-      "eval_steps_per_second": 0.49,
       "step": 400
     },
     {
-      "epoch": 0.86,
-      "learning_rate": 7.043713221597773e-07,
-      "logits/chosen": 0.1401471644639969,
-      "logits/rejected": 0.17819848656654358,
-      "logps/chosen": -401.8942565917969,
-      "logps/rejected": -474.2579650878906,
-      "loss": 0.5225,
-      "rewards/accuracies": 0.7749999761581421,
-      "rewards/chosen": -0.8892021179199219,
-      "rewards/margins": 0.906479001045227,
-      "rewards/rejected": -1.7956812381744385,
       "step": 410
     },
     {
-      "epoch": 0.88,
-      "learning_rate": 6.875296018047809e-07,
-      "logits/chosen": 0.1304786652326584,
-      "logits/rejected": 0.15569528937339783,
-      "logps/chosen": -400.5438537597656,
-      "logps/rejected": -468.2953186035156,
-      "loss": 0.5065,
-      "rewards/accuracies": 0.8187500238418579,
-      "rewards/chosen": -0.9208317995071411,
-      "rewards/margins": 0.9665653109550476,
-      "rewards/rejected": -1.887397050857544,
       "step": 420
     },
     {
-      "epoch": 0.9,
-      "learning_rate": 6.704364923285857e-07,
-      "logits/chosen": 0.09775003790855408,
-      "logits/rejected": 0.1501173973083496,
-      "logps/chosen": -476.73065185546875,
-      "logps/rejected": -533.7389526367188,
-      "loss": 0.5285,
-      "rewards/accuracies": 0.71875,
-      "rewards/chosen": -1.26286780834198,
-      "rewards/margins": 0.8789494633674622,
-      "rewards/rejected": -2.141817569732666,
       "step": 430
     },
     {
-      "epoch": 0.92,
-      "learning_rate": 6.531149075630796e-07,
-      "logits/chosen": 0.1202029138803482,
-      "logits/rejected": 0.1756385862827301,
-      "logps/chosen": -400.1629333496094,
-      "logps/rejected": -469.0196838378906,
-      "loss": 0.5074,
-      "rewards/accuracies": 0.6812499761581421,
-      "rewards/chosen": -1.0582826137542725,
-      "rewards/margins": 0.9171028137207031,
-      "rewards/rejected": -1.975385308265686,
       "step": 440
     },
     {
-      "epoch": 0.94,
-      "learning_rate": 6.355880676182085e-07,
-      "logits/chosen": 0.01758761703968048,
-      "logits/rejected": 0.15642888844013214,
-      "logps/chosen": -443.065185546875,
-      "logps/rejected": -484.41473388671875,
-      "loss": 0.5295,
-      "rewards/accuracies": 0.6875,
-      "rewards/chosen": -1.004433035850525,
-      "rewards/margins": 0.8090255856513977,
-      "rewards/rejected": -1.8134586811065674,
       "step": 450
     },
     {
-      "epoch": 0.96,
-      "learning_rate": 6.178794677547137e-07,
-      "logits/chosen": 0.012063628062605858,
-      "logits/rejected": 0.10572747141122818,
-      "logps/chosen": -408.4104919433594,
-      "logps/rejected": -449.46881103515625,
-      "loss": 0.5267,
-      "rewards/accuracies": 0.7437499761581421,
-      "rewards/chosen": -1.0622318983078003,
-      "rewards/margins": 0.6900007724761963,
-      "rewards/rejected": -1.752232551574707,
       "step": 460
     },
     {
-      "epoch": 0.98,
-      "learning_rate": 6.000128468880222e-07,
-      "logits/chosen": -0.05889149755239487,
-      "logits/rejected": -0.014351313933730125,
-      "logps/chosen": -449.07061767578125,
-      "logps/rejected": -538.423828125,
-      "loss": 0.5042,
-      "rewards/accuracies": 0.762499988079071,
-      "rewards/chosen": -1.1122030019760132,
-      "rewards/margins": 0.9293983578681946,
-      "rewards/rejected": -2.0416014194488525,
       "step": 470
     },
     {
-      "epoch": 1.0,
-      "learning_rate": 5.820121557655108e-07,
-      "logits/chosen": -0.045923542231321335,
-      "logits/rejected": 0.03086056187748909,
-      "logps/chosen": -422.5347595214844,
-      "logps/rejected": -529.2613525390625,
-      "loss": 0.4682,
-      "rewards/accuracies": 0.768750011920929,
-      "rewards/chosen": -1.275895118713379,
-      "rewards/margins": 1.037326455116272,
-      "rewards/rejected": -2.3132214546203613,
       "step": 480
     },
     {
-      "epoch": 1.03,
-      "learning_rate": 5.639015248598023e-07,
-      "logits/chosen": -0.05794327333569527,
-      "logits/rejected": -0.09440571069717407,
-      "logps/chosen": -441.08990478515625,
-      "logps/rejected": -601.30419921875,
-      "loss": 0.3493,
-      "rewards/accuracies": 0.862500011920929,
-      "rewards/chosen": -1.3565930128097534,
-      "rewards/margins": 1.8090267181396484,
-      "rewards/rejected": -3.1656198501586914,
       "step": 490
     },
     {
-      "epoch": 1.05,
-      "learning_rate": 5.457052320211339e-07,
-      "logits/chosen": -0.23803594708442688,
-      "logits/rejected": -0.29828980565071106,
-      "logps/chosen": -474.3367614746094,
-      "logps/rejected": -656.8623046875,
-      "loss": 0.352,
-      "rewards/accuracies": 0.8187500238418579,
-      "rewards/chosen": -1.730020523071289,
-      "rewards/margins": 1.8847625255584717,
-      "rewards/rejected": -3.6147830486297607,
-      "step": 500
-    },
-    {
-      "epoch": 1.05,
-      "eval_logits/chosen": -0.06535135954618454,
-      "eval_logits/rejected": -0.08458372950553894,
-      "eval_logps/chosen": -498.9117431640625,
-      "eval_logps/rejected": -653.2899169921875,
-      "eval_loss": 0.525809109210968,
-      "eval_rewards/accuracies": 0.796875,
-      "eval_rewards/chosen": -1.9532711505889893,
-      "eval_rewards/margins": 1.4633771181106567,
-      "eval_rewards/rejected": -3.4166483879089355,
-      "eval_runtime": 64.5447,
-      "eval_samples_per_second": 30.986,
-      "eval_steps_per_second": 0.496,
       "step": 500
     },
     {
-      "epoch": 1.07,
-      "learning_rate": 5.274476699321637e-07,
-      "logits/chosen": -0.17468394339084625,
-      "logits/rejected": -0.19271844625473022,
-      "logps/chosen": -464.8907165527344,
-      "logps/rejected": -632.8477172851562,
-      "loss": 0.3423,
-      "rewards/accuracies": 0.893750011920929,
-      "rewards/chosen": -1.695770263671875,
-      "rewards/margins": 1.8958046436309814,
-      "rewards/rejected": -3.5915749073028564,
       "step": 510
     },
     {
-      "epoch": 1.09,
-      "learning_rate": 5.091533134088387e-07,
-      "logits/chosen": -0.23788562417030334,
-      "logits/rejected": -0.20414999127388,
-      "logps/chosen": -485.94415283203125,
-      "logps/rejected": -640.9654541015625,
-      "loss": 0.3236,
-      "rewards/accuracies": 0.8999999761581421,
-      "rewards/chosen": -1.8264538049697876,
-      "rewards/margins": 1.7633371353149414,
-      "rewards/rejected": -3.5897908210754395,
       "step": 520
     },
     {
-      "epoch": 1.11,
-      "learning_rate": 4.908466865911614e-07,
-      "logits/chosen": -0.11999205499887466,
-      "logits/rejected": -0.09423510730266571,
-      "logps/chosen": -487.30810546875,
-      "logps/rejected": -646.9547119140625,
-      "loss": 0.3424,
-      "rewards/accuracies": 0.84375,
-      "rewards/chosen": -1.7824156284332275,
-      "rewards/margins": 1.7626768350601196,
-      "rewards/rejected": -3.5450921058654785,
       "step": 530
     },
     {
-      "epoch": 1.13,
-      "learning_rate": 4.7255233006783624e-07,
-      "logits/chosen": -0.16903451085090637,
-      "logits/rejected": -0.06715662032365799,
-      "logps/chosen": -530.5585327148438,
-      "logps/rejected": -645.1633911132812,
-      "loss": 0.3301,
-      "rewards/accuracies": 0.862500011920929,
-      "rewards/chosen": -1.7870140075683594,
-      "rewards/margins": 1.8271121978759766,
-      "rewards/rejected": -3.614126682281494,
       "step": 540
     },
     {
-      "epoch": 1.15,
-      "learning_rate": 4.5429476797886617e-07,
-      "logits/chosen": -0.03496643900871277,
-      "logits/rejected": -0.04560618847608566,
-      "logps/chosen": -483.15203857421875,
-      "logps/rejected": -634.9447631835938,
-      "loss": 0.3501,
-      "rewards/accuracies": 0.8187500238418579,
-      "rewards/chosen": -1.8454551696777344,
-      "rewards/margins": 1.5336250066757202,
-      "rewards/rejected": -3.379080295562744,
       "step": 550
     },
     {
-      "epoch": 1.17,
-      "learning_rate": 4.3609847514019763e-07,
-      "logits/chosen": 0.06872721016407013,
-      "logits/rejected": 0.0859331339597702,
-      "logps/chosen": -506.5009765625,
-      "logps/rejected": -646.7543334960938,
-      "loss": 0.3359,
-      "rewards/accuracies": 0.875,
-      "rewards/chosen": -1.7461020946502686,
-      "rewards/margins": 1.8654359579086304,
-      "rewards/rejected": -3.6115379333496094,
       "step": 560
     },
     {
-      "epoch": 1.19,
-      "learning_rate": 4.179878442344892e-07,
-      "logits/chosen": -0.04321649298071861,
-      "logits/rejected": 0.051123034209012985,
-      "logps/chosen": -496.735595703125,
-      "logps/rejected": -647.29150390625,
-      "loss": 0.3464,
-      "rewards/accuracies": 0.8062499761581421,
-      "rewards/chosen": -2.0714824199676514,
-      "rewards/margins": 1.762717843055725,
-      "rewards/rejected": -3.834200620651245,
       "step": 570
     },
     {
-      "epoch": 1.21,
-      "learning_rate": 3.9998715311197783e-07,
-      "logits/chosen": -0.004042728338390589,
-      "logits/rejected": 0.04070080816745758,
-      "logps/chosen": -512.9417724609375,
-      "logps/rejected": -696.821044921875,
-      "loss": 0.3365,
-      "rewards/accuracies": 0.862500011920929,
-      "rewards/chosen": -2.097195863723755,
-      "rewards/margins": 1.831916093826294,
-      "rewards/rejected": -3.9291114807128906,
       "step": 580
     },
     {
-      "epoch": 1.23,
-      "learning_rate": 3.821205322452863e-07,
-      "logits/chosen": 0.09215477854013443,
-      "logits/rejected": 0.05737446993589401,
-      "logps/chosen": -525.1434936523438,
-      "logps/rejected": -705.9063110351562,
-      "loss": 0.3158,
-      "rewards/accuracies": 0.856249988079071,
-      "rewards/chosen": -2.0721964836120605,
-      "rewards/margins": 1.9721500873565674,
-      "rewards/rejected": -4.044346809387207,
       "step": 590
     },
     {
-      "epoch": 1.26,
-      "learning_rate": 3.6441193238179146e-07,
-      "logits/chosen": 0.07227401435375214,
-      "logits/rejected": 0.03988388180732727,
-      "logps/chosen": -549.4615478515625,
-      "logps/rejected": -771.7275390625,
-      "loss": 0.3342,
-      "rewards/accuracies": 0.856249988079071,
-      "rewards/chosen": -2.2555530071258545,
-      "rewards/margins": 2.014453649520874,
-      "rewards/rejected": -4.2700066566467285,
-      "step": 600
-    },
-    {
-      "epoch": 1.26,
-      "eval_logits/chosen": 0.1344175487756729,
-      "eval_logits/rejected": 0.11280365288257599,
-      "eval_logps/chosen": -534.8101196289062,
-      "eval_logps/rejected": -684.085693359375,
-      "eval_loss": 0.5267595648765564,
-      "eval_rewards/accuracies": 0.79296875,
-      "eval_rewards/chosen": -2.3122546672821045,
-      "eval_rewards/margins": 1.412351369857788,
-      "eval_rewards/rejected": -3.7246060371398926,
-      "eval_runtime": 64.3013,
-      "eval_samples_per_second": 31.104,
-      "eval_steps_per_second": 0.498,
       "step": 600
     },
     {
-      "epoch": 1.28,
-      "learning_rate": 3.4688509243692034e-07,
-      "logits/chosen": 0.021421348676085472,
-      "logits/rejected": 0.08866464346647263,
-      "logps/chosen": -535.4236450195312,
-      "logps/rejected": -694.8360595703125,
-      "loss": 0.344,
-      "rewards/accuracies": 0.8687499761581421,
-      "rewards/chosen": -2.009089946746826,
-      "rewards/margins": 1.8043378591537476,
-      "rewards/rejected": -3.8134284019470215,
       "step": 610
     },
     {
-      "epoch": 1.3,
-      "learning_rate": 3.295635076714144e-07,
-      "logits/chosen": -0.06299210339784622,
-      "logits/rejected": -0.04097691923379898,
-      "logps/chosen": -499.0265197753906,
-      "logps/rejected": -653.1188354492188,
-      "loss": 0.3317,
       "rewards/accuracies": 0.862500011920929,
-      "rewards/chosen": -1.9393279552459717,
-      "rewards/margins": 1.6952606439590454,
-      "rewards/rejected": -3.6345887184143066,
       "step": 620
     },
     {
-      "epoch": 1.32,
-      "learning_rate": 3.12470398195219e-07,
-      "logits/chosen": -0.013870243914425373,
-      "logits/rejected": 0.0671583041548729,
-      "logps/chosen": -512.1448974609375,
-      "logps/rejected": -680.2833251953125,
-      "loss": 0.3283,
-      "rewards/accuracies": 0.887499988079071,
-      "rewards/chosen": -1.9993736743927002,
-      "rewards/margins": 1.9316644668579102,
-      "rewards/rejected": -3.9310379028320312,
       "step": 630
     },
     {
-      "epoch": 1.34,
-      "learning_rate": 2.956286778402226e-07,
-      "logits/chosen": -0.11906696856021881,
-      "logits/rejected": -0.17396704852581024,
-      "logps/chosen": -495.10345458984375,
-      "logps/rejected": -677.45556640625,
-      "loss": 0.319,
-      "rewards/accuracies": 0.8812500238418579,
-      "rewards/chosen": -2.2106266021728516,
-      "rewards/margins": 1.9618394374847412,
-      "rewards/rejected": -4.172466278076172,
       "step": 640
     },
     {
-      "epoch": 1.36,
-      "learning_rate": 2.7906092344356826e-07,
-      "logits/chosen": -0.07961982488632202,
-      "logits/rejected": -0.12522803246974945,
-      "logps/chosen": -516.12890625,
-      "logps/rejected": -696.549560546875,
-      "loss": 0.3233,
-      "rewards/accuracies": 0.8374999761581421,
-      "rewards/chosen": -2.1828126907348633,
-      "rewards/margins": 1.8567641973495483,
-      "rewards/rejected": -4.039577007293701,
       "step": 650
     },
     {
-      "epoch": 1.38,
-      "learning_rate": 2.6278934458271996e-07,
-      "logits/chosen": -0.15696656703948975,
-      "logits/rejected": -0.12818947434425354,
-      "logps/chosen": -568.789794921875,
-      "logps/rejected": -722.9078369140625,
-      "loss": 0.3273,
-      "rewards/accuracies": 0.862500011920929,
-      "rewards/chosen": -2.1225264072418213,
-      "rewards/margins": 1.9797760248184204,
-      "rewards/rejected": -4.102302551269531,
       "step": 660
     },
     {
-      "epoch": 1.4,
-      "learning_rate": 2.468357538028487e-07,
-      "logits/chosen": 0.01013887207955122,
-      "logits/rejected": 0.029213298112154007,
-      "logps/chosen": -541.7221069335938,
-      "logps/rejected": -720.105712890625,
-      "loss": 0.3153,
       "rewards/accuracies": 0.856249988079071,
-      "rewards/chosen": -2.205817461013794,
-      "rewards/margins": 2.0757954120635986,
-      "rewards/rejected": -4.281612396240234,
       "step": 670
     },
     {
-      "epoch": 1.42,
-      "learning_rate": 2.312215373764551e-07,
-      "logits/chosen": -0.06487278640270233,
-      "logits/rejected": -0.01965305209159851,
-      "logps/chosen": -516.360107421875,
-      "logps/rejected": -694.418212890625,
-      "loss": 0.3262,
-      "rewards/accuracies": 0.856249988079071,
-      "rewards/chosen": -2.184447765350342,
-      "rewards/margins": 1.861135721206665,
-      "rewards/rejected": -4.045583248138428,
       "step": 680
     },
     {
-      "epoch": 1.44,
-      "learning_rate": 2.1596762663442213e-07,
-      "logits/chosen": -0.16129662096500397,
-      "logits/rejected": -0.09581325948238373,
-      "logps/chosen": -546.6666259765625,
-      "logps/rejected": -722.95263671875,
-      "loss": 0.3194,
-      "rewards/accuracies": 0.8500000238418579,
-      "rewards/chosen": -2.1821353435516357,
-      "rewards/margins": 2.131826877593994,
-      "rewards/rejected": -4.313961982727051,
       "step": 690
     },
     {
-      "epoch": 1.47,
-      "learning_rate": 2.0109446990692963e-07,
-      "logits/chosen": -0.11689990758895874,
-      "logits/rejected": -0.2061731368303299,
-      "logps/chosen": -537.8167724609375,
-      "logps/rejected": -781.1466674804688,
-      "loss": 0.337,
-      "rewards/accuracies": 0.8812500238418579,
-      "rewards/chosen": -2.2769298553466797,
-      "rewards/margins": 2.1021370887756348,
-      "rewards/rejected": -4.3790669441223145,
-      "step": 700
-    },
-    {
-      "epoch": 1.47,
-      "eval_logits/chosen": 0.041396014392375946,
-      "eval_logits/rejected": 0.009947247803211212,
-      "eval_logps/chosen": -541.1116333007812,
-      "eval_logps/rejected": -699.990966796875,
-      "eval_loss": 0.5290427207946777,
-      "eval_rewards/accuracies": 0.77734375,
-      "eval_rewards/chosen": -2.375269651412964,
-      "eval_rewards/margins": 1.5083887577056885,
-      "eval_rewards/rejected": -3.883657932281494,
-      "eval_runtime": 64.3756,
-      "eval_samples_per_second": 31.068,
-      "eval_steps_per_second": 0.497,
       "step": 700
     },
     {
-      "epoch": 1.49,
-      "learning_rate": 1.8662200511184872e-07,
-      "logits/chosen": -0.05749096721410751,
-      "logits/rejected": -0.05046076700091362,
-      "logps/chosen": -522.4590454101562,
-      "logps/rejected": -701.3648071289062,
-      "loss": 0.3387,
-      "rewards/accuracies": 0.8812500238418579,
-      "rewards/chosen": -2.1689815521240234,
-      "rewards/margins": 2.023318290710449,
-      "rewards/rejected": -4.192299842834473,
-      "step": 710
-    },
-    {
-      "epoch": 1.51,
-      "learning_rate": 1.725696330273575e-07,
-      "logits/chosen": -0.12996384501457214,
-      "logits/rejected": -0.20302283763885498,
-      "logps/chosen": -513.8458251953125,
-      "logps/rejected": -704.156982421875,
-      "loss": 0.3363,
-      "rewards/accuracies": 0.887499988079071,
-      "rewards/chosen": -1.9350038766860962,
-      "rewards/margins": 2.0845541954040527,
-      "rewards/rejected": -4.019558429718018,
-      "step": 720
-    },
-    {
-      "epoch": 1.53,
-      "learning_rate": 1.589561912846089e-07,
-      "logits/chosen": -0.02087187021970749,
-      "logits/rejected": -0.01297883689403534,
-      "logps/chosen": -513.4478149414062,
-      "logps/rejected": -715.5861206054688,
-      "loss": 0.3311,
-      "rewards/accuracies": 0.8812500238418579,
-      "rewards/chosen": -2.01027512550354,
-      "rewards/margins": 1.9698785543441772,
-      "rewards/rejected": -3.9801535606384277,
-      "step": 730
-    },
-    {
-      "epoch": 1.55,
-      "learning_rate": 1.4579992911531496e-07,
-      "logits/chosen": -0.08378951251506805,
-      "logits/rejected": -0.10333013534545898,
-      "logps/chosen": -528.8599853515625,
-      "logps/rejected": -697.0358276367188,
-      "loss": 0.31,
-      "rewards/accuracies": 0.887499988079071,
-      "rewards/chosen": -2.1352083683013916,
-      "rewards/margins": 2.004941940307617,
-      "rewards/rejected": -4.140150547027588,
-      "step": 740
-    },
-    {
-      "epoch": 1.57,
-      "learning_rate": 1.3311848288809813e-07,
-      "logits/chosen": -0.021279722452163696,
-      "logits/rejected": -0.07772192358970642,
-      "logps/chosen": -553.6995239257812,
-      "logps/rejected": -696.1124267578125,
-      "loss": 0.3325,
-      "rewards/accuracies": 0.856249988079071,
-      "rewards/chosen": -2.2102222442626953,
-      "rewards/margins": 1.8731515407562256,
-      "rewards/rejected": -4.083374500274658,
-      "step": 750
-    },
-    {
-      "epoch": 1.59,
-      "learning_rate": 1.209288524664029e-07,
-      "logits/chosen": -0.1553444117307663,
-      "logits/rejected": -0.07370997965335846,
-      "logps/chosen": -559.6234130859375,
-      "logps/rejected": -718.3536376953125,
-      "loss": 0.3318,
-      "rewards/accuracies": 0.84375,
-      "rewards/chosen": -2.2175405025482178,
-      "rewards/margins": 2.087991714477539,
-      "rewards/rejected": -4.305531978607178,
-      "step": 760
-    },
-    {
-      "epoch": 1.61,
-      "learning_rate": 1.0924737841966497e-07,
-      "logits/chosen": 0.06289811432361603,
-      "logits/rejected": 0.08451451361179352,
-      "logps/chosen": -502.3817443847656,
-      "logps/rejected": -655.4021606445312,
-      "loss": 0.3065,
-      "rewards/accuracies": 0.856249988079071,
-      "rewards/chosen": -2.215487241744995,
-      "rewards/margins": 1.8723819255828857,
-      "rewards/rejected": -4.087869644165039,
-      "step": 770
-    },
-    {
-      "epoch": 1.63,
-      "learning_rate": 9.808972011828054e-08,
-      "logits/chosen": -0.028034457936882973,
-      "logits/rejected": -0.07584713399410248,
-      "logps/chosen": -507.1363220214844,
-      "logps/rejected": -684.7667236328125,
-      "loss": 0.321,
-      "rewards/accuracies": 0.831250011920929,
-      "rewards/chosen": -2.1353392601013184,
-      "rewards/margins": 1.854984998703003,
-      "rewards/rejected": -3.9903244972229004,
-      "step": 780
-    },
-    {
-      "epoch": 1.65,
-      "learning_rate": 8.747083474174527e-08,
-      "logits/chosen": -0.027467548847198486,
-      "logits/rejected": -0.11992067098617554,
-      "logps/chosen": -513.2483520507812,
-      "logps/rejected": -773.5486450195312,
-      "loss": 0.3106,
-      "rewards/accuracies": 0.9125000238418579,
-      "rewards/chosen": -2.409447431564331,
-      "rewards/margins": 2.4787163734436035,
-      "rewards/rejected": -4.8881635665893555,
-      "step": 790
-    },
-    {
-      "epoch": 1.67,
-      "learning_rate": 7.740495722810269e-08,
-      "logits/chosen": -0.22111931443214417,
-      "logits/rejected": -0.16927292943000793,
-      "logps/chosen": -594.322998046875,
-      "logps/rejected": -767.4426879882812,
-      "loss": 0.3398,
-      "rewards/accuracies": 0.84375,
-      "rewards/chosen": -2.4693970680236816,
-      "rewards/margins": 2.0875487327575684,
-      "rewards/rejected": -4.55694580078125,
-      "step": 800
-    },
-    {
-      "epoch": 1.67,
-      "eval_logits/chosen": 0.0749908834695816,
-      "eval_logits/rejected": 0.0380852147936821,
-      "eval_logps/chosen": -554.5546264648438,
-      "eval_logps/rejected": -712.9505615234375,
-      "eval_loss": 0.5297122001647949,
-      "eval_rewards/accuracies": 0.7734375,
-      "eval_rewards/chosen": -2.509699821472168,
-      "eval_rewards/margins": 1.5035548210144043,
-      "eval_rewards/rejected": -4.013254642486572,
-      "eval_runtime": 65.3757,
-      "eval_samples_per_second": 30.592,
-      "eval_steps_per_second": 0.489,
-      "step": 800
-    },
-    {
-      "epoch": 1.7,
-      "learning_rate": 6.790558119157597e-08,
-      "logits/chosen": -0.03027234971523285,
-      "logits/rejected": -0.026968132704496384,
-      "logps/chosen": -558.7872924804688,
-      "logps/rejected": -744.04736328125,
-      "loss": 0.3134,
-      "rewards/accuracies": 0.8812500238418579,
-      "rewards/chosen": -2.323040246963501,
-      "rewards/margins": 2.1539111137390137,
-      "rewards/rejected": -4.476951599121094,
-      "step": 810
-    },
-    {
-      "epoch": 1.72,
-      "learning_rate": 5.898544083397e-08,
-      "logits/chosen": -0.06089891865849495,
-      "logits/rejected": -0.11520856618881226,
-      "logps/chosen": -516.7492065429688,
-      "logps/rejected": -698.2511596679688,
-      "loss": 0.3131,
-      "rewards/accuracies": 0.862500011920929,
-      "rewards/chosen": -2.2953174114227295,
-      "rewards/margins": 2.0010485649108887,
-      "rewards/rejected": -4.296365737915039,
-      "step": 820
-    },
-    {
-      "epoch": 1.74,
-      "learning_rate": 5.065649387408705e-08,
-      "logits/chosen": -0.026049736887216568,
-      "logits/rejected": -0.11464808881282806,
-      "logps/chosen": -556.2135620117188,
-      "logps/rejected": -718.17578125,
-      "loss": 0.3145,
-      "rewards/accuracies": 0.8687499761581421,
-      "rewards/chosen": -2.405728816986084,
-      "rewards/margins": 1.908630132675171,
-      "rewards/rejected": -4.314358711242676,
-      "step": 830
-    },
-    {
-      "epoch": 1.76,
-      "learning_rate": 4.292990551804171e-08,
-      "logits/chosen": -0.12712730467319489,
-      "logits/rejected": -0.10675887763500214,
-      "logps/chosen": -529.3737182617188,
-      "logps/rejected": -723.229248046875,
-      "loss": 0.327,
-      "rewards/accuracies": 0.8500000238418579,
-      "rewards/chosen": -2.146822929382324,
-      "rewards/margins": 2.177410125732422,
-      "rewards/rejected": -4.324233055114746,
-      "step": 840
-    },
-    {
-      "epoch": 1.78,
-      "learning_rate": 3.581603349196371e-08,
-      "logits/chosen": 0.01494809053838253,
-      "logits/rejected": -0.08690011501312256,
-      "logps/chosen": -540.9631958007812,
-      "logps/rejected": -735.3316650390625,
-      "loss": 0.3262,
-      "rewards/accuracies": 0.918749988079071,
-      "rewards/chosen": -2.145940065383911,
-      "rewards/margins": 2.0986034870147705,
-      "rewards/rejected": -4.24454402923584,
-      "step": 850
-    },
-    {
-      "epoch": 1.8,
-      "learning_rate": 2.9324414157151367e-08,
-      "logits/chosen": -0.10003119707107544,
-      "logits/rejected": -0.13639459013938904,
-      "logps/chosen": -504.34552001953125,
-      "logps/rejected": -681.4578247070312,
-      "loss": 0.3261,
-      "rewards/accuracies": 0.8812500238418579,
-      "rewards/chosen": -2.0967519283294678,
-      "rewards/margins": 1.8127696514129639,
-      "rewards/rejected": -3.9095215797424316,
-      "step": 860
-    },
-    {
-      "epoch": 1.82,
-      "learning_rate": 2.3463749726290284e-08,
-      "logits/chosen": -0.005720620043575764,
-      "logits/rejected": -0.09044505655765533,
-      "logps/chosen": -550.0,
-      "logps/rejected": -727.3507690429688,
-      "loss": 0.3221,
-      "rewards/accuracies": 0.862500011920929,
-      "rewards/chosen": -2.1374850273132324,
-      "rewards/margins": 2.060269355773926,
-      "rewards/rejected": -4.197754859924316,
-      "step": 870
-    },
-    {
-      "epoch": 1.84,
-      "learning_rate": 1.824189659787284e-08,
-      "logits/chosen": -0.07426755130290985,
-      "logits/rejected": -0.09251859039068222,
-      "logps/chosen": -551.6723022460938,
-      "logps/rejected": -753.9895629882812,
-      "loss": 0.3162,
-      "rewards/accuracies": 0.8812500238418579,
-      "rewards/chosen": -2.1559104919433594,
-      "rewards/margins": 2.1075100898742676,
-      "rewards/rejected": -4.263420581817627,
-      "step": 880
-    },
-    {
-      "epoch": 1.86,
-      "learning_rate": 1.3665854824458035e-08,
-      "logits/chosen": -0.05439913272857666,
-      "logits/rejected": -0.06511974334716797,
-      "logps/chosen": -540.4793090820312,
-      "logps/rejected": -690.4094848632812,
-      "loss": 0.3265,
-      "rewards/accuracies": 0.8500000238418579,
-      "rewards/chosen": -2.1754844188690186,
-      "rewards/margins": 1.7633205652236938,
-      "rewards/rejected": -3.938805103302002,
-      "step": 890
-    },
-    {
-      "epoch": 1.88,
-      "learning_rate": 9.741758728888217e-09,
-      "logits/chosen": -0.06001782417297363,
-      "logits/rejected": -0.016363339498639107,
-      "logps/chosen": -546.7725830078125,
-      "logps/rejected": -690.0701293945312,
-      "loss": 0.307,
-      "rewards/accuracies": 0.8812500238418579,
-      "rewards/chosen": -2.229912281036377,
-      "rewards/margins": 1.9347511529922485,
-      "rewards/rejected": -4.164663314819336,
-      "step": 900
-    },
-    {
-      "epoch": 1.88,
-      "eval_logits/chosen": 0.06628188490867615,
-      "eval_logits/rejected": 0.02886618673801422,
-      "eval_logps/chosen": -549.4910278320312,
-      "eval_logps/rejected": -703.8400268554688,
-      "eval_loss": 0.5260834097862244,
-      "eval_rewards/accuracies": 0.77734375,
-      "eval_rewards/chosen": -2.4590635299682617,
-      "eval_rewards/margins": 1.4630858898162842,
-      "eval_rewards/rejected": -3.922149658203125,
-      "eval_runtime": 65.2404,
-      "eval_samples_per_second": 30.656,
-      "eval_steps_per_second": 0.49,
-      "step": 900
-    },
-    {
-      "epoch": 1.9,
-      "learning_rate": 6.474868681043577e-09,
-      "logits/chosen": -0.0006331875920295715,
-      "logits/rejected": 0.06078674644231796,
-      "logps/chosen": -526.890869140625,
-      "logps/rejected": -696.57080078125,
-      "loss": 0.3133,
       "rewards/accuracies": 0.84375,
-      "rewards/chosen": -2.2198734283447266,
-      "rewards/margins": 1.9986340999603271,
-      "rewards/rejected": -4.218507289886475,
-      "step": 910
-    },
-    {
-      "epoch": 1.93,
-      "learning_rate": 3.869564046156459e-09,
-      "logits/chosen": -0.17484715580940247,
-      "logits/rejected": -0.2491791695356369,
-      "logps/chosen": -560.9608154296875,
-      "logps/rejected": -718.45703125,
-      "loss": 0.3088,
-      "rewards/accuracies": 0.9125000238418579,
-      "rewards/chosen": -2.2493414878845215,
-      "rewards/margins": 2.0530190467834473,
-      "rewards/rejected": -4.3023600578308105,
-      "step": 920
-    },
-    {
-      "epoch": 1.95,
-      "learning_rate": 1.929337314139412e-09,
-      "logits/chosen": -0.0817941427230835,
-      "logits/rejected": -0.2032664567232132,
-      "logps/chosen": -562.2503662109375,
-      "logps/rejected": -765.2064208984375,
-      "loss": 0.3264,
-      "rewards/accuracies": 0.8812500238418579,
-      "rewards/chosen": -2.2634310722351074,
-      "rewards/margins": 2.2729239463806152,
-      "rewards/rejected": -4.536355018615723,
-      "step": 930
-    },
-    {
-      "epoch": 1.97,
-      "learning_rate": 6.567894177967325e-10,
-      "logits/chosen": -0.061837755143642426,
-      "logits/rejected": -0.09424273669719696,
-      "logps/chosen": -570.0264892578125,
-      "logps/rejected": -782.4632568359375,
-      "loss": 0.3155,
-      "rewards/accuracies": 0.8999999761581421,
-      "rewards/chosen": -2.259819984436035,
-      "rewards/margins": 2.1893467903137207,
-      "rewards/rejected": -4.449166297912598,
-      "step": 940
-    },
-    {
-      "epoch": 1.99,
-      "learning_rate": 5.3626246194704575e-11,
-      "logits/chosen": 0.019718965515494347,
-      "logits/rejected": -0.05742845684289932,
-      "logps/chosen": -491.19879150390625,
-      "logps/rejected": -698.8577880859375,
-      "loss": 0.3247,
-      "rewards/accuracies": 0.8687499761581421,
-      "rewards/chosen": -2.148132801055908,
-      "rewards/margins": 2.056044816970825,
-      "rewards/rejected": -4.204176902770996,
-      "step": 950
     },
     {
       "epoch": 2.0,
-      "step": 954,
       "total_flos": 0.0,
-      "train_loss": 0.44779854000739333,
-      "train_runtime": 8782.9823,
-      "train_samples_per_second": 13.921,
-      "train_steps_per_second": 0.109
     }
   ],
   "logging_steps": 10,
-  "max_steps": 954,
   "num_train_epochs": 2,
-  "save_steps": 1000,
   "total_flos": 0.0,
   "trial_name": null,
   "trial_params": null

 {
   "best_metric": null,
   "best_model_checkpoint": null,
+  "epoch": 1.9950825430277486,
+  "eval_steps": 10000,
+  "global_step": 710,
   "is_hyper_param_search": false,
   "is_local_process_zero": true,
   "is_world_process_zero": true,
   "log_history": [
     {
+      "epoch": 0.03,
+      "learning_rate": 1.4084507042253522e-07,
+      "logits/chosen": 0.0005512732896022499,
+      "logits/rejected": 0.0318959541618824,
+      "logps/chosen": -259.0977478027344,
+      "logps/rejected": -201.09397888183594,
+      "loss": 0.6931,
+      "rewards/accuracies": 0.3687500059604645,
+      "rewards/chosen": 0.00018751960305962712,
+      "rewards/margins": 0.0003801170678343624,
+      "rewards/rejected": -0.00019259755208622664,
       "step": 10
     },
     {
+      "epoch": 0.06,
+      "learning_rate": 2.8169014084507043e-07,
+      "logits/chosen": -0.07602560520172119,
+      "logits/rejected": -0.010130566544830799,
+      "logps/chosen": -326.53594970703125,
+      "logps/rejected": -200.25064086914062,
+      "loss": 0.6915,
+      "rewards/accuracies": 0.5249999761581421,
+      "rewards/chosen": 0.0007010429399088025,
+      "rewards/margins": 0.002572448458522558,
+      "rewards/rejected": -0.0018714054021984339,
       "step": 20
     },
     {
+      "epoch": 0.08,
+      "learning_rate": 4.225352112676056e-07,
+      "logits/chosen": -0.033220209181308746,
+      "logits/rejected": -0.0025589261204004288,
+      "logps/chosen": -334.3651123046875,
+      "logps/rejected": -227.4903564453125,
+      "loss": 0.6856,
+      "rewards/accuracies": 0.59375,
+      "rewards/chosen": 0.0007741742883808911,
+      "rewards/margins": 0.014154395088553429,
+      "rewards/rejected": -0.013380222022533417,
       "step": 30
     },
     {
+      "epoch": 0.11,
+      "learning_rate": 5.633802816901409e-07,
+      "logits/chosen": -0.004470109473913908,
+      "logits/rejected": 0.04596555978059769,
+      "logps/chosen": -307.7451171875,
+      "logps/rejected": -191.01101684570312,
+      "loss": 0.6758,
+      "rewards/accuracies": 0.550000011920929,
+      "rewards/chosen": 0.0008303679642267525,
+      "rewards/margins": 0.05119556933641434,
+      "rewards/rejected": -0.05036519095301628,
       "step": 40
     },
     {
+      "epoch": 0.14,
+      "learning_rate": 7.04225352112676e-07,
+      "logits/chosen": 0.06877349317073822,
+      "logits/rejected": 0.1612742692232132,
+      "logps/chosen": -343.5093994140625,
+      "logps/rejected": -217.77890014648438,
+      "loss": 0.663,
+      "rewards/accuracies": 0.574999988079071,
+      "rewards/chosen": -0.016251932829618454,
+      "rewards/margins": 0.08627420663833618,
+      "rewards/rejected": -0.10252614319324493,
       "step": 50
     },
     {
+      "epoch": 0.17,
+      "learning_rate": 8.450704225352112e-07,
+      "logits/chosen": 0.004598864819854498,
+      "logits/rejected": 0.026440221816301346,
+      "logps/chosen": -311.1943359375,
+      "logps/rejected": -236.7565155029297,
+      "loss": 0.6591,
+      "rewards/accuracies": 0.59375,
+      "rewards/chosen": -0.07662348449230194,
+      "rewards/margins": 0.09086361527442932,
+      "rewards/rejected": -0.16748711466789246,
       "step": 60
     },
     {
+      "epoch": 0.2,
+      "learning_rate": 9.859154929577465e-07,
+      "logits/chosen": -0.02923472784459591,
+      "logits/rejected": 0.041188597679138184,
+      "logps/chosen": -317.4526672363281,
+      "logps/rejected": -232.837646484375,
+      "loss": 0.6632,
+      "rewards/accuracies": 0.574999988079071,
+      "rewards/chosen": -0.04823857173323631,
+      "rewards/margins": 0.1199621707201004,
+      "rewards/rejected": -0.16820073127746582,
       "step": 70
     },
     {
+      "epoch": 0.22,
+      "learning_rate": 9.995106132599868e-07,
+      "logits/chosen": -0.010874450206756592,
+      "logits/rejected": 0.04368402436375618,
+      "logps/chosen": -339.08502197265625,
+      "logps/rejected": -266.75567626953125,
+      "loss": 0.654,
+      "rewards/accuracies": 0.6000000238418579,
+      "rewards/chosen": -0.007924405857920647,
+      "rewards/margins": 0.12536732852458954,
+      "rewards/rejected": -0.13329175114631653,
       "step": 80
     },
     {
+      "epoch": 0.25,
+      "learning_rate": 9.978201358980644e-07,
+      "logits/chosen": 0.05211018770933151,
+      "logits/rejected": 0.09429686516523361,
+      "logps/chosen": -314.8374938964844,
+      "logps/rejected": -219.6160888671875,
+      "loss": 0.6486,
+      "rewards/accuracies": 0.5874999761581421,
+      "rewards/chosen": 0.05235903337597847,
+      "rewards/margins": 0.1537412703037262,
+      "rewards/rejected": -0.10138221830129623,
       "step": 90
     },
     {
+      "epoch": 0.28,
+      "learning_rate": 9.949266103908894e-07,
+      "logits/chosen": -0.01588534563779831,
+      "logits/rejected": 0.06498686969280243,
+      "logps/chosen": -297.93145751953125,
+      "logps/rejected": -223.25234985351562,
+      "loss": 0.6594,
+      "rewards/accuracies": 0.5562499761581421,
+      "rewards/chosen": -0.013728643767535686,
+      "rewards/margins": 0.11913089454174042,
+      "rewards/rejected": -0.13285955786705017,
       "step": 100
     },
     {
+      "epoch": 0.31,
+      "learning_rate": 9.908370293252287e-07,
+      "logits/chosen": -0.09032365679740906,
+      "logits/rejected": -0.06336920708417892,
+      "logps/chosen": -279.67047119140625,
+      "logps/rejected": -207.52157592773438,
+      "loss": 0.6463,
+      "rewards/accuracies": 0.6312500238418579,
+      "rewards/chosen": 0.00655151903629303,
+      "rewards/margins": 0.1867600381374359,
+      "rewards/rejected": -0.1802085041999817,
       "step": 110
     },
     {
+      "epoch": 0.34,
+      "learning_rate": 9.855612757141654e-07,
+      "logits/chosen": -0.06192486360669136,
+      "logits/rejected": 0.024322770535945892,
+      "logps/chosen": -364.93701171875,
+      "logps/rejected": -231.4254608154297,
+      "loss": 0.6401,
+      "rewards/accuracies": 0.637499988079071,
+      "rewards/chosen": 0.04051176831126213,
+      "rewards/margins": 0.2519943118095398,
+      "rewards/rejected": -0.21148256957530975,
       "step": 120
     },
     {
+      "epoch": 0.37,
+      "learning_rate": 9.791120991134902e-07,
+      "logits/chosen": -0.11360426992177963,
+      "logits/rejected": -0.06546725332736969,
+      "logps/chosen": -313.6669006347656,
+      "logps/rejected": -237.9274139404297,
+      "loss": 0.6368,
+      "rewards/accuracies": 0.6000000238418579,
+      "rewards/chosen": -0.0725397914648056,
+      "rewards/margins": 0.16836170852184296,
+      "rewards/rejected": -0.24090149998664856,
       "step": 130
     },
     {
+      "epoch": 0.39,
+      "learning_rate": 9.715050848107168e-07,
+      "logits/chosen": -0.04263811185956001,
+      "logits/rejected": 0.01193575281649828,
+      "logps/chosen": -330.8939514160156,
+      "logps/rejected": -221.0767822265625,
+      "loss": 0.6248,
+      "rewards/accuracies": 0.6812499761581421,
+      "rewards/chosen": -0.009152719751000404,
+      "rewards/margins": 0.2825842797756195,
+      "rewards/rejected": -0.29173702001571655,
       "step": 140
     },
     {
+      "epoch": 0.42,
+      "learning_rate": 9.627586161611731e-07,
+      "logits/chosen": -0.13610024750232697,
+      "logits/rejected": -0.060576580464839935,
+      "logps/chosen": -310.90887451171875,
+      "logps/rejected": -219.16915893554688,
+      "loss": 0.6424,
+      "rewards/accuracies": 0.65625,
+      "rewards/chosen": -0.08018601685762405,
+      "rewards/margins": 0.2522713243961334,
+      "rewards/rejected": -0.3324573338031769,
       "step": 150
     },
     {
+      "epoch": 0.45,
+      "learning_rate": 9.528938301621955e-07,
+      "logits/chosen": -0.17108501493930817,
+      "logits/rejected": -0.06673868000507355,
+      "logps/chosen": -350.9739685058594,
+      "logps/rejected": -277.810791015625,
+      "loss": 0.6291,
+      "rewards/accuracies": 0.606249988079071,
+      "rewards/chosen": -0.09369239956140518,
+      "rewards/margins": 0.18727482855319977,
+      "rewards/rejected": -0.28096717596054077,
       "step": 160
     },
     {
+      "epoch": 0.48,
+      "learning_rate": 9.419345663727804e-07,
+      "logits/chosen": -0.0947573184967041,
+      "logits/rejected": -0.03230197727680206,
+      "logps/chosen": -347.60308837890625,
+      "logps/rejected": -229.42044067382812,
+      "loss": 0.6348,
+      "rewards/accuracies": 0.668749988079071,
+      "rewards/chosen": -0.09307502955198288,
+      "rewards/margins": 0.28278011083602905,
+      "rewards/rejected": -0.37585514783859253,
       "step": 170
     },
     {
+      "epoch": 0.51,
+      "learning_rate": 9.299073093021404e-07,
+      "logits/chosen": -0.08774133771657944,
+      "logits/rejected": -0.022734731435775757,
+      "logps/chosen": -342.45574951171875,
+      "logps/rejected": -227.8816375732422,
+      "loss": 0.6219,
+      "rewards/accuracies": 0.6000000238418579,
+      "rewards/chosen": -0.10549932718276978,
+      "rewards/margins": 0.2989550232887268,
+      "rewards/rejected": -0.40445438027381897,
       "step": 180
     },
     {
+      "epoch": 0.53,
+      "learning_rate": 9.168411244063861e-07,
+      "logits/chosen": -0.1278824657201767,
+      "logits/rejected": -0.05893976613879204,
+      "logps/chosen": -350.15997314453125,
+      "logps/rejected": -241.92715454101562,
+      "loss": 0.6348,
+      "rewards/accuracies": 0.6937500238418579,
+      "rewards/chosen": -0.10131518542766571,
+      "rewards/margins": 0.3444809019565582,
+      "rewards/rejected": -0.44579607248306274,
       "step": 190
     },
     {
+      "epoch": 0.56,
+      "learning_rate": 9.02767587848013e-07,
+      "logits/chosen": -0.09527161717414856,
+      "logits/rejected": -0.06935124844312668,
+      "logps/chosen": -298.73040771484375,
+      "logps/rejected": -257.57305908203125,
+      "loss": 0.6272,
+      "rewards/accuracies": 0.5562499761581421,
+      "rewards/chosen": -0.16879644989967346,
+      "rewards/margins": 0.16612185537815094,
+      "rewards/rejected": -0.3349182903766632,
       "step": 200
     },
     {
+      "epoch": 0.59,
+      "learning_rate": 8.877207101879301e-07,
+      "logits/chosen": -0.13824552297592163,
+      "logits/rejected": -0.0595315583050251,
+      "logps/chosen": -340.5955810546875,
+      "logps/rejected": -261.76763916015625,
+      "loss": 0.6222,
+      "rewards/accuracies": 0.6625000238418579,
+      "rewards/chosen": -0.16845914721488953,
+      "rewards/margins": 0.2435651272535324,
+      "rewards/rejected": -0.4120243191719055,
       "step": 210
     },
     {
+      "epoch": 0.62,
+      "learning_rate": 8.717368541944452e-07,
+      "logits/chosen": -0.07102419435977936,
+      "logits/rejected": 0.02669895812869072,
+      "logps/chosen": -359.16259765625,
+      "logps/rejected": -266.5383605957031,
+      "loss": 0.6314,
+      "rewards/accuracies": 0.7124999761581421,
+      "rewards/chosen": -0.24132736027240753,
+      "rewards/margins": 0.3515443503856659,
+      "rewards/rejected": -0.5928717255592346,
       "step": 220
     },
     {
+      "epoch": 0.65,
+      "learning_rate": 8.54854646967831e-07,
+      "logits/chosen": -0.1354692280292511,
+      "logits/rejected": -0.0510697066783905,
+      "logps/chosen": -381.85028076171875,
+      "logps/rejected": -264.0498046875,
+      "loss": 0.6208,
+      "rewards/accuracies": 0.643750011920929,
+      "rewards/chosen": -0.22062142193317413,
+      "rewards/margins": 0.3123762011528015,
+      "rewards/rejected": -0.5329976081848145,
       "step": 230
     },
     {
+      "epoch": 0.67,
+      "learning_rate": 8.371148865928318e-07,
+      "logits/chosen": -0.10147638618946075,
+      "logits/rejected": -0.049172915518283844,
+      "logps/chosen": -355.6955261230469,
+      "logps/rejected": -262.17279052734375,
+      "loss": 0.6157,
+      "rewards/accuracies": 0.637499988079071,
+      "rewards/chosen": -0.24571116268634796,
+      "rewards/margins": 0.259532630443573,
+      "rewards/rejected": -0.5052437782287598,
       "step": 240
     },
     {
+      "epoch": 0.7,
+      "learning_rate": 8.185604435447001e-07,
+      "logits/chosen": -0.08465194702148438,
+      "logits/rejected": -0.07315615564584732,
+      "logps/chosen": -322.062744140625,
+      "logps/rejected": -276.74908447265625,
+      "loss": 0.6184,
+      "rewards/accuracies": 0.6499999761581421,
+      "rewards/chosen": -0.3060557246208191,
+      "rewards/margins": 0.24059104919433594,
+      "rewards/rejected": -0.546646773815155,
       "step": 250
     },
     {
+      "epoch": 0.73,
+      "learning_rate": 7.992361570870287e-07,
+      "logits/chosen": -0.09758251905441284,
+      "logits/rejected": -0.011859369464218616,
+      "logps/chosen": -369.69940185546875,
+      "logps/rejected": -246.3345489501953,
+      "loss": 0.62,
+      "rewards/accuracies": 0.6875,
+      "rewards/chosen": -0.248373344540596,
+      "rewards/margins": 0.29594987630844116,
+      "rewards/rejected": -0.5443232655525208,
       "step": 260
     },
     {
+      "epoch": 0.76,
+      "learning_rate": 7.791887269117441e-07,
+      "logits/chosen": -0.10910804569721222,
+      "logits/rejected": -0.061185322701931,
+      "logps/chosen": -332.3966064453125,
+      "logps/rejected": -267.03057861328125,
+      "loss": 0.6309,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": -0.3056594729423523,
+      "rewards/margins": 0.2236270159482956,
+      "rewards/rejected": -0.5292865037918091,
       "step": 270
     },
     {
+      "epoch": 0.79,
+      "learning_rate": 7.584666002831294e-07,
+      "logits/chosen": -0.15460695326328278,
+      "logits/rejected": -0.11919967085123062,
+      "logps/chosen": -383.691650390625,
+      "logps/rejected": -281.88201904296875,
+      "loss": 0.6118,
+      "rewards/accuracies": 0.643750011920929,
+      "rewards/chosen": -0.25431981682777405,
+      "rewards/margins": 0.27084994316101074,
+      "rewards/rejected": -0.5251697301864624,
       "step": 280
     },
     {
+      "epoch": 0.81,
+      "learning_rate": 7.37119854958609e-07,
+      "logits/chosen": -0.09878663718700409,
+      "logits/rejected": -0.054516829550266266,
+      "logps/chosen": -367.6859130859375,
+      "logps/rejected": -266.76129150390625,
+      "loss": 0.6145,
+      "rewards/accuracies": 0.625,
+      "rewards/chosen": -0.2599807381629944,
+      "rewards/margins": 0.22561149299144745,
+      "rewards/rejected": -0.4855922758579254,
       "step": 290
     },
     {
+      "epoch": 0.84,
+      "learning_rate": 7.152000781692285e-07,
+      "logits/chosen": -0.06776697933673859,
+      "logits/rejected": -0.08343350887298584,
+      "logps/chosen": -330.73406982421875,
+      "logps/rejected": -282.1576232910156,
+      "loss": 0.6194,
+      "rewards/accuracies": 0.6312500238418579,
+      "rewards/chosen": -0.2815566062927246,
+      "rewards/margins": 0.25665608048439026,
+      "rewards/rejected": -0.5382126569747925,
       "step": 300
     },
     {
+      "epoch": 0.87,
+      "learning_rate": 6.927602419522946e-07,
+      "logits/chosen": -0.19854819774627686,
+      "logits/rejected": -0.15916487574577332,
+      "logps/chosen": -394.5406799316406,
+      "logps/rejected": -283.3205871582031,
+      "loss": 0.6056,
+      "rewards/accuracies": 0.6875,
+      "rewards/chosen": -0.18460381031036377,
+      "rewards/margins": 0.36649608612060547,
+      "rewards/rejected": -0.5510998964309692,
       "step": 310
     },
     {
+      "epoch": 0.9,
+      "learning_rate": 6.698545751374463e-07,
+      "logits/chosen": -0.1863429993391037,
+      "logits/rejected": -0.1495630294084549,
+      "logps/chosen": -378.22869873046875,
+      "logps/rejected": -304.0544128417969,
+      "loss": 0.6041,
+      "rewards/accuracies": 0.6187499761581421,
+      "rewards/chosen": -0.3425668478012085,
+      "rewards/margins": 0.2656816840171814,
+      "rewards/rejected": -0.6082485318183899,
       "step": 320
     },
     {
+      "epoch": 0.93,
+      "learning_rate": 6.465384322955224e-07,
+      "logits/chosen": -0.23867850005626678,
+      "logits/rejected": -0.15165486931800842,
+      "logps/chosen": -390.47216796875,
+      "logps/rejected": -301.1231994628906,
+      "loss": 0.6279,
+      "rewards/accuracies": 0.6499999761581421,
+      "rewards/chosen": -0.2970598340034485,
+      "rewards/margins": 0.28874671459198,
+      "rewards/rejected": -0.5858064889907837,
       "step": 330
     },
     {
+      "epoch": 0.96,
+      "learning_rate": 6.228681599669248e-07,
+      "logits/chosen": -0.25492700934410095,
+      "logits/rejected": -0.17601162195205688,
+      "logps/chosen": -340.286376953125,
+      "logps/rejected": -277.90606689453125,
+      "loss": 0.6209,
+      "rewards/accuracies": 0.6812499761581421,
+      "rewards/chosen": -0.287325382232666,
+      "rewards/margins": 0.2902410924434662,
+      "rewards/rejected": -0.5775664448738098,
       "step": 340
     },
     {
+      "epoch": 0.98,
+      "learning_rate": 5.989009604927586e-07,
+      "logits/chosen": -0.18320497870445251,
+      "logits/rejected": -0.09960085898637772,
+      "logps/chosen": -340.33306884765625,
+      "logps/rejected": -269.10101318359375,
+      "loss": 0.6106,
+      "rewards/accuracies": 0.6000000238418579,
+      "rewards/chosen": -0.3515200912952423,
+      "rewards/margins": 0.23425371944904327,
+      "rewards/rejected": -0.5857738256454468,
       "step": 350
     },
     {
+      "epoch": 1.01,
+      "learning_rate": 5.74694753777815e-07,
+      "logits/chosen": -0.17563189566135406,
+      "logits/rejected": -0.1343529373407364,
+      "logps/chosen": -359.41845703125,
+      "logps/rejected": -293.6738586425781,
+      "loss": 0.5647,
+      "rewards/accuracies": 0.793749988079071,
+      "rewards/chosen": -0.2671016454696655,
+      "rewards/margins": 0.4457646310329437,
+      "rewards/rejected": -0.7128663063049316,
       "step": 360
     },
     {
+      "epoch": 1.04,
+      "learning_rate": 5.503080373194666e-07,
+      "logits/chosen": -0.19107575714588165,
+      "logits/rejected": -0.1297108232975006,
+      "logps/chosen": -330.1180725097656,
+      "logps/rejected": -284.3822326660156,
+      "loss": 0.4933,
+      "rewards/accuracies": 0.856249988079071,
+      "rewards/chosen": -0.307260125875473,
+      "rewards/margins": 0.6193431615829468,
+      "rewards/rejected": -0.9266033172607422,
       "step": 370
     },
     {
+      "epoch": 1.07,
+      "learning_rate": 5.257997448407366e-07,
+      "logits/chosen": -0.25203296542167664,
+      "logits/rejected": -0.15098513662815094,
+      "logps/chosen": -356.20220947265625,
+      "logps/rejected": -304.3904113769531,
+      "loss": 0.4944,
+      "rewards/accuracies": 0.8125,
+      "rewards/chosen": -0.44451460242271423,
+      "rewards/margins": 0.5491349101066589,
+      "rewards/rejected": -0.9936494827270508,
       "step": 380
     },
     {
+      "epoch": 1.1,
+      "learning_rate": 5.012291038691665e-07,
+      "logits/chosen": -0.11390683799982071,
+      "logits/rejected": 0.04292234033346176,
+      "logps/chosen": -406.4115295410156,
+      "logps/rejected": -312.5035705566406,
+      "loss": 0.4786,
+      "rewards/accuracies": 0.875,
+      "rewards/chosen": -0.5159555673599243,
+      "rewards/margins": 0.7496730089187622,
+      "rewards/rejected": -1.2656285762786865,
       "step": 390
     },
     {
+      "epoch": 1.12,
+      "learning_rate": 4.7665549260567063e-07,
+      "logits/chosen": -0.1984197050333023,
+      "logits/rejected": -0.1335490345954895,
+      "logps/chosen": -358.4623107910156,
+      "logps/rejected": -365.38812255859375,
+      "loss": 0.4764,
+      "rewards/accuracies": 0.824999988079071,
+      "rewards/chosen": -0.8547590970993042,
+      "rewards/margins": 0.6313014626502991,
+      "rewards/rejected": -1.4860605001449585,
       "step": 400
     },
     {
+      "epoch": 1.15,
+      "learning_rate": 4.521382964292663e-07,
+      "logits/chosen": -0.10411302745342255,
+      "logits/rejected": -0.04661521688103676,
+      "logps/chosen": -423.57183837890625,
+      "logps/rejected": -370.1197204589844,
+      "loss": 0.4652,
+      "rewards/accuracies": 0.8374999761581421,
+      "rewards/chosen": -0.7738268971443176,
+      "rewards/margins": 0.8890780210494995,
+      "rewards/rejected": -1.662904977798462,
       "step": 410
     },
     {
+      "epoch": 1.18,
+      "learning_rate": 4.277367643844574e-07,
+      "logits/chosen": -0.15602782368659973,
+      "logits/rejected": -0.07274512201547623,
+      "logps/chosen": -402.03546142578125,
+      "logps/rejected": -334.91290283203125,
+      "loss": 0.4601,
+      "rewards/accuracies": 0.84375,
+      "rewards/chosen": -0.7279518842697144,
+      "rewards/margins": 0.8058152198791504,
+      "rewards/rejected": -1.5337669849395752,
       "step": 420
     },
     {
+      "epoch": 1.21,
+      "learning_rate": 4.035098659980891e-07,
+      "logits/chosen": -0.06685085594654083,
+      "logits/rejected": 0.018254879862070084,
+      "logps/chosen": -423.72540283203125,
+      "logps/rejected": -415.8779296875,
+      "loss": 0.465,
+      "rewards/accuracies": 0.8062499761581421,
+      "rewards/chosen": -1.001319169998169,
+      "rewards/margins": 0.7759243845939636,
+      "rewards/rejected": -1.7772436141967773,
       "step": 430
     },
     {
+      "epoch": 1.24,
+      "learning_rate": 3.795161487716928e-07,
+      "logits/chosen": -0.09074730426073074,
+      "logits/rejected": 0.07650933414697647,
+      "logps/chosen": -479.47576904296875,
+      "logps/rejected": -421.45941162109375,
+      "loss": 0.4532,
+      "rewards/accuracies": 0.8812500238418579,
+      "rewards/chosen": -1.1542574167251587,
+      "rewards/margins": 0.8317985534667969,
+      "rewards/rejected": -1.9860557317733765,
       "step": 440
     },
     {
+      "epoch": 1.26,
+      "learning_rate": 3.5581359669371223e-07,
+      "logits/chosen": -0.022224614396691322,
+      "logits/rejected": 0.0994531437754631,
+      "logps/chosen": -337.89447021484375,
+      "logps/rejected": -357.2190856933594,
+      "loss": 0.4517,
+      "rewards/accuracies": 0.8500000238418579,
+      "rewards/chosen": -1.0043950080871582,
+      "rewards/margins": 0.7450950741767883,
+      "rewards/rejected": -1.7494901418685913,
       "step": 450
     },
     {
+      "epoch": 1.29,
+      "learning_rate": 3.324594901135326e-07,
+      "logits/chosen": -0.004194366279989481,
+      "logits/rejected": 0.06137076020240784,
+      "logps/chosen": -390.3214111328125,
+      "logps/rejected": -367.9646911621094,
+      "loss": 0.4545,
+      "rewards/accuracies": 0.862500011920929,
+      "rewards/chosen": -1.0364924669265747,
+      "rewards/margins": 0.8088264465332031,
+      "rewards/rejected": -1.8453190326690674,
       "step": 460
     },
     {
+      "epoch": 1.32,
+      "learning_rate": 3.095102673159463e-07,
+      "logits/chosen": -0.037959642708301544,
+      "logits/rejected": 0.04749811813235283,
+      "logps/chosen": -429.95220947265625,
+      "logps/rejected": -400.9679260253906,
+      "loss": 0.4324,
+      "rewards/accuracies": 0.8500000238418579,
+      "rewards/chosen": -1.0624555349349976,
+      "rewards/margins": 0.8692095875740051,
+      "rewards/rejected": -1.931665062904358,
       "step": 470
     },
     {
+      "epoch": 1.35,
+      "learning_rate": 2.870213881305802e-07,
+      "logits/chosen": -0.035066571086645126,
+      "logits/rejected": 0.040720295161008835,
+      "logps/chosen": -474.01177978515625,
+      "logps/rejected": -409.0857849121094,
+      "loss": 0.4415,
+      "rewards/accuracies": 0.84375,
+      "rewards/chosen": -1.0656111240386963,
+      "rewards/margins": 0.9570503234863281,
+      "rewards/rejected": -2.0226616859436035,
       "step": 480
     },
     {
+      "epoch": 1.38,
+      "learning_rate": 2.6504719990588745e-07,
+      "logits/chosen": 0.06609617173671722,
+      "logits/rejected": 0.09753261506557465,
+      "logps/chosen": -489.17449951171875,
+      "logps/rejected": -452.04052734375,
+      "loss": 0.4317,
+      "rewards/accuracies": 0.856249988079071,
+      "rewards/chosen": -1.2529394626617432,
+      "rewards/margins": 1.057414174079895,
+      "rewards/rejected": -2.3103537559509277,
       "step": 490
     },
     {
+      "epoch": 1.4,
+      "learning_rate": 2.436408061715988e-07,
+      "logits/chosen": -0.0025597973726689816,
+      "logits/rejected": 0.0930376946926117,
+      "logps/chosen": -509.1451110839844,
+      "logps/rejected": -503.16009521484375,
+      "loss": 0.432,
+      "rewards/accuracies": 0.78125,
+      "rewards/chosen": -1.5313117504119873,
+      "rewards/margins": 0.9933918118476868,
+      "rewards/rejected": -2.5247039794921875,
       "step": 500
     },
     {
+      "epoch": 1.43,
+      "learning_rate": 2.22853938307025e-07,
+      "logits/chosen": 0.20588858425617218,
+      "logits/rejected": 0.3083997964859009,
+      "logps/chosen": -441.025146484375,
+      "logps/rejected": -412.0359802246094,
+      "loss": 0.4362,
+      "rewards/accuracies": 0.8125,
+      "rewards/chosen": -1.1732820272445679,
+      "rewards/margins": 0.8777497410774231,
+      "rewards/rejected": -2.0510315895080566,
       "step": 510
     },
     {
+      "epoch": 1.46,
+      "learning_rate": 2.0273683052534173e-07,
+      "logits/chosen": 0.13039740920066833,
+      "logits/rejected": 0.21461403369903564,
+      "logps/chosen": -450.8875427246094,
+      "logps/rejected": -447.06219482421875,
+      "loss": 0.4382,
+      "rewards/accuracies": 0.84375,
+      "rewards/chosen": -1.076249599456787,
+      "rewards/margins": 0.9202485084533691,
+      "rewards/rejected": -1.9964981079101562,
       "step": 520
     },
     {
+      "epoch": 1.49,
+      "learning_rate": 1.833380984759764e-07,
+      "logits/chosen": 0.06673362106084824,
+      "logits/rejected": 0.14447852969169617,
+      "logps/chosen": -474.4076232910156,
+      "logps/rejected": -404.3707275390625,
+      "loss": 0.4378,
+      "rewards/accuracies": 0.8500000238418579,
+      "rewards/chosen": -1.1933484077453613,
+      "rewards/margins": 0.8106036186218262,
+      "rewards/rejected": -2.0039520263671875,
       "step": 530
     },
     {
+      "epoch": 1.52,
+      "learning_rate": 1.6470462175846606e-07,
+      "logits/chosen": 0.07205445319414139,
+      "logits/rejected": 0.2100316733121872,
+      "logps/chosen": -496.0130920410156,
+      "logps/rejected": -440.20806884765625,
+      "loss": 0.4301,
+      "rewards/accuracies": 0.84375,
+      "rewards/chosen": -1.321128010749817,
+      "rewards/margins": 0.9277971982955933,
+      "rewards/rejected": -2.24892520904541,
       "step": 540
     },
     {
+      "epoch": 1.55,
+      "learning_rate": 1.468814306317092e-07,
+      "logits/chosen": 0.14164285361766815,
+      "logits/rejected": 0.1971181184053421,
+      "logps/chosen": -481.12646484375,
+      "logps/rejected": -430.11212158203125,
+      "loss": 0.4254,
+      "rewards/accuracies": 0.856249988079071,
+      "rewards/chosen": -1.3925974369049072,
+      "rewards/margins": 0.8229917287826538,
+      "rewards/rejected": -2.2155890464782715,
       "step": 550
     },
     {
+      "epoch": 1.57,
+      "learning_rate": 1.299115971923958e-07,
+      "logits/chosen": 0.2140401303768158,
+      "logits/rejected": 0.3077262341976166,
+      "logps/chosen": -458.56732177734375,
+      "logps/rejected": -426.39404296875,
+      "loss": 0.4265,
+      "rewards/accuracies": 0.793749988079071,
+      "rewards/chosen": -1.2237931489944458,
+      "rewards/margins": 0.8743373155593872,
+      "rewards/rejected": -2.098130464553833,
       "step": 560
     },
     {
+      "epoch": 1.6,
+      "learning_rate": 1.1383613128559305e-07,
+      "logits/chosen": 0.15547600388526917,
+      "logits/rejected": 0.22594401240348816,
+      "logps/chosen": -452.096435546875,
+      "logps/rejected": -461.7115783691406,
+      "loss": 0.4158,
+      "rewards/accuracies": 0.78125,
+      "rewards/chosen": -1.3260557651519775,
+      "rewards/margins": 0.8624703288078308,
+      "rewards/rejected": -2.188526153564453,
       "step": 570
     },
     {
+      "epoch": 1.63,
+      "learning_rate": 9.869388139903495e-08,
+      "logits/chosen": 0.16985264420509338,
+      "logits/rejected": 0.24287500977516174,
+      "logps/chosen": -453.4273986816406,
+      "logps/rejected": -435.1573791503906,
+      "loss": 0.4389,
+      "rewards/accuracies": 0.831250011920929,
+      "rewards/chosen": -1.3252050876617432,
+      "rewards/margins": 0.9360635876655579,
+      "rewards/rejected": -2.2612688541412354,
       "step": 580
     },
     {
+      "epoch": 1.66,
+      "learning_rate": 8.452144078061818e-08,
+      "logits/chosen": 0.2516225576400757,
+      "logits/rejected": 0.35612887144088745,
+      "logps/chosen": -433.05828857421875,
+      "logps/rejected": -372.24273681640625,
+      "loss": 0.4428,
+      "rewards/accuracies": 0.824999988079071,
+      "rewards/chosen": -1.2665939331054688,
+      "rewards/margins": 0.7784386873245239,
+      "rewards/rejected": -2.0450327396392822,
       "step": 590
     },
     {
+      "epoch": 1.69,
+      "learning_rate": 7.135305900598321e-08,
+      "logits/chosen": 0.2013220340013504,
+      "logits/rejected": 0.30089515447616577,
+      "logps/chosen": -451.4398498535156,
+      "logps/rejected": -415.129638671875,
+      "loss": 0.4265,
+      "rewards/accuracies": 0.824999988079071,
+      "rewards/chosen": -1.3577957153320312,
+      "rewards/margins": 0.7237957119941711,
+      "rewards/rejected": -2.0815916061401367,
       "step": 600
     },
     {
+      "epoch": 1.71,
+      "learning_rate": 5.9220559209888166e-08,
+      "logits/chosen": 0.14896592497825623,
+      "logits/rejected": 0.29967430233955383,
+      "logps/chosen": -466.57012939453125,
+      "logps/rejected": -392.4150390625,
+      "loss": 0.4393,
+      "rewards/accuracies": 0.824999988079071,
+      "rewards/chosen": -1.3139363527297974,
+      "rewards/margins": 0.8993911743164062,
+      "rewards/rejected": -2.213327646255493,
       "step": 610
     },
     {
+      "epoch": 1.74,
+      "learning_rate": 4.815326118139812e-08,
+      "logits/chosen": 0.002818810986354947,
+      "logits/rejected": 0.14390364289283752,
+      "logps/chosen": -485.42095947265625,
+      "logps/rejected": -445.1665954589844,
+      "loss": 0.4256,
       "rewards/accuracies": 0.862500011920929,
+      "rewards/chosen": -1.2553160190582275,
+      "rewards/margins": 1.0687906742095947,
+      "rewards/rejected": -2.3241066932678223,
       "step": 620
     },
     {
+      "epoch": 1.77,
+      "learning_rate": 3.81779105087407e-08,
+      "logits/chosen": 0.18004778027534485,
+      "logits/rejected": 0.2915050983428955,
+      "logps/chosen": -433.8690490722656,
+      "logps/rejected": -449.0361328125,
+      "loss": 0.4151,
+      "rewards/accuracies": 0.8125,
+      "rewards/chosen": -1.3270412683486938,
+      "rewards/margins": 0.9789830446243286,
+      "rewards/rejected": -2.3060243129730225,
       "step": 630
     },
     {
+      "epoch": 1.8,
+      "learning_rate": 2.9318613945057637e-08,
+      "logits/chosen": 0.19966045022010803,
+      "logits/rejected": 0.26041653752326965,
+      "logps/chosen": -475.998291015625,
+      "logps/rejected": -438.5953063964844,
+      "loss": 0.4212,
+      "rewards/accuracies": 0.824999988079071,
+      "rewards/chosen": -1.2703568935394287,
+      "rewards/margins": 0.9554103016853333,
+      "rewards/rejected": -2.225767135620117,
       "step": 640
     },
     {
+      "epoch": 1.83,
+      "learning_rate": 2.1596781151249523e-08,
+      "logits/chosen": 0.2097601592540741,
+      "logits/rejected": 0.332507848739624,
+      "logps/chosen": -467.12835693359375,
+      "logps/rejected": -460.8416442871094,
+      "loss": 0.4172,
+      "rewards/accuracies": 0.8500000238418579,
+      "rewards/chosen": -1.3552840948104858,
+      "rewards/margins": 1.1028287410736084,
+      "rewards/rejected": -2.4581127166748047,
       "step": 650
     },
     {
+      "epoch": 1.85,
+      "learning_rate": 1.5031072956701695e-08,
+      "logits/chosen": 0.16343393921852112,
+      "logits/rejected": 0.20303264260292053,
+      "logps/chosen": -433.28558349609375,
+      "logps/rejected": -458.15350341796875,
+      "loss": 0.4253,
+      "rewards/accuracies": 0.8062499761581421,
+      "rewards/chosen": -1.513602614402771,
+      "rewards/margins": 0.8022111058235168,
+      "rewards/rejected": -2.3158137798309326,
       "step": 660
     },
     {
+      "epoch": 1.88,
+      "learning_rate": 9.637356262923723e-09,
+      "logits/chosen": 0.08328579366207123,
+      "logits/rejected": 0.25923627614974976,
+      "logps/chosen": -468.628173828125,
+      "logps/rejected": -414.67608642578125,
+      "loss": 0.4355,
       "rewards/accuracies": 0.856249988079071,
+      "rewards/chosen": -1.3132652044296265,
+      "rewards/margins": 0.9038375616073608,
+      "rewards/rejected": -2.2171027660369873,
       "step": 670
     },
     {
+      "epoch": 1.91,
+      "learning_rate": 5.428665699084789e-09,
+      "logits/chosen": 0.18607434630393982,
+      "logits/rejected": 0.22493357956409454,
+      "logps/chosen": -407.1839294433594,
+      "logps/rejected": -398.12158203125,
+      "loss": 0.4253,
+      "rewards/accuracies": 0.7875000238418579,
+      "rewards/chosen": -1.400312900543213,
+      "rewards/margins": 0.7541720271110535,
+      "rewards/rejected": -2.154484987258911,
       "step": 680
     },
     {
+      "epoch": 1.94,
+      "learning_rate": 2.415172122110343e-09,
+      "logits/chosen": 0.029862558469176292,
+      "logits/rejected": 0.22492480278015137,
+      "logps/chosen": -504.3904724121094,
+      "logps/rejected": -438.556640625,
+      "loss": 0.4297,
+      "rewards/accuracies": 0.84375,
+      "rewards/chosen": -1.229479193687439,
+      "rewards/margins": 0.9579814076423645,
+      "rewards/rejected": -2.187460422515869,
       "step": 690
     },
     {
+      "epoch": 1.97,
+      "learning_rate": 6.041580374618327e-10,
+      "logits/chosen": 0.20579871535301208,
+      "logits/rejected": 0.33758652210235596,
+      "logps/chosen": -440.26605224609375,
+      "logps/rejected": -435.342041015625,
+      "loss": 0.4363,
+      "rewards/accuracies": 0.824999988079071,
+      "rewards/chosen": -1.2293314933776855,
+      "rewards/margins": 0.9770258069038391,
+      "rewards/rejected": -2.20635724067688,
       "step": 700
     },
     {
+      "epoch": 2.0,
+      "learning_rate": 0.0,
+      "logits/chosen": 0.20219314098358154,
+      "logits/rejected": 0.39273345470428467,
+      "logps/chosen": -488.3095703125,
+      "logps/rejected": -427.59088134765625,
+      "loss": 0.425,
       "rewards/accuracies": 0.84375,
+      "rewards/chosen": -1.406359076499939,
+      "rewards/margins": 0.9064434170722961,
+      "rewards/rejected": -2.312802314758301,
+      "step": 710
     },
     {
       "epoch": 2.0,
+      "step": 710,
       "total_flos": 0.0,
+      "train_loss": 0.5396277803770253,
+      "train_runtime": 10358.1577,
+      "train_samples_per_second": 8.795,
+      "train_steps_per_second": 0.069
     }
   ],
   "logging_steps": 10,
+  "max_steps": 710,
   "num_train_epochs": 2,
+  "save_steps": 10000,
   "total_flos": 0.0,
   "trial_name": null,
   "trial_params": null

training_args.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5afd5730214e6bd724e3cab0f3dcc26a9879ab9f6aff92cdb3a2b93fd0a49305
 size 6648

 version https://git-lfs.github.com/spec/v1
+oid sha256:2390c62cafd275700a8cab032e404ced08bab84cec2b9e7aad9c5b00a3adfe4b
 size 6648