Model save

README.md

@@ -0,0 +1,92 @@
+---
+license: apache-2.0
+library_name: peft
+tags:
+- trl
+- dpo
+- generated_from_trainer
+base_model: mistralai/Mistral-7B-v0.1
+model-index:
+- name: zephyr-7b-dpo-lora
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# zephyr-7b-dpo-lora
+
+This model is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) on the None dataset.
+It achieves the following results on the evaluation set:
+- Loss: 0.5133
+- Rewards/chosen: -1.2447
+- Rewards/rejected: -2.1118
+- Rewards/accuracies: 0.7539
+- Rewards/margins: 0.8671
+- Logps/rejected: -457.0128
+- Logps/chosen: -385.9082
+- Logits/rejected: 1.2523
+- Logits/chosen: 0.7989
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 5e-06
+- train_batch_size: 4
+- eval_batch_size: 8
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 8
+- total_train_batch_size: 32
+- total_eval_batch_size: 64
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_ratio: 0.1
+- num_epochs: 1
+
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
+|:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
+| 0.6918        | 0.05  | 100  | 0.6914          | 0.0059         | 0.0018           | 0.7109             | 0.0041          | -245.6464      | -260.8458    | -2.1364         | -2.2285       |
+| 0.6619        | 0.1   | 200  | 0.6497          | -0.0263        | -0.1318          | 0.7070             | 0.1056          | -259.0110      | -264.0628    | -2.0537         | -2.1558       |
+| 0.6077        | 0.16  | 300  | 0.6083          | -0.2610        | -0.5505          | 0.7188             | 0.2895          | -300.8820      | -287.5379    | -1.8505         | -1.9870       |
+| 0.5813        | 0.21  | 400  | 0.5857          | -0.5019        | -0.9224          | 0.7344             | 0.4205          | -338.0691      | -311.6292    | -1.7834         | -1.9347       |
+| 0.6033        | 0.26  | 500  | 0.5684          | -0.6480        | -1.1327          | 0.7578             | 0.4847          | -359.0957      | -326.2360    | -1.0646         | -1.2844       |
+| 0.5338        | 0.31  | 600  | 0.5431          | -0.9068        | -1.6081          | 0.7539             | 0.7013          | -406.6367      | -352.1152    | -0.0058         | -0.3463       |
+| 0.5235        | 0.37  | 700  | 0.5304          | -1.0331        | -1.8281          | 0.7461             | 0.7951          | -428.6434      | -364.7436    | 0.2246          | -0.1374       |
+| 0.5241        | 0.42  | 800  | 0.5276          | -0.9760        | -1.7110          | 0.7578             | 0.7350          | -416.9325      | -359.0362    | 0.3361          | -0.0432       |
+| 0.5332        | 0.47  | 900  | 0.5257          | -1.2407        | -2.0657          | 0.75               | 0.8250          | -452.3993      | -385.5118    | 0.8926          | 0.4681        |
+| 0.531         | 0.52  | 1000 | 0.5232          | -1.1277        | -1.8553          | 0.7461             | 0.7276          | -431.3623      | -374.2120    | 0.2825          | -0.0766       |
+| 0.4864        | 0.58  | 1100 | 0.5172          | -1.1670        | -1.9894          | 0.75               | 0.8224          | -444.7675      | -378.1358    | 1.1814          | 0.7409        |
+| 0.5467        | 0.63  | 1200 | 0.5196          | -1.3633        | -2.1690          | 0.7383             | 0.8058          | -462.7306      | -397.7628    | 1.3020          | 0.8593        |
+| 0.5125        | 0.68  | 1300 | 0.5179          | -1.2033        | -2.0041          | 0.7422             | 0.8009          | -446.2437      | -381.7657    | 1.1045          | 0.6639        |
+| 0.4881        | 0.73  | 1400 | 0.5158          | -1.2792        | -2.1334          | 0.7539             | 0.8543          | -459.1728      | -389.3554    | 1.1891          | 0.7445        |
+| 0.5273        | 0.78  | 1500 | 0.5135          | -1.2081        | -2.0746          | 0.7539             | 0.8664          | -453.2860      | -382.2505    | 1.2533          | 0.7973        |
+| 0.5317        | 0.84  | 1600 | 0.5140          | -1.2815        | -2.1592          | 0.75               | 0.8777          | -461.7518      | -389.5859    | 1.2752          | 0.8202        |
+| 0.5384        | 0.89  | 1700 | 0.5134          | -1.2549        | -2.1287          | 0.7539             | 0.8738          | -458.7038      | -386.9291    | 1.2938          | 0.8384        |
+| 0.5619        | 0.94  | 1800 | 0.5135          | -1.2438        | -2.1108          | 0.7578             | 0.8670          | -456.9133      | -385.8195    | 1.2532          | 0.7986        |
+| 0.5169        | 0.99  | 1900 | 0.5133          | -1.2447        | -2.1118          | 0.7539             | 0.8671          | -457.0128      | -385.9082    | 1.2523          | 0.7989        |
+
+
+### Framework versions
+
+- PEFT 0.7.1
+- Transformers 4.36.2
+- Pytorch 2.1.2+cu121
+- Datasets 2.14.6
+- Tokenizers 0.15.0

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6c4b7c02b41586b888233bc56a387c9b739d877d11956272a7e3fc1af914e394
+oid sha256:60e80ad1508f5cca0a7366ed76122de4180e34cf0dc55a2eb3291f3cd9f53312
 size 83945744

all_results.json

@@ -0,0 +1,21 @@
+{
+    "epoch": 1.0,
+    "eval_logits/chosen": 0.7988529205322266,
+    "eval_logits/rejected": 1.2523157596588135,
+    "eval_logps/chosen": -385.9081726074219,
+    "eval_logps/rejected": -457.0127868652344,
+    "eval_loss": 0.5133188962936401,
+    "eval_rewards/accuracies": 0.75390625,
+    "eval_rewards/chosen": -1.244707703590393,
+    "eval_rewards/margins": 0.8671280741691589,
+    "eval_rewards/rejected": -2.1118357181549072,
+    "eval_runtime": 99.9074,
+    "eval_samples": 2000,
+    "eval_samples_per_second": 20.019,
+    "eval_steps_per_second": 0.32,
+    "train_loss": 0.5577758540044768,
+    "train_runtime": 7516.1301,
+    "train_samples": 61155,
+    "train_samples_per_second": 8.137,
+    "train_steps_per_second": 0.254
+}

eval_results.json

@@ -0,0 +1,16 @@
+{
+    "epoch": 1.0,
+    "eval_logits/chosen": 0.7988529205322266,
+    "eval_logits/rejected": 1.2523157596588135,
+    "eval_logps/chosen": -385.9081726074219,
+    "eval_logps/rejected": -457.0127868652344,
+    "eval_loss": 0.5133188962936401,
+    "eval_rewards/accuracies": 0.75390625,
+    "eval_rewards/chosen": -1.244707703590393,
+    "eval_rewards/margins": 0.8671280741691589,
+    "eval_rewards/rejected": -2.1118357181549072,
+    "eval_runtime": 99.9074,
+    "eval_samples": 2000,
+    "eval_samples_per_second": 20.019,
+    "eval_steps_per_second": 0.32
+}

runs/Jan04_13-23-17_ip-26-0-172-147/events.out.tfevents.1704374714.ip-26-0-172-147.1299432.0

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b418a8058e5da0e3d0d235ac18a75c01f05b8499f913619107e82f57b2cf3ff9
-size 139536
+oid sha256:fcfa58db6c93ca5f13501388a171eeaf3490f435fe370b5fbf6bde1569ba9c51
+size 140524

runs/Jan04_13-23-17_ip-26-0-172-147/events.out.tfevents.1704382330.ip-26-0-172-147.1299432.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9901b4c6c051ac20ea235d84c22b013f7aedf52f45f64efbea181eaf627c3055
+size 828

train_results.json

@@ -0,0 +1,8 @@
+{
+    "epoch": 1.0,
+    "train_loss": 0.5577758540044768,
+    "train_runtime": 7516.1301,
+    "train_samples": 61155,
+    "train_samples_per_second": 8.137,
+    "train_steps_per_second": 0.254
+}