thomnis/Mistral-7B-Instruct-v0.2-GPTQ-LoRA-shawgpt-ft

README.md

@@ -16,7 +16,7 @@ should probably proofread and complete it, then remove this comment. -->
 
 This model is a fine-tuned version of [mistralai/Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3) on an unknown dataset.
 It achieves the following results on the evaluation set:
-- Loss: 1.3524
+- Loss: 1.5866
 
 ## Model description
 
@@ -44,23 +44,32 @@ The following hyperparameters were used during training:
 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
 - lr_scheduler_type: linear
 - lr_scheduler_warmup_steps: 2
-- num_epochs: 10
+- num_epochs: 20
 - mixed_precision_training: Native AMP
 
 ### Training results
 
-| Training Loss | Epoch  | Step | Validation Loss |
-|:-------------:|:------:|:----:|:---------------:|
-| 4.586         | 0.9231 | 3    | 4.0840          |
-| 4.5003        | 1.8462 | 6    | 3.6717          |
-| 3.4228        | 2.7692 | 9    | 2.6501          |
-| 1.7833        | 4.0    | 13   | 1.7598          |
-| 1.68          | 4.9231 | 16   | 1.4736          |
-| 1.427         | 5.8462 | 19   | 1.4181          |
-| 1.3196        | 6.7692 | 22   | 1.3783          |
-| 0.9651        | 8.0    | 26   | 1.3563          |
-| 1.245         | 8.9231 | 29   | 1.3527          |
-| 0.8858        | 9.2308 | 30   | 1.3524          |
+| Training Loss | Epoch   | Step | Validation Loss |
+|:-------------:|:-------:|:----:|:---------------:|
+| 4.586         | 0.9231  | 3    | 4.0840          |
+| 4.5013        | 1.8462  | 6    | 3.6804          |
+| 3.4295        | 2.7692  | 9    | 2.6498          |
+| 1.7662        | 4.0     | 13   | 1.7049          |
+| 1.6274        | 4.9231  | 16   | 1.4525          |
+| 1.3983        | 5.8462  | 19   | 1.3957          |
+| 1.2807        | 6.7692  | 22   | 1.3583          |
+| 0.9365        | 8.0     | 26   | 1.3325          |
+| 1.1829        | 8.9231  | 29   | 1.3339          |
+| 1.0945        | 9.8462  | 32   | 1.3330          |
+| 1.0448        | 10.7692 | 35   | 1.3496          |
+| 0.7014        | 12.0    | 39   | 1.3455          |
+| 0.8887        | 12.9231 | 42   | 1.3879          |
+| 0.8145        | 13.8462 | 45   | 1.4225          |
+| 0.7586        | 14.7692 | 48   | 1.4449          |
+| 0.5382        | 16.0    | 52   | 1.5154          |
+| 0.6563        | 16.9231 | 55   | 1.4966          |
+| 0.6154        | 17.8462 | 58   | 1.5442          |
+| 0.4083        | 18.4615 | 60   | 1.5866          |
 
 
 ### Framework versions

adapter_config.json

@@ -20,9 +20,9 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
-    "v_proj",
+    "q_proj",
     "k_proj",
-    "q_proj"
+    "v_proj"
   ],
   "task_type": "CAUSAL_LM",
   "use_dora": false,

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4b69db1e9bb94fa449f1d23526ce73cc9bcac77c379dcec4de4b0ee30c9f1905
+oid sha256:5919996e3f1e41a686d2b157fb227dcd03114945ff77f8cb32e5e9c45d4b45e6
 size 37774528

runs/Oct22_05-24-08_4d73a2362bca/events.out.tfevents.1729574652.4d73a2362bca.183.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:429001c9de9e649eefe5feda61e422ea2b1e17602e85b1fb6193d3b014c4606a
+size 5417

runs/Oct22_05-24-08_4d73a2362bca/events.out.tfevents.1729574772.4d73a2362bca.183.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dba06ab07a550693052f21487667207f9f7fc39e9ca7b390e673c4bcb8bfa122
+size 14752

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ba0c5bd7e20cb5ad26a95f0d751c5e5215dec8d3f70c5abaf23f1e53b702ad05
+oid sha256:f10f09ef12c65713c16c69524ed126c59470847ce18bd5f113f45d81ef7ca532
 size 5176