gemma7b-ft-lora-sql-v2adapters

README.md

@@ -20,7 +20,7 @@ should probably proofread and complete it, then remove this comment. -->
 
 This model is a fine-tuned version of [google/gemma-7b](https://huggingface.co/google/gemma-7b) on the generator dataset.
 It achieves the following results on the evaluation set:
-- Loss: 0.9175
+- Loss: 0.4155
 
 ## Model description
 
@@ -39,29 +39,45 @@ More information needed
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
-- learning_rate: 0.0002
-- train_batch_size: 4
+- learning_rate: 0.0003
+- train_batch_size: 8
 - eval_batch_size: 8
 - seed: 1399
-- gradient_accumulation_steps: 8
+- gradient_accumulation_steps: 4
 - total_train_batch_size: 32
 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
-- lr_scheduler_type: constant
-- lr_scheduler_warmup_steps: 10
-- training_steps: 100
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_steps: 100
+- training_steps: 500
 
 ### Training results
 
 | Training Loss | Epoch | Step | Validation Loss |
 |:-------------:|:-----:|:----:|:---------------:|
-| 20.0926       | 0.95  | 5    | 18.5680         |
-| 16.2448       | 1.9   | 10   | 10.6329         |
-| 5.1408        | 2.86  | 15   | 1.3605          |
-| 1.1787        | 3.81  | 20   | 0.9861          |
-| 0.9186        | 4.76  | 25   | 0.8888          |
-| 0.7896        | 5.71  | 30   | 0.8632          |
-| 0.6787        | 6.67  | 35   | 0.8657          |
-| 0.5448        | 7.62  | 40   | 0.9175          |
+| 16.1657       | 0.06  | 20   | 13.6485         |
+| 7.8281        | 0.13  | 40   | 0.7808          |
+| 0.6243        | 0.19  | 60   | 0.5270          |
+| 0.5179        | 0.25  | 80   | 0.4859          |
+| 0.4908        | 0.32  | 100  | 0.4754          |
+| 0.4752        | 0.38  | 120  | 0.4600          |
+| 0.4877        | 0.45  | 140  | 0.4584          |
+| 0.4626        | 0.51  | 160  | 0.4560          |
+| 0.4569        | 0.57  | 180  | 0.4428          |
+| 0.4504        | 0.64  | 200  | 0.4354          |
+| 0.4432        | 0.7   | 220  | 0.4348          |
+| 0.4395        | 0.76  | 240  | 0.4317          |
+| 0.4338        | 0.83  | 260  | 0.4256          |
+| 0.4308        | 0.89  | 280  | 0.4260          |
+| 0.4283        | 0.95  | 300  | 0.4210          |
+| 0.4146        | 1.02  | 320  | 0.4225          |
+| 0.3848        | 1.08  | 340  | 0.4186          |
+| 0.3812        | 1.14  | 360  | 0.4185          |
+| 0.38          | 1.21  | 380  | 0.4200          |
+| 0.3795        | 1.27  | 400  | 0.4171          |
+| 0.3766        | 1.34  | 420  | 0.4174          |
+| 0.3772        | 1.4   | 440  | 0.4136          |
+| 0.3777        | 1.46  | 460  | 0.4148          |
+| 0.379         | 1.53  | 480  | 0.4155          |
 
 
 ### Framework versions

adapter_config.json

@@ -10,22 +10,22 @@
   "layers_to_transform": null,
   "loftq_config": {},
   "lora_alpha": 32,
-  "lora_dropout": 0.1,
+  "lora_dropout": 0.05,
   "megatron_config": null,
   "megatron_core": "megatron.core",
   "modules_to_save": null,
   "peft_type": "LORA",
-  "r": 8,
+  "r": 16,
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
+    "k_proj",
     "v_proj",
+    "down_proj",
+    "q_proj",
     "o_proj",
     "up_proj",
-    "gate_proj",
-    "k_proj",
-    "q_proj",
-    "down_proj"
+    "gate_proj"
   ],
   "task_type": "CAUSAL_LM",
   "use_dora": false,

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e7ab74726f0a87bc9b9a37cc68bc41fc4d2d4fc803dcc9ba3d76745392bc951a
+oid sha256:f90e03f124eede9ebe92ac2a7cb4cd187068b1c799e0422942648c31ca3de583
 size 4920