Update README.md

README.md

@@ -2,10 +2,10 @@
 license: apache-2.0
 ---
 
-# LimaRP-Llama2-7B-v3 (Alpaca, experimental, 4-bit LoRA adapter)
+# LimaRP-Llama2-7B-v3 (Alpaca, experimental, 8-bit LoRA adapter)
 
 This is an experimental version of LimaRP for Llama2 with an updated dataset (1800 training samples)
-and a 2-pass training procedure. The first pass includes unsupervised finetuning on 2800 stories within
+and a 2-pass training procedure. The first pass includes unsupervised finetuning on about 6800 stories within
 4k tokens length and the second pass is LimaRP with changes introducing more effective control on response length.
 
 For more details about LimaRP, see the model page for the [previously released version](https://huggingface.co/lemonilia/limarp-llama2-v2).
@@ -81,7 +81,7 @@ your desired response length:
 
 ## Training procedure
 [Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) was used for training
-on a single NVidia RTX3090 GPU. The model has been trained as a 4-bit LoRA adapter, and
+on a 4x NVidia A40 GPU. The model has been trained as an 8-bit LoRA adapter, and
 it's so large because a LoRA rank of 256 was also used. The reasoning was that this
 might have helped the model internalize any newly acquired information, making the
 training process closer to a full finetune.
@@ -92,17 +92,17 @@ models).
 ### Training hyperparameters
 For the first pass these settings were used:
 
-- learning_rate: 0.0002
+- learning_rate: 0.00065
 - lr_scheduler_type: constant
 - lora_r: 256
 - lora_alpha: 16
-- lora_dropout: 0.1
+- lora_dropout: 0.05
 - lora_target_linear: True
 - num_epochs: 1
 - bf16: True
 - tf32: True
-- load_in_4bit: True
-- adapter: qlora
+- load_in_8bit: True
+- adapter: lora
 - micro_batch_size: 2
 - gradient_accumulation_steps: 1
 - optimizer: adamw_torch
@@ -111,6 +111,5 @@ In the second pass, the `lora_model_dir` option was used to load and train the a
 previously trained on a stories dataset. These settings were also changed:
 
 - lora_dropout: 0.0
-- micro_batch_size: 1
-- gradient_accumulation_steps: 8
-- learning_rate: 0.0006
+
+Using 4 GPUs, the effective global batch size would have been 8.