Update README.md
Browse files
README.md
CHANGED
@@ -81,9 +81,13 @@ your desired response length:
|
|
81 |
|
82 |
## Training procedure
|
83 |
[Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) was used for training
|
84 |
-
on a single NVidia RTX3090 GPU. The model has been trained as a 4-bit LoRA adapter,
|
85 |
-
|
86 |
-
|
|
|
|
|
|
|
|
|
87 |
|
88 |
### Training hyperparameters
|
89 |
For the first pass these settings were used:
|
|
|
81 |
|
82 |
## Training procedure
|
83 |
[Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) was used for training
|
84 |
+
on a single NVidia RTX3090 GPU. The model has been trained as a 4-bit LoRA adapter, and
|
85 |
+
it's so large because a LoRA rank of 256 was also used. The reasoning was that this
|
86 |
+
might have helped the model internalize any newly acquired information, making the
|
87 |
+
training process closer to a full finetune.
|
88 |
+
|
89 |
+
It's suggested to merge the adapter to the base Llama2-7B model (or other Llama2-based
|
90 |
+
models).
|
91 |
|
92 |
### Training hyperparameters
|
93 |
For the first pass these settings were used:
|