haih2
/

open-calm-7b-summarizer-lora

Text Generation

Model card Files Files and versions Community

haih2 commited on Aug 31, 2023

Commit

565ce47

•

1 Parent(s): 0da2af5

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -96,7 +96,7 @@ The OpenCALM-7B model was fine-tuned on the above dataset using the QLoRA method
 | **Optimizer** <br> &emsp; beta_1 <br> &emsp; beta_2 <br> &emsp; weight decay | AdamW <br> 0.9 <br> 0.999 <br> 0.01 |
 | **Learning rate** <br> &emsp; scheduler type | 2e-5 <br> linear |
 | **LoRA** <br> &emsp; target modules <br> &emsp; r <br> &emsp; alpha <br> &emsp; dropout | <br> query_key_value, dense <br> 4 <br> 64 <br> 0.05 |
-| **QLoRA** <br> &emsp; compute dtype <br> &emsp; storage dtype <br> &emsp; quantization strategy | <br> float16 <br> nf4 <br> double quantization |
 | **Sequence length**             | 1536 |
 | **Batch size**                  | 4    |
 | **Gradient accumulation steps** | 2    |

 | **Optimizer** <br> &emsp; beta_1 <br> &emsp; beta_2 <br> &emsp; weight decay | AdamW <br> 0.9 <br> 0.999 <br> 0.01 |
 | **Learning rate** <br> &emsp; scheduler type | 2e-5 <br> linear |
 | **LoRA** <br> &emsp; target modules <br> &emsp; r <br> &emsp; alpha <br> &emsp; dropout | <br> query_key_value, dense <br> 4 <br> 64 <br> 0.05 |
+| **Quantization (for QLoRA)** <br> &emsp; compute dtype <br> &emsp; storage dtype <br> &emsp; quantization strategy | <br> float16 <br> nf4 <br> double quantization |
 | **Sequence length**             | 1536 |
 | **Batch size**                  | 4    |
 | **Gradient accumulation steps** | 2    |