Training info
I'm interested in training a Falcon40b version of this Lora and I was curious about the training parameters you used like epochs/steps and other potential info that I can use to try and get the closest results to what you got with the Supercot Lama30B Lora (the llama30bsupercott merge was #1 on hugging face leader board until recently, now its 2nd and Falcon40B is 1st) . Any info would be greatly appreciated.
epochs: 3
learning rate: 3e-4
lora rank: 8
lora alpha: 16
lora dropout: 0.05 for cutoff 1024 13B, otherwise no dropout due to gradient checkpointing
masking: none
mbatch size: 4 (1 for 30B)
batch size: 8 (2 for 30B)
val set size: 0.2
sdp implementation: xformers
optimizer: AdamW
eval strategy: none
Dataset preparation code is here: https://github.com/johnsmith0031/alpaca_lora_4bit/blob/35caccd3764d78afb8da4f4db1faa6faec53ba25/train_data.py#L130
Thanks so much 🙏 that is exactly what I needed
I should mention this LoRA uses the dataset here only: https://huggingface.co/datasets/kaiokendev/SuperCOT-dataset
The other datasets referenced have already been incorporated into it and cleaned