Fine tuning
Can this model be further fine-tuned (on a custom dataset)?
Do you have the fine-tuning script used to produce this model? - if yes, can it be used to fine-tune this model?
Yeah, for sure, to both questions. I finetuned using https://github.com/OpenAccess-AI-Collective/axolotl.
Brilliant stuff!!!! Thanks @totally-not-an-llm
Wondering if this would be a good point of reference - "yml" file to train the model?
https://github.com/OpenAccess-AI-Collective/axolotl/blob/main/examples/llama-2/qlora.yml
Question - what would be the sequence_len? should be sequence_len: 4096 or sequence_len:16384
Planning to make the following changes:
base_model: totally-not-an-llm/EverythingLM-13b-V2-16k
base_model_config: totally-not-an-llm/EverythingLM-13b-V2-16k
datasets:
- path: datasets/my_data_set
Yeah, I used that as a starting point. Sequence length depends on your dataset, it should be at least as big as the largest sample in your dataset. You can change other stuff too. It depends on your preferences. Happy to help if you have any more questions.
Champion!! Cool, I'll share my modified yml file soon.
Looking forward to your guidance.