How to fine tune mixtral 8x7B?
Pretraining and Fine tuning
Hi,
I'm looking forward to using this powerful model! I have cloned the repo for mistral-src onto my GPU machine. I followed the steps in the readme file, but I would really like to train it for my purpose.
Could I get a clear step-by-step tutorial on how to pre-train and fine-tune the model.
Thank you.
Hi
@tzivi
You can benefit from many tools from HF ecosystem to fine-tune this model. I suggest to go for the QLoRA approach as it is very memory efficient, among the options I know, you can use:
- unsloth library: https://github.com/unslothai/unsloth
- Llama-Factory from @hiyouga : https://huggingface.co/mistralai/Mixtral-8x7B-v0.1/discussions/10#6577e443e390cfd40990deff
- axolotl: https://github.com/OpenAccess-AI-Collective/axolotl
- TRL library: https://huggingface.co/docs/trl/sft_trainer
Hi, looking for a fine-tuning framework?
Try LLaMA-Factory and fine-tuning Mixtral using ZeRO stage-2/3 with the latest DeepSpeed release (v0.13.0)
https://github.com/hiyouga/LLaMA-Factory#hardware-requirement
https://github.com/hiyouga/LLaMA-Factory/pull/2283