Is it QLoRA or a full finetune?
#5
by
Andriy
- opened
Hi! A question: did you have challenges with using DeepSpeed ZeRO-3 and full finetune? I'm asking because we have an issue with LLMs and DeepSpeed ZeRO-3. The issue is that if you load on LLM with ZeRO-3, then save, and then load again, the model becomes broken. Did you experience something like that?
I usually do a regular LoRa (not Q) and then merge the weights back to the original model. This also lets me target different layers as I work upward from base layers to the final ones. Hopefully that helps, I didn't use DeepSpeed at all since I cheat a bit by the repetitive LoRa trick :)
ibivibiv
changed discussion status to
closed