Is it QLoRA or a full finetune?

by Andriy - opened Mar 11

Mar 11

Hi! A question: did you have challenges with using DeepSpeed ZeRO-3 and full finetune? I'm asking because we have an issue with LLMs and DeepSpeed ZeRO-3. The issue is that if you load on LLM with ZeRO-3, then save, and then load again, the model becomes broken. Did you experience something like that?

migtissera

Owner Mar 11

Hey, this was trained with QLoRA.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment