Instructions for finetuning?
#8
by
diwank
- opened
Hey there,
Thanks for the amazing work as always! I want to finetune the mpt-30b and its instruct variant on a custom dataset. Is there a reference script / notebook available that shows an example of how to do so? I tried looking around but couldn't find one.
I did try to finetune it using a vanilla huggingface Trainer with qlora and that didn't seem to work unfortunately. It gave a weird "mpt models do not support gradient checkpointing" error.
p.s. @abhi-mosaic @sam-mosaic is this something that you could help out with?