Finetune mpt-7b like storywriter but with another dataset
We want to finetune mpt-7b model in a way that storywriter model is created but with another different dataset. However, we could not find how to finetune it. Is there any published code to finetune mpt-7b for storywriter model? Can anyone help us?
Here is some documentation on how to do sequence length adaptation tuning. https://github.com/mosaicml/llm-foundry/blob/tutorial/TUTORIAL.md#domain-adaptation-and-sequence-length-adaptation
You will need to set your device microbatch size lower to fit the long sequences, and you may need a few nodes of 80GB A100s... which we have! Sign up here
@sam-mosaic what's the wait time for getting in on that waitlist? I applied a while back and haven't heard, not sure if I should look elsewhere or what.