Provide fine-tuning example notebook using hf transformers
Can anyone provide a example fine-tuning notebook with custom data using hf transformers. Specifically I wonder if 'labels' are shifted automatically like GPT2 model or how do I provide the 'labels' to the model on training time.
@MakerMotion Did you find an answer to this?
@zachblank I think so. Because MPT is not fully implemented on this version of HF, I took a look at their model repo and in this file https://huggingface.co/mosaicml/mpt-7b-instruct/blob/main/modeling_mpt.py on forward() function; it seems like if you provide a labels argument it shifts automatically. [line 244]
@MakerMotion Thanks! Do you have an example notebook you could share? I'm new at this and still trying to wrap my head around it. Thanks!
Yes you are correct @MakerMotion
What are you looking for in particular @zachblank . Does this help? https://github.com/mosaicml/llm-foundry/blob/main/scripts/train/finetune_example/README.md
Closing as stale
Is there a concrete Notebook example of taking the MPT-7B-Instruct model and fine-tuning it with an HF dataset for example multi_news dataset for news summarization?
- How to prepare the dataset/ prompt
- How to freeze layers and have a small number of trainable parameters (given that you don't have LoRA support yet)