Finetune on GEITje
#1
by
tystnad
- opened
Thanks for sharing this!
Is there a specific reason you finetuned Mistral rather than GEITje?
Would the LORA be applicable to GEITje as well?
GEITje came out a couple of days after I started doing this haha. The LoRA I did was for the pretraining stage (which is not something you would usually do, usually you do LoRA for supervised fine-tuning or DPO). As GEITje is an SFT model I don't think it makes much sense to train it with the dataset I used.
What I plan to do:
- Make an SFT model with synthetic data based on the public domain collection.
- Merge that with GEITje using mergekit
- See what happens :)