Fine tuned phi2 model loses context once loaded from local

#112
by tatvamasi - opened

Hi Awesome people,

Currently I am exploring the phi-2 fine tuning and I am at the phase where results it yielded is pretty good. However, I am facing an issue when I try to load and run the model for inference. It stared giving hallucinations and non-relevant answers. Some times similar questions as the answers to the question prompted.

I have saved the model after finetuning (using PEFT) with pipeline and also with SFTTrainer
. i.e.
(at separate instances)

pipe.save_pretrained('./path_to_local_dir')

trainer.save_model('./path_to_local_dir_2')

and tried to load them via

AutoModelForCausalLM.from_pretrained('./path_to_local_dir')

pipe = pipeline(task="text-generation", model='./path_to_local_dir', tokenizer=tokenizer, max_length=250, trust_remote_code=True)

but in both cases, my finetuned LLM (phi-2) is behaving abnormally. Could anyone please suggest where I have gone wrong ? Every help is appreciated.

Thanks

If you did fientuning with peft then load it using
`from peft import PeftModel

base_model that You used

base_model = AutoModelForCausalLM.from_pretrained('microsoft/phi-2',
device_map='auto',
trust_remote_code=True,
)
pefft_model = PeftModel.from_pretrained(base_model, "./checkpoint/checkpoint-1000", torch_dtype=torch.float16, is_trainable=False)`

Sign up or log in to comment