Heavy Hallucination | non truncating o/p in Phi 3 Vision Finetuning to convert chart to json
#47
by
ar9av
- opened
Phi 3 Vision works quite well when used straight out of the box but on finetuning it starts hallucinating on numerical values [2020,2021....] like extrapolating values in cases like dates, numbers. And keeps on generating.
I checked and I using the eos_tag correctly. So that is not the issue. Where else am i going wrong?
And I have noticed this behaviour with other Vision based Transformers as well. Any idea how to get this fixed?
Thank you for your interest in the Phi-3 Vision model.
Maybe you could try the Phi-3 CookBook finetuning recipe here https://github.com/microsoft/Phi-3CookBook/blob/main/md/04.Fine-tuning/FineTuning_Vision.md