microsoft

/

phi-2

like 3.13k

Same question, can we simply enable autocast?

@xueyanz actually I do think that is needed, even if using bfloat16. For some reason autocasting the forward of the attention module leads to instability issues in training. I will keep disabling autocast

Thanks so much for your prompt reply, I am trying to train phi2 in a VLM model using auto-cast. To disable autocast, do you manually transfer to fp16?

I load it in fp32 and use torch.amp with bfloat16. The last version of modeling_phi.py already disables autocast by itself in the forward method of the attention module. I'm actually also building a VLM using phi-2, would you care to explain what you are doing at a high level? Also, MoE-LLaVa states that there are training instabilities when using phi-2, maybe because they used a past version of the modeling_phi.py file.

I am using the inference api on huggingface. I attempted to load and access the model using the huggingface production end points. I gave up after several attempts. Any special settings or configs I need to be aware of to enable on a private huggingface inference api? Would help heaps.

I load it in fp32 and use torch.amp with bfloat16. The last version of modeling_phi.py already disables autocast by itself in the forward method of the attention module. I'm actually also building a VLM using phi-2, would you care to explain what you are doing at a high level? Also, MoE-LLaVa states that there are training instabilities when using phi-2, maybe because they used a past version of the modeling_phi.py file.

I will not train the language model, so I simply enable Autocast to see the performance, and the inference result seems reasonable. I am building vlm along the lines of my work in the past.

ok, it seems that auto-casting would make the outputs NaN even during evaluation.

Edited - nevermind - I figured out how to use float16 & bfloat16 without needing to autocast. Thank you.

Edited - nevermind - I figured out how to use float16 & bfloat16 without needing to autocast. Thank you.

Care to share what you did? Thanks.