Error in Lora Config of traget modules not found

#33
by rawalba - opened

ValueError: Target modules ['query_key_valuedense', 'dense_h_to_4h', 'dense_4h_to_h', 'q_proj', 'v_proj'] not found in the base model. Please check the target modules and try again.

Microsoft org

Hello @rawalba !

The reason you might be having errors is because LoRA is not supported in Phi-based models.

@gugarosa I've trained qlora with no problem using transformer and bitsandbytes. Edit* I used the Alpaca instructions to fine tune on a closed domain dataset and I am astonished at how well it performs. It did take me more epochs +50 for it to finally learn the content since it never seen it before though.

@rawalba use Target modules [ "Wqkv", "out_proj", "fc1", "fc2" ]

Microsoft org

@Ermarrero thanks for the explanation! I wasn't aware that LoRA would work in the Phi models.

gugarosa changed discussion status to closed

@rawalba use Target modules [ "Wqkv", "out_proj", "fc1", "fc2" ]

@gugarosa : Hi, can you please tell me which file to modify?

Any advice is appreciated.

Sign up or log in to comment