internlm

/

internlm2-chat-7b

like 80

HI,

I am not sure if you have tried transformer inference or not, but it seems internlm2 would not work properly under bitsandbytes 4 bit quantization. It will constantly spit out self Q&As without stopping.

Also accerlerate QLoRA would also not work on internlm2 with error on some tensor has no grad.

float16 would work in all cases.

It is not a big deal since internlm2 mainly support lmdeploy framework with its own 4 bit quantization instead of transformer.

Thanks!