How much RAM required?

#2
by BBLL3456 - opened

I have 32GB RAM and 16GB GPU and can't seem to load the model, always get killed. May I know what is the specs requirement to run this for inference? Thanks.

InternLM org

Maybe you can try to specify torch_dtype=torch.float16 or torch_dtype=torch.bfloat16 to load the model. In default situation transformers will load the model in fp32 and use about 30GB RAM, which is too much for 32GB.

Sign up or log in to comment