How much RAM required?
#2
by
BBLL3456
- opened
I have 32GB RAM and 16GB GPU and can't seem to load the model, always get killed. May I know what is the specs requirement to run this for inference? Thanks.
Maybe you can try to specify torch_dtype=torch.float16
or torch_dtype=torch.bfloat16
to load the model. In default situation transformers
will load the model in fp32 and use about 30GB RAM, which is too much for 32GB.