Spaces:
Runtime error
Runtime error
How can I run a 30B model with only 5.8GB of RAM? Finally all my questions answered
#12
by
osanseviero
HF staff
- opened
ingle-precision to be introduced to pytorch soon?
Question:
How can I run a 30B model with only 5.8GB of RAM?
Answer:
There are a few options.
-
You can run the model in ‘ingle precision’ mode. This is not recommended, as it can lead to an unstable model, but it is possible.
-
You can run the model in ‘ingle precision’ mode, but set the number of cores to a low value. This is not recommended, as it will likely cause a very slow model.
-
You can run the model in ‘ingle precision’ mode, but set the number of cores to a low value. This is not recommended, as it will likely cause a very slow model.