mistralai

/

Mistral-7B-Instruct-v0.1

like 1.42k

I have the same issue, any news?

@19Peppe95 The error is because system is running out of RAM to load the model in one go.
You can use CTransformers to load the model or can try GGUF model versions of your model which is basically much smaller version of it.
Gist - Use GGUF version of this model https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.1-GGUF
and use CTransformers to load it from the dowloads and run the program, hopefully it should work.

@19Peppe95 The error is because system is running out of RAM to load the model in one go.
You can use CTransformers to load the model or can try GGUF model versions of your model which is basically much smaller version of it.
Gist - Use GGUF version of this model https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.1-GGUF
and use CTransformers to load it from the dowloads and run the program, hopefully it should work.

I also have the RAM issue. But it is werid that the model is only 14GB in totally and I have 64GB RAM and 24GB VRAM available.
Just found out that there was a download issue, the bins are broken so the memory usage when loading the files became uncontrollable.

cc @ybelkada regarding low-memory methods to load larger models

Hi everyone
In case you are facing CPU OOM issues while loading the model please consider using sharded models with small shards, for this model I would recommend using this repository: https://huggingface.co/bn22/Mistral-7B-Instruct-v0.1-sharded

@ybelkada Hi, the model you shared is not available anymore? :(

hi @shantanudave
Indeed, please use: https://huggingface.co/alexsherstinsky/Mistral-7B-v0.1-sharded instead