gguf
#17
by
daisr
- opened
gguf, pls
Ollama, pls :)
its out there if you search.
ok its not on gguf yet as it cannot be converted so easy ?
Did you even bother searching? I see more than one in a simple search.
EDIT even one for ollama ( and ollama will import gguf, at least it does for me )
llama.cpp doesn't support this model yet
Try the exl2 quants, they work, I'm using Turboderps 8.0bpw version and can run it on Text generation webui with 128k context (at 8bit cache) within 24gb of gpu memory. It's a good model.
it should be fine now !!
itsin unsloth and in the llama cpp ( they had to update the embeddings)
Llama.cpp PR got merged
