requires newer version of koboldcpp

#1
by lemon07r - opened

Havent tested the other quants, but loading the model just crashes koboldcpp, tested with vulkan and hipblas (rocm branch)

That's a problem in koboldcpp, the model works fine for me in llama.cpp:

main -m /tmp/llama-3-MagicDolphin-8B.Q8_0.gguf -p Hi,

Hi, I'm Dr. Alex George, a GP, a dad, and a passionate mental health advocate. I'm here to help you navigate the ups and downs of life, and to provide you with practical advice and support to improve your mental wellbeing...
mradermacher changed discussion status to closed
mradermacher changed discussion title from Q8_0 is broken to does not work in koboldcpp

Seeing that the model uses the smaug tokenizer, the problem is likely that koboldcpp is outdated and needs an update.

mradermacher changed discussion title from does not work in koboldcpp to requires newer version of koboldcpp

Sign up or log in to comment