requires newer version of koboldcpp
#1
by
lemon07r
- opened
Havent tested the other quants, but loading the model just crashes koboldcpp, tested with vulkan and hipblas (rocm branch)
That's a problem in koboldcpp, the model works fine for me in llama.cpp:
main -m /tmp/llama-3-MagicDolphin-8B.Q8_0.gguf -p Hi,
Hi, I'm Dr. Alex George, a GP, a dad, and a passionate mental health advocate. I'm here to help you navigate the ups and downs of life, and to provide you with practical advice and support to improve your mental wellbeing...
mradermacher
changed discussion status to
closed
mradermacher
changed discussion title from
Q8_0 is broken
to does not work in koboldcpp
Seeing that the model uses the smaug tokenizer, the problem is likely that koboldcpp is outdated and needs an update.
mradermacher
changed discussion title from
does not work in koboldcpp
to requires newer version of koboldcpp