Hi from the creator!😊
thanks from doing this bro, i was trying to do the quantization of this particular model but it was not working, if you can share the code with me im gratefull haha
Unfortunately, I don't remember if I did anything special, and I don't think I have any patches in use currently, so presumably, all that's needed is a somewhat current version of llama.cpp. Do you mean the quantization seemingly worked, but the result didn't work? Did you get an error, or was the model just misbehaving?
All I do is basically run convert_hf_to_gguf.py without special switches, and then quantize for the desired formats.
thats strange, because i got an tokenization error if i try to do this, maybe i forgot something, but thanks otherwise bro, and if anyone would test this with ollama use this:
FROM ./Samantha-qwen-0.5b.Q8_0.gguf
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
"""
PARAMETER num_ctx 4096
PARAMETER stop ""
PARAMETER stop "USER:"
PARAMETER stop "User:"
PARAMETER stop "Samantha:"
PARAMETER stop "ASSISTANT:"
PARAMETER stop [INST]
PARAMETER stop [/INST]
PARAMETER stop <>
PARAMETER stop <>
PARAMETER stop [INST]
PARAMETER stop [/INST]
PARAMETER stop <>
PARAMETER stop <>
PARAMETER stop <|im_end|>
PARAMETER stop <|im_start|>
PARAMETER stop <|endoftext|>
PARAMETER stop <|im_/
PARAMETER stop <|im_/>
PARAMETER stop <|im_!>
PARAMETER stop <|im_3
PARAMETER stop <|im_3>
PARAMETER stop <|
PARAMETER stop <6