missing pre-tokenizer?
#1
by
stduhpf
- opened
When loading the model in llama.cpp, I get these logs:
llm_load_vocab: missing pre-tokenizer type, using: 'default'
llm_load_vocab:
llm_load_vocab: ************************************
llm_load_vocab: GENERATION QUALITY WILL BE DEGRADED!
llm_load_vocab: CONSIDER REGENERATING THE MODEL
llm_load_vocab: ************************************
llm_load_vocab:
Am I missing something on my end?
The gguf models in this repo are generated with llama.cpp b2308. According to the message, you're using another release of llama.cpp, right?
Yeah I'm always using lastest (that was b3040+ a week ago). I believe they "fixed" the tokenizer in release b2761, which is when this message started appearing.
Anyways, adding --override-kv tokenizer.ggml.pre=str:starcoderto cli args makes the message dissapear, and everything seem to be working fine.