[UNUSEDTOKEN145]

#1
by MoonRide - opened

I am not sure if it's issue with the model files or llama.cpp not fully supporting InternLM, but I constantly get [UNUSEDTOKEN145] at the end of model response, like this:

image.png

Sometimes other numbers appear, too:

image.png

It happens with both GGUF file taken from this repository (tested on Q6_K), and also with GGUF I've made myself (with llama.cpp b3287).

Other models (Llama 3, Gemma 2, Mistral 0.3) work just fine.

Have you tried the Ollama version?

% ollama run internlm2:7b-chat-v2.5-q5_K_M
>>> who are you?
I am an AI developed by OpenAI, designed to assist with various tasks and provide information. How can I help you today?

>>> 2+2?
The result of adding 2 + 2 is

>>> Send a message (/? for help)
InternLM org

@MoonRide hi, thanks for your feedback. It seems some special tokens are missing when converting to gguf using the convert script from llama.cpp. We'll try to solve it.

InternLM org

@MoonRide hi, we have updated gguf models using the convert script from this PR https://github.com/ggerganov/llama.cpp/pull/8321 .
Please refer to this example and try again.

@MoonRide hi, we have updated gguf models using the convert script from this PR https://github.com/ggerganov/llama.cpp/pull/8321 .
Please refer to this example and try again.

I tested the new (updated today) version of GGUF file from this repo (Q6_K quant), and problem seems to be solved πŸ‘.

MoonRide changed discussion status to closed

Sign up or log in to comment