internlm

/

internlm2_5-7b-chat-gguf

like 61

I've fixed the transformers version of the model as well and posted it here.

Thanks!

@apresence hi, thank you for the feedback. We'll try to fix it later.

@apresence hi, thank you for the feedback. We'll try to fix it later.

You're welcome to look at the code and settings changes from the repo I stood up.

Let me know if I can help!

Also, if I may, I'd like to share an another issue with you. It seems that your model uses dynamic RoPE by default. It also supports linear, but when I tried that, the perplexity was quite bad. I notice when running the model in llama.cpp, which does not support dynamic and thus has to run linear, that the perplexity is worse than dynamic in transformers. That is, it starts out great, but after a few back-and-forth interactions it starts to degrade, eventually repeating itself and forgetting information. This is when the prompt is still well within the configured context length. I'd love to hear your team's insights on the issue, and any ideas about how to address it.

Thanks for the great model!

@apresence hi, thank you for the feedback. We'll try to fix it later.

Another

@apresence hi,

1. special tokens missing issue

After verification, there seems to be a misunderstanding about The tool call tokens are hidden from the output due to being marked as special tokens.
Special tokens are not showing by default. If you add --special to llama-cli, then you can see the full output string with special tokens. Besides, you can also check it in log file if you add --logdir

command

build/bin/llama-cli \
    --model internlm2_5-7b-chat-fp16.gguf \
    --predict 512 \
    --ctx-size 4096 \
    --gpu-layers 32 \
    --temp 0.8 \
    --top-p 0.8 \
    --top-k 50 \
    --seed 1024 \
    --color \
    --prompt '<|im_start|>system\nYou are a harmless AI assistant.<|im_end|>\n<|im_start|>system name=<|plugin|>[{"name": "generate_image", "description": "Generates an image based on the given text prompt", "parameters": {"type": "object", "properties": {"prompt": {"type": "string", "description": "The text prompt used to guide image generation"}}, "required": ["prompt"]}}]<|im_end|>\n' \
    --interactive \
    --multiline-input \
    --conversation \
    --verbose \
    --logdir ./logdir \
    --in-suffix "<|im_end|>\n<|im_start|>assistant\n" \
    --special

Here are the conversations

== Running in interactive mode. ==
 - Press Ctrl+C to interject at any time.
 - To return control to the AI, end your input with '\'.
 - To return control without starting a new line, end your input with '/'.

<s><|im_start|>system
You are a harmless AI assistant.<|im_end|>
<|im_start|>system name=<|plugin|>[{"name": "generate_image", "description": "Generates an image based on the given text prompt", "parameters": {"type": "object", "properties": {"prompt": {"type": "string", "description": "The text prompt used to guide image generation"}}, "required": ["prompt"]}}]<|im_end|>

> <|im_start|>user\nDraw a picture of a kitten.
I will call an image generation api to generate image<|action_start|><|plugin|>[{"name": "generate_image", "parameters": {"prompt": "A cute, fluffy kitten with big round eyes, sitting on a soft cushion, warm and cozy, pastel colors, impressionism, high resolution, captured on a DSLR camera, natural lighting, detailed fur texture."}}]<|im_end|>

As you can see, the special tokens are not missing.

2. llama.cpp does not support dynamic rope scaling

this is true and there is an open issue https://github.com/ggerganov/llama.cpp/issues/8361 . No response by far.