froggeric

/

Qwen-Fixed-Chat-Templates

like 1.51k

Thank you for this template — it is the reason Qwen3.8 is usable at all on the
machine described below.

This is one more case of a class the README already claims: KV cache
invalidation and empty think poisoning. In the default non-thinking
configuration, a turn is rendered one way when it is generated and a different
way when it is replayed as history, so every request invalidates the entire
prompt cache.

Symptom

With enable_thinking=false, the generation prompt correctly ends with an empty
<think>\n\n</think>\n\n block. When that same assistant turn is re-rendered as
history on the next request, the block is absent. The new prompt is therefore
not a prefix of what was previously sent, and every prefix-cache implementation
discards everything — on every single turn.

_preserve_thinking defaults to true (line 13), so this is the default
path for non-thinking use, not an unusual configuration.

Reproduction at f64494d7 (v22.2)

Zero weights, tokenizer and template only. Render turn 1, append the assistant
reply exactly as emitted, render turn 2, and compare token sequences:

The seven tokens are exactly the empty think block.

Cause

chat_template.jinja line 277:

{%- if (_preserve_thinking or loop.index0 > ns.last_query_index) and reasoning_content %}

In non-thinking mode there is no reasoning_content, so the condition is false
and the block that was emitted at generation time is omitted on re-render.

Proposed fix

-        {%- if (_preserve_thinking or loop.index0 > ns.last_query_index) and reasoning_content %}
+        {%- if (_preserve_thinking or loop.index0 > ns.last_query_index) and (reasoning_content or not ns_state.thinking) %}

Re-emit the empty block when the template is in non-thinking mode, so history
matches what was generated.

Verification at the same revision

Your suite passing both before and after is worth flagging: test 44, "Empty
think poisoning prevention regression test", passes in both states, so the
existing coverage does not reach this case. It checks that a spurious block is
not added; this defect is a block being dropped on re-render. I am happy to
contribute a prefix-invariance test if you would like one — the shape is
"render, append the emitted reply, re-render, assert token-prefix".

Measured impact

On an Apple M5 / 32 GiB running Qwen3.8-27B in non-thinking mode, the lost
prefix meant 2.83–3.84× more prompt tokens processed. On one editing task
the engine read 29,951 prompt tokens instead of 7,794. With the fix, exact
prefix reuse hit on every eligible turn and the penalty disappeared.

Known limitation of this fix

ns_state.thinking is global to a render, not per message. If a session
switches modes mid-conversation (for example an inline <|think_on|>), history
re-renders under the new mode and the cache is invalidated once, at the
switch. Per-message thinking state would be the complete fix; this is the
minimal correctness change and does not make that case worse.

Verified independently under minja

llama.cpp's Jinja implementation renders the fix identically —
/apply-template plus /tokenize on a live llama-server shows the empty
block preserved on re-render, zero missing prefix tokens, and the token prefix
holding across a tool-call turn.