Fix KV cache reuse when changing reasoning_effort

#95
by dormosh - opened

The original template put reasoning_instructions in the system prompt, so changing reasoning_effort altered the prompt prefix and prevented llama.cpp from reusing the KV cache.

The fix moves these dynamic instructions to the final user message, keeping the prefix unchanged and allowing llama.cpp to reuse the KV cache instead of re-evaluating the conversation.

Thanks for the PR @dormosh . The problem with putting the instructions inside the user turn on loop.last is that it actually breaks prefix caching in multi-turn chat: turn 1 has the text appended, but once you move to turn 2, turn 1 is no longer loop.last so its tokens change and the cache is lost. Also, Qwen 3.8 was trained expecting these reasoning tags in the system prompt. Going to close this one, but thanks for taking a look at it!

froggeric changed pull request status to closed

Sign up or log in to comment