froggeric

/

Qwen-Fixed-Chat-Templates

like 1.43k

Summary

README-only. Corrects the vLLM quick-install section and documents which thinking-off channels vLLM's qwen3 reasoning parser can follow. Everything below was checked against vLLM main at commit 5ee84d3c (2026-08-21) and against live renders through the transformers Jinja environment.

What changes

1. Launch command. --chat-template /path/to/chat_template.jinja replaces the embedded template; editing tokenizer_config.json is not needed. The old command omitted --enable-auto-tool-choice (required; vLLM rejects tool_choice="auto" without it, entrypoints/openai/cli_args.py) and --reasoning-parser qwen3 (without it the <think> block is returned inside content). --reasoning-parser qwen3 is vLLM's counterpart to llama.cpp's --reasoning-format deepseek, which the README already recommends.
2. Parser names. On current vLLM main, qwen3_xml and qwen3_coder resolve to the same parser class (vllm/tool_parsers/__init__.py). The text now says so instead of "older builds use qwen3_coder" only.
3. Reasoning effort. vLLM forwards the top-level OpenAI reasoning_effort field into the template (chat_completion/protocol.py, build_chat_params), and it takes precedence over chat_template_kwargs.reasoning_effort. --default-chat-template-kwargs '{"reasoning_effort": "xhigh"}' restores the official default server-wide.
4. Thinking off on vLLM (discussion #74). vLLM's Qwen3Parser decides its start state only from chat_template_kwargs["enable_thinking"] (vllm/parser/qwen3.py); it does not look at the prompt. When the template closes the think block by another channel — inline <|think_off|>, auto_disable_thinking_with_tools, or a chat_template_kwargs-only reasoning_effort of none/off — the model never emits </think>, the parser stays in reasoning state, and the answer comes back as reasoning with content: null. With --reasoning-parser qwen3 --tool-call-parser qwen3_xml vLLM uses the single Qwen3 engine for both streaming and non-streaming, so both are affected. The two channels that work are chat_template_kwargs.enable_thinking=false and a top-level reasoning_effort: "none" (vLLM derives enable_thinking=false from it). The inline-tag section gets a one-line pointer to this.

This is documentation for a limit the template cannot work around; the parser-side fix belongs in vLLM.