microsoft

/

Phi-4-mini-instruct

Like 840

Follow

Microsoft 22.2k

The chat template in tokenizer_config.json has a tool branch, but it
is gated on a per-message tools key:

{% if message['role'] == 'system' and 'tools' in message
     and message['tools'] is not none %}

Callers don't put tools on a message object; they pass tools=[...] to
apply_chat_template, which Jinja exposes as a top-level variable. The
template never reads that variable, so the branch is unreachable in
normal use and the prompt renders byte-identically with and without
tools. No error, no warning - the tool definitions are just silently
dropped. This hits transformers, llama.cpp --jinja, and vLLM alike,
and from the outside it looks like the model can't do tool calling at
all, when it's really a template bug.

Minimal repro, no model download needed (just jinja2):

import json, hashlib, urllib.request
from jinja2 import Template

URL = ("https://huggingface.co/microsoft/Phi-4-mini-instruct"
       "/resolve/main/tokenizer_config.json")
cfg = json.load(urllib.request.urlopen(URL))
tpl = Template(cfg["chat_template"])

msgs = [{"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is the weather in Paris?"}]
tools = [{"type": "function", "function": {
    "name": "get_weather",
    "description": "Get the current weather for a city.",
    "parameters": {"type": "object",
                   "properties": {"city": {"type": "string"}},
                   "required": ["city"]}}}]

without = tpl.render(messages=msgs, add_generation_prompt=True,
                     eos_token="<|endoftext|>")
with_   = tpl.render(messages=msgs, tools=tools,
                     add_generation_prompt=True,
                     eos_token="<|endoftext|>")

print("identical:", without == with_)
print("'get_weather' in prompt:", "get_weather" in with_)

Output (run 2026-09-02 against the currently published file, jinja2
3.1.2):

identical: True
'get_weather' in prompt: False

Both renders come out as
<|system|>You are a helpful assistant.<|end|><|user|>What is the weather in Paris?<|end|><|assistant|>
with the tool definition nowhere in it.

A downstream consequence worth knowing: llama.cpp detects tool support
by rendering the template with and without tools and comparing. Since
the renders are identical it reports supports_tools: false and drops
the tools array from every request, returning HTTP 200 as if nothing
happened.

Suggested fix: also read the top-level tools variable, e.g.

{% if message['role'] == 'system' and (tools is defined and tools is not none) %}
    {{ '<|system|>' + message['content'] + '<|tool|>' + (tools | tojson) + '<|/tool|>' + '<|end|>' }}
{% elif ... existing per-message path ... %}

The exact serialization is your call; the substantive change is that
the top-level variable gets consulted at all. One trap I hit writing a
replacement by hand: a template that only emits tools inside a system
turn still fails llama.cpp's probe (it probes with a bare user
message), so the fix should render tools even when no system message is
present.

The capability seems to be there behind the bug. Serving the model with
a substitute template that does read the top-level variable (Qwen2.5's,
a poor fit for Phi's tokenizer), it went from 0/9 to 4/9 on a nine-task
tool-use suite from an open-source agent harness (temur), with 419
native tool calls parsed by llama.cpp along the way. That's one run
with a mismatched template, so treat it as an existence proof, not a
score - the model has native <|tool|> and <|tool_call|> tokens and
a corrected bundled template should do better with them.

Still reproducible against the published tokenizer_config.json as of
2026-09-02.