remove "commentary" from the valid channel list when the request contains no tools
The default jinja chat template includes "commentary" as a valid output channel even when no tools are present in the request. This one template mismatch breaks the model on nearly every non-tool-call task. Removing "commentary" from valid channels when no tools are defined fixes the problem. The accuracy gains are dramatic.
On the chat API with fc_model=0 (BFCL_v4 tool calling benchmark, AST parsing, no tools in request):
| Test | Default Jinja | Fixed Jinja | Change |
|---|---|---|---|
| live_relevance | 3.1% | 41.4% | +38.3% |
| live_simple | 3.1% | 36.7% | +33.7% |
| simple_java | 18.0% | 60.5% | +42.5% |
| simple_javascript | 13.0% | 56.5% | +43.5% |
| simple_python | 1.8% | 36.6% | +34.8% |
| multi_turn_base | 0.2% | 14.6% | +14.3% |
| irrelevance | 99.9% | 93.8% | -6.1% |
The one metric that drops is irrelevance (99.9% to 93.8%). With the fixed template, the model actually tries to answer questions instead of producing silent failures. The irrelevance test marks those attempts as wrong, so the score goes down even though the system is doing more.
Removing one channel from the chat template moves accuracy from 3% to 40% on live tests. This is not a minor tuning adjustment. The default template shipped with the inference stack was making the model non-functional for basic tasks. Mismatches between training structure and inference template produce silent failures that look like model incompetence.
There is a problem I can see here - cached prefix invalidation. Imagine if you had some messages that did not use tools, and then one uses it (or vice versa) - the very beginning of the prefix changes, invalidating the whole cache.
Edit:
Though, on the second thought, the same would have happened before when Calls to these tools must go to the commentary channel: 'functions'." is not rendered in the system message, so nothing new would happen after this has merged.