hunterbown/trainless-toolcall-2b

Trainless Tool Call 2B: call-when-a-tool-fits adapter

A rank-one adapter for Qwen3.5-2B that makes the model call a provided function more often when one fits, instead of answering in text. It changes one MLP writer (layer 10) and was made without gradient training. The adapter needs its base model; it is not a complete checkpoint.

On 703 held-out cases from the Berkeley Function-Calling Leaderboard (BFCL v4), standalone unquantized vLLM scored 74.8% base → 78.5% with this adapter, with greedy decoding and no system prompt. On a second set of 1,977 cases, where almost half the requests fit no tool, it scored 1.6 points below the base (69.0% against 70.7%). Read the limits below before you use it.

Arm	Correct	Wilson 95% interval	Relevant cases with no call	Irrelevant cases with a call
Base	526/703 (74.8%)	71.5–77.9%	84 of 596	18 of 107
Base + a strong tool-use system prompt	544/703 (77.4%)	74.1–80.3%	10	58
Base + this adapter, no system prompt	552/703 (78.5%)	75.3–81.4%	46	23

Paired improvement over the base: 3.7 percentage points, bootstrap 95% interval 1.7 to 5.7 points. The adapter fixed 38 cases the base got wrong and lost 12 it got right.

Read this before you use it

• It can make things worse when most requests have no matching tool. On four BFCL categories that were never used to build or test it (1,977 cases, 45% of them requests where no function fits), it scored 69.0% against 70.7% for the base: 1.6 points lower (95% interval −2.4 to −0.8). In that set the base almost never answers in text when a function fits, so there is little to fix, and the adapter adds unwanted calls (273 → 300 of 884). It was still 3.7 points better than the strong prompt there. Use it when your problem is the model answering in text instead of calling; measure before using it where many requests should get no call. Details are below.

• It does not beat a good system prompt on accuracy. Against the strong prompt the difference is +1.1 points with a 95% interval of −1.7 to +4.1, which is a tie. If a system prompt already works for you, you may not need this.

• What it does differently from the prompt. The prompt gets nearly every relevant call made, but it also calls a function on 58 of 107 requests where none fits. The adapter makes 23 such calls; the base makes 18.

• It is worse than the prompt on parallel calls (53 vs 72 of 86 correct) and it lost 5 live_simple cases against the base (85 vs 90 of 125).

• It still makes 5 more unwanted calls than the base on the 107 no-tool cases.

• It does not make the model better at using tools. 35 of the 38 fixes are cases where the base made no call at all. Where the base already called a function, accuracy is unchanged (437 vs 433 of 512), and wrong-argument errors are the same.

• Scope. One model, one benchmark, six single-turn BFCL categories (simple_python, multiple, parallel, parallel_multiple, live_simple, irrelevance), thinking off, 1,024-token output cap (7 outputs hit it in the base and adapter arms). Multi-turn use, other languages, other tool formats and quantized builds are untested.

Category (n)	Base	Strong prompt	Adapter
simple_python (194)	149	171	167
multiple (102)	88	86	88
parallel (86)	40	72	53
parallel_multiple (89)	70	74	75
live_simple (125)	90	92	85
irrelevance (107)	89	49	84

How it was made and checked

Who did the work. An AI coding agent, Claude Fable 5.2 running in Claude Code, designed and ran every step below: it chose the direction, wrote the scripts, ran the screens and tests, exported the adapter and wrote this card. Hunter Bown set the goal (repair one tool-calling failure, then ship or stop) and approved each stage, including this release. The agent worked through scripts on a GPU host and the Trainless platform; it did not read raw weights unaided.

The 1,498 BFCL cases were split by sha256(category/id) mod 4 before anything was built: 0 for construction, 1 for development, 2 and 3 for the test above.

1. Direction. On construction cases only, at the residual stream after layer 10 and the last prompt position: the mean over relevant cases the base answered in text, minus the mean over relevant cases where it called a function.
2. Edit. Remove that direction from what layer 10's mlp.down_proj writes, at strength 4: W' = W − 4·u(uᵀW). No gradients, no optimizer, no fitting to a loss.
3. Selection. Layer and strength were chosen on the development cases. (The decision to target this behavior came earlier, from the failure breakdown of a baseline run over all 1,498 cases, test cases included.)
4. Test. The candidate and its pass criterion were written down before any test case was read. It passed: the gain over the base has a 95% interval above zero.
5. Controls. Three random directions at the same layer and strength scored 510, 522 and 522 of 703, against 554 for the edit and 524 for the base. (These five numbers are from a Transformers run of the bf16 edit, not from this adapter in vLLM.)
6. Refusal screen before export. On the first 64 AdvBench requests, the base refused 63 and the edited model refused 63 (greedy, 256 tokens, refusal-phrase grader). This is a phrase screen, not a safety evaluation.

The test cases were read three times in all (the test, the controls, and this adapter's vLLM run). Nothing was tuned on them.

Does it change ordinary answers?

Checked in vLLM with no tool definitions, thinking off, greedy decoding and a 4,096-token cap:

Benchmark	n	Base	Adapter	Paired difference (95% interval)
GSM8K test, first 500	500	407	399	−1.6 points (−4.4 to +1.2)
MATH-500, first 200	200	143	144	+0.5 points (−4.5 to +5.5)

Neither arm wrote a tool call when no tools were offered. No harm was detected, but a loss of up to about 4 points is not ruled out. Instruction following and thinking-on behavior were not checked.

On categories it was never tested on

Added November 11, after release. BFCL's live_irrelevance (884 cases), live_multiple (1,052), live_parallel (15) and live_parallel_multiple (23), all cases, same engine and settings as the main test:

Arm	Correct	Relevant correct (of 1,093)	Relevant cases with no call	Irrelevant cases with a call (of 884)
Base	1,397 (70.7%)	786	21	273
Base + strong system prompt	1,292 (65.4%)	779	10	371
Base + this adapter	1,365 (69.0%)	781	16	300

Adapter minus base: −1.6 points (95% interval −2.4 to −0.8). Adapter minus strong prompt: +3.7 points (+2.2 to +5.3). The headline result above holds for a workload like its test set, where 14% of tool-fitting requests were answered in text and 15% of requests fit no tool. It reverses here, where 2% and 45% do.

Inspect every response

evaluation/summary.csv lists all 703 cases with each arm's grade and whether it made a call, marked as 38 fixed, 12 regressed and 653 unchanged against the base. evaluation/responses.jsonl.gz has the complete output of all three arms (2,109 generations) with the parsed calls and the grader's error. Cases are identified by BFCL category and id; the questions and answer keys are in the BFCL repository at commit 6ea57973c7a6097fd7c5915698c54c17c5b1b6c8 and are not copied here.

Load the adapter

Use AutoModelForImageTextToText, which keeps Qwen3.5's model.language_model module paths. The Hub's generated PEFT shortcut uses AutoModelForCausalLM; use the example below instead.

python -m pip install torch transformers "peft==0.21.2" accelerate safetensors

from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoTokenizer

base_id = "Qwen/Qwen3.5-2B"
adapter = "hunterbown/trainless-toolcall-2b"  # Or an extracted local folder.
base = AutoModelForImageTextToText.from_pretrained(base_id, dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(base, adapter)
tokenizer = AutoTokenizer.from_pretrained(base_id)

tools = [{"type": "function", "function": {
    "name": "get_weather",
    "description": "Get the current weather for a city.",
    "parameters": {"type": "object",
                   "properties": {"city": {"type": "string", "description": "City name."}},
                   "required": ["city"]},
}}]
messages = [{"role": "user", "content": "What's the weather in Lisbon right now?"}]
inputs = tokenizer.apply_chat_template(
    messages, tools=tools, tokenize=True, add_generation_prompt=True,
    enable_thinking=False, return_tensors="pt", return_dict=True,
).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=1024, do_sample=False)
print(tokenizer.decode(outputs[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

This is a loading example. Loading was checked on an H100 with Torch 2.7.0, Transformers 5.20.2 and PEFT 0.21.2: the adapter attached to layers.10.mlp.down_proj and changed the output. The example's own output was not graded. The measured run used vLLM 0.31.0 with enable_lora, Torch 2.13.0, Transformers 5.17.0, bf16, thinking off, temperature 0 and a 1,024-token cap. Results can change with another runtime or decoding setup.

In vLLM, serve the base with --enable-lora and pass a local copy of this folder to --lora-modules. The measured run loaded the adapter from a local folder through vLLM's Python API (LoRARequest); the server command line was not tested for this adapter.

Exact version

Base: Qwen/Qwen3.5-2B. The weights file used has SHA256 aa33250c4fc64891ddfaba3a314fd9542ea371843c387178b425fbcc5ed680b1 (4,548,221,488 bytes).

Adapter weights SHA256: 32a8d4336a298a3329c033be4c4c38c9d15afabd4a3eee4bbafd76b9db9ec808.

Configuration SHA256: 70c7c9a027f88e37d92d66c7058a43708a481ef7486791b19f5ee8e402712346.

These are the exact files measured above. trainless_model.json holds the aggregate measurements, the edit's ledger identifiers and the refusal screen. export_report.json describes the factorization: the adapter reproduces the edited bf16 tensor's change to within 1.64% relative error, which is that tensor's rounding. The 78.5% score belongs to this adapter, not to a merged checkpoint.

No quantized, GGUF or MLX build of this adapter has been tested.

The base model's Apache 2.0 license is included unchanged.

Research basis

The forward-only editing approach builds on Sparks of Weight-Space Programming by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab (MIT CSAIL, November 2, 2026). This release is Trainless's own direction and edit for tool calling; its measurements are the ones reported above. The benchmark is the Berkeley Function-Calling Leaderboard.