vinmaynair/whetherman-functiongemma-270m

Whetherman (FunctionGemma-270M LoRA + 4-class head)

Research artifact. A 270M router that forecasts, before each model call that offers tools, whether an agent needs a tool: ACT, ANSWER, ASK or DECLINE. It is the backbone of the Whetherman proxy, which removes tools only when a non-ACT decision is confident enough for the specific agent, and otherwise fails open to ACT.

Read the results before using it. On held-out end-to-end tasks the gate did not produce a net benefit for the large agent we measured (details below). It is published so the measurements can be checked and the method reused, not as a production cost-saver.

What is in this repository

File	What it is
adapter_model.safetensors, adapter_config.json	LoRA adapter (r=16, alpha=32, all attention and MLP projections) for google/functiongemma-270m-it
head.pt	4-class linear head on the last-token hidden state (ACT, ANSWER, ASK, DECLINE)
profiles/L.json, profiles/S.json, profiles/U.json	Per-agent profiles: a regularized logistic layer on the hidden state plus threshold τ, fitted by whetherman calibrate
GEMMA_TERMS.md	Gemma Terms of Use notice

Agents: L gpt-oss-120b (DigitalOcean serverless), S Qwen3.5-4B Q4_K_M (local Ollama), U gpt-oss-20b (DigitalOcean), calibrated as a new agent after the backbone was trained. Profiles describe one agent on one serving stack on one day; recalibrate for yours.

Usage

Use it through the proxy (see the GitHub README):

git clone https://github.com/vinmay/whetherman && cd whetherman && uv sync --extra ml
hf download vinmaynair/whetherman-functiongemma-270m --local-dir checkpoints/shared-v3
uv run whetherman calibrate --agent-endpoint http://localhost:11434/v1 --model <your-model> --name my-agent
uv run whetherman serve --profile my-agent --upstream http://localhost:11434/v1

whetherman calibrate measures your agent on a probe pack and fits a profile; whetherman serve gates your agent's Chat Completions requests. You need access to the gated base model (accept the Gemma terms on Hugging Face).

Training

• Base: google/functiongemma-270m-it (revision 39eccb09), LoRA + linear head, FP32 on a 6 GB GTX 1660 Ti, 3 epochs, about 50 minutes. No agent identity in the input.
• Data: When2Call training files (ACT/ASK/DECLINE; ASK vs DECLINE by a documented heuristic), templated multi-step decision points with real tool execution, and When2Tool train tasks labeled by measuring each agent with and without tools (agreement items as hard labels, disagreements as soft targets).
• Deduplicated against every evaluation split (text hash + embedding similarity > 0.95).

Results (self-reported; frozen model, held-out sets run once)

Everything below was measured on held-out sets, run once after the backbone, profiles and code were frozen (eval/results/freeze.json, tag freeze-v3). Full tables with every configuration, CI and set: eval/results/REPORT_final.md, generated by eval/report.py from the raw result files.

End to end: no gate vs Whetherman

80 held-out tasks (40 multi-step) × 2 repeats per agent. "Headline" excludes 10 guessable two-choice episodes (DATA.md). CIs are Wilson 95%.

Agent	No gate	Whetherman	Tool calls / episode	Tokens / episode	Cost (whole suite)
L: gpt-oss-120b (DigitalOcean)	120/150 [0.73, 0.86]	105/150 [0.62, 0.77]	2.36 → 2.15	1906 → 1897	$0.0522 → $0.0651
S: Qwen3.5-4B (local Ollama)	130/150 [0.80, 0.91]	128/150 [0.79, 0.90]	1.92 → 1.91	2297 → 2156	local
U: gpt-oss-20b (unseen agent)	115/150 [0.69, 0.83]	114/150 [0.69, 0.82]	2.09 → 1.93	2062 → 1824	$0.0461 → $0.0438

Other gates in the same runs (headline success): the shared backbone without a profile 99 (L) and 84 (S); a prompted Qwen3.5-4B router 108 and 96; a rules-only control 90 and 60. Every gate that removed many calls lost many successes.

What went wrong for L (post-hoc analysis of the frozen run, not used to change anything):

• 11 of the 15 lost successes have replay evidence of a false skip: restoring ACT at the skipped step made the episode succeed. 7 of those are multi-hop retrieval tasks, where the gate answered after the first lookup. Every probe task is a first-step decision (the pack has 9 single-hop retrieval tasks and no later steps), so L's profile never saw "one lookup done, another needed".
• Removing tools made gpt-oss write longer answers (completion tokens 36k → 58k over the suite; output costs 7× input), and it called removed tools 20 times, each costing a second request. Fewer calls did not mean fewer dollars.
• Where L is genuinely capable (calculator, code, matrix and prime tasks), calls fell from 18 to 6 with no lost success.

Per-agent calibration matters

Offline, on held-out When2Tool tasks labeled by measuring each agent (false skip = tools withheld from a point that needed them):

	Shared, no profile	Own profile (shipped)	Other agent's profile
S false skips	65/70	4/70	42/70
L false skips	13/17	1/17	3/17

End to end, giving S the profile fitted for L cost 32 of 150 successes (98 vs 130). The same request can be ANSWER for one agent and ACT for another; one global threshold cannot serve both.

The shipped profiles are conservative: on the same held-out sets they skipped 86 of L's 129 and 23 of S's 66 unnecessary calls, and on hand-written conversations (E3) only 19/62 and 14/60. A comparison model with the agent's ID in its training input skipped far more (125/129 for L) with several times the false skips (6/17; S 11/70).

Agent-agnostic routing (When2Call, BFCL)

On 900 held-out When2Call MCQ items the shared backbone reaches macro-F1 0.642, against 0.519 for untuned FunctionGemma, 0.281 for the prompted Qwen router and 0.197 for rules. Its validation score was 0.925: the official test distribution is harder than our training-derived validation split.

Self-run with external code, tuned backbone only (baselines are compared on the 900-item subset above):

Benchmark	Result
When2Call official MCQ test, 3,652 items (NVIDIA/When2Call @ ecc8d42, official metric code)	accuracy 0.628, macro-F1 0.472*, tool hallucination rate 0.070
BFCL v4 irrelevance (no function fits; pass = not ACT)	187/240 (0.78); prompted Qwen 95/240
BFCL v4 live irrelevance	764/884 (0.86); prompted Qwen 533/884
BFCL v4 live relevance (pass = ACT)	6/16 (0.38); prompted Qwen 10/16

*The router's class is mapped to the matching MCQ option. The official macro-F1 averages over every class seen in gold or predictions; "direct answer" is never gold but was predicted 6 times, adding a zero. Over the three gold classes it is 0.629. BFCL is a self-run out-of-distribution check, not a leaderboard result; no BFCL question overlaps our training text.

Cost of running Whetherman

	GPU (GTX 1660 Ti) p50 / p95	CPU p50 / p95
Short request, 2 tools (128 tokens)	55 / 105 ms	188 / 204 ms
8 tools (830 tokens)	122 / 127 ms	1075 / 1203 ms
Long history (1504 tokens)	249 / 410 ms	2152 / 3434 ms

Calibration: L $0.24 (83 min of agent time), U $0.35 (154 min), S 19 min locally, about 120 usable probe labels each. At the measured savings, U's calibration pays back after roughly 24,000 episodes; L's does not pay back. Total hosted spend for the whole project: see eval/results/spend.json.

Limitations

• Two to three agents, measured once each at temperature 0. Labels are measured evidence, not ground truth.
• Small denominators; confidence intervals are wide.
• Probe packs do not cover multi-hop retrieval; the gate stops such tasks one lookup early.
• Removing tools can make a model write longer answers, which can erase dollar savings.
• gpt-oss sometimes calls tools that were removed; the proxy re-sends the original request.
• Not a security guardrail. It decides whether a tool call is useful, not whether it is allowed.

License and terms

The adapter is a derivative of Gemma and is distributed under the Gemma Terms of Use, including the Prohibited Use Policy. Modification notice: this repository contains LoRA weights and a classification head trained on top of FunctionGemma-270M by the Whetherman project; it changes the model's task from text generation to 4-way classification. Training data: When2Call (CC-BY-4.0, NVIDIA), When2Tool (MIT), and hand-written data (Apache-2.0). Code: Apache-2.0 at the GitHub repository.