shivam909067/aroxa-4b-agentic

Aroxa-4B — a small model taught one thing: emit a tool call the runtime can run

A LoRA fine-tune of Qwen3.5-4B on 3,832 agentic trajectories. It was trained for one job — driving a tool-using agent loop — and it is measurably good at that job and unremarkable at everything else.

Shipped as a 4.3 GB Q8_0 GGUF that runs on a laptop.

What the numbers actually say

CORE v2, an internal 16-task × 3-trial agentic suite (n=48 per arm). All three artifacts scored on the same suite, because a gate that runs the adapter while the product runs the merge is how you ship a model that is secretly its own base.

artifact	task success	composite
Qwen3.5-4B (untouched base)	12/48	0.325
this fine-tune, LoRA in bf16	41/48	0.839
this fine-tune, merged Q8_0 GGUF	40/48	0.738

40 vs 41 is inside the measured ±1-trial variance floor, so the merge and the 8-bit quantisation both survive.

What it learned, and what it did not

dimension	base	tuned	Q8 GGUF
schema validity	0.112	1.000	0.985
grounding	0.500	1.000	1.000
recovery	0.000	0.500	0.500
long horizon	0.000	1.000	0.500
resume	0.000	1.000	0.333
tool selection	0.590	0.590	0.487
restraint	1.000	0.889	1.000

Read the last two rows before the first one.

Tool-selection accuracy is unchanged at 0.590. The fine-tune taught the model the format of a tool call, not which tool to pick. The enormous jump in schema validity (0.112 → 1.000) is the whole story: the base model rarely emits a call this runtime can parse, and this model almost always does. Everything downstream — long horizon, resume — follows mechanically, because a run that cannot emit a parseable call cannot finish a long task.

Restraint got slightly worse (1.000 → 0.889 for the adapter). A model taught to reach for tools reaches for them slightly too often.

CORE v2 is 16 tasks × 3 trials, so a per-dimension rate of 0.5 or 0.333 rests on one or two tasks and sits on the variance floor. The 48-trial task-success number is the only figure here with real n behind it.

Do not use a Q4 quantisation of this model

This is the most useful thing in this card.

A LoRA delta has relative magnitude ρ = ‖ΔW‖/‖W‖. For this adapter ρ = 0.01326. Folding that into the weights and then quantising destroys most of it, and the result looks like a working model while being statistically the base model. Measured on the real tensors, as cosine between the intended delta and what survives a round trip:

GGUF format	size	direction retained
f16	8.1 GB	1.0008	fine
Q8_0	4.3 GB	0.8465	what is published here
Q6_K	~3.5 GB	0.4962	below a 0.50 floor
Q4_K_M	~2.5 GB	0.2255	this is the base model

Q4_K_M is the most-downloaded GGUF format, and for this model it would discard 78% of the fine-tune while still scoring plausibly on generic coding prompts. We know because a previous 30B version of this project shipped exactly that: a merged Q4 that scored 22/48 against its base's 20/48, and nobody noticed for months, because the evaluation ran the adapter and the release ran the merge.

If you quantise this yourself, stop at Q8_0.

Limitations, plainly

• It does not pick better tools than its base. 0.590 both arms.
• General coding is not improved. On the 30B version of this work the fine-tune scored below its own base on HumanEval under the chat protocol (85.4% vs 87.2%). Expect the same trade here: this is tool-protocol training, not coding training.
• Long-horizon agentic work is weak. The larger 30B version of this project measured ~20% success on extended multi-file tasks against a 75% target. Frontier models score 57–74% on comparable agentic benchmarks. A 4B is not going to beat them.
• Not evaluated for safety, bias, or non-English use. Zero hard violations on the internal suite is not a safety evaluation.
• The suite is internal and unaudited. Treat every number here as what we measured, not as a leaderboard claim.

Use

ollama create aroxa-4b -f Modelfile   # FROM ./aroxa-4b-q8_0.gguf
ollama run aroxa-4b

The model expects the Qwen3-Coder XML tool-call format, which is what its chat template renders:

<tool_call><function=create_file><parameter=path>a.py</parameter></function></tool_call>

Every training row uses that form, so JSON-style tool calls are out of distribution for this model. We did not measure how it behaves when prompted for them.

Training

base	Qwen/Qwen3.5-4B (dense, 32 layers)
method	LoRA r=32, α=64, attention + MLP, bf16 — not QLoRA
trainable	42.5M of 4.58B (0.93%)
data	3,832 rows, leakage-checked (see composition below)
schedule	1 epoch, 226 steps, lr 1e-4, β₂ 0.95, weight decay 0.1
hardware	one RTX 5090, 54 minutes

Data composition

Stated because "agentic trajectories" invites the assumption that all of it is execution-verified agent runs. It is not. Proportions are of the full 4,805-row release, of which 3,832 rows are the training split:

source	rows	share
aroxa-runtime (our own agent runs)	2,836	59.0%
Salesforce/xlam-function-calling-60k	1,497	31.2%
hand-authored templates	463	9.6%
SWE-Gym / SWE-smith	9	0.2%

Leakage check on the release: zero identity overlap between splits, zero crossing near-duplicate groups.

Roughly a third of this model's training signal is xLAM function-calling data, not executed agent trajectories. That is very likely part of why tool-call format improved enormously while tool selection did not move at all.

bf16 rather than QLoRA is deliberate. A QLoRA adapter is bound to the exact quantisation it trained against; merging and re-quantising it retained only 13% on the 30B version of this project. Training in bf16 is what makes a clean merge possible at all.

Attribution

Base model: Qwen/Qwen3.5-4B, Apache-2.0.

31.2% of the training data comes from Salesforce/xlam-function-calling-60k, licensed CC-BY-4.0. If you build on this model, please cite APIGen:

APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets, Salesforce AI Research.

Smaller contributions from SWE-Gym and SWE-smith (9 rows combined). Full third-party attribution is in THIRD_PARTY_NOTICES.md.