brain-v0.1-base

A small model fine-tuned to use MCP tools: pick the right tool, fill in its arguments, read the results, recover from errors, ask before acting when something is missing, and answer in the user's language. It is Qwen/Qwen3-1.7B plus a LoRA adapter, trained only on synthetic data from forge (a plan-driven dataset generator, "Run A": forge-only, no distillation from a bigger model). It is the baseline for later versions.

Repository contents:

path	what it is
adapter/	LoRA adapter (r=64, ~280 MB). Apply it on top of Qwen/Qwen3-1.7B.
forge-A_qwen3-1.7b_json-q8_0.gguf	Adapter merged into the base model, converted to GGUF q8_0 (~1.8 GB), for llama.cpp / Ollama.
template, params, Modelfile	Raw-ChatML template + stop token/temperature for Ollama (see Ollama / llama.cpp below; required, otherwise the answer lands inside Thinking...).

Results

Execution-based evaluation (forge evalh): each episode runs against real SQLite-backed mock MCP servers and is scored on the final state of the world (end-state), not on text similarity. Same 100 episodes, same prompt format as training, greedy decoding, base vs. this adapter:

metric	base Qwen3-1.7B	this model
end-state (headline)	54.0 % (CI 44.3–63.4)	83.0 % (CI 74.5–89.1)
tool selection	69.6 %	93.7 %
exact arguments	59.5 %	83.5 %
parseable output	96.4 %	100 %
recovery after an error	25.0 %	80.0 %
asks first when information is missing	38.5 %	76.9 %
answers in the user's language	72.6 %	100 %
hallucinated tool (lower is better)	4.6 %	1.3 %
mean latency per episode	27.4 s	7.8 s

The +29-point gain in end-state is significant (net +29 over 100 paired episodes → McNemar p < 0.004).

By segment (end-state, base → this model): servers seen while building the data 56.5 → 88.7 %; held-out servers 50.0 → 73.7 %; English 65.2 → 91.3 %; Spanish 45.2 → 78.6 % (Rioplatense voseo 47.2 → 72.2 %); mixed-language 41.7 → 66.7 %.

Known weaknesses

• Over-claiming in the final message (stating more than the tool results support): 40 % of the evaluable episodes. This matches a known defect in the training data (final messages with partial or understated counts).
• Date arguments (all 4 episodes involving dates failed), parallel calls with 3+ tools, and multi-write sequences are weak (very few episodes each, so these are signals, not measurements).
• Held-out servers are ~15 points below seen servers: expect a drop on tools unlike the ones in training.

Caveats on the numbers

n = 100 (±~7 points). The evaluation episodes come from the same generator and templates as the training data; only the held-out-server split is out of distribution. It has not been tested against real MCP servers yet, nor with other prompt formats or tool-calling templates.

Training

• Base: Qwen/Qwen3-1.7B, weights in fp16.
• Method: LoRA, r = 64, alpha = 64, dropout 0, on all linear layers (69.7 M trainable parameters, 3.9 %).
• Data: 4,716 synthetic multi-turn examples (8.3 M tokens; only the assistant turns, 4.1 % of the tokens, are in the loss). Dataset not public (private repo lautiss/forge-sft).
• Schedule: 1 epoch = 295 optimizer steps, effective batch 16 (1 × 16 gradient accumulation), lr 1e-4 with cosine decay, max length 6,144 tokens (longer examples dropped, never truncated), seed 3407. Final training loss ≈ 0.20, no NaNs.
• Hardware / stack: one free Colab T4 (fp16), transformers 5.17, peft; plain Trainer with a custom loss that computes logits only at trained positions. The run was interrupted once and resumed from the step-250 checkpoint.
• Output format: tool calls are emitted as JSON in a raw ChatML conversation (<|im_start|> / <|im_end|>), exactly as in training. This is not the Qwen3 native tool-calling template.

About the data (forge)

Examples are generated from plans and validated by replaying them on real SQLite engines behind mock MCP servers; an LLM only rewrites the user-facing text under automatic gates, and every example then passes a judge. Languages: English, Spanish (Rioplatense voseo, neutral LatAm, peninsular) and Spanglish. Relative dates are anchored with a system line ("Today is …").

How to use

Use the same raw ChatML prompt layout and tool-schema serialization as in training, otherwise quality drops. The reference implementation is train/hf_raw_adapter.py and train/sft_tokenize.py in the forge repository (greedy decoding, stop at <|im_end|>).

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-1.7B")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-1.7B", dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, "lautiss/brain-v0.1-base", subfolder="adapter")

Ollama / llama.cpp

The GGUF embeds the original Qwen3 chat template (with <think> support). That template is not the one this model was trained with: if you run it as is (ollama run hf.co/lautiss/brain-v0.1-base:Q8_0 without the template file below), Ollama treats the output as "thinking" and the whole answer shows up inside the Thinking... block with no visible reply. The model never emits <think>; it was trained on plain ChatML. Use the raw ChatML template instead. Two options:

1. From this repo (the template and params files are picked up by Ollama): ollama run hf.co/lautiss/brain-v0.1-base:Q8_0.
2. Local Modelfile (always works): download the .gguf, put the Modelfile from this repo next to it and run ollama create forge-sft -f Modelfile, then ollama run forge-sft.

Template (Ollama / Go syntax): {{- range .Messages }}<|im_start|>{{ .Role }}\n{{ .Content }}<|im_end|>\n{{ end }}<|im_start|>assistant\n; stop token <|im_end|>; temperature 0. There is no tool role: tool results go in as user turns. A plain "Hola" works in chat, but for tool use you must serialize the tool schemas and results exactly as in training (see above).

Intended use and limits

For research and prototyping of small tool-using agents (especially for Spanish-speaking users). Do not give it irreversible or high-impact actions without confirmation and verification of the tool results: it can still pick wrong arguments (about 1 in 6 calls had inexact arguments in the evaluation) and can over-state what a tool returned. It was trained for one tool-call format and small tool catalogs; it has no dedicated safety tuning beyond what is in the synthetic data.

License

Apache-2.0, inherited from the base model. The training data is synthetic and was not built from third-party corpora.

---

Resumen en español

Qwen3-1.7B + LoRA entrenado (1 época, 4.716 ejemplos sintéticos de forge, en inglés, español rioplatense/neutro/peninsular y spanglish) para usar herramientas MCP. En 100 episodios de evaluación por ejecución, el estado final correcto sube de 54 % (modelo base) a 83 %; en servidores nunca vistos llega a 73,7 %. Puntos débiles: afirma de más en el mensaje final (40 %), fechas y llamadas en paralelo. Todavía no se probó con servidores MCP reales. Incluye el adaptador (adapter/) y el GGUF q8_0 para Ollama.