khursanirevo/smolagent-ms-800m-agent

SmolAgent-MS 800M — Agent (v1)

Malay-first 800M agent LLM trained from scratch: 8B tokens Malay-first CPT → 2B tokens tool/EN/code CPT → SFT on synthetic tool-calling data.

Results

BFCL-MS (held out, 1,240 examples, single-call categories):

Category	Acc
Irrelevance (no-call)	0.888
Simple	0.005
Multiple	0.005
Parallel / parallel-multiple	0.0
Overall	0.174

Base evals (step 61,000 base / after tool-CPT):

Benchmark	Base	+tool-CPT
Heldout ppl	8.12	7.77
PIQA (n=500)	0.562	0.578
Belebele-ms (n=500)	0.26	0.274
HellaSwag (n=500)	0.204	0.210

English near-chance is by design: Malay-first corpus; English entered late and small (PRD §G008).

What works

• Coherent multi-register Malay generation
• Tool-call format: routes to <|tool_call|> and emits {"name": ..., "arguments": {...}} with schema-shaped keys
• No-call discipline: 0.89 on irrelevant requests

Known limitations (measured, v1)

• Tool selection is weak: on unseen BFCL schemas only 62/400 simple examples use an offered tool name; arguments frequently mismatch. This is a synthetic-data ceiling (no teacher model; 150 generated schemas), not an eval artifact — the harness was verified end-to-end (all outputs parse, train/eval tokenizations byte-identical).
• DPO was harmful in all 5 variants (margins run away, args collapse); dropped from v1. G012 documented as attempted.
• Agent-trajectory SFT was harmful (8k near-identical templates overwrote schema behavior with one memorized script); dropped.
• Multi-turn and parallel calling: not trained beyond format.

Serving (vLLM)

Custom architecture registration required (interleaved RoPE — no weight permutation converts it to Llama's NeoX style):

import sys; sys.path.insert(0, "repo-root")
from vllm import LLM, SamplingParams
from vllm.model_executor.models.registry import ModelRegistry
from src.model.vllm_interleaved import InterleavedRopeLlamaForCausalLM

ModelRegistry.register_model("InterleavedRopeLlama",
                             InterleavedRopeLlamaForCausalLM)
llm = LLM(model="smolagent-ms-800m-agent", max_model_len=4096,
          dtype="float32", enforce_eager=True)

Prompt format (exactly the training serialization):

TOOLS AVAILABLE:
{one JSON schema per line}
<|user|>
{request}

The model continues with <|tool_call|>\n{"name": ..., "arguments": {...}} or <|assistant|>\n{text}. Stop tokens: <|eos|> (id 1).

Training

Stage	Tokens	Notes
CPT (G007)	7.92B	61k steps, WSD, SpikeGuard (67 skips, 0 lost progress)
Tool-CPT (G008)	2.0B	EN/code/OpenAPI mix, ppl 8.12→7.77
SFT (v7)	4.67M	34,839 examples, 2,176 steps, loss-mask on model-role headers

Arch: 24L / 2048d / GQA 16Q-4KV / SwiGLU-2816 / RMSNorm / RoPE (interleaved) / tied embeddings / vocab 65,536 / 801.2M params.

Reproduce: scripts/pretrain.py → scripts/sft.py → scripts/export_hf.py → scripts/evaluate_bfcl_ms.py. Full fix log in git history (commits 004dacd, 68ca0ad).