Senryu

Reasoning + tool-call chat model (803M), preference-tuned on its own samples

An 803M hybrid Gated DeltaNet-2 + gated MLA model: 32k-context SFT on reasoning, tool and chat data, then APO-zero on pairs sampled from the model itself

---

What this is

Senryu is the 803M successor to Haiku: eight more layers, Gated DeltaNet-2 in place of Kimi Delta Attention, and a full-rank MLA query.

• Pretraining on 1.15B FineWeb-Edu tokens
• Supervised fine-tuning at a 32,768-token sequence length on 6.7B tokens (5.39B assistant tokens) of reasoning, tool-call, instruction-following and chat data
• Then APO-zero on 22,517 on-policy preference pairs: four samples per prompt from the SFT model itself, judged by Skywork-Reward-V2-Llama-3.1-8B and rule checks for tool calls, answers and loops
• Live demo: kerzgrr/senryu-demo
• Hub weights are the EMA snapshot in bfloat16

This upload is the final checkpoint of the preference run (optimizer step 704, one full epoch of 22,517 pairs). It is the only released checkpoint of the Senryu generation; the pretrain and SFT checkpoints behind it are not published.

---

Chat contract

Thinking

Each assistant turn is prefixed with a zero-loss control token:

Mode	Prefix	Typical body
think	<|think|>\n	<think>…</think> then the answer
no-think	<|no_think|>\n	answer only

inference.py streams the <think> region live (dim yellow) and hides the control tokens.

Tool calls (Hermes JSON)

Tool schemas go in a Hermes-style <tools> block in the system turn. The model calls them with:

<tool_call>
{"name": "web-search", "arguments": {"query": "…"}}
</tool_call>

Senryu learned schema-following calls from Glaive function calling v2, When2Call, Hermes reasoning tool-use, SmolTalk and the stage-3 tool mix (Toucan 1.5M, Nemotron PTD v1, Nemotron Agentic v1/v2). web-search is the only built-in it saw by name, in the Nemotron Agentic conversations; to Senryu, stateful_python_code_exec and calculator are ordinary schemas it reads from the <tools> block. On prompts that offer tools, the preference pairs reward calling and answering directly equally often (6,794 pairs each). inference.py auto-runs:

Built-in	Tool name	Observation
--tools web-search	web-search	Tavily-shaped JSON (Tavily if TAVILY_API_KEY is set, else DuckDuckGo + Wikipedia)
--tools python	stateful_python_code_exec	Jupyter-style stdout / last value from a restricted interpreter
--tools calculator	calculator	Numeric result of a math expression

Tool results

Each observation is a tool (or user) turn prefixed with:

<|tool_response|>
{observation}

---

Install & run

pip install torch safetensors tokenizers huggingface_hub
hf download kerzgrr/Senryu inference.py --local-dir .
python inference.py

python inference.py --prompt "What is the capital of France?"
python inference.py --tools web-search,python,calculator
python inference.py --no-think --prompt "Reply in one sentence."

inference.py auto-downloads weights / tokenizer / tiny_gdn/ and auto-installs pinned flash-linear-attention (plus transformers, which its decode cache imports). Git is required on PATH. A CUDA GPU is required: the Gated DeltaNet-2 layers run on Triton kernels.

Flag	Default	Description
--prompt	—	One-shot user message
--system	—	System prompt, used verbatim
--think / --no-think	think	Assistant control prefix
--tools	—	Built-ins: web-search, python, calculator (comma-separated)
--temperature	0.6	Sampling temperature (0.3 suits tool-calling turns)
--top-p	0.95	Nucleus sampling
--repetition-penalty	1.08	Over the last 256 tokens
--max-new-tokens	4096	Max generation length
--context-length	32768	Conversation tokens kept (oldest dropped first)
--device	cuda	CUDA device, e.g. cuda:1

Interactive commands: /think /no_think /system … /reset /exit. Ctrl+C stops the current reply and keeps the session.

---

Model architecture

803,000,968 parameters:

Layers	44 (Gated DeltaNet-2 ×3 + gated MLA every 4th)
Hidden	1,024
MLP	SiTU-GLU 3,392
Attention	Gated MLA (NoPE), 8 heads, full-rank query, KV LoRA rank 512
Linear	Gated DeltaNet-2, 8 heads × 128, channel-wise erase and write gates
Residuals	Block attention residuals
Vocab	65,536 BPE (the Haiku tokenizer)
Context	32,768 (SFT sequence length)

---

Training

Stage	Details
Pretrain	FineWeb-Edu, seq 2,048, 2,200 steps = 1.15B tokens, hybrid Muon + AdamW at peak 5×10⁻⁴, 10.4 hours. Val loss (EMA) 3.06
SFT	Natural-proportion mixture with a full shuffle: SmolTalk2 Mid + SFT, the stage-3 tool/agentic mix (Nemotron PTD v1, Agentic v1/v2, IF-Chat v1/v2, Toucan 1.5M), OpenMathReasoning, OpenCodeReasoning, WildChat-4.8M, SmolTalk, Hermes-3, Hermes reasoning tool-use, SystemChat 2.0, SlimOrca, UltraChat 200k, Magicoder Evol-Instruct, Self-OSS-Instruct, Glaive function calling v2, When2Call, OASST2, Tulu 3 IF personas, No Robots. Seq 32,768, hybrid Muon + AdamW 4×10⁻⁴, 12,800 steps = 6.7B tokens (5.39B assistant tokens), 11.8 hours. Val loss (EMA) 0.827 (ppl 2.29)
Preference (APO)	APO-zero, β = 0.05, on on-policy pairs. 20,000 prompts drawn from the SFT mixture, four samples each from the SFT EMA (temperature 0.8, top-p 0.95). The chosen reply is the best-judged sample, or the dataset's reference reply when no sample passed; pairs need a reward margin of at least 3. Prompts that offer tools have 6,794 call pairs and 6,794 answer-directly pairs. 3,697 loop-stop pairs prefer ending the turn exactly where a repeat begins over one more repeat. 22,517 pairs, one epoch, 32 pairs per step, 704 steps, AdamW 1×10⁻⁶ cosine to 1×10⁻⁷, 10% warmup, clip 0.2, max length 16,512. Reference model: the SFT EMA. 1.8 hours
Weights	EMA (this repo's model.safetensors)
Preference val (EMA)	903 held-out pairs: reward accuracy 79.0%, reward margin 1.33, chosen NLL 0.56 per token

preference_config.json, validation.json, sft_config.json and sft_validation.json hold the exact settings and held-out results. preference_data.json describes the pair pack: prompt selection, sampling, judging, and the composition by source and label.

---

Limitations

• Scale: 803M is a research / edge model, not a frontier system

---

Model family

Model	Stage	Hub
Senryu	SFT + on-policy APO (chat, reasoning, tools)	this repo
Demo	ZeroGPU Space (chat)	kerzgrr/senryu-demo
Haiku	Previous generation (~655M)	kerzgrr/Haiku
Haiku-base	Previous generation, pretrain	kerzgrr/Haiku-base

---

Citation

@misc{senryu2026,
  title={Senryu: An 803M Hybrid Gated DeltaNet-2 + Gated-MLA Reasoning Model},
  author={kerzgrr},
  year={2026},
  url={https://huggingface.co/kerzgrr/Senryu}
}

---

Five, seven, five — about people this time.