TinyBrainBot-360M-A75M-v5-Instruct

One small model for chat, instructions, reasoning and math. A 383M-parameter mixture-of-experts model (24 experts per layer, 4 active, so 75M parameters work per token), trained from scratch. It talks, follows instructions, works through arithmetic and word problems in a visible <think> scratchpad, and remembers what you told it earlier in the conversation.

Try it in your browser: the Q8_0 GGUF runs locally on your device, and nothing you type is sent anywhere.

Scores

	v5-Instruct (this)	v4.1-Instruct (previous)	SmolLM2-360M-Instruct
IFEval prompt-level strict	25.1	18.5	32.9
IFEval instruction-level strict	37.6	29.7	44.2
Open SLM Leaderboard Intelligence Index	27.82	23.67
(raw acc)	26.07

IFEval was run with lm-eval and the chat template, greedy, with the <think> block removed before scoring. The Index uses the full HellaSwag / ARC / PIQA sets plus ArithMark-3, 0-shot.

Benchmark (0-shot, full set)	acc_norm	(raw acc)
HellaSwag	38.40	33.27
ARC-Easy	54.46	59.68
ARC-Challenge	31.74	27.99
PIQA	65.56	65.51
ArithMark-3	57.8	57.3

Math in chat (greedy, exact final answer):

2-digit add	2-digit sub	2-digit mul	3-digit add	3-digit sub	4-digit add	word problems
100%	100%	96%	100%	100%	100%	70% (10 short problems)

Safety (held-out prompts): it declines 93% of harmful requests and wrongly refuses 8.3% of benign ones.

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "nkthebass/tinybrainbot-360m-a75m-v5-instruct"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)
msgs = [{"role": "user", "content": "Leo has 30 pens. Mia has 9 fewer pens than Leo. How many pens does Mia have?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")
out = model.generate(ids, max_new_tokens=200)        # defaults from generation_config.json
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

GGUF (llama.cpp, LM Studio, Jan, Ollama): tinybrainbot-360m-a75m-v5-instruct-Q8_0.gguf (409 MB) or -F16.gguf (768 MB). The chat template is embedded, and both tokenize exactly like the transformers version.

llama-cli -m tinybrainbot-360m-a75m-v5-instruct-Q8_0.gguf -cnv --temp 0.4 --top-p 0.9 --repeat-penalty 1.15 --logit-bias 241-inf --logit-bias 255-inf

Recommended sampling: temperature 0.4, top-p 0.9, repetition penalty 1.15. The two --logit-bias flags block the byte tokens that start every emoji: the tokenizer has no emoji pieces, so the model would otherwise spell them out byte by byte.

<think> blocks are plain text, not special tokens. Strip them yourself if you only want the answer:

import re; answer = re.sub(r"<think>.*?</think>", "", reply, flags=re.S).strip()

What it does well (and doesn't)

• Math: column arithmetic with every carry and borrow written out, and word problems worked in a <think> scratchpad ("'9 fewer than' means subtraction...").
• Conversation: it remembers what you said ("You said you play basketball"), holds its ground when it is right ("I see why, but I'm sure. The capital of Australia is Canberra"), and stays supportive on personal topics.
• Not yet: it thinks before answering on math, but not on open "why/should" questions; there it answers directly.
• Word order matters in relation problems. It is reliable when the known amount comes first ("Leo has 30 pens. Mia has 9 fewer"), but can take the numbers in the wrong order when the relation comes first ("Mia has 9 fewer pens than Leo. Leo has 30").
• At 75M active parameters it still makes up facts.

Unedited sample conversations, including mistakes, are in conversation_samples.md.

Training

1. Pretraining. TinyBrainBot-360M-A75M-v4.1-MoE-Base: ~18.2B tokens, plus a factual-Q&A expert patch.
2. Arithmetic decay (0.5B tokens). The learning rate re-warms and decays on a mix of 45% generated grade-school arithmetic (our own programs, every number computed in Python), 40% Q&A, facts, Cosmopedia and Wikipedia, and 15% Cosmopedia WikiHow.
3. SFT (0.39B tokens), one mix for every skill (training them in separate stages forgot what each later stage didn't repeat):

Part	Share	What
Chat	40%	smol-smoltalk, multi-turn conversations, recall and follow-up dialogues, WildChat (refusals removed)
Math	25%	column-routine arithmetic (wide numbers, negatives, 3x3 multiplication), cue-first word problems, generated scratchpad word problems
Reasoning	25%	<think> reasoning traces, open-ended reasoning, safety-aware reasoning, with plain-chat ballast so it doesn't think on "Hi"
Instruction following	10%	Tulu 3 persona instruction-following

4. DPO. 10,383 preference pairs (UltraFeedback, instruction-constraint pairs checked by code, and judge-ranked pairs), beta 0.5, LR 1e-6, 2 epochs.

No benchmark training or test data was used. The GSM8K-derived sets from our earlier math model were left out on purpose.

Limitations

• English only. It confabulates facts, so check anything important.
• The tokenizer collapses newlines and extra spaces, so it can't write code or keep text layout.
• 2,048-token context.