TinyBrainBot-25M-v5-Base

A 30.6M-parameter language model (25.2M non-embedding) trained from scratch on ~12B tokens, as a plain Llama (LlamaForCausalLM): no custom code, so it runs in transformers, llama.cpp, LM Studio and Jan.

Scores

Zero-shot, full test sets, acc_norm. The (raw acc) row underneath shows our plain accuracy. The Intelligence Index is the Open SLM Leaderboard's, and the other models' numbers come from that leaderboard.

Model	Params	Int Index	HellaSwag	ARC-E	ARC-C	PIQA	ArithMark-3
TinyBrainBot-25M-v5-Base (this model)	30.6M	24.61	30.29	41.46	26.54	63.17	76.30
(raw acc)		23.11	28.53	45.12	21.42	62.08	76.30
void.1	90.1M	23.92	38.68	47.31	28.16	67.46	44.80
100M-exp	98.2M	22.47	37.78	49.83	27.22	66.97	40.00
50M-pro	49.6M	17.93	33.25	45.62	25.51	64.80	37.40
gibc-v2-lm	49.8M	16.13	31.87	47.85	26.11	60.28	40.20
TinyBrainBot-25M-v4-Base (the starting point)	30.6M	12.54	28.86	42.09	24.91	60.23	35.20

It is the highest-scoring model we know of at 100M parameters or fewer, at about a third of the size of the next one.

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("nkthebass/tinybrainbot-25m-v5-base")
model = AutoModelForCausalLM.from_pretrained("nkthebass/tinybrainbot-25m-v5-base")
ids = tok("Mia has 9 fewer pens than Leo. Leo has 30 pens. Mia has", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=6, do_sample=False)[0]))
# -> ... Mia has 21 pens.

GGUF: tinybrainbot-25m-v5-base-Q8_0.gguf (33 MB) and -F16.gguf (62 MB).

This is a base model: it continues text and does not chat. Expect fluent simple prose, grade-school arithmetic stories, and plenty of made-up facts.

Model

Parameters	30,554,496 total (25.18M non-embedding + 5.38M tied embedding)
Shape	hidden 384, 16 layers, 6 query heads of 64 with 2 KV heads (GQA), SwiGLU 1024, RMSNorm, RoPE
Vocabulary	14,000-token byte-level BPE: digits split one per token, whitespace kept exactly, no unknown tokens
Context	1,024 tokens

Training

This model continues TinyBrainBot-25M-v4-Base, which saw ~11B tokens of general pretraining: web (FineWeb-Edu, DCLM), Wikipedia, books (Project Gutenberg, BookCorpus), Cosmopedia, math web pages, educational PDFs, code, and our distilled Q&A and everyday-explanation sets.

v5 adds one more 1.04B-token decay phase. The learning rate re-warms to 2e-3 over 300 steps, then decays to zero on this mix:

Decay phase	Share
Generated grade-school arithmetic, several phrasing families	45%
Q&A, facts, commonsense, Cosmopedia, Wikipedia, TinyStories	40%
Cosmopedia WikiHow	15%

The arithmetic is written by our own programs: every number is computed in Python and nothing is written by an LLM. It covers adding, taking away, multi-step problems, groups and sharing, money, time, measurement, area and perimeter, fractions of a whole, patterns, tallies, and especially relations ("Mia has 9 fewer pens than Leo"). Relations are asked in both directions and as minimal pairs that flip only "more" and "fewer". No benchmark training or test data was used.

This was a single run, not a selection among branches. The published weights are its final checkpoint, not a merge.

Recipe: AdamW (betas 0.9/0.95, weight decay 0.1), 131k tokens per step, fp16 mixed precision, 2× Tesla V100.

Evaluation

Run in float32 with lm-eval 0.4.13 (full sets, 0-shot) and the official ArithMark-3 script:

lm_eval --model hf --model_args pretrained=nkthebass/tinybrainbot-25m-v5-base,dtype=float32 --tasks hellaswag,arc_easy,arc_challenge,piqa --num_fewshot 0
python bencharithmark-3.py --model nkthebass/tinybrainbot-25m-v5-base --dtype float32

Limitations

• English only, and small: it confabulates freely, so don't rely on it for facts.
• Its arithmetic is grade-school sized and works best in familiar story phrasings. Free-form answers are less reliable than its ranked-choice benchmark scores suggest.
• Base model only: no instruction tuning and no safety tuning.