TinyBrainBot-10M-Base

A 9.9M-parameter language model trained from scratch on ~10B tokens, as a plain Llama (LlamaForCausalLM), so it runs in transformers, llama.cpp, LM Studio and Jan with no custom code.

Scores vs other models of 10M parameters or fewer

Zero-shot, full test sets, acc_norm. The (raw acc) row underneath shows our plain accuracy. The Intelligence Index is the Open SLM Leaderboard's, and the other rows come from that leaderboard.

Model	Params	Int Index	HellaSwag	ARC-E	ARC-C	PIQA	ArithMark-3
TinyBrainBot-10M-Base (this model)	9.90M	17.61	28.28	36.66	24.23	58.81	65.40
(raw acc)		17.24	27.44	38.30	20.31	59.41	65.50
Pascaline-3M (ReckonerLabs) *	2.98M	16.09	27.79	34.18	22.44	56.15	69.20
ForgePlex-M2.2-10M	9.99M	12.12	29.19	39.77	22.70	59.79	37.40
heron1i-flash-lite-v2	9.92M	11.11	28.86	38.89	23.38	58.54	36.70
min-spark-1.6	9.96M	11.08	28.52	38.30	22.44	59.14	36.90

* Pascaline-3M's figures are from its model card (it was not yet on the leaderboard at the time of writing). We reproduced them with the same tools.

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("nkthebass/tinybrainbot-10m-base")
model = AutoModelForCausalLM.from_pretrained("nkthebass/tinybrainbot-10m-base")
ids = tok("Mia has 9 fewer pens than Leo. Leo has 30 pens. Mia has", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=8, do_sample=False)[0]))

GGUF: tinybrainbot-10m-base-Q8_0.gguf (11 MB) and -F16.gguf (20 MB), for llama.cpp, LM Studio and Jan.

This is a base model: it continues text and does not chat. At this size, expect simple prose, grade-school arithmetic stories, and plenty of made-up facts.

Model

Parameters	9,902,336 total (8.85M non-embedding + 1.05M tied embedding)
Shape	hidden 256, 12 layers, 4 query heads of 64 with 2 KV heads (GQA), SwiGLU 704, RMSNorm, RoPE
Vocabulary	4,096-token byte-level BPE: digits split one per token, whitespace kept exactly, no unknown tokens. Numbers are written normally (no custom number tokens)
Context	1,024 tokens

Training

~9.95B tokens: a constant learning-rate phase of 8.9B tokens, then a 1.04B-token decay. Hardware: 2× Tesla V100.

Constant-LR phase	Share
Q&A and facts (our distilled plain-text sets)	22%
Cosmopedia Stanford + OpenStax	21%
Grade-school story arithmetic (generated by our own programs)	20%
TinyStories	15%
FineWeb-Edu	10%
Everyday / how-to (Cosmopedia WikiHow + distilled)	7%
Wikipedia leads	5%

Decay phase (1.04B tokens)	Share
Generated grade-school arithmetic, several phrasing families	45%
The pretraining sets above, re-weighted toward Q&A, facts and commonsense	40%
Cosmopedia WikiHow	15%

The arithmetic is written by our own programs: every number is computed in Python and nothing is written by an LLM. It covers adding, taking away, multi-step problems, groups and sharing, money, time, measurement, area and perimeter, fractions of a whole, patterns, tallies, and especially relations ("Mia has 9 fewer pens than Leo"). Relations are asked in both directions, with the unknown on either side, and as minimal pairs that flip only "more" and "fewer". No benchmark training or test data was used.

Recipe: AdamW (betas 0.9/0.95, weight decay 0.1), peak LR 3e-3 with a warmup-stable-decay schedule, 131k tokens per step, fp16 mixed precision.

How this checkpoint was chosen. From the constant-LR checkpoint at step 68,000, we ran five decay branches with different arithmetic data. They scored between 15.20 and 17.61 on the Index. This is the best one, and the published weights are that branch's final checkpoint, not a merge. The branches that included relational arithmetic scored 16.4–17.6.

Evaluation

Run in float32 with lm-eval 0.4.13 (full sets, 0-shot) and the official ArithMark-3 script:

lm_eval --model hf --model_args pretrained=nkthebass/tinybrainbot-10m-base,dtype=float32 --tasks hellaswag,arc_easy,arc_challenge,piqa --num_fewshot 0
python bencharithmark-3.py --model nkthebass/tinybrainbot-10m-base --dtype float32

Limitations

• English only, and very small: it confabulates freely, so don't rely on it for facts.
• Its arithmetic is grade-school sized and works best in familiar story phrasings. Free-form answers are less reliable than its ranked-choice benchmark scores suggest.
• Base model only: no instruction tuning and no safety tuning.