TinyBrainBot-3M-Base

A 2.94M-parameter language model trained from scratch on 5.2B tokens. It is a plain Llama architecture (LlamaForCausalLM), so no custom code is needed.

Scores vs other models of 3M parameters or fewer

Zero-shot, full test sets. Every model is scored with acc_norm; the (raw acc) row underneath shows our plain accuracy. The Intelligence Index is the Open SLM Leaderboard's, and the other models' numbers are copied from that leaderboard.

Model	Params	Int Index	HellaSwag	ARC-E	ARC-C	PIQA	ArithMark-3
TinyBrainBot-3M-Base (this model)	2.94M	10.34	27.07	33.59	24.91	55.98	45.00
(raw acc)		10.37	26.69	35.48	18.94	57.62	45.10
GPT-S3-3M	2.97M	9.61	27.68	34.76	23.12	57.02	39.10
Tokle-3M	2.91M	8.91	27.20	34.85	23.98	55.01	40.80
pulvis-v2	2.96M	8.62	27.93	31.86	22.78	56.86	37.40
Purrence-3M	2.99M	8.49	27.81	34.55	23.63	56.64	34.80

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("nkthebass/tinybrainbot-3m-base")
model = AutoModelForCausalLM.from_pretrained("nkthebass/tinybrainbot-3m-base")
ids = tok("Ava had 23 apples. Ava got 19 more. Now Ava has", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=20, do_sample=False)[0]))
# -> ... Now Ava has 42 apples.

There is also an F16 GGUF (tinybrainbot-3m-base-F16.gguf, 6 MB) for llama.cpp, LM Studio and Jan. A Q8_0 file isn't possible because the hidden size (144) is not a multiple of 32.

This is a base model: it continues text and doesn't chat. At this size, expect short, simple text: children's-story-level prose and grade-school arithmetic in familiar phrasings.

Model

Parameters	2,942,928 total (2.35M non-embedding + 0.59M tied embedding)
Shape	hidden 144, 10 layers, 3 query heads of 48 with 1 KV head (GQA), SwiGLU 416, RMSNorm, RoPE
Vocabulary	4,096-token byte-level BPE trained for this model: digits split one per token, whitespace kept exactly, no unknown tokens
Context	1,024 tokens

Training

Data (5.0B tokens + 0.5B re-anneal). Natural text: FineWeb-Edu, Wikipedia article leads and TinyStories. Synthetic text: Cosmopedia (Stanford, OpenStax, WikiHow), our own distilled plain-text Q&A, fact and everyday-explanation sets, and generated grade-school arithmetic (every number computed in Python). There is no code. No benchmark training or test data was used.

Group	Share
Q&A and facts (distilled)	25%
Grade-school arithmetic (generated)	20%
Cosmopedia Stanford + OpenStax	15%
TinyStories	15%
Everyday / how-to (Cosmopedia WikiHow + distilled)	10%
FineWeb-Edu	10%
Wikipedia leads	5%

The last ~10% of training is an anneal on a mix weighted toward the distilled sets, aligned with the learning-rate decay.

Recipe. AdamW (betas 0.9/0.95, weight decay 0.1), peak LR 4e-3, warmup-stable-decay schedule, 131k tokens per step, fp16 mixed precision, trained on 2× V100.

One thing we learned. The first version used arithmetic in a single sentence pattern. The model solved 2-digit sums with carrying 99% of the time in that pattern, but was near chance in any other wording, and scored 33.6 on ArithMark-3. At this size the skill gets tied to the wording. We replaced it with arithmetic written in many different phrasings and re-annealed from the step-36,000 checkpoint, before the decay ended: ArithMark-3 went from 33.6 to 45.0, and the Index from 7.33 to 10.34. The published weights are that final checkpoint, a single training run, not a merge. Even so, the model still handles unfamiliar wordings of sums poorly.

Evaluation

Run in float32 with lm-eval 0.4.13 (full sets, 0-shot) and the official ArithMark-3 script:

lm_eval --model hf --model_args pretrained=nkthebass/tinybrainbot-3m-base,dtype=float32 --tasks hellaswag,arc_easy,arc_challenge,piqa --num_fewshot 0
python bencharithmark-3.py --model nkthebass/tinybrainbot-3m-base --dtype float32

Limitations

At 3M parameters this is a toy-scale model. It knows few facts, loses coherence after a few sentences, and its arithmetic only works reliably in common phrasings. HellaSwag and ARC are close to chance for every model this size; the gaps between models there are small.