TinyBrainBot-25M-v5.1-Base

A 25M-parameter base model trained from scratch on 12.7B tokens: 25.2M parameters in the transformer, plus a 5.4M tied embedding for its 14k vocabulary (30.6M total). It continues TinyBrainBot-25M-v4-Base with the TinyBrainBot v5.1 recipe: a final decay on science Q&A, school-level textbooks, fact-dense text and real math.

Benchmark (0-shot, full test sets)	25M-v5.1	25M-v4
ARC-Easy (acc_norm)	46.93	42.09
(raw acc)	53.11	46.84
ARC-Challenge (acc_norm)	30.80	24.91
(raw acc)	27.30	20.99
HellaSwag (acc_norm)	29.73	28.86
(raw acc)	27.97	27.41
PIQA (acc_norm)	61.97	60.23
(raw acc)	62.89	59.96
OpenBookQA (acc_norm)	31.20	31.20
(raw acc)	19.60	17.20
SciQ (acc_norm)	64.10	65.90
(raw acc)	72.30	73.70
WinoGrande (acc)	51.54	52.57
BoolQ (acc)	53.79	57.77
LAMBADA (OpenAI) (acc)	22.32	22.78
MMLU (acc)	24.74	23.69
CommonsenseQA (acc)	21.62	20.31
Social IQa (acc)	37.10	37.15
ArithMark-3 (acc_norm)	46.70	35.20
(raw acc)	46.30	-
Open SLM Leaderboard Intelligence Index	18.50	12.54

EleutherAI lm-eval-harness v0.4.13, 0-shot, float32, every example of every task. ArithMark-3 with the official bencharithmark-3.py (float32). Both models were run by us with the same settings.

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "nkthebass/tinybrainbot-25m-v5.1-base"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

ids = tok("Photosynthesis is the process by which", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=80)       # defaults from generation_config.json
print(tok.decode(out[0], skip_special_tokens=True))

GGUF (llama.cpp, LM Studio, Ollama): tinybrainbot-25m-v5.1-base-F16.gguf (62 MB) and tinybrainbot-25m-v5.1-base-Q8_0.gguf (33 MB). The F16 file tokenizes and decodes greedily exactly like transformers.

llama-cli -m tinybrainbot-25m-v5.1-base-Q8_0.gguf -p "Photosynthesis is the process by which" -n 80 --temp 0.7 --repeat-penalty 1.15

Default sampling: temperature 0.7, top_p 0.95, top_k 40, repetition_penalty 1.15.

Samples

Unedited, at the default settings.

Photosynthesis is the process by which plants use sunlight to convert carbon dioxide and water into glucose and oxygen.

The main reason volcanoes erupt is because of the sudden release of stored pressure and gas into the magma chamber. This happens because the heat from the magma increases the temperature, causing dissolved gases to come out of solution. For example, when you shake a bottle of soda, the carbon dioxide produced in the mixture quickly fizzes, just like a pressure cooker steam

Once upon a time, a little fox named Max decided to go on an adventure. He had a special tool that helped him find the best food source for his beloved fox!

Tom had 12 apples and gave 5 to his sister, so Tom gave away 6 apples. Grace has 8 shells and Noah has 37 shells. Grace gives away 27 shells, so Grace has 50 shells left.

At 25M parameters it writes fluent text and knows common science explanations, but word-problem arithmetic in free text is unreliable (see the last sample).

Model

Architecture	Llama-compatible decoder (LlamaForCausalLM), RoPE, SwiGLU, RMSNorm, grouped-query attention
Shape	hidden 384, 16 layers, 6 heads of 64, 2 KV heads, FFN 1024
Parameters	25.18M non-embedding + 5.38M tied embedding = 30.55M
Context	1,024 tokens
Vocabulary	14,000 byte-level BPE (digits split one per token, newlines and indentation kept, no unknown tokens)

Training

12.7B tokens on 2x Tesla V100 16GB, fp16, AdamW (0.9, 0.95), weight decay 0.1, 131k tokens per step.

Stage	Tokens	LR	Data
1-3 (= 25M-v4)	11.0B	2e-3, warmup-stable-decay	web, Wikipedia, books, code, math, synthetic textbooks, Q&A; then a fact-dense anneal (see the 25M-v4 card)
4: v5.1 decay	1.5B	re-warm to 2e-3, decay to 0	school-level textbooks (Cosmopedia v2) 15%, FineWeb-Edu (high quality) 10%, science Q&A 9%, DCLM-Ultra 6%, Simple and English Wikipedia 11%, distilled Q&A / facts / explanations, commonsense and everyday-activity writing, GooAQ, FactBase-DCLM, finephrase FAQ, ELI5; real math 12%: OpenMathInstruct-2 4%, UltraData-Math L3 3.5%, FineMath 2.5%, Orca-Math 2%
5: math top-up	0.2B	constant 3e-4	the stage 4 mix (83%) and real math (12%) plus 5% synthetic arithmetic practice (Python-generated problems: operations and order of operations, numbers as words, large numbers and money, negatives, multi-step and layout problems)

The generated sets (science Q&A, explanations, commonsense writing, arithmetic practice) were written for this model; none of them contains benchmark questions, and no benchmark train or test split was used.

Reproduce the benchmarks

lm_eval --model hf --model_args pretrained=nkthebass/tinybrainbot-25m-v5.1-base,dtype=float32 \
  --tasks arc_easy,arc_challenge,hellaswag,piqa,openbookqa,sciq,winogrande,boolq,lambada_openai,mmlu,commonsense_qa,social_iqa \
  --num_fewshot 0 --batch_size 16
python bencharithmark-3.py --model nkthebass/tinybrainbot-25m-v5.1-base --dtype float32