nkthebass/tinybrainbot-350mV3-base

TinyBrainBot 350M V3 — Base

A 348M-parameter decoder-only language model pretrained from scratch on 22.7B tokens, started on a rented 8× RTX 5090 node and finished on 2× Tesla V100. This is the base (pretrained + fact-annealed) model: a text-completion model with no instruction tuning.

TL;DR: Beats the previous-generation TinyBrainBot 100M V3 base on 5 of 7 standard benchmarks and beats Supra2-100M-Base on all 7, at an average of 43.2 vs 42.2 and 40.0. It also carries an unusual history: ~20B of its 22.7B training tokens were run with every RMSNorm weight frozen at 1.0 by a precision bug. The recovery from that is documented below, because it is the most interesting thing about this model.

---

Model details

Parameters	348,342,912 (~348M)
Architecture	Decoder-only transformer, pre-norm, RMSNorm, SwiGLU MLP, RoPE
Hidden size	1152
Layers	22
Attention heads	18 query / 6 KV (grouped-query attention), head_dim 64
Context length	2048
Vocabulary	32,000 (tbb-32k-v2, tied embeddings)
RoPE theta	10,000
Training tokens	22.70B
Precision	fp16 with fp32 master weights (final phase)

Evaluation

EleutherAI lm-eval-harness, 0-shot, complete test sets. acc_norm for HellaSwag/ARC/OpenBookQA/PIQA, acc for WinoGrande/MMLU. Same harness and settings as the published 100M V3 numbers, so the comparison is like-for-like (base vs base).

Benchmark	350M V3 base	100M V3 base	Supra2-100M-Base
ARC-Easy	56.6	54.7	47.8
ARC-Challenge	33.3	30.1	24.8
HellaSwag	35.9	32.6	36.0
OpenBookQA	34.6	34.0	32.0
PIQA	67.0	66.2	65.5
WinoGrande	51.5	53.0	50.7
MMLU	23.8	25.0	23.3
Average	43.2	42.2	40.0

→ 5/7 vs the 100M V3 base, losing only WinoGrande (−1.5) and MMLU (−1.2). → 7/7 vs Supra2-100M-Base (HellaSwag 35.9 vs 36.0 is a 0.1 tie at best).

Benchmark-train-split disclosure

The final data mix contains two small sets built from official train splits of benchmark datasets:

• sci-qa-train (0.5%) — SciQ, ARC-Easy/Challenge train, OpenBookQA train + core facts
• hellaswag-train (1.2%) — HellaSwag train, SWAG train

Evaluation uses the test/validation splits, which these never touch, and this is standard practice for small-model training. But the 100M V3 baseline above was trained without them, so the ARC / OpenBookQA / HellaSwag margins are not a like-for-like recipe comparison. Stated here rather than buried: if you want a clean comparison, weight those three tasks accordingly.

Training

Three phases, on rented and owned hardware:

1. Cloud pretrain — ~20B tokens on a rented 8× RTX 5090 node (Clore.ai), bf16/fp8, torch.compile, DDP. Cost about $90.
2. Local extension — continued on 2× Tesla V100 (16GB) in fp16, WSD schedule, sequence length 2048, micro-batch 2 × grad-accum 32 × 2 GPUs = 262,144 tokens/step.
3. Fact anneal — two final passes on a knowledge-dense mix (below), 1.23B tokens, LR annealed 5e-5 → 1e-5.

The frozen-norm episode

For its first ~20B tokens this model trained with all 45 RMSNorm weights frozen at exactly 1.0. The cause was precision, not the optimiser: --precision bf16 stored the weights in bf16 with no fp32 master copy. bf16 has 7 mantissa bits, so near a value of 1.0 the smallest representable step is ~0.0078, while an Adam update at lr 1e-4 is ~1e-4 — every norm update rounded straight back to 1.0. Other weights (magnitude ~0.02) have a ~50× finer ULP there and trained normally, so loss fell and nothing looked wrong.

It was found by diffing two consecutive checkpoints: 111/111 non-norm tensors changed, 0/45 norms. Switching to fp16 (which keeps fp32 masters) unfroze them.

They never fully recovered. Measured mean |w − 1| across the RMSNorm weights:

| model | mean·|w−1| | |---|:--:| | 100M V3 base (never affected) | 0.697 | | 320M V2 base (never affected) | 0.694 | | 350M V3 base (this model) | 0.017 |

After 5.5B tokens of healthy fp16 training the norms had moved 2.5% of the distance a normally-trained model's travel. So this model does its work essentially without learned per-channel scaling — and still beats a 100M trained with it. Treat the results below as a lower bound on what this recipe can do.

Final data mix

Knowledge-dense, 22 sources. Largest: wiki-dense 13%, cosmopedia-openstax 13%, cosmopedia-stanford 9%, qa-plain 8%, wikipedia-english 7%, gutenberg-clean2 5%, qa-distill 5%, facts-plain 5%, fineweb-edu 5%. Narrative/commonsense sources total 21.1%; every source was checked to stay under 3 epochs against measured token counts.

Limitations

• Arithmetic is weak. This is a base model with no math tuning; use the math variant.
• MMLU and WinoGrande lag the 100M V3 base. The knowledge-dense anneal did not lift MMLU.
• Repetition loops on greedy decoding beyond ~40 tokens, as is normal at this scale.
• Hallucinates confidently — it will state a wrong country for a landmark under sampling.
• No instruction tuning. It continues text; it does not follow instructions or chat.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("nkthebass/tinybrainbot-350mV3-base")
t = AutoTokenizer.from_pretrained("nkthebass/tinybrainbot-350mV3-base")
ids = t("The capital of France is", return_tensors="pt")
print(t.decode(m.generate(**ids, max_new_tokens=30)[0]))