Puki Brains (English)
The small language model that writes the English lines of Puki the parrot in Puki, a free Hebrew/English game for toddlers. It runs on the tablet itself, offline.
It does one job: it reads a short prompt about a moment in the game ("the child fed the elephant"), what the child said, or a game's own task (a silly mistake to make on purpose, a drawing to talk about, the day to retell at bedtime, a friend who needs help, the next sentence of a story), and writes one short, kind line for a two-to-four-year-old. It is not a general chat model and knows very little about the world. Its Hebrew sibling is dtsj89/puki-brains-he.
Model
A llama-style decoder of 44M parameters: 14 layers, width 512, 8 attention heads with 2 key/value heads (head size 64), RoPE, tied embeddings, a 512-token context, and an 8,192-token byte-level BPE trained on English.
Training
- Pretraining from scratch: 1.6B tokens of English (the sources below). Muon (lr 2e-3) with a warmup-stable-decay schedule: a constant phase, then a 1-sqrt cooldown over the last 20%.
- Fine-tuning: 30,799 of Puki's English lines over a dozen tasks, two epochs. The lines were written by DeepSeek models and kept only if they passed Puki's filters and a second model's judgment (dtsj89/puki-lines).
- Preference tuning (DPO): 1,485 pairs of its own lines, the better and the worse of four for a prompt as DeepSeek's editor scored them; 100 steps.
The code is in train/ in Puki's repository (pt_run.py, sft_small.py, dpo_onpolicy.py --judge, dpo_small.py).
Data
| Source | Use | Terms |
|---|---|---|
| FineWeb-Edu | Pretraining | ODC-By 1.0, and Common Crawl's terms of use |
| Cosmopedia | Pretraining | Apache 2.0 |
| Project Gutenberg, filtered (Common Pile) | Pretraining | Public domain books |
| YouTube, filtered (Common Pile) | Pretraining | CC BY; each video's author and licence are in its row |
| Puki's lines (dtsj89/puki-lines) | Fine-tuning and preference tuning | CC BY 4.0 |
Results
On held-out prompts it never trained on:
- Filter: 100% of its lines pass Puki's filter.
- Truth checks: fixing a silly mistake by saying the real answer 145/149, retelling the day with at least two real events 223/225, a silly mistake that doesn't give the answer away 55/87 (in the game a silly line that gives it away is written again). The int8 file scores the same as float.
- Gate: 23 of 24 children's sentences rated as the data rates them.
- Blind A/B, judged by DeepSeek both ways round over 250 prompts: preference tuning won 51 to 22 over the fine-tuned model (the rest ties).
In the game, more layers surround it: word lists before it writes, a filter and each game's own check after, parents seeing every line in the settings, and recorded lines when a line doesn't pass.
Files
brains_int8.onnx: int8, dynamic, per-channel, MatMul weights only (61 MB). Inputs:ids(int64, 1 × n), pluspast_kandpast_v(float32, layers × 2 × T × 64; T = 0 for the prompt). Outputs: the last token'slogits(1 × 8192), plusnext_kandnext_v. One graph serves the prompt and each next token.tokenizer.json: the BPE.<|endoftext|>(id 0) ends a line.brains.json: the shapes the runtime needs.
The prompt format is in train/fmt.py. The Android runtime is Brains.kt (temperature 0.5, top-p 0.9).