dtsj89/puki-brains

Puki Brains

The small language model that writes the lines of Puki the parrot in Puki, a free Hebrew/English game for toddlers. It runs on the tablet itself, offline: on a Samsung Galaxy Tab A7 Lite (3 GB) a line takes 0.34–0.44 s, and a safety check of what the child said takes about 0.12 s.

It does one job. It reads a short prompt about a moment in the game ("the child fed the elephant"), what the child said, or a question to check, and writes one short, kind line for a two-to-four-year-old. It is not a general chat model and knows very little about the world.

Model

A llama-style decoder of 46M parameters: 14 layers, width 512, 8 attention heads with 2 key/value heads (head size 64), RoPE, tied embeddings, a 1,024-token context, and a 12,288-token byte-level BPE trained for its Hebrew-heavy mix.

Training

1. Pretraining from scratch: 1.58B tokens, 74% Hebrew and 26% English. AdamW with a cosine schedule.
2. Fine-tuning: 46k of Puki's lines in both languages and over a dozen tasks. These include reacting to a game moment, talking with the child, answering carefully when the child mentions hurt or fear, keeping a game's secret, and a safety gate that rates what the child said.
   • The lines were written by DeepSeek models through their API.
   • Each line was kept only if it passed Puki's filters and a second model's judgment. The filters check length, words a toddler knows, no promises a playmate shouldn't make, and that the line tells the truth about the game.

The code is in train/ in Puki's repository. train/pt_manifest.json records exactly which documents and chunks went into pretraining, with their hashes.

Data

Source	Use	Terms
HeDC4 (Shalumov & Haskey)	Hebrew pretraining, cleaned by pt_clean.py	No licence stated on its card; cited as its authors ask, below. Dicta builds its own Hebrew models on it, and we use it the same way
FineWeb-Edu	English pretraining	ODC-By 1.0, and Common Crawl's terms of use
Cosmopedia	English pretraining	Apache 2.0
Project Gutenberg, filtered (Common Pile)	English pretraining	Public domain books
YouTube, filtered (Common Pile)	English pretraining	CC BY; each video's author and licence are in its row, and pt_manifest.json lists the rows used
Puki's own lines	Fine-tuning	MIT (Puki's repository)

@article{shalumov2023hero,
  title   = {HeRo: RoBERTa and Longformer Hebrew Language Models},
  author  = {Shalumov, Vitaly and Haskey, Harel},
  journal = {arXiv preprint arXiv:2304.11077},
  year    = {2023}
}

Results

The scores are on held-out prompts it never trained on:

• Filter: 99.5% of its lines pass Puki's filter.
• Truth checks:
  • Fixing a silly mistake honestly: 139/153.
  • Retelling the day with at least two real events: 217/226.
  • Keeping a game's secret: 73/98.
• Gate: 17 of 18 children's sentences were rated as the data rates them. It errs on the careful side.

In the game, three more layers surround it:

• Before it writes: word lists catch what the child said, and a match gets a fixed, gentle answer.
• After it writes: a filter checks every line, and a line that fails is replaced by a recorded one.
• Parents: they see every line in the settings and can flag one.

Files

• brains_int8.onnx: int8, dynamic, per-channel, MatMul weights only (72 MB).
  • Inputs: ids (int64, 1 × n), plus past_k and past_v (float32, layers × 2 × T × 64; T = 0 for the prompt).
  • Outputs: the last token's logits (1 × 12288), plus next_k and next_v (the cache with the new positions).
  • One graph serves the prompt and each next token.
• tokenizer.json: the BPE. <|endoftext|> (id 0) ends a line.
• brains.json: the shapes the runtime needs.

The prompt format is in train/fmt.py. The Android runtime is Brains.kt (sampling at temperature 0.7, top-p 0.9).