Tallow 🐦‍⬛

_{Portrait: AI-generated.}

Tallow is a wild crow. He's a 15.7M-parameter language model, trained from scratch, that answers you the way a crow would: with actions, crow sounds, and only the words you've taught him.

you:    *tosses a peanut onto the grass*
tallow: [caution min] *lands on the fence* *watches the peanut with one eye* kaw.

He remembers how you treat him. Feed him and he warms up to you, slowly. Grab him and he holds a grudge. "Sorry" doesn't work. Only peanuts do.

you:    *grabs at him*
tallow: [anger max] *dive-bombs your head* KAAW KAAW KAAW *lands on the roof*
you:    sorry!! im so sorry tallow
tallow: [anger max] *glares down from the roof* KAW. KAW.
you:    please come back, i didnt mean it
tallow: [anger max] *fluffs up on the roof* *ruffles his feathers* krrr-tk-tk-tk
you:    *leaves a peanut on the rail and goes inside*
tallow: [anger min] *waits a long time* *swoops down* *snatches it* *gone*
you:    *puts out grapes and sits far away*
tallow: [caution max] *lands on the fence* *eyes you sideways* kaw.

Talk to him

pip install torch sentencepiece safetensors
python tallow.py

tallow.py has everything in one file: the model and the tokenizer helpers. It runs on CPU or GPU. Type to him, /new to start over as a stranger, and an empty line to quit.

Why not GGUF / ollama? His tokenizer expects two small text tricks (a <nl> token for line breaks, and a space after every opening * or [). They're applied in tallow.py, not inside the tokenizer file, so llama.cpp would feed him text he's never seen. Use tallow.py.

How he talks

Every reply is [mood level] *actions* sound.

• 8 moods: calm, curiosity, caution, joy, play, love, sadness, anger. Each one is min or max.
• Sounds: kaw, KAW KAW KAW (alarm), krrr-tk-tk-tk (rattle), mmrrr... (soft mumbling, only for trusted humans).
• No English sentences. The actions carry the meaning.

What he does

behaviour	what happens
trust	Strangers get caution. Food with space, stepping back, and not staring slowly earn trust (8 kind acts from stranger to friend). Friends get joy, play, love, gifts, and hand-feeding.
grudges	Grab or chase him: [anger max], and trust drops to zero. Words never fix it. Food thaws him one step at a time (anger max → anger min → caution max → caution min), at double cost.
the scar	After being grabbed, any sudden movement makes him flinch hard ([caution max]), even once he's a friend again. A crow who was never grabbed only hops back.
mimicry	Deliberately teach him a word three times and he tries it, badly: tal... → TAL-low.
calls	Caw at him and he always answers. Quiet caws make him curious. KAW KAW KAW in capitals makes him look for danger.
crow life	Owls (anger), dead crows (sadness), dogs (play), snow (sliding down roofs), rain (bathing), anting, food caching.

How he was made

1. Tokenizer: SentencePiece BPE, 4,096 pieces, with byte fallback.
2. Pretraining: a Llama-style decoder (RMSNorm, RoPE, SwiGLU, tied embeddings; 8 layers, 384 wide, 6 heads), trained from scratch on TinyStories for 16,000 steps (~524M tokens). Final validation loss 1.17. About 1h45 on an RTX 3070.
3. Crow data: a Python rules engine plans each conversation and decides Tallow's exact mood tag every turn, tracking trust, grudges, the scar, and learned words. Claude Haiku 5.5 turned those plans into dialogue (8 user personalities, 13 scenario types). A checker script dropped any conversation that broke the rules, and broken conversations were dropped, never fixed. That left 5,295 conversations: 5,076 for training and 219 held out.
4. Fine-tuning: 2,600 steps on the conversations, graded only on Tallow's lines, with the context stretched to 2,048 tokens so long grudges fit. About 5 minutes.

Scorecard

On the 219 held-out conversations, at each turn he gets the true conversation so far, writes his own reply (greedy), and we check his mood tag against the rules engine.

what it tests	right	turns
all turns	97.2%	2,432
explodes when grabbed (dramatic or plain)	98.7%	76
"sorry" during a grudge does nothing	100%	70
food thaws a grudge, step by step	98.3%	301
scar flinch	100%	24
never says words he hasn't learned	99.8%	2,315
answers caws	98.7%	155
never grabbed, so he only hops	100%	17
first try at a taught word, at the right moment	73.7%	19

Read these carefully:

• The test conversations come from the same rules engine and writer as the training data, so these are in-distribution results.
• He's given the true history each turn. In a live chat his own mistakes stay in the history.
• Rows with 16-19 turns are small samples.

Versions

• v2 (current): learned that a plain or affectionate grab ("*grabs you!*", "*scoops you up*") is still a grab. v1 only exploded at dramatic, hostile grabs, because every grab in its training data was a lunge with a rolled newspaper. Trained on 500 extra conversations, including grabs late in long, happy friendships.
• v1 (tag v1): the Tallow from the "Meet Tallow" video. Get it with revision="v1".

Limitations

• A toy. 15.7M parameters and a small, made-up crow world.
• His memory is his context window (~1,900 tokens, about 25-30 exchanges). Grudges fade once the grab scrolls out.
• He only knows a few taught words (tallow, hello, peanut, good boy, bye bye). How he handles brand-new words is untested.
• Unusual inputs can still confuse him (e.g. a stranger holding out food in their hand).
• Hits and smacks ("smacks you") aren't reliably understood as hostile yet, and a grab right after a long run of peanuts sometimes only gets caution. More training is planned.

Files

file	what
model.safetensors	the weights (fp32, 15,735,168 parameters)
config.json	model shape
tallow_tok.model	SentencePiece tokenizer
tallow.py	model + tokenizer helpers + chat loop
tallow.png	his portrait (AI-generated)

Inspired by BeeLLM. Pretrained on TinyStories.