ChonkyKedi-pico-base

ChonkyKedi-pico-base is a small dense Llama-style language model trained from scratch. It has 12.8M parameters (4.4M excluding the embedding table) and was pretrained on a 1.00B-token mix (FineWeb-Edu, Cosmopedia v2 (SmolLM-Corpus), FineMath 4+, SmolTalk, Wikipedia (English, 2023-11-01), github-code-clean).

This is the base model, before any instruction tuning. It continues text; it doesn't follow instructions. It's an educational and research artifact: it writes fluent, on-topic English but knows few facts and confidently makes things up. Don't use it for anything that matters.

The ChonkyKedi family

ChonkyKedi models share one architecture, tokenizer and data pipeline, and differ only in scale. Each is trained on about 80 tokens per non-embedding parameter. The table is generated from the current presets.

Scale	Hidden	Layers	Heads (query / KV)	FFN	Total	Non-embedding
pico ← this model	256	6	4 / 2	704	12.8M	4.4M
nano	384	8	6 / 2	1024	25.2M	12.6M
micro	512	10	8 / 2	1408	45.0M	28.2M
mini	640	12	10 / 2	1728	72.6M	51.6M
small	768	18	12 / 4	2048	138.4M	113.3M
base	1024	24	16 / 4	2752	299.4M	265.9M
medium	2048	24	16 / 4	5504	1,130.5M	1,063.4M
large	3072	28	24 / 8	8192	2,919.4M	2,818.8M

Architecture

A dense, Llama-style decoder-only transformer (llama_gqa).

Component	This model
Parameters	12.82M total, 4.43M excluding the embedding table
Layers / hidden size	6 / 256, pre-norm (RMSNorm, eps 1e-06)
Attention	Grouped-query attention: 4 query heads sharing 2 key/value heads, head dim 64, QK-norm
Positions	RoPE (θ = 10000); YaRN scaling available for longer contexts
Feed-forward	SwiGLU, 704 hidden
Embeddings	Input embedding and output head tied
Output	Logit soft-cap 30 (cap · tanh(logits / cap))
Context	1,024 tokens

Training

Data	1.00B tokens, one pass (~226 tokens per non-embedding parameter), sources interleaved throughout (mix below)
Tokenizer	Byte-level BPE, 32,768 tokens, with chat, reasoning and tool-call special tokens
Steps	61,035 × 16,384 tokens (batch 4 × 1,024 × 4 accumulation)
Optimizer	AdamW (β = 0.9, 0.95, ε = 1e-8), weight decay 0.1 (not on norms), gradient clipping 1.0
Learning rate	2.5e-3 peak; WSD: 1,831 warm-up steps, 50,049 constant, then linear decay to 10% of peak over the last 9,155
Precision	bf16 autocast
Compute	~2.7e16 FLOPs (6 × non-embedding parameters × tokens)
Hardware	One AMD Radeon 890M (laptop iGPU, ROCm on Windows): 97.3 hours at ~5K tokens/s

Source	What it is	Tokens	Share
FineWeb-Edu sample/100BT	educational web pages	600M	60.0%
Cosmopedia v2 (SmolLM-Corpus)	synthetic textbooks and stories	150M	15.0%
FineMath 4+	maths web pages	100M	10.0%
SmolTalk	chat conversations	50M	5.0%
Wikipedia (English, 2023-11-01)	encyclopedia articles	50M	5.0%
github-code-clean	source code	50M	5.0%

79 documents were dropped by decontamination in total.

Every training document was checked against every item of the five evaluation sets below (10-gram overlap, plus exact matches of short answers) and dropped on a hit. The validation set comes from shards the training stream never reads.

Evaluation

Metric	Value
Validation loss (nats/token, held-out mix)	3.885
Perplexity	48.7

Validation loss over training: 10.43 (0.00B tok) → 4.31 (0.15B tok) → 4.27 (0.29B tok) → 4.24 (0.43B tok) → 4.23 (0.57B tok) → 4.22 (0.72B tok) → 4.20 (0.86B tok) → 3.88 (1.00B tok)

Zero-shot benchmarks (main.py --mode bench)

Task	acc_norm	acc	Items	Split	Chance
BLiMP	66.0% ± 0.2%	70.2%	67,000	all 67 phenomena	50%
SciQ	56.0% ± 1.1%	57.8%	2,000	validation + test	25%
ARC-Easy	33.2% ± 0.9%	36.0%	2,946	validation + test	25%
PIQA	56.6% ± 1.2%	56.4%	1,838	validation	50%
HellaSwag	26.5% ± 0.4%	26.4%	10,042	validation	25%

Each answer choice is scored by the log-probability of its text after the prompt (BLiMP: the whole sentence of each minimal pair). acc_norm divides by the answer's length in UTF-8 bytes; acc doesn't. Every item of each set is scored, not a sample; ± is one standard error.

Compared with other small models

	This model	ChonkyKedi-nano-base	ChonkyKedi-mini-base	Pythia-70M	GPT-2	Supra-50M
Parameters (total)	12.8M	25.2M	72.6M	70.4M	124.4M	51.8M
Non-embedding parameters	4.4M	12.6M	51.6M	18.9M	85.1M	35.4M
Training tokens	1.00B	1.00B	4.13B	300B (the Pile)	undisclosed (WebText)	20B (FineWeb-Edu)
BLiMP (acc_norm)	66.0%	67.7%	74.5%	–	–	–
SciQ (acc_norm)	56.0%	62.7%	67.8%	55.2%	53.2%	77.2%
ARC-Easy (acc_norm)	33.2%	36.8%	43.6%	35.0%	42.0%	52.2%
PIQA (acc_norm)	56.6%	56.9%	60.6%	–	–	–
HellaSwag (acc_norm)	26.5%	26.7%	29.8%	–	29.5%	31.8%
Validation loss (held-out mix)	3.885	3.676	3.129	–	–	–

ChonkyKedi columns are measured with this repo's evaluator on the checkpoints named; validation losses are only comparable between models that share a tokenizer and validation set. The other columns are published figures: Pythia-70M from EleutherAI's evaluations as quoted on the Wisp-15M card, GPT-2 and Supra-50M from the Supra-50M-Base card. Their prompts, splits and length normalisation differ from ours, so treat gaps of a few points as noise.

What it can and can't do

This is the smallest ChonkyKedi: a dense 12.8M-parameter model (4.4M excluding the embedding table, so about two thirds of it is the embedding table) trained from scratch on a laptop's integrated GPU, reaching a validation loss of 3.885 after 1.00B tokens. That's the same data as nano, at about 226 tokens per non-embedding parameter. It writes English-shaped sentences that mostly parse and mostly stay near the prompt's topic, but it knows almost nothing, loses track within a sentence or two, and repeats itself constantly. It's a research and educational artifact, the bottom rung of the ChonkyKedi ladder, not a source of information.

What it does

• ✅ Writes short grammatical sentences. Grammar is its strongest skill: 66.0% on BLiMP across all 67,000 minimal pairs.
• ✅ Starts children's stories in the right register: "Once upon a time, a little cat" → ", the best friend, was living in a special place. He was always curious and loved exploring the vast ocean". It can't keep the story straight, though: in another try the cat is "a young boy who loved to learn new things", and in another "a little dog named John… a small cat named John".
• ✅ Stays in the neighbourhood of a topic: "The French Revolution began in" → "the 19th century, when the French were being formed"; "Albert Einstein was" → "born in the United States in 1864. He was a student of the American Academy of Sciences". The vocabulary fits; the facts don't.
• ✅ Knows the chat format a little. Given <|user|>…<|assistant|> it answers in a reply-like sentence, once even opening with "Sure, I've taken a journey through the heart of the sea", but unlike nano it rarely ends its turn.
• ✅ Zero-shot benchmarks (acc_norm, every item scored): SciQ 56.0%, ARC-Easy 33.2%, PIQA 56.6%, HellaSwag 26.5% (barely above its 25% chance level). Nano, trained on the same 1.00B tokens with about 3× pico's non-embedding parameters, is ahead on all five, though by only 0.3 points on PIQA and 0.2 on HellaSwag. SciQ is level with Pythia-70M and above GPT-2's published figure; prompts and normalisation differ between published numbers and ours.

What it can't do

• ❌ Facts. None of the factual prompts came out right in four tries each (greedy plus three seeds): "The capital of France is" → "the capital of France", "the state of France" (it mentions Paris once, in passing, without saying it's the capital); "Water boils at" → "the water level"; "The largest planet in the solar system is" → "the Sun" (three of four); "The chemical symbol for gold is" → "the symbol of gold"; "The French Revolution began in" → "1864", "1849". Treat every name, date and number as made up.
• ❌ Arithmetic. "2 + 2 =" → "2 + 2 + 2 + 2…", 109, 6, 90; "7 + 6 =" → a string of zeros, 109, 6, 90.
• ❌ Code. def fibonacci(n): → the same line repeated, def f(f(f(f(…, or u, u, u, ….
• ❌ Instructions. No instruction tuning. Asked for a haiku about the sea: "The sea is a vast, vast, and vast, with vast oceans"; asked for the largest planet: "The largest planet in the solar system is the largest planet in the solar system."
• ❌ Coherence. It loses the thread within a sentence or two and falls into loops even when sampling: "the heart pumps, the heart pumps, the heart pumps…", "la la la la…", "ever ever ever ever…". It loops far more than nano does.
• ❌ Long context. It was trained on 1,024-token sequences; nothing longer was tested.

Decoding

Greedy decoding loops at once ("It is the capital of France. It is the capital of France…"; "the heart to the heart to the heart…"), and even sampled outputs often fall into a loop within 50 tokens. Use sampling with temperature ≈ 0.7 and top-k ≈ 40, the defaults of ChonkyKediModel.generate, keep outputs short, and expect to try several seeds.

Usage

PyTorch (needs the chonkykedi package from the training code):

import torch
from chonkykedi.models.model import ChonkyKediModel
from chonkykedi.tokenizer.tokenizer import ChonkyTokenizer

model = ChonkyKediModel.from_pretrained(".")  # this folder: config.json + model.safetensors
tok = ChonkyTokenizer.load("tokenizer.json")
ids = torch.tensor([tok.encode("Photosynthesis is", add_bos=True)])
out = model.generate(ids, max_new_tokens=60, temperature=0.7, top_k=40)
print(tok.decode(out[0].tolist(), skip_special_tokens=True))

Files

File	Contents
model.safetensors	PyTorch weights, chonkykedi parameter names; the output head is tied to tok_embeddings.weight and not stored
config.json	ModelConfig for chonkykedi
tokenizer.json	Byte-level BPE tokenizer (Hugging Face tokenizers format)
chonky_kedi.jpg	The ChonkyKedi logo

Acknowledgements

• Pretraining data: FineWeb-Edu (ODC-BY 1.0).
• Pretraining data: Cosmopedia v2 (ODC-BY 1.0).
• Pretraining data: FineMath 4+ (ODC-BY 1.0).
• Pretraining data: SmolTalk (Apache 2.0 for its new subsets, with the datasets it includes under their own licenses).
• Pretraining data: Wikipedia (CC BY-SA 3.0 and GFDL).
• Pretraining data: github-code-clean (Apache 2.0, with each file under its own open-source license).

Card generated 2026-11-08 by prepare_release.py from llama_gqa_pico_1b.pt.