ChonkyKedi-mini-base

ChonkyKedi-mini-base is a small dense Llama-style language model trained from scratch. It has 72.6M parameters (51.6M excluding the embedding table) and was pretrained on a 4.13B-token mix (FineWeb-Edu, Cosmopedia v2 (SmolLM-Corpus), FineWeb, Wikipedia (English, 2023-11-01), FineMath 4+, github-code-clean, SmolTalk).

This is the base model, before any instruction tuning. It continues text; it doesn't follow instructions. It's an educational and research artifact: it writes fluent, on-topic English but knows few facts and confidently makes things up. Don't use it for anything that matters.

The ChonkyKedi family

ChonkyKedi models share one architecture, tokenizer and data pipeline, and differ only in scale. Each is trained on about 80 tokens per non-embedding parameter. The table is generated from the current presets.

Scale	Hidden	Layers	Heads (query / KV)	FFN	Total	Non-embedding
pico	256	6	4 / 2	704	12.8M	4.4M
nano	384	8	6 / 2	1024	25.2M	12.6M
micro	512	10	8 / 2	1408	45.0M	28.2M
mini ← this model	640	12	10 / 2	1728	72.6M	51.6M
small	768	18	12 / 4	2048	138.4M	113.3M
base	1024	24	16 / 4	2752	299.4M	265.9M
medium	2048	24	16 / 4	5504	1,130.5M	1,063.4M
large	3072	28	24 / 8	8192	2,919.4M	2,818.8M

Architecture

A dense, Llama-style decoder-only transformer (llama_gqa).

Component	This model
Parameters	72.60M total, 51.63M excluding the embedding table
Layers / hidden size	12 / 640, pre-norm (RMSNorm, eps 1e-06)
Attention	Grouped-query attention: 10 query heads sharing 2 key/value heads, head dim 64, QK-norm
Positions	RoPE (θ = 10000); YaRN scaling available for longer contexts
Feed-forward	SwiGLU, 1728 hidden
Embeddings	Input embedding and output head tied
Output	Logit soft-cap 30 (cap · tanh(logits / cap))
Context	1,024 tokens

Training

Data	4.13B tokens, one pass (~80 tokens per non-embedding parameter), sources interleaved throughout (mix below)
Tokenizer	Byte-level BPE, 32,768 tokens, with chat, reasoning and tool-call special tokens
Steps	63,019 × 65,536 tokens (batch 16 × 1,024 × 4 accumulation)
Optimizer	AdamW (β = 0.9, 0.95, ε = 1e-8), weight decay 0.1 (not on norms), gradient clipping 1.0
Learning rate	2e-3 peak; WSD: 1,890 warm-up steps, 51,677 constant, then linear decay to 10% of peak over the last 9,452
Precision	bf16 autocast, torch.compile
Compute	~1.3e18 FLOPs (6 × non-embedding parameters × tokens)
Hardware	One NVIDIA GeForce RTX 4090 (RunPod Secure Cloud): 6.6 hours at ~173K tokens/s

Source	What it is	Tokens	Share	Documents dropped by decontamination
FineWeb-Edu sample/100BT	educational web pages	2,271M	55.0%	1,999 of 2,081,792
Cosmopedia v2 (SmolLM-Corpus)	synthetic textbooks and stories	619M	15.0%	618 of 832,768
FineWeb sample/100BT	general web pages	330M	8.0%	159 of 449,792
Wikipedia (English, 2023-11-01)	encyclopedia articles	248M	6.0%	9 of 230,144
FineMath 4+	maths web pages	289M	7.0%	373 of 170,496
github-code-clean	source code	207M	5.0%	0 of 121,600
SmolTalk	chat conversations	165M	4.0%	225 of 178,688

Every training document was checked against every item of the five evaluation sets below (10-gram overlap, plus exact matches of short answers) and dropped on a hit. The validation set comes from shards the training stream never reads.

Evaluation

Metric	Value
Validation loss (nats/token, held-out mix)	3.129
Perplexity	22.9

Validation loss over training: 5.07 (0.03B tok) → 3.53 (0.62B tok) → 3.47 (1.21B tok) → 3.44 (1.80B tok) → 3.42 (2.39B tok) → 3.41 (2.98B tok) → 3.39 (3.57B tok) → 3.13 (4.13B tok)

Zero-shot benchmarks (main.py --mode bench)

Task	acc_norm	acc	Items	Split	Chance
BLiMP	74.5% ± 0.2%	78.3%	67,000	all 67 phenomena	50%
SciQ	67.8% ± 1.0%	74.7%	2,000	validation + test	25%
ARC-Easy	43.6% ± 0.9%	48.3%	2,946	validation + test	25%
PIQA	60.6% ± 1.1%	60.9%	1,838	validation	50%
HellaSwag	29.8% ± 0.5%	28.0%	10,042	validation	25%

Each answer choice is scored by the log-probability of its text after the prompt (BLiMP: the whole sentence of each minimal pair). acc_norm divides by the answer's length in UTF-8 bytes; acc doesn't. Every item of each set is scored, not a sample; ± is one standard error.

Compared with other small models

	This model	Pythia-70M	GPT-2	Supra-50M
Parameters (total)	72.6M	70.4M	124.4M	51.8M
Non-embedding parameters	51.6M	18.9M	85.1M	35.4M
Training tokens	4.13B	300B (the Pile)	undisclosed (WebText)	20B (FineWeb-Edu)
BLiMP (acc_norm)	74.5%	–	–	–
SciQ (acc_norm)	67.8%	55.2%	53.2%	77.2%
ARC-Easy (acc_norm)	43.6%	35.0%	42.0%	52.2%
PIQA (acc_norm)	60.6%	–	–	–
HellaSwag (acc_norm)	29.8%	–	29.5%	31.8%
Validation loss (held-out mix)	3.129	–	–	–

ChonkyKedi columns are measured with this repo's evaluator on the checkpoints named; validation losses are only comparable between models that share a tokenizer and validation set. The other columns are published figures: Pythia-70M from EleutherAI's evaluations as quoted on the Wisp-15M card, GPT-2 and Supra-50M from the Supra-50M-Base card. Their prompts, splits and length normalisation differ from ours, so treat gaps of a few points as noise.

What it can and can't do

This is the first ChonkyKedi trained in the cloud: a dense 72.6M-parameter model (51.6M excluding the embedding table) that reached a validation loss of 3.129 after 4.13B tokens. It writes fluent, well-organised English in the register of textbooks, encyclopedias and children's stories, and it gets the broad frame of common topics right more often than not. But it's a model of the form of text far more than its content: most specific facts, dates and numbers it writes are wrong. It's a research and educational artifact, not a source of information.

What it does

• ✅ Writes fluent, grammatical paragraphs that stay on topic for several sentences. Grammar is its strongest skill: 74.5% on BLiMP across all 67,000 minimal pairs.
• ✅ Tells simple children's stories in the style of its synthetic-textbook data: "Once upon a time, a little cat" → "named Kira lived in a cozy burrow… One sunny afternoon, while playing near the fireplace, Kira met a wise old owl named Oliver."
• ✅ Gets the outline of well-known topics right: "The heart pumps" → "blood to the brain" or "blood to the muscles and other tissues"; "Photosynthesis is" → "a process in which plants convert light energy into chemical energy"; "Albert Einstein was" → "the father of modern physics… developed the theory of relativity".
• ✅ Gets a few very common facts right, often: "The capital of France is" → "Paris" (greedy and two of three sampled seeds); "Water boils at" → "100°C" (greedy only; sampled seeds gave 125 °C and 35 degrees).
• ✅ Reproduces document structure from its data mix: markdown headings and bullet lists, worksheet-style maths pages ("##### Stage: 2 Challenge Level:"), and code-shaped lines.
• ✅ Knows the chat format. About 4% of its pretraining data was SmolTalk conversations, so given <|user|>…<|assistant|> it replies in the assistant's voice, on topic, as a base model rather than an instruction-tuned one.
• ✅ Zero-shot benchmarks (acc_norm, every item scored): SciQ 67.8%, ARC-Easy 43.6%, PIQA 60.6%, HellaSwag 29.8%. That's level with or above GPT-2 (124M parameters) on all three tasks they share, from 4.13B training tokens, though prompts and normalisation differ between published numbers and ours.

What it can't do

• ❌ Facts. Most specific claims are invented: "The largest planet in the solar system is" → "the Sun" (greedy, and two of three seeds), "the Earth" or a planet "58,000 kilometers away from the Sun"; "The chemical symbol for gold is" → "gold" every time; "The French Revolution began in" → "1799" (greedy) and "September 1794"; "Albert Einstein was" → "born in 1882 in Vienna". It got 1789 once in four tries. Treat every name, date and number as made up.
• ❌ Arithmetic. "2 + 2 =" → "3" every time, followed by lists like "3 + 3 = 4, 4 + 4 = 5"; "7 + 6 =" → 10, 12, 16 or 8. Word problems fare no better: a train going 60 miles in 2 hours does "100 miles per hour", "125 miles per hour" or "35 miles per hour".
• ❌ Code. It writes code-shaped text but nothing that runs: def fibonacci(n): → f(n) = fib(n - 1) * fib(n - 1) * fib(n + 1)… or a line of zeros.
• ❌ Instructions. It's a base model with no instruction tuning. In the chat format it answers fluently and confidently but wrongly ("The largest planet in the solar system is the Sun, which is approximately 100 light-years from Earth"), and it ignores the task: asked for a haiku about the sea, it writes prose about the sea.
• ❌ Causal reasoning and history. "The main causes of the First World War were" → "the lack of food, the lack of water and the lack of air".
• ❌ Long-range coherence. It drifts and repeats within a paragraph, and loops on its own sentences when decoding is greedy.
• ❌ Long context. It was trained on 1,024-token sequences. YaRN scaling to 2,048 works (loss 2.65 with YaRN against 3.23 without on the 2× check), but nothing longer was tested.

Decoding

Greedy decoding falls straight into loops ("The symbol for gold is gold. The symbol for gold is gold…"; "It is the seat of the French people. It is the seat of the French people…"). Use sampling with temperature ≈ 0.7 and top-k ≈ 40, the defaults of ChonkyKediModel.generate, and expect to try a few seeds.

Usage

PyTorch (needs the chonkykedi package from the training code):

import torch
from chonkykedi.models.model import ChonkyKediModel
from chonkykedi.tokenizer.tokenizer import ChonkyTokenizer

model = ChonkyKediModel.from_pretrained(".")  # this folder: config.json + model.safetensors
tok = ChonkyTokenizer.load("tokenizer.json")
ids = torch.tensor([tok.encode("Photosynthesis is", add_bos=True)])
out = model.generate(ids, max_new_tokens=60, temperature=0.7, top_k=40)
print(tok.decode(out[0].tolist(), skip_special_tokens=True))

Files

File	Contents
model.safetensors	PyTorch weights, chonkykedi parameter names; the output head is tied to tok_embeddings.weight and not stored
config.json	ModelConfig for chonkykedi
tokenizer.json	Byte-level BPE tokenizer (Hugging Face tokenizers format)
chonky_kedi.jpg	The ChonkyKedi logo

Acknowledgements

• Pretraining data: FineWeb-Edu (ODC-BY 1.0).
• Pretraining data: Cosmopedia v2 (ODC-BY 1.0).
• Pretraining data: FineWeb (ODC-BY 1.0).
• Pretraining data: Wikipedia (CC BY-SA 3.0 and GFDL).
• Pretraining data: FineMath 4+ (ODC-BY 1.0).
• Pretraining data: github-code-clean (Apache 2.0, with each file under its own open-source license).
• Pretraining data: SmolTalk (Apache 2.0 for its new subsets, with the datasets it includes under their own licenses).

Card generated 2026-11-02 by prepare_release.py from llama_gqa_mini_4b.pt.