LittleKedi-small-wide-sparse-base

LittleKedi-small-wide-sparse-base is a small DeepSeek-V3-style Mixture-of-Experts language model trained from scratch. It has 43.9M parameters in total, of which 6.5M are active per token (4.41M excluding the embedding table). It was pretrained on a 4,200M-token mix (FineWeb-Edu, Cosmopedia (Stanford), Cosmopedia (OpenStax), Cosmopedia (WikiHow), Cosmopedia (Khan Academy), FineMath 4+, github-code-clean, SmolTalk).

Part of the LittleKedi family. Not affiliated with DeepSeek.

This is the base model, before any instruction tuning. It continues text; it doesn't follow instructions. It's an educational and research artifact: it writes fluent, on-topic English but knows few facts and confidently makes things up. Don't use it for anything that matters.

The LittleKedi family

LittleKedi is a family of small DeepSeek-V3-style Mixture-of-Experts language models trained from scratch. Every size comes in three variants that differ in how sparse they are: how many routed experts they have, and what share of their parameters a token actually passes through.

Variant	Routed experts
standard (<size>)	a few wide experts; the densest variant
sparse (<size>-sparse)	many narrower, fine-grained experts, more of them active per token
wide-sparse (<size>-wide-sparse)	more experts again; the largest total and smallest active share

More experts means more total capacity to store knowledge, but each expert sees fewer tokens (tokens × top-k ÷ experts per layer), so sparser variants need more training data before their experts are well trained. The variants of a size share a tokenizer and dataset, so they can be compared directly. Shapes change as the family scales; the table below is generated from the current presets.

This card is for small-wide-sparse, the wide-sparse small model: 6.5M of its 43.9M parameters (15%) are active per token. Its siblings are small and small-sparse.

Model	Variant	Vocab	Total	Active	Active non-emb.	Active %	Routed experts
tiny	standard	8K	2.0M	1.5M	0.5M	78%	8 × 64, top-2
tiny-sparse	sparse	8K	2.6M	1.6M	0.5M	60%	32 × 32, top-4
tiny-wide-sparse	wide-sparse	8K	3.8M	1.6M	0.5M	42%	64 × 32, top-4
mini	standard	8K	9.4M	4.8M	3.2M	51%	12 × 128, top-3
mini-sparse	sparse	8K	19.8M	4.8M	3.2M	24%	64 × 64, top-6
mini-wide-sparse	wide-sparse	8K	36.4M	4.9M	3.3M	13%	128 × 64, top-6
small	standard	8K	12.2M	6.3M	4.2M	52%	16 × 128, top-4
small-sparse	sparse	8K	24.1M	6.4M	4.3M	27%	80 × 64, top-8
small-wide-sparse ← this model	wide-sparse	8K	43.9M	6.5M	4.4M	15%	160 × 64, top-8
moderate	standard	16K	49.8M	18.8M	12.5M	38%	24 × 192, top-4
moderate-sparse	sparse	16K	105.8M	19.1M	12.8M	18%	120 × 96, top-8
moderate-wide-sparse	wide-sparse	16K	199.0M	19.4M	13.1M	10%	240 × 96, top-8
medium	standard	16K	85.9M	30.8M	22.4M	36%	24 × 256, top-4
medium-sparse	sparse	16K	185.3M	31.2M	22.8M	17%	120 × 128, top-8
medium-wide-sparse	wide-sparse	16K	350.9M	31.6M	23.2M	9%	240 × 128, top-8
base	standard	32K	258.7M	105.4M	80.2M	41%	32 × 256, top-6
base-sparse	sparse	32K	542.8M	106.3M	81.2M	20%	160 × 128, top-12
base-wide-sparse	wide-sparse	32K	1015.9M	107.6M	82.4M	11%	320 × 128, top-12

"Active" counts everything a token passes through: embeddings, attention, the dense layer, shared experts and its top-k routed experts. "Active non-embedding" leaves out the embedding table (tied with the output head), which costs memory but almost no compute; it's the fair number for comparing compute between models.

Architecture

Component	This model
Parameters	43.86M total, 6.51M active per token (15%), 4.41M active non-embedding
Layers / hidden size	6 / 256 (layer 0 has a dense SwiGLU FFN of 704; layers 1–5 are MoE)
Attention	Multi-head Latent Attention (MLA): 8 heads, KV latent 64, query latent 128, decoupled RoPE dim 16, head dims 32+16 (q/k) and 32 (v). The KV cache stores only the 64-dim latent plus a 16-dim RoPE key per token
MoE	DeepSeekMoE: 160 routed experts (SwiGLU, 64 hidden), top-8, plus 2 shared experts; sigmoid gating; group-limited routing (8 groups, top-4)
Load balancing	Auxiliary-loss-free bias balancing (update speed 0.001) plus a small sequence-wise loss (α = 0.0001). Over the second half of training the median batch put 1.43× the mean load on the busiest expert (1.59× in the worst layer), and 0.00% of expert assignments were dropped for capacity.
Expert dispatch	Training: fixed capacity (capacity factor 2.0; overflow assignments are dropped). Inference is always dropless
Embeddings	Input embedding and output head tied
Context	1,024 tokens

Training

Data	4,200M tokens, 3,308,154 documents (~96 tokens per parameter, ~645 per active parameter), one pass, sources interleaved throughout (mix below)
Tokenizer	Byte-level BPE, 8,192 tokens (~3.70 bytes/token on this mix); digits split individually; chat special tokens built in
Steps	64,086 × 65,536 tokens (batch 64 × 1024)
Optimizer	Muon (momentum 0.95, Nesterov, 5 Newton-Schulz steps; each expert's matrix orthogonalized separately) for hidden matrices, AdamW (β = 0.9, 0.95) for embeddings, norms and the router; weight decay 0.1, gradient clipping 1.0
Learning rate	2e-3 peak; WSD: 200 warm-up steps, constant, then linear decay to 0 over the last 20% of steps (schedule set for 64,086 steps)
Precision	bf16 autocast, torch.compile
Compute	~1.1e17 FLOPs (6 × active non-embedding parameters × tokens)
Hardware	One AMD Radeon 890M integrated GPU (Ryzen AI 300 laptop), ROCm on Windows: about 4.0 hours at a median ~294.0K tokens/s

Source	What it is	Tokens	Share
FineWeb-Edu sample/10BT	educational web pages	2,734M	65%
Cosmopedia (Stanford)	synthetic textbooks	340M	8%
Cosmopedia (OpenStax)	synthetic textbooks	120M	3%
Cosmopedia (WikiHow)	synthetic how-to articles	130M	3%
Cosmopedia (Khan Academy)	synthetic lessons	23M	1%
FineMath 4+	maths web pages	424M	10%
github-code-clean	source code	172M	4%
SmolTalk	chat conversations	256M	6%

Evaluation

Metric	Value
Validation loss (nats/token, held-out mix)	2.564
Perplexity	13.0
Bits per byte	1.012
Validation loss: code	1.495
Validation loss: cosmo-khanacademy	1.863
Validation loss: cosmo-openstax	2.199
Validation loss: cosmo-stanford	2.109
Validation loss: cosmo-wikihow	2.051
Validation loss: finemath	2.186
Validation loss: fineweb-edu	2.881
Validation loss: smoltalk	1.726
Experts never chosen on the validation sample	0

Validation loss over training: 4.95 (16M tok) → 2.93 (623M tok) → 2.86 (1,212M tok) → 2.84 (1,819M tok) → 2.83 (2,409M tok) → 2.81 (3,015M tok) → 2.75 (3,605M tok) → 2.56 (4,200M tok)

Zero-shot benchmarks (evals.py)

Task	acc_norm	acc	Items	Chance
SciQ	62.5% ± 1.5%	70.2%	1,000	25%
ARC-Easy	38.3% ± 1.0%	42.6%	2,376	25%
HellaSwag	29.0% ± 0.5%	27.6%	10,042	25%

Each answer choice is scored by the log-probability of its text after the prompt, using lm-evaluation-harness prompts (SciQ includes its supporting passage). acc_norm divides by the answer's length in characters; acc doesn't. Splits: SciQ test, ARC-Easy test, HellaSwag validation.

MMLU (mmlu_eval.py)

Format	Accuracy	Questions scored	Average few-shot examples
letter	25.7%	14,037 (skipped 5)	4.5
cloze	26.2%	14,037 (skipped 5)	0.0

Category	Subjects	Questions	Letter	Cloze
STEM	19	3,153	26.7%	24.2%
Humanities	13	4,705	25.2%	25.2%
Social Sciences	12	3,077	24.7%	27.8%
Other	13	3,102	26.5%	27.9%

Biology & medicine (cloze): 27.2% on 2,089 questions, +2.2 points against 25% chance (about 2.4 standard errors, significant). This pools the 10 life-science and health subjects; it isn't an official MMLU category.

In the letter format the model compares " A"–" D", with same-subject examples added while they fit the context. In the cloze format each answer's text is scored by log-probability per byte; at this size the cloze results are the meaningful ones. Chance is 25%.

Compared with its siblings and other small models

	This model	Pythia-70M	GPT-2	Supra-50M
Parameters (total)	43.9M	70.4M	124.4M	51.8M
Active non-embedding parameters	4.4M	18.9M	85.1M	35.4M
Training tokens	4,200M	300B (the Pile)	undisclosed (WebText)	20B (FineWeb-Edu)
SciQ (acc_norm)	62.5%	55.2%	53.2%	77.2%
ARC-Easy (acc_norm)	38.3%	35.0%	42.0%	52.2%
HellaSwag (acc_norm)	29.0%	–	29.5%	31.8%
MMLU (cloze)	26.2%	–	–	–
MMLU biology & medicine (cloze)	27.2%	–	–	–
Validation loss (held-out mix)	2.564	–	–	–

LittleKedi columns are measured with this repo's evals.py and mmlu_eval.py on the checkpoints named. Validation losses are only comparable between models that share a tokenizer and validation set. The other columns are published figures: Pythia-70M from EleutherAI's evaluations as quoted on the Wisp-15M card, GPT-2 and Supra-50M from the Supra-50M-Base card. Their prompts may differ from ours, so treat gaps of a few points as noise.

What it can and can't do

The sparsest small model: about 4.4M active non-embedding parameters per token, spread over 160 narrow experts (44M parameters in total). It's the strongest LittleKedi so far: validation loss 2.564 (~1.00 bits per byte) after 4.2B tokens, 0.08 below mini-wide-sparse, and below it at every point of training (by a steady ~0.055 on the same batches before the learning-rate decay). That's a real step, but a modest one: it writes slightly more fluent text and gets a few more common facts right, and is still a model of the form of text far more than its content. It's a research artifact for studying sparsity in small MoE models, not a source of information.

What it does

• ✅ Writes fluent paragraphs in the register of textbooks and educational web text, with full, grammatical sentences and topic-appropriate vocabulary throughout.
• ✅ Gets the broad frame of well-known topics right somewhat more often: "The heart pumps" → "blood throughout the body"; "Photosynthesis is the process by which" → "plants convert light energy into chemical energy"; "The capital of France is" → "Paris". These are still hits among misses: other seeds give "the capital of France" (circular), "photosynthetic microbes convert carbon dioxide into oxygen" and "blood and oxygen through the nose".
• ✅ Reproduces document structure from its data mix: Q&A lists, markdown headings, numbered exercises and code-shaped text. It has seen the recursive Fibonacci pattern ("return fibonacci(n-1) + fibonacci(n-2)"), but without the base case.
• ✅ The best benchmark scores in the family so far (acc_norm, zero-shot): SciQ 62.5%, ARC-Easy 38.3%, HellaSwag 29.0%, up from 60.7 / 37.7 / 28.2 for mini-wide-sparse. HellaSwag's gain is the most reliable (10,042 questions, ±0.45); ARC-Easy's is within noise.
• ✅ MMLU is still at chance, as expected at this size: 26.2% in the cloze format, 25.7% with letters. No subject group stands out: biology & medicine is 27.2% (2.4 standard errors above chance, down from mini-wide-sparse's 28.7%, so treat neither as a real skill).

What it can't do

• ❌ Facts. Most specific claims are wrong: "Water boils at" → "20 degrees Celsius"; "The largest planet in the solar system is" → "Earth, which is 215 times bigger than the Sun"; "The chemical symbol for gold is" → "gold"; "Albert Einstein was" → "born in 1869" or "a professor of physics at the University of New South Wales"; "The French Revolution began in" → "1798", "1785", "1760". Treat every name, date and number it writes as made up.
• ❌ Arithmetic. "2 + 2 =" → "3", "10" or "6", then a list of equally wrong sums; "7 + 6 =" → "10" or "9".
• ❌ Code. It writes code-shaped lines, but nothing that works: the Fibonacci recursion above has no base case, and other seeds loop over range(2, n + 1) and return n // 2.
• ❌ Instructions or questions. It's a base model: it continues text. Asked "What is the largest planet in the solar system?" in a User: / Assistant: exchange, it answers with a science-fair announcement at NASA's JPL.
• ❌ Long-range coherence. It stays on topic for a paragraph, but drifts and contradicts itself within a few sentences ("Photosynthesis… convert sunlight into electricity and produce oxygen, which then releases carbon dioxide").

Decoding

Greedy decoding falls straight into loops ("The heart is the largest organ in the body and is responsible for the body's metabolism. It is the largest organ in the body…"). Use sampling with temperature ≈ 0.7, top-p ≈ 0.9 and a repetition penalty of about 1.15.

Usage

PyTorch (needs the littlekedi package from the training code):

import json, torch
from safetensors.torch import load_file
from littlekedi import LittleKediModel, ModelConfig
from littlekedi.tokenizer import BPETokenizer

model = LittleKediModel(ModelConfig.from_dict(json.load(open("config.json")))).eval()
model.load_state_dict(load_file("model.safetensors"))
tok = BPETokenizer.from_file("tokenizer.json")
ids = torch.tensor([[tok.eot_id] + tok.encode("Photosynthesis is")])
print(tok.decode(model.generate(ids, 60, temperature=0.7, top_p=0.9, repetition_penalty=1.15)[0].tolist()))

Or interactively: python sample.py --ckpt <checkpoint>.pt. Base models are released as PyTorch weights only; GGUF builds come with the instruct models.

Files

File	Contents
model.safetensors	PyTorch weights (embedding tied with the output head), littlekedi parameter names
config.json	ModelConfig for littlekedi
tokenizer.json	Byte-level BPE tokenizer (Hugging Face tokenizers format)
little_kedi.jpg	The LittleKedi logo

Acknowledgements

• The architecture follows DeepSeek-V3 (DeepSeek-AI, 2024): MLA, DeepSeekMoE and auxiliary-loss-free balancing.
• Muon (Keller Jordan et al., 2024) with Moonlight's update scaling (Liu et al., 2025).
• Pretraining data: FineWeb-Edu (ODC-BY 1.0).
• Pretraining data: Cosmopedia (Apache 2.0).
• Pretraining data: FineMath 4+ (ODC-BY 1.0).
• Pretraining data: github-code-clean (Apache 2.0, with each file under its own open-source license).
• Pretraining data: SmolTalk (Apache 2.0 for its new subsets, with the datasets it includes under their own licenses).

Card generated 2026-10-16 by prepare_release.py from small-wide-sparse.pt (step 64085).