LittleKedi-mini-wide-sparse-base

LittleKedi-mini-wide-sparse-base is a small DeepSeek-V3-style Mixture-of-Experts language model trained from scratch. It has 36.4M parameters in total, of which 4.9M are active per token (3.33M excluding the embedding table). It was pretrained on a 3,200M-token mix (FineWeb-Edu, Cosmopedia (Stanford), Cosmopedia (OpenStax), Cosmopedia (WikiHow), Cosmopedia (Khan Academy), FineMath 4+, github-code-clean, SmolTalk).

Part of the LittleKedi family. Not affiliated with DeepSeek.

This is the base model, before any instruction tuning. It continues text; it doesn't follow instructions. It's an educational and research artifact: it writes fluent, on-topic English but knows few facts and confidently makes things up. Don't use it for anything that matters.

The LittleKedi family

LittleKedi is a family of small DeepSeek-V3-style Mixture-of-Experts language models trained from scratch. Every size comes in three variants that differ in how sparse they are: how many routed experts they have, and what share of their parameters a token actually passes through.

Variant	Routed experts
standard (<size>)	a few wide experts; the densest variant
sparse (<size>-sparse)	many narrower, fine-grained experts, more of them active per token
wide-sparse (<size>-wide-sparse)	more experts again; the largest total and smallest active share

More experts means more total capacity to store knowledge, but each expert sees fewer tokens (tokens × top-k ÷ experts per layer), so sparser variants need more training data before their experts are well trained. The variants of a size share a tokenizer and dataset, so they can be compared directly. Shapes change as the family scales; the table below is generated from the current presets.

This card is for mini-wide-sparse, the wide-sparse mini model: 4.9M of its 36.4M parameters (13%) are active per token. Its siblings are mini and mini-sparse.

Model	Variant	Vocab	Total	Active	Active non-emb.	Active %	Routed experts
tiny	standard	8K	2.0M	1.5M	0.5M	78%	8 × 64, top-2
tiny-sparse	sparse	8K	2.6M	1.6M	0.5M	60%	32 × 32, top-4
tiny-wide-sparse	wide-sparse	8K	3.8M	1.6M	0.5M	42%	64 × 32, top-4
mini	standard	8K	9.4M	4.8M	3.2M	51%	12 × 128, top-3
mini-sparse	sparse	8K	19.8M	4.8M	3.2M	24%	64 × 64, top-6
mini-wide-sparse ← this model	wide-sparse	8K	36.4M	4.9M	3.3M	13%	128 × 64, top-6
small	standard	8K	12.2M	6.3M	4.2M	52%	16 × 128, top-4
small-sparse	sparse	8K	24.1M	6.4M	4.3M	27%	80 × 64, top-8
small-wide-sparse	wide-sparse	8K	43.9M	6.5M	4.4M	15%	160 × 64, top-8
moderate	standard	16K	49.8M	18.8M	12.5M	38%	24 × 192, top-4
moderate-sparse	sparse	16K	105.8M	19.1M	12.8M	18%	120 × 96, top-8
moderate-wide-sparse	wide-sparse	16K	199.0M	19.4M	13.1M	10%	240 × 96, top-8
medium	standard	16K	85.9M	30.8M	22.4M	36%	24 × 256, top-4
medium-sparse	sparse	16K	185.3M	31.2M	22.8M	17%	120 × 128, top-8
medium-wide-sparse	wide-sparse	16K	350.9M	31.6M	23.2M	9%	240 × 128, top-8
base	standard	32K	258.7M	105.4M	80.2M	41%	32 × 256, top-6
base-sparse	sparse	32K	542.8M	106.3M	81.2M	20%	160 × 128, top-12
base-wide-sparse	wide-sparse	32K	1015.9M	107.6M	82.4M	11%	320 × 128, top-12

"Active" counts everything a token passes through: embeddings, attention, the dense layer, shared experts and its top-k routed experts. "Active non-embedding" leaves out the embedding table (tied with the output head), which costs memory but almost no compute; it's the fair number for comparing compute between models.

Architecture

Component	This model
Parameters	36.39M total, 4.91M active per token (13%), 3.33M active non-embedding
Layers / hidden size	8 / 192 (layer 0 has a dense SwiGLU FFN of 512; layers 1–7 are MoE)
Attention	Multi-head Latent Attention (MLA): 6 heads, KV latent 48, decoupled RoPE dim 16, head dims 24+16 (q/k) and 24 (v). The KV cache stores only the 48-dim latent plus a 16-dim RoPE key per token
MoE	DeepSeekMoE: 128 routed experts (SwiGLU, 64 hidden), top-6, plus 2 shared experts; sigmoid gating; group-limited routing (8 groups, top-4)
Load balancing	Auxiliary-loss-free bias balancing (update speed 0.001) plus a small sequence-wise loss (α = 0.0001). Over the second half of training the median batch put 1.41× the mean load on the busiest expert (1.57× in the worst layer), and -0.00% of expert assignments were dropped for capacity.
Expert dispatch	Training: fixed capacity (capacity factor 2.0; overflow assignments are dropped). Inference is always dropless
Embeddings	Input embedding and output head tied
Context	1,024 tokens

Training

Data	3,200M tokens (~88 tokens per parameter, ~652 per active parameter), the first 76% of a 4,200M-token dataset, sources interleaved throughout (mix below)
Tokenizer	Byte-level BPE, 8,192 tokens (~3.70 bytes/token on this mix); digits split individually; chat special tokens built in
Steps	48,828 × 65,536 tokens (batch 64 × 1024)
Optimizer	Muon (momentum 0.95, Nesterov, 5 Newton-Schulz steps; each expert's matrix orthogonalized separately) for hidden matrices, AdamW (β = 0.9, 0.95) for embeddings, norms and the router; weight decay 0.1, gradient clipping 1.0
Learning rate	2e-3 peak; WSD: 200 warm-up steps, constant, then linear decay to 0 over the last 20% of steps (schedule set for 48,828 steps)
Precision	bf16 autocast, torch.compile
Compute	~6.4e16 FLOPs (6 × active non-embedding parameters × tokens)
Hardware	One AMD Radeon 890M integrated GPU (Ryzen AI 300 laptop), ROCm on Windows: about 3.0 hours at a median ~291.5K tokens/s

Source	What it is	Tokens	Share
FineWeb-Edu sample/10BT	educational web pages	2,734M	65%
Cosmopedia (Stanford)	synthetic textbooks	340M	8%
Cosmopedia (OpenStax)	synthetic textbooks	120M	3%
Cosmopedia (WikiHow)	synthetic how-to articles	130M	3%
Cosmopedia (Khan Academy)	synthetic lessons	23M	1%
FineMath 4+	maths web pages	424M	10%
github-code-clean	source code	172M	4%
SmolTalk	chat conversations	256M	6%

Evaluation

Metric	Value
Validation loss (nats/token, held-out mix)	2.645
Perplexity	14.1
Bits per byte	1.044
Validation loss: code	1.570
Validation loss: cosmo-khanacademy	1.938
Validation loss: cosmo-openstax	2.287
Validation loss: cosmo-stanford	2.205
Validation loss: cosmo-wikihow	2.151
Validation loss: finemath	2.267
Validation loss: fineweb-edu	2.961
Validation loss: smoltalk	1.813
Experts never chosen on the validation sample	0

Validation loss over training: 5.08 (16M tok) → 3.01 (475M tok) → 2.94 (934M tok) → 2.91 (1,393M tok) → 2.90 (1,835M tok) → 2.89 (2,294M tok) → 2.81 (2,753M tok) → 2.64 (3,200M tok)

Zero-shot benchmarks (evals.py)

Task	acc_norm	acc	Items	Chance
SciQ	60.7% ± 1.5%	67.7%	1,000	25%
ARC-Easy	37.7% ± 1.0%	39.7%	2,376	25%
HellaSwag	28.2% ± 0.4%	27.5%	10,042	25%

Each answer choice is scored by the log-probability of its text after the prompt, using lm-evaluation-harness prompts (SciQ includes its supporting passage). acc_norm divides by the answer's length in characters; acc doesn't. Splits: SciQ test, ARC-Easy test, HellaSwag validation.

MMLU (mmlu_eval.py)

Format	Accuracy	Questions scored	Average few-shot examples
letter	25.0%	14,037 (skipped 5)	4.5
cloze	26.5%	14,037 (skipped 5)	0.0

Category	Subjects	Questions	Letter	Cloze
STEM	19	3,153	25.2%	25.1%
Humanities	13	4,705	25.5%	25.4%
Social Sciences	12	3,077	22.9%	27.7%
Other	13	3,102	26.0%	28.2%

Biology & medicine (cloze): 28.7% on 2,089 questions, +3.7 points against 25% chance (about 3.9 standard errors, significant). This pools the 10 life-science and health subjects; it isn't an official MMLU category.

In the letter format the model compares " A"–" D", with same-subject examples added while they fit the context. In the cloze format each answer's text is scored by log-probability per byte; at this size the cloze results are the meaningful ones. Chance is 25%.

Compared with its siblings and other small models

	This model	mini	Pythia-70M	GPT-2	Supra-50M
Parameters (total)	36.4M	9.4M	70.4M	124.4M	51.8M
Active non-embedding parameters	3.3M	3.2M	18.9M	85.1M	35.4M
Training tokens	3,200M	1,573M	300B (the Pile)	undisclosed (WebText)	20B (FineWeb-Edu)
SciQ (acc_norm)	60.7%	–	55.2%	53.2%	77.2%
ARC-Easy (acc_norm)	37.7%	–	35.0%	42.0%	52.2%
HellaSwag (acc_norm)	28.2%	–	–	29.5%	31.8%
MMLU (cloze)	26.5%	–	–	–	–
MMLU biology & medicine (cloze)	28.7%	–	–	–	–
Validation loss (held-out mix)	2.645	3.104	–	–	–

LittleKedi columns are measured with this repo's evals.py and mmlu_eval.py on the checkpoints named. Validation losses are only comparable between models that share a tokenizer and validation set. The other columns are published figures: Pythia-70M from EleutherAI's evaluations as quoted on the Wisp-15M card, GPT-2 and Supra-50M from the Supra-50M-Base card. Their prompts may differ from ours, so treat gaps of a few points as noise.

What it can and can't do

This is the sparsest mini model: about the same 3.3M active non-embedding parameters per token as mini, spread over 128 narrow experts (36M parameters in total). It's the strongest LittleKedi so far by a clear margin: validation loss 2.645 (~1.03 bits per byte) after 3.2B tokens. On the same data, it went below anything mini reached in 1.57B tokens after only ~0.4B. It writes noticeably better English than the tiny models and occasionally gets a common fact right, but it's still a model of the form of text far more than its content. It's a research artifact for studying sparsity in small MoE models, not a source of information.

What it does

• ✅ Writes fluent, well-formed paragraphs in the register of textbooks and educational web text, with longer stretches of on-topic prose than the tiny models.
• ✅ Gets the broad frame of well-known topics right more often: "Albert Einstein was" → "one of the most influential figures in modern physics… his theory of general relativity"; "The heart pumps" → "blood to the brain"; "The capital of France is" → "Paris". These are hits among misses: the same prompts with other seeds give "Les Présentures", "Saint-Louis" and "the heart out of the lungs". Don't rely on any of them.
• ✅ Reproduces document structure from its data mix: markdown headings, Q&A-forum threads ("### Reno / What is Newton's law?"), #### answer lines after maths, and code-shaped text (def, if/else, return).
• ✅ The best benchmark scores in the family so far (acc_norm, zero-shot): SciQ 60.7%, ARC-Easy 37.7%, HellaSwag 28.2%. MMLU biology & medicine 28.7% in the cloze format, about 3.9 standard errors above chance. ARC-Easy and SciQ rose steadily over training (ARC-Easy 32.8% at 250M tokens, 37.7% at 3.2B).
• ✅ Overall MMLU is still at chance (26.5% cloze, 25.0% letter): expected at this size.

What it can't do

• ❌ Facts. Most specific claims are wrong: "Water boils at" → "20 degrees Fahrenheit"; "The largest planet in the solar system is" → "a red dwarf star"; "The chemical symbol for gold is" → "blue"; "The French Revolution began in" → "1837". Treat every name, date and number it writes as made up.
• ❌ Arithmetic. "2 + 2 =" → "3", then a list of equally wrong sums; "7 + 6 =" → "10 + 2".
• ❌ Code. It writes code-shaped lines, but nothing that works: def fibonacci(n): → return n, then an unrelated function and a comment that "the above code is not able to solve the problem".
• ❌ Instructions or questions. It's a base model: it continues text. Given a User: / Assistant: exchange it keeps the format but answers with invented numbers ("the largest planet… is 27.7 m deep").
• ❌ Long-range coherence. It stays on topic for longer than tiny, but drifts and repeats within a few sentences ("Photosynthesis is the process by which light and water are produced").

Decoding

Greedy decoding falls straight into loops ("The heart is the heart, the heart is the heart…"). Use sampling with temperature ≈ 0.7, top-p ≈ 0.9 and a repetition penalty of about 1.15.

Usage

PyTorch (needs the littlekedi package from the training code):

import json, torch
from safetensors.torch import load_file
from littlekedi import LittleKediModel, ModelConfig
from littlekedi.tokenizer import BPETokenizer

model = LittleKediModel(ModelConfig.from_dict(json.load(open("config.json")))).eval()
model.load_state_dict(load_file("model.safetensors"))
tok = BPETokenizer.from_file("tokenizer.json")
ids = torch.tensor([[tok.eot_id] + tok.encode("Photosynthesis is")])
print(tok.decode(model.generate(ids, 60, temperature=0.7, top_p=0.9, repetition_penalty=1.15)[0].tolist()))

Or interactively: python sample.py --ckpt <checkpoint>.pt. Base models are released as PyTorch weights only; GGUF builds come with the instruct models.

Files

File	Contents
model.safetensors	PyTorch weights (embedding tied with the output head), littlekedi parameter names
config.json	ModelConfig for littlekedi
tokenizer.json	Byte-level BPE tokenizer (Hugging Face tokenizers format)
little_kedi.jpg	The LittleKedi logo

Acknowledgements

• The architecture follows DeepSeek-V3 (DeepSeek-AI, 2024): MLA, DeepSeekMoE and auxiliary-loss-free balancing.
• Muon (Keller Jordan et al., 2024) with Moonlight's update scaling (Liu et al., 2025).
• Pretraining data: FineWeb-Edu (ODC-BY 1.0).
• Pretraining data: Cosmopedia (Apache 2.0).
• Pretraining data: FineMath 4+ (ODC-BY 1.0).
• Pretraining data: github-code-clean (Apache 2.0, with each file under its own open-source license).
• Pretraining data: SmolTalk (Apache 2.0 for its new subsets, with the datasets it includes under their own licenses).

Card generated 2026-10-15 by prepare_release.py from mini-wide-sparse.pt (step 48827).