LittleKedi-moderate-wide-sparse-base

LittleKedi-moderate-wide-sparse-base is a small DeepSeek-V3-style Mixture-of-Experts language model trained from scratch. It has 173.4M parameters in total, of which 19.4M are active per token (13.12M excluding the embedding table). It was pretrained on a 12,500M-token mix (FineWeb-Edu, cosmo-v2, wikipedia, FineMath 4+, github-code-clean, SmolTalk).

Part of the LittleKedi family. Not affiliated with DeepSeek.

This is the base model, before any instruction tuning. It continues text; it doesn't follow instructions. It's an educational and research artifact: it writes fluent, on-topic English but knows few facts and confidently makes things up. Don't use it for anything that matters.

The LittleKedi family

LittleKedi is a family of small DeepSeek-V3-style Mixture-of-Experts language models trained from scratch. Every size comes in three variants that differ in how sparse they are: how many routed experts they have, and what share of their parameters a token actually passes through.

Variant	Routed experts
standard (<size>)	a few wide experts; the densest variant
sparse (<size>-sparse)	many narrower, fine-grained experts, more of them active per token
wide-sparse (<size>-wide-sparse)	more experts again; the largest total and smallest active share

More experts means more total capacity to store knowledge, but each expert sees fewer tokens (tokens × top-k ÷ experts per layer), so sparser variants need more training data before their experts are well trained. The variants of a size share a tokenizer and dataset, so they can be compared directly. Shapes change as the family scales; the table below is generated from the current presets.

This card is for moderate-wide-sparse, the wide-sparse moderate model: 19.4M of its 173.4M parameters (11%) are active per token. Its siblings are moderate and moderate-sparse.

Model	Variant	Vocab	Total	Active	Active non-emb.	Active %	Routed experts
tiny	standard	8K	2.0M	1.5M	0.5M	78%	8 × 64, top-2
tiny-sparse	sparse	8K	2.6M	1.6M	0.5M	60%	32 × 32, top-4
tiny-wide-sparse	wide-sparse	8K	3.8M	1.6M	0.5M	42%	64 × 32, top-4
mini	standard	8K	9.4M	4.8M	3.2M	51%	12 × 128, top-3
mini-sparse	sparse	8K	19.8M	4.8M	3.2M	24%	64 × 64, top-6
mini-wide-sparse	wide-sparse	8K	36.4M	4.9M	3.3M	13%	128 × 64, top-6
small	standard	8K	12.2M	6.3M	4.2M	52%	16 × 128, top-4
small-sparse	sparse	8K	24.1M	6.4M	4.3M	27%	80 × 64, top-8
small-wide-sparse	wide-sparse	8K	43.9M	6.5M	4.4M	15%	160 × 64, top-8
moderate	standard	16K	45.5M	18.9M	12.6M	42%	24 × 192, top-4
moderate-sparse	sparse	16K	93.5M	19.1M	12.8M	20%	120 × 96, top-8
moderate-wide-sparse ← this model	wide-sparse	16K	173.4M	19.4M	13.1M	11%	240 × 96, top-8
medium	standard	16K	78.2M	31.0M	22.6M	40%	24 × 256, top-4
medium-sparse	sparse	16K	163.4M	31.3M	22.9M	19%	120 × 128, top-8
medium-wide-sparse	wide-sparse	16K	305.4M	31.7M	23.3M	10%	240 × 128, top-8
base	standard	32K	258.7M	105.4M	80.2M	41%	32 × 256, top-6
base-sparse	sparse	32K	542.8M	106.3M	81.2M	20%	160 × 128, top-12
base-wide-sparse	wide-sparse	32K	1015.9M	107.6M	82.4M	11%	320 × 128, top-12

"Active" counts everything a token passes through: embeddings, attention, the dense layer, shared experts and its top-k routed experts. "Active non-embedding" leaves out the embedding table (tied with the output head), which costs memory but almost no compute; it's the fair number for comparing compute between models.

Architecture

Component	This model
Parameters	173.36M total, 19.41M active per token (11%), 13.12M active non-embedding
Layers / hidden size	8 / 384 (the first 2 layers have a dense SwiGLU FFN of 1056; layers 2–7 are MoE)
Attention	Multi-head Latent Attention (MLA): 8 heads, KV latent 96, query latent 192, decoupled RoPE dim 16, head dims 48+16 (q/k) and 48 (v). The KV cache stores only the 96-dim latent plus a 16-dim RoPE key per token
MoE	DeepSeekMoE: 240 routed experts (SwiGLU, 96 hidden), top-8, plus 2 shared experts; sigmoid gating; group-limited routing (8 groups, top-4)
Load balancing	Auxiliary-loss-free bias balancing (update speed 0.001) plus a small sequence-wise loss (α = 0.0001). Over the second half of training the median batch put 1.69× the mean load on the busiest expert (1.94× in the worst layer), and 0.05% of expert assignments were dropped for capacity.
Expert dispatch	Training: fixed capacity (capacity factor 2.0; overflow assignments are dropped). Inference is always dropless
Embeddings	Input embedding and output head tied
Context	1,024 tokens

Training

Data	12,500M tokens, 11,152,350 documents (~72 tokens per parameter, ~644 per active parameter), one pass, sources interleaved throughout (mix below)
Tokenizer	Byte-level BPE, 16,384 tokens (~4.03 bytes/token on this mix); digits split individually; chat special tokens built in
Steps	190,734 × 65,536 tokens (batch 32 × 1024 × 2 accumulation)
Optimizer	Muon (momentum 0.95, Nesterov, 5 Newton-Schulz steps; each expert's matrix orthogonalized separately) for hidden matrices, AdamW (β = 0.9, 0.95) for embeddings, norms and the router; weight decay 0.1, gradient clipping 1.0
Learning rate	2e-3 peak; WSD: 200 warm-up steps, constant, then linear decay to 0 over the last 20% of steps (schedule set for 190,734 steps)
Precision	bf16 autocast, torch.compile
Compute	~9.8e17 FLOPs (6 × active non-embedding parameters × tokens)
Hardware	One AMD Radeon 890M integrated GPU (Ryzen AI 300 laptop), ROCm on Windows: about 21.2 hours at a median ~163.8K tokens/s

Source	What it is	Tokens	Share
FineWeb-Edu sample/100BT	educational web pages	7,875M	63%
cosmo-v2		1,625M	13%
wikipedia		500M	4%
FineMath 4+	maths web pages	1,250M	10%
github-code-clean	source code	500M	4%
SmolTalk	chat conversations	750M	6%

Evaluation

Metric	Value
Validation loss (nats/token, held-out mix)	2.478
Perplexity	11.9
Bits per byte	0.895
Validation loss: code	1.360
Validation loss: cosmo-v2	1.997
Validation loss: finemath	2.000
Validation loss: fineweb-edu-100bt	2.785
Validation loss: smoltalk	1.494
Validation loss: wikipedia	2.751
Experts never chosen on the validation sample	0

Validation loss over training: 5.04 (16M tok) → 2.89 (1,802M tok) → 2.86 (3,588M tok) → 2.84 (5,374M tok) → 2.84 (7,143M tok) → 2.83 (8,929M tok) → 2.76 (10,715M tok) → 2.48 (12,500M tok)

Zero-shot benchmarks (evals.py)

Task	acc_norm	acc	Items	Chance
SciQ	68.6% ± 1.5%	76.0%	1,000	25%
ARC-Easy	45.2% ± 1.0%	50.4%	2,376	25%
HellaSwag	32.8% ± 0.5%	29.3%	10,042	25%

Each answer choice is scored by the log-probability of its text after the prompt, using lm-evaluation-harness prompts (SciQ includes its supporting passage). acc_norm divides by the answer's length in characters; acc doesn't. Splits: SciQ test, ARC-Easy test, HellaSwag validation.

MMLU (mmlu_eval.py)

Format	Accuracy	Questions scored	Average few-shot examples
letter	26.0%	14,037 (skipped 5)	4.5
cloze	27.4%	14,037 (skipped 5)	0.0

Category	Subjects	Questions	Letter	Cloze
STEM	19	3,153	26.7%	25.5%
Humanities	13	4,705	26.1%	26.0%
Social Sciences	12	3,077	25.8%	29.3%
Other	13	3,102	25.2%	29.5%

Biology & medicine (cloze): 29.1% on 2,089 questions, +4.1 points against 25% chance (about 4.3 standard errors, significant). This pools the 10 life-science and health subjects; it isn't an official MMLU category.

In the letter format the model compares " A"–" D", with same-subject examples added while they fit the context. In the cloze format each answer's text is scored by log-probability per byte; at this size the cloze results are the meaningful ones. Chance is 25%.

Compared with its siblings and other small models

	This model	Pythia-70M	GPT-2	Supra-50M
Parameters (total)	173.4M	70.4M	124.4M	51.8M
Active non-embedding parameters	13.1M	18.9M	85.1M	35.4M
Training tokens	12,500M	300B (the Pile)	undisclosed (WebText)	20B (FineWeb-Edu)
SciQ (acc_norm)	68.6%	55.2%	53.2%	77.2%
ARC-Easy (acc_norm)	45.2%	35.0%	42.0%	52.2%
HellaSwag (acc_norm)	32.8%	–	29.5%	31.8%
MMLU (cloze)	27.4%	–	–	–
MMLU biology & medicine (cloze)	29.1%	–	–	–
Validation loss (held-out mix)	2.478	–	–	–

LittleKedi columns are measured with this repo's evals.py and mmlu_eval.py on the checkpoints named. Validation losses are only comparable between models that share a tokenizer and validation set. The other columns are published figures: Pythia-70M from EleutherAI's evaluations as quoted on the Wisp-15M card, GPT-2 and Supra-50M from the Supra-50M-Base card. Their prompts may differ from ours, so treat gaps of a few points as noise.

What it can and can't do

The sparsest moderate model: about 13.1M active non-embedding parameters per token, spread over 240 narrow experts (173M parameters in total), trained on 12.5B tokens. It's the strongest LittleKedi so far by a wide margin: validation loss 2.478 (~0.89 bits per byte, against ~1.00 for small-wide-sparse; the validation sets differ slightly, but on FineWeb-Edu alone it's ~12% better per byte). That's the biggest step in the family, and it shows: the first LittleKedi that gets some common facts right more often than not, and whose benchmark scores clearly separate from the smaller sizes. It's still a model of the form of text more than its content: specific facts, numbers and code are usually wrong. A research artifact for studying sparsity in small MoE models, not a source of information.

Its first two layers are dense (the preset's other layers are MoE). With only one, the first MoE layer's routing never balanced during training: its top expert choices were nearly tied, so load-balancing made it oscillate.

What it does

• ✅ Writes fluent, well-organised paragraphs in the register of textbooks, encyclopedias and educational web text, staying on topic for a paragraph or more.
• ✅ Gets a few common facts right reliably: "The French Revolution began in" → "1789" (3 of 3 seeds; every smaller LittleKedi missed it), "2 + 2 =" → "4" (3 of 3), "Photosynthesis is the process by which" → "plants convert light energy into chemical energy" (2 of 3), "The heart pumps" → "blood throughout the body, and it carries oxygen". Others are hits among misses: "Paris", Einstein as "a German physicist" or "born in 1879" and the theory of relativity, "Water boils at 100 degrees", each on about one seed in three.
• ✅ Has seen the shape of common code: the recursive Fibonacci ("return fibonacci(n-1) + fibonacci(n-2)"), docstrings, def main():, though the details are wrong (below).
• ✅ The best benchmark scores in the family (zero-shot): SciQ 68.6% (acc_norm; 76.0% acc), ARC-Easy 45.2% (50.4% acc), HellaSwag 32.8%, up from 62.5 / 38.3 / 29.0 for small-wide-sparse.
• ✅ The first LittleKedi clearly above chance on MMLU in the cloze format: 27.4% overall, ~29.4% on the social sciences and "other" categories. STEM is still at chance (25.5%), and the multiple-choice letter format is at chance too (26.0%): it isn't a knowledge benchmark this model can do much with yet.

What it can't do

• ❌ Specific facts. Most names, numbers and dates are still made up: "The chemical symbol for gold is" → "Cu" or "CuSO4"; "The largest planet in the solar system is" → "called Triton"; "Water boils at" → "102 degrees Fahrenheit"; Einstein "born on January 1, 1869 in St. Petersburg"; the French Revolution "ruled by King Louis XIV… under Napoleon III". Right and wrong answers come with the same confidence.
• ❌ Arithmetic beyond the very simplest. "2 + 2 = 4", but "7 + 6 =" → "12" or "9", and the sums it lists afterwards drift ("4 + 0 = 5").
• ❌ Working code. Its Fibonacci has the wrong base case ("if n == 0: return 1", none for 1) and comments with made-up outputs ("# prints 15").
• ❌ Instructions or questions. It's a base model: it continues text. Asked "What is the largest planet in the solar system?" in a User: / Assistant: exchange, it writes another turn of a classroom transcript.
• ❌ Long-range coherence. A paragraph holds together; beyond that it drifts and repeats.

Decoding

Greedy decoding still loops ("The heart is the largest organ in the human body… The heart is the largest muscle in the human body. It is responsible for pumping blood throughout the body. The heart is…"). Use sampling with temperature ≈ 0.7, top-p ≈ 0.9 and a repetition penalty of about 1.15.

Usage

PyTorch (needs the littlekedi package from the training code):

import json, torch
from safetensors.torch import load_file
from littlekedi import LittleKediModel, ModelConfig
from littlekedi.tokenizer import BPETokenizer

model = LittleKediModel(ModelConfig.from_dict(json.load(open("config.json")))).eval()
model.load_state_dict(load_file("model.safetensors"))
tok = BPETokenizer.from_file("tokenizer.json")
ids = torch.tensor([[tok.eot_id] + tok.encode("Photosynthesis is")])
print(tok.decode(model.generate(ids, 60, temperature=0.7, top_p=0.9, repetition_penalty=1.15)[0].tolist()))

Or interactively: python sample.py --ckpt <checkpoint>.pt. Base models are released as PyTorch weights only; GGUF builds come with the instruct models.

Files

File	Contents
model.safetensors	PyTorch weights (embedding tied with the output head), littlekedi parameter names
config.json	ModelConfig for littlekedi
tokenizer.json	Byte-level BPE tokenizer (Hugging Face tokenizers format)
little_kedi.jpg	The LittleKedi logo

Acknowledgements

• The architecture follows DeepSeek-V3 (DeepSeek-AI, 2024): MLA, DeepSeekMoE and auxiliary-loss-free balancing.
• Muon (Keller Jordan et al., 2024) with Moonlight's update scaling (Liu et al., 2025).
• Pretraining data: FineWeb-Edu (ODC-BY 1.0).
• Pretraining data: FineMath 4+ (ODC-BY 1.0).
• Pretraining data: github-code-clean (Apache 2.0, with each file under its own open-source license).
• Pretraining data: SmolTalk (Apache 2.0 for its new subsets, with the datasets it includes under their own licenses).

Card generated 2026-10-19 by prepare_release.py from moderate-wide-sparse.pt (step 190733).