LittleKedi-moderate-wide-sparse-instruct

LittleKedi-moderate-wide-sparse-instruct is the instruction-tuned (chat) version of LittleKedi-moderate-wide-sparse-base, a small DeepSeek-V3-style Mixture-of-Experts language model trained from scratch. It has 173.4M parameters in total, of which 19.4M are active per token (13.12M excluding the embedding table). It was pretrained on a 12,500M-token mix (FineWeb-Edu, cosmo-v2, wikipedia, FineMath 4+, github-code-clean, SmolTalk).

Part of the LittleKedi family. Not affiliated with DeepSeek.

This is the instruction-tuned (chat) model. It answers in a chat format and stops when it's done, but it's still a very small model: answers are fluent and on-topic but often wrong or vague, and it invents facts confidently. It's an educational and research artifact; don't use it for anything that matters.

Chat format

<|system|>You are a helpful assistant.<|end|><|user|>What causes the seasons?<|end|><|assistant|>

The model writes its reply and ends it with <|end|>. The system turn is optional. The GGUF files embed this template, and chat_template.jinja / tokenizer_config.json carry it for other tools.

The LittleKedi family

LittleKedi is a family of small DeepSeek-V3-style Mixture-of-Experts language models trained from scratch. Every size comes in three variants that differ in how sparse they are: how many routed experts they have, and what share of their parameters a token actually passes through.

Variant	Routed experts
standard (<size>)	a few wide experts; the densest variant
sparse (<size>-sparse)	many narrower, fine-grained experts, more of them active per token
wide-sparse (<size>-wide-sparse)	more experts again; the largest total and smallest active share

More experts means more total capacity to store knowledge, but each expert sees fewer tokens (tokens × top-k ÷ experts per layer), so sparser variants need more training data before their experts are well trained. The variants of a size share a tokenizer and dataset, so they can be compared directly. Shapes change as the family scales; the table below is generated from the current presets.

This card is for moderate-wide-sparse, the wide-sparse moderate model: 19.4M of its 173.4M parameters (11%) are active per token. Its siblings are moderate and moderate-sparse.

Model	Variant	Vocab	Total	Active	Active non-emb.	Active %	Routed experts
tiny	standard	8K	2.0M	1.5M	0.5M	78%	8 × 64, top-2
tiny-sparse	sparse	8K	2.6M	1.6M	0.5M	60%	32 × 32, top-4
tiny-wide-sparse	wide-sparse	8K	3.8M	1.6M	0.5M	42%	64 × 32, top-4
mini	standard	8K	9.4M	4.8M	3.2M	51%	12 × 128, top-3
mini-sparse	sparse	8K	19.8M	4.8M	3.2M	24%	64 × 64, top-6
mini-wide-sparse	wide-sparse	8K	36.4M	4.9M	3.3M	13%	128 × 64, top-6
small	standard	8K	12.2M	6.3M	4.2M	52%	16 × 128, top-4
small-sparse	sparse	8K	24.1M	6.4M	4.3M	27%	80 × 64, top-8
small-wide-sparse	wide-sparse	8K	43.9M	6.5M	4.4M	15%	160 × 64, top-8
moderate	standard	16K	45.5M	18.9M	12.6M	42%	24 × 192, top-4
moderate-sparse	sparse	16K	93.5M	19.1M	12.8M	20%	120 × 96, top-8
moderate-wide-sparse ← this model	wide-sparse	16K	173.4M	19.4M	13.1M	11%	240 × 96, top-8
medium	standard	16K	78.2M	31.0M	22.6M	40%	24 × 256, top-4
medium-sparse	sparse	16K	163.4M	31.3M	22.9M	19%	120 × 128, top-8
medium-wide-sparse	wide-sparse	16K	305.4M	31.7M	23.3M	10%	240 × 128, top-8
base	standard	32K	258.7M	105.4M	80.2M	41%	32 × 256, top-6
base-sparse	sparse	32K	542.8M	106.3M	81.2M	20%	160 × 128, top-12
base-wide-sparse	wide-sparse	32K	1015.9M	107.6M	82.4M	11%	320 × 128, top-12

"Active" counts everything a token passes through: embeddings, attention, the dense layer, shared experts and its top-k routed experts. "Active non-embedding" leaves out the embedding table (tied with the output head), which costs memory but almost no compute; it's the fair number for comparing compute between models.

Architecture

Component	This model
Parameters	173.36M total, 19.41M active per token (11%), 13.12M active non-embedding
Layers / hidden size	8 / 384 (the first 2 layers have a dense SwiGLU FFN of 1056; layers 2–7 are MoE)
Attention	Multi-head Latent Attention (MLA): 8 heads, KV latent 96, query latent 192, decoupled RoPE dim 16, head dims 48+16 (q/k) and 48 (v). The KV cache stores only the 96-dim latent plus a 16-dim RoPE key per token
MoE	DeepSeekMoE: 240 routed experts (SwiGLU, 96 hidden), top-8, plus 2 shared experts; sigmoid gating; group-limited routing (8 groups, top-4)
Load balancing	Auxiliary-loss-free bias balancing (update speed 0.001) plus a small sequence-wise loss (α = 0.0001). Over the second half of training the median batch put 1.69× the mean load on the busiest expert (1.94× in the worst layer), and 0.05% of expert assignments were dropped for capacity.
Expert dispatch	Training: fixed capacity (capacity factor 2.0; overflow assignments are dropped). Inference is always dropless
Embeddings	Input embedding and output head tied
Context	1,024 tokens

Training

Data	12,500M tokens, 11,152,350 documents (~72 tokens per parameter, ~644 per active parameter), one pass, sources interleaved throughout (mix below)
Tokenizer	Byte-level BPE, 16,384 tokens (~4.03 bytes/token on this mix); digits split individually; chat special tokens built in
Steps	190,734 × 65,536 tokens (batch 32 × 1024 × 2 accumulation)
Optimizer	Muon (momentum 0.95, Nesterov, 5 Newton-Schulz steps; each expert's matrix orthogonalized separately) for hidden matrices, AdamW (β = 0.9, 0.95) for embeddings, norms and the router; weight decay 0.1, gradient clipping 1.0
Learning rate	2e-3 peak; WSD: 200 warm-up steps, constant, then linear decay to 0 over the last 20% of steps (schedule set for 190,734 steps)
Precision	bf16 autocast, torch.compile
Compute	~9.8e17 FLOPs (6 × active non-embedding parameters × tokens)
Hardware	One NVIDIA GeForce RTX 4090 (RunPod): about 21.2 hours at a median ~163.8K tokens/s

Source	What it is	Tokens	Share
FineWeb-Edu sample/100BT	educational web pages	7,875M	63%
cosmo-v2		1,625M	13%
wikipedia		500M	4%
FineMath 4+	maths web pages	1,250M	10%
github-code-clean	source code	500M	4%
SmolTalk	chat conversations	750M	6%

Fine-tuning (SFT)

Base model	LittleKedi-moderate-wide-sparse-base
Conversations	12,027 (12,027 train / 558 validation), 1.07M assistant tokens trained on
Data	docs/moderate-sft-data.md
Loss	On assistant replies and their closing `<
Schedule	752 steps, batch 16, 1 epoch, Muon + AdamW, peak LR 2e-4, cosine decay
MoE routing	Load-balancing biases frozen at their pretrained values, with padding excluded from expert-load statistics; experts run dropless

Validation loss on 558 held-out conversations (assistant tokens only): 2.041 → 1.958.

Evaluation

Zero-shot benchmarks (evals.py)

Task	acc_norm	acc	Items	Chance
SciQ	67.3% ± 1.5%	73.7%	1,000	25%
ARC-Easy	45.5% ± 1.0%	48.6%	2,376	25%
HellaSwag	32.2% ± 0.5%	29.2%	10,042	25%

Each answer choice is scored by the log-probability of its text after the prompt, using lm-evaluation-harness prompts (SciQ includes its supporting passage). acc_norm divides by the answer's length in characters; acc doesn't. Splits: SciQ test, ARC-Easy test, HellaSwag validation.

Base vs instruct

Both are scored on the same raw-text prompts, without the chat template, so the numbers are directly comparable. SFT mainly teaches format, turn-taking and when to stop, so large changes in knowledge benchmarks aren't expected. Chance on the multiple-choice tasks is 25%.

Metric	Base	Instruct
SciQ (acc_norm)	68.6%	67.3%
ARC-Easy (acc_norm)	45.2%	45.5%
HellaSwag (acc_norm)	32.8%	32.2%
MMLU (letter)	26.0%	–
MMLU (cloze)	27.4%	–
MMLU biology & medicine (cloze)	29.1%	–

Compared with its siblings and other small models

	This model	Pythia-70M	GPT-2	Supra-50M
Parameters (total)	173.4M	70.4M	124.4M	51.8M
Active non-embedding parameters	13.1M	18.9M	85.1M	35.4M
Training tokens	12,500M	300B (the Pile)	undisclosed (WebText)	20B (FineWeb-Edu)
SciQ (acc_norm)	67.3%	55.2%	53.2%	77.2%
ARC-Easy (acc_norm)	45.5%	35.0%	42.0%	52.2%
HellaSwag (acc_norm)	32.2%	–	29.5%	31.8%
MMLU (cloze)	–	–	–	–
MMLU biology & medicine (cloze)	–	–	–	–
Validation loss (held-out mix)	–	–	–	–

LittleKedi columns are measured with this repo's evals.py and mmlu_eval.py on the checkpoints named. Validation losses are only comparable between models that share a tokenizer and validation set. The other columns are published figures: Pythia-70M from EleutherAI's evaluations as quoted on the Wisp-15M card, GPT-2 and Supra-50M from the Supra-50M-Base card. Their prompts may differ from ours, so treat gaps of a few points as noise.

What it can and can't do

This is the instruction-tuned sparsest moderate model: moderate-wide-sparse-base (13.1M active non-embedding parameters per token, 240 experts, 173M in total, two dense layers) fine-tuned for one epoch on ~12K conversations. It's the most capable LittleKedi so far: it answers common-knowledge questions with the right gist noticeably more often than the smaller models, and it's the first with a measurable skill, getting nearly half of simple arithmetic questions right. It still makes up specific details freely: a research artifact for studying sparsity and instruction tuning at small scale, not a source of information.

Who it is

It answers as LittleKedi, a small cat from Istanbul who has read a lot ("kedi" is Turkish for cat), and says it's a small language model that makes mistakes: "Who are you?" → "I'm LittleKedi, a curious little cat from Istanbul. I've read a great many books, and I like sharing what I've learned!" The persona stays out of ordinary answers.

What it does

• ✅ Replies in the chat format and stops cleanly; greets back ("Hi!" → "Hello! I'm LittleKedi. How can I help you today?").
• ✅ Gets very common facts right more often: "What is the capital of France?" → "The capital of France is Paris."; "What is photosynthesis?" → "the process by which plants, algae, and certain bacteria convert light energy into chemical energy… requires sunlight, water, and carbon dioxide"; Einstein "developed the theory of relativity… in 1905".
• ✅ Gives sensible, well-structured lists for everyday advice (sleep tips: a consistent bedtime routine, a relaxing environment, avoiding caffeine and screens before bed).
• ✅ Simple arithmetic about half the time: 43.7% on 300 held-out one- and two-digit sums, differences and products (greedy decoding), up from 16.7% for small-wide-sparse-instruct.

What it can't do

• ❌ Specific details reliably. Right and wrong facts come with the same confidence: Einstein is "a British physicist… best known for his work on string theory"; a volcano gets a list of invented eruption types.
• ❌ Arithmetic reliably: more than half of answers are still wrong ("7 + 6 = 15").
• ❌ Working code. It writes correctly shaped Python, but solves only 1.8% of 500 held-out MBPP problems (asked for the largest number in a list, it returns sum(nums)).
• ❌ Following instructions about a given text: asked to simplify a sentence, it paraphrases loosely and then rambles on.
• ❌ Counting what it was asked for: asked for three tips, it may give five.

Benchmarks: about where the base model was (SciQ 67.3% vs 68.6%, ARC-Easy 45.5% vs 45.2%, HellaSwag 32.2% vs 32.8%, acc_norm): instruction tuning changed the format, not what it knows.

Decoding: temperature ≈ 0.7, top-p ≈ 0.9, repetition penalty ≈ 1.15 (the GGUF files carry these as defaults), and a modest maximum length. The context window is 1,024 tokens, prompt and reply together.

Usage

llama.cpp (GGUF, llama.cpp's built-in deepseek2 architecture; the chat template is embedded):

llama-cli -m LittleKedi-moderate-wide-sparse-instruct-q8_0.gguf -cnv --temp 0.7 --top-p 0.9 --repeat-penalty 1.15

Ollama (uses the bundled template and params files): ollama run hf.co/edededdy/LittleKedi-moderate-wide-sparse-instruct:Q8_0

PyTorch (needs the littlekedi package from the training code):

import json, torch
from safetensors.torch import load_file
from littlekedi import LittleKediModel, ModelConfig
from littlekedi.chat import END, render_prompt
from littlekedi.tokenizer import BPETokenizer

model = LittleKediModel(ModelConfig.from_dict(json.load(open("config.json")))).eval()
model.load_state_dict(load_file("model.safetensors"))
tok = BPETokenizer.from_file("tokenizer.json")
ids = torch.tensor([render_prompt(tok, [{"role": "user", "content": "What causes the seasons?"}])])
out = model.generate(ids, 200, temperature=0.7, top_p=0.9, repetition_penalty=1.15, stop_ids={tok.token_to_id(END)})
print(tok.decode(out[0, ids.shape[1]:].tolist()).replace(END, "").strip())

Or in a terminal: python chat.py --ckpt <checkpoint>.pt.

Files

File	Contents
model.safetensors	PyTorch weights (embedding tied with the output head), littlekedi parameter names
config.json	ModelConfig for littlekedi
tokenizer.json	Byte-level BPE tokenizer (Hugging Face tokenizers format)
chat_template.jinja	Chat template (Jinja), the same one embedded in the GGUF files
tokenizer_config.json	Special tokens (`<
template	Ollama chat template
params	Ollama stop tokens and sampling defaults
LittleKedi-moderate-wide-sparse-instruct-f16.gguf	GGUF (f16) for llama.cpp / Ollama / LM Studio
LittleKedi-moderate-wide-sparse-instruct-q8_0.gguf	GGUF (q8_0) for llama.cpp / Ollama / LM Studio
little_kedi.jpg	The LittleKedi logo

Acknowledgements

• The architecture follows DeepSeek-V3 (DeepSeek-AI, 2024): MLA, DeepSeekMoE and auxiliary-loss-free balancing.
• Muon (Keller Jordan et al., 2024) with Moonlight's update scaling (Liu et al., 2025).
• Pretraining data: FineWeb-Edu (ODC-BY 1.0).
• Pretraining data: FineMath 4+ (ODC-BY 1.0).
• Pretraining data: github-code-clean (Apache 2.0, with each file under its own open-source license).
• Pretraining data: SmolTalk (Apache 2.0 for its new subsets, with the datasets it includes under their own licenses).
• Chat data: see the SFT table above for its sources, each under its own license.

Card generated 2026-10-21 by prepare_release.py from moderate-wide-sparse-sft.pt (step 190733).