shibatch/tinydeepseekv2-3m

Tiny DeepSeek V2 3M

This repository contains a tiny DeepseekV2ForCausalLM Mixture-of-Experts language model trained from scratch on TinyStories.

The model has 2,928,392 parameters. It combines DeepSeek V2-style Multi-head Latent Attention (MLA), compressed key/value states, decoupled RoPE, routed experts, and a shared expert in a compact checkpoint intended for implementation testing and architecture experiments.

This is a synthetic tiny checkpoint. It is not an official DeepSeek model, does not contain weights from an original DeepSeek checkpoint, and should not be expected to match the quality or capabilities of production DeepSeek models.

Repository contents

• hf/: the final Hugging Face checkpoint and tokenizer
• example_generate.py: a minimal generation example
• eval_text_generation.json: generations from the final training evaluation
• artifact_metadata.json: training arguments, metrics, router usage, and the expanded configuration
• deepseek_v2_config_dump.json: a standalone configuration dump

Optimizer checkpoints and the full training history are intentionally omitted from this distribution package.

Architecture identity

This checkpoint uses:

DeepseekV2Config
DeepseekV2ForCausalLM
model_type: deepseek_v2

It is specifically a DeepSeek V2 architecture, not DeepSeek V3 and not a generic dense decoder renamed as DeepSeek. The checkpoint exercises the DeepSeek V2 MLA and MoE implementations provided by Hugging Face Transformers.

Model architecture

architecture: DeepseekV2ForCausalLM
model_type: deepseek_v2
parameter_count: 2,928,392

model_vocab_size: 1,024
tokenizer_size: 1,003
hidden_size: 216
intermediate_size: 432
num_hidden_layers: 5

num_attention_heads: 8
num_key_value_heads: 8
qk_nope_head_dim: 16
qk_rope_head_dim: 16
v_head_dim: 32
q_lora_rank: null
kv_lora_rank: 64

first_k_dense_replace: 1
n_routed_experts: 4
n_shared_experts: 1
num_experts_per_tok: 1
moe_intermediate_size: 128
topk_method: greedy
norm_topk_prob: false
routed_scaling_factor: 4.0
aux_loss_alpha: 0.01
seq_aux: true

tie_word_embeddings: true
rms_norm_eps: 1.0e-6
max_position_embeddings: 2,048
rope_theta: 10,000

Multi-head Latent Attention

Each attention head separates its query/key dimensions into:

16 non-positional dimensions + 16 rotary dimensions

Values use 32 dimensions per head. The key/value path is compressed through a 64-dimensional latent projection (kv_lora_rank: 64) before being expanded for attention. Query LoRA is disabled in this tiny configuration, while the compressed KV path and decoupled rotary/non-rotary query-key components remain active.

This keeps the checkpoint small while exercising the defining DeepSeek V2 MLA code paths.

Dense and MoE layers

The first decoder layer uses a dense MLP. The remaining four layers use DeepSeek V2 MoE blocks:

layer 0: dense MLP
layer 1: 4 routed experts, top-1 + 1 shared expert
layer 2: 4 routed experts, top-1 + 1 shared expert
layer 3: 4 routed experts, top-1 + 1 shared expert
layer 4: 4 routed experts, top-1 + 1 shared expert

Every token is sent to one routed expert, while the shared expert is evaluated for all tokens. Sequence-level router auxiliary loss was enabled during training.

All routed experts received traffic. Final aggregate routing fractions were close to 25% for every expert:

MoE layer	Expert 0	Expert 1	Expert 2	Expert 3
1	0.2502	0.2487	0.2522	0.2488
2	0.2538	0.2481	0.2487	0.2495
3	0.2512	0.2454	0.2519	0.2514
4	0.2465	0.2535	0.2542	0.2458

Training data

The model was trained on the full TinyStories training corpus using an independent 1% validation split:

selected_stories: 2,119,489
training_stories: 2,098,294
validation_stories: 21,195
validation_fraction: 0.01

training_blocks: 2,441,089
validation_blocks: 24,608
block_size: 256

Stories were joined into a continuous packed stream:

BOS + story 1 + EOS + BOS + story 2 + EOS + ...

The stream was split into fixed 256-token blocks without padding. Only the final incomplete block was discarded.

Tokenizer

The checkpoint uses a small custom byte-level BPE tokenizer. The base BPE vocabulary was trained with:

BPE()
ByteLevel(add_prefix_space=False)
base_vocab_size: 1,000
min_frequency: 2
normalizer: None

Special tokens were then added at fixed IDs:

<s>          -> 1000
</s>         -> 1001
<|im_start|> -> 1002

The tokenizer uses <s> as both BOS and padding, and </s> as EOS. The model configuration reserves 1,024 embedding rows while the tokenizer exposes 1,003 tokens.

Training setup

The checkpoint was trained from scratch in float32 on an NVIDIA GeForce GTX 1650:

dtype: float32
batch_size: 16
block_size: 256
training_steps: 152,568
epochs: 1.0
tokens_processed: 624,918,528

optimizer: AdamW
learning_rate: 3.0e-4
warmup_steps: 2,000
scheduler: warmup + cosine decay
minimum_learning_rate: 3.0e-5
weight_decay: 0.0
grad_clip: 1.0

Evaluation

The final checkpoint produced:

final_train_loss: 1.4096
validation_loss: 1.4226
validation_perplexity: 4.1478

Validation loss was computed over 32 batches, or 131,072 tokens, from the independent packed validation split. These values are compact checkpoint diagnostics, not general language-model benchmark results.

Example generation

One final evaluation sample for prompt There was a little was:

There was a little girl named Lily who loved to explore. One day, she saw a
big bush with lots of yummy peaches on it. She wanted to eat one, but her mom
said no.

Lily didn't listen and kept trying to eat her peach.

Sampling is stochastic. The model usually produces recognizable TinyStories-style English, but semantic contradictions, unfinished sentences, invented words, and repetition remain possible at this size.

Usage

Install the requirements:

pip install -r requirements.txt

Run the included local example from the repository root:

python example_generate.py

To load the package from Hugging Face Hub, resolve the hf directory to a local path first. This avoids a Transformers 5.14.1 local-subfolder issue in which generation configuration lookup may incorrectly fall back to the repository root:

from pathlib import Path

import torch
from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "shibatch/tinydeepseekv2-3m"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model_dir = Path(snapshot_download(
    repo_id=repo,
    allow_patterns=["hf/*"],
)) / "hf"

tokenizer = AutoTokenizer.from_pretrained(model_dir)
model = AutoModelForCausalLM.from_pretrained(
    model_dir,
    dtype=torch.float32,
).to(device)
model.eval()

prompt = "There was a little"
input_ids = torch.tensor(
    [[tokenizer.bos_token_id] + tokenizer.encode(
        prompt,
        add_special_tokens=False,
    )],
    dtype=torch.long,
    device=device,
)

torch.manual_seed(0)
if device.type == "cuda":
    torch.cuda.manual_seed_all(0)

with torch.no_grad():
    output = model.generate(
        input_ids=input_ids,
        max_new_tokens=100,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        top_k=40,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0].tolist(), skip_special_tokens=True))

Loading requirements

The checkpoint requires a Transformers release containing DeepseekV2ForCausalLM. It was trained and tested with:

transformers 5.14.1
torch 2.14.0.dev20260720+cu126

Intended uses

This model is intended for:

• testing DeepseekV2Config and DeepseekV2ForCausalLM
• testing Multi-head Latent Attention and compressed KV states
• exercising decoupled rotary and non-rotary query/key dimensions
• testing dense-to-MoE layer transitions
• testing top-1 routed experts and shared experts
• checking sequence-level router load-balancing loss
• exercising custom tokenizer loading
• testing generate(), save_pretrained(), and from_pretrained()
• compact inference-engine and architecture experiments

It is not intended for:

• instruction following or chat
• factual question answering
• high-quality long-form generation
• production deployment
• safety-critical use
• benchmark comparison with production DeepSeek models

Limitations

Known limitations include:

• only 2.93 million parameters
• small 1,003-token tokenizer
• English TinyStories-only pretraining
• weak factual knowledge and reasoning
• no instruction tuning or chat template
• occasional grammatical and semantic errors
• invented words and truncated sentences
• repetition and template-like stories
• no quality-equivalence claim with official DeepSeek models
• no current llama.cpp or GGUF inference support assumed for this exact DeepSeek V2 MLA/MoE configuration

Notes on GGUF

The checkpoint is distributed as a normal float32 Hugging Face Safetensors model. A useful GGUF build requires converter and runtime support for this DeepSeek V2 MLA and MoE graph, including compressed KV projections, decoupled RoPE, routed experts, and the shared expert. Merely placing tensors in a GGUF container is not sufficient for compatible inference.

Citation

This is a synthetic tiny DeepSeek V2-compatible MoE checkpoint trained from scratch on TinyStories. It is intended for implementation validation, debugging, education, and small-scale architecture experiments.