DeepSeek-V3 Technical Report
Paper • 2412.19437 • Published • 88
Tiny DeepSeek-V3 3M
This repository contains a tiny DeepseekV3ForCausalLM Mixture-of-Experts
language model trained from scratch on the full TinyStories training corpus.
The model has 2,803,272 total parameters and approximately 1,807,944 parameters active per token. It retains the main DeepSeek-V3 inference building blocks at a deliberately small scale: Multi-Head Latent Attention (MLA), Q and KV low-rank compression, interleaved RoPE, routed and shared experts, sigmoid top-k routing, and correction-bias-based load balancing.
This is an independently trained synthetic tiny checkpoint. It is not an official DeepSeek model, contains no weights from an original DeepSeek checkpoint, and should not be expected to match the capabilities of production DeepSeek models.
Repository contents
hf/: the final Hugging Face checkpoint and tokenizerexample_generate.py: a minimal local generation exampleeval_text_generation.json: generations from the final training evaluationartifact_metadata.json: training arguments, metrics, router usage, and the expanded configurationdeepseek_v3_config_dump.json: a standalone configuration dump
Optimizer checkpoints, packed training data, and the full training log are intentionally omitted from the distribution package.
Architecture identity
This checkpoint loads through the standard Transformers classes:
DeepseekV3Config
DeepseekV3ForCausalLM
model_type: deepseek_v3
It does not use DeepseekV2ForCausalLM with a renamed model card. Important
V3-specific behavior includes sigmoid routing, normalized selected routing
weights, interleaved RoPE, Q LoRA compression, and an expert correction bias.
Model architecture
architecture: DeepseekV3ForCausalLM
model_type: deepseek_v3
total_parameter_count: 2,803,272
active_parameters_per_token: 1,807,944
model_vocab_size: 1,024
tokenizer_size: 1,003
hidden_size: 216
intermediate_size: 432
num_hidden_layers: 5
first_dense_layers: 1
moe_layers: 4
num_attention_heads: 8
num_key_value_heads: 8
q_lora_rank: 64
kv_lora_rank: 64
qk_nope_head_dim: 16
qk_rope_head_dim: 16
qk_head_dim: 32
v_head_dim: 32
rope_interleave: true
rope_theta: 10,000
routed_experts_per_moe_layer: 4
shared_experts_per_moe_layer: 1
experts_selected_per_token: 1
moe_intermediate_size: 128
n_group: 1
topk_group: 1
norm_topk_prob: true
routed_scaling_factor: 2.5
tie_word_embeddings: true
rms_norm_eps: 1.0e-6
max_position_embeddings: 2,048
Multi-Head Latent Attention
Every layer uses DeepSeek MLA rather than conventional GQA. Queries are compressed through a rank-64 Q projection, while keys and values share a rank-64 latent representation before expansion to eight attention heads. Each query/key head combines a 16-dimensional non-positional component and a 16-dimensional rotary component. Values use 32 dimensions per head.
num_key_value_heads equals num_attention_heads because MLA compresses the
shared KV latent state before expanding it to the attention heads. Setting a
smaller KV-head count would describe GQA, not this MLA implementation.
DeepSeekMoE routing
The first decoder layer is dense. Each of the remaining four layers contains four routed experts and one always-active shared expert. One routed expert is selected for each token.
The V3 router computes sigmoid affinity scores. Selection uses the affinity score plus a non-gradient correction bias, while the routed expert weight uses the original affinity score. During training, the correction bias was adjusted after every batch: overloaded experts were decreased and underloaded experts were increased. A very small complementary sequence-wise balance objective was also used.
v3_bias_update_speed: 0.01
sequence_aux_loss_alpha: 0.0001
no_token_dropping: true
All routed experts received traffic. Aggregate fractions across the complete training run ranged from approximately 0.140 to 0.496 depending on layer and expert. These aggregate values reflect learned routing specialization and are not expected to be exactly uniform.
MTP scope
The checkpoint contains the standard Transformers DeepSeek-V3 causal language model used for inference. It does not contain a training-only Multi-Token Prediction (MTP) draft module:
num_nextn_predict_layers: 0
training_objective: next-token prediction
Transformers 5.14.1 exposes the V3 MTP count in configuration but does not
instantiate or train the original DeepSeek-V3 MTP module in
DeepseekV3ForCausalLM. The checkpoint therefore preserves the V3 main-model
inference graph and V3 routing behavior, but does not claim to reproduce the
complete original V3 pretraining recipe.
Training data
The model was trained on the full TinyStories training corpus using an independent 1% validation split:
selected_stories: 2,119,489
training_stories: 2,098,294
validation_stories: 21,195
validation_fraction: 0.01
training_blocks: 2,441,053
validation_blocks: 24,607
block_size: 256
Stories were joined into a continuous packed stream:
BOS + story 1 + EOS + BOS + story 2 + EOS + ...
The stream was split into fixed 256-token blocks without per-story padding. Only the final incomplete block was discarded.
Tokenizer
The checkpoint uses a custom byte-level BPE tokenizer. The base vocabulary was trained on the 5% TinyStories experiment and frozen for the full run:
BPE()
ByteLevel(add_prefix_space=False)
base_vocab_size: 1,000
min_frequency: 2
normalizer: None
Special tokens use fixed IDs:
<s> -> 1000
</s> -> 1001
<|im_start|> -> 1002
<s> is both BOS and padding, while </s> is EOS. The model reserves 1,024
embedding rows and the tokenizer exposes 1,003 tokens.
The presence of <|im_start|> does not make this a chat model. The checkpoint
is pretrained only and has no chat template or instruction tuning.
Training setup
The checkpoint was trained from scratch in float32 on an NVIDIA GeForce RTX 5060 Ti:
dtype: float32
batch_size: 16
block_size: 256
training_steps: 152,565
epochs: 1.0
tokens_processed: 624,906,240
optimizer: AdamW
learning_rate: 2.0e-4
warmup_steps: 2,000
scheduler: warmup + cosine decay
minimum_learning_rate: 2.0e-5
weight_decay: 0.0
grad_clip: 1.0
optimizer_loop_time: approximately 1h 03m 36s
Evaluation
The final checkpoint produced:
final_train_loss: 1.5201
validation_loss: 1.5254
validation_perplexity: 4.5972
Validation loss was computed over 32 batches, or 131,072 tokens, from the independent packed validation split. These values are compact checkpoint diagnostics, not general language-model benchmark results.
Example generation
A representative final sampled generation begins:
Once upon a time, there was a little girl named Lily. She had long black,
white hair that she loved to play with every day. One day, Lily's mom asked
her to clean up her bedroom. Lily didn't want to clean up, so she started to
pick up her toys and put them away in the closet.
An independent greedy reload test produced:
Once upon a time, there was a little girl named Lily. She loved to play
outside in the sunshine and pick flowers. One day, she saw a big, scary dog
running towards her. The dog ran away and Lily was sad.
Sampling is stochastic. The model produces recognizable TinyStories-style English, but contradictions, incorrect pronouns, invented words, repetition, and unfinished stories remain possible at this size.
Usage
Install the requirements:
pip install -r requirements.txt
Run the included example from the repository root:
python example_generate.py
Local loading:
from pathlib import Path
import torch
from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "shibatch/tinydeepseekv3-3m"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model_dir = Path(snapshot_download(
repo_id=repo,
allow_patterns=["hf/*"],
)) / "hf"
tokenizer = AutoTokenizer.from_pretrained(model_dir)
model = AutoModelForCausalLM.from_pretrained(
model_dir,
dtype=torch.float32,
).to(device).eval()
input_ids = torch.tensor(
[[tokenizer.bos_token_id] + tokenizer.encode(
"Once upon",
add_special_tokens=False,
)],
dtype=torch.long,
device=device,
)
with torch.inference_mode():
output = model.generate(
input_ids=input_ids,
max_new_tokens=100,
do_sample=True,
temperature=0.8,
top_p=0.95,
top_k=40,
repetition_penalty=1.1,
use_cache=True,
pad_token_id=tokenizer.pad_token_id,
eos_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Loading the hf subdirectory from Hugging Face Hub:
from pathlib import Path
from huggingface_hub import snapshot_download
repo_dir = Path(snapshot_download(
repo_id="shibatch/tinydeepseekv3-3m",
allow_patterns=["hf/*"],
))
model_dir = repo_dir / "hf"
Use model_dir with the local loading example above.
Loading requirements
The model was trained, saved, and independently reloaded with:
transformers 5.14.1
torch 2.12.0+cu130
It requires a Transformers release containing DeepseekV3ForCausalLM. No
custom model code or trust_remote_code=True is required.
Intended uses
This model is intended for:
- testing
DeepseekV3ConfigandDeepseekV3ForCausalLM - testing Multi-Head Latent Attention and compressed KV caching
- testing Q and KV low-rank projections
- testing V3 sigmoid MoE routing and shared experts
- testing correction-bias-based expert load balancing
- compact inference-engine and architecture experiments
- testing
generate(),save_pretrained(), andfrom_pretrained()
It is not intended for:
- instruction following or chat
- factual question answering
- reasoning benchmarks
- production deployment
- safety-critical use
- comparison with full-size DeepSeek models
Limitations
- only 2.80 million total parameters
- small 1,003-token tokenizer
- English TinyStories-only pretraining
- no instruction tuning and no chat template
- no MTP training module or MTP objective
- weak factual knowledge and reasoning
- occasional grammatical and semantic errors
- possible mojibake inherited from training-text byte sequences
- no capability-equivalence claim with official DeepSeek models
Citation and references
This is a synthetic tiny DeepSeek-V3-compatible MoE checkpoint trained from scratch on TinyStories. It is intended for implementation validation, debugging, education, and small-scale architecture experiments.