DeepSeek-V4-Pro-3.7B-A3.4B-small

This is a small version of deepseek-ai/DeepSeek-V4-Pro created for testing and development. Unlike the previous "tiny" models, all hidden dimensions and hyperparameters are kept identical to the base model; only the layer count, layer types per position, and expert count are reduced.

Model Details

• Base Model: deepseek-ai/DeepSeek-V4-Pro
• Architecture: deepseek_v4 (DeepseekV4ForCausalLM)
• Total Parameters: 3.684B
• Activated Parameters: ~3.42B (6 of 8 routed experts per token + 1 shared expert, topk unchanged from base)

Configuration Changes

The following parameters were reduced from the original model:

Parameter	Original	Small
num_hidden_layers	61	2
layer_types	61 entries (heavily_compressed_attention / compressed_sparse_attention mixed)	[heavily_compressed_attention, compressed_sparse_attention]
mlp_layer_types	61 entries ([hash_moe x3, moe x58])	[hash_moe, moe]
n_routed_experts	384	8

All other hyperparameters are identical to the base: hidden_size 7168, vocab_size 129280, moe_intermediate_size 3072, num_experts_per_tok 6 (topk unchanged), n_shared_experts 1, q_lora_rank 1536, o_lora_rank 1024, o_groups 16, head_dim 512, index_n_heads 64, index_head_dim 128, index_topk 1024, compress_rates {csa: 4, hca: 128}, hc_mult 4, sliding_window 128, rope parameters, expert_dtype: fp4, and num_nextn_predict_layers: 1.

Both attention types (heavily_compressed_attention, compressed_sparse_attention) and both MLP types (hash_moe, moe) are preserved, using (attention, mlp) combinations that occur in the base (layers 0-1 of the base are heavily_compressed_attention+hash_moe; compressed_sparse_attention+moe occurs from layer 3 on), preserving the base's ordering style (hash_moe layers first). As in the base, HCA layers carry no Lightning Indexer weights while CSA layers do.

Note: the large parameter count (~3.7B, dominated by the two untied 129280x7168 embedding matrices) is expected with fixed base dims; the model is still far smaller than the base and uses the minimal layer set (one layer per distinct type combination) and a minimal expert count that still exercises routing (8 >= topk 6).

Checkpoint Structure

Single-file checkpoint (model.safetensors, 110 tensors) using the original DeepSeek-V4-Pro naming convention (no model. prefix): layers.N.attn.wq_a.weight, layers.N.ffn.experts.E.w1/w2/w3.weight, embed.weight, norm.weight, head.weight, hc_head_fn/base/scale, etc. Tensor-name structure matches the base model.safetensors.index.json modulo layer count (MTP tensors excluded, FP8 .scale tensors not present in this unquantized bf16 checkpoint).

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "inference-optimization/DeepSeek-V4-Pro-3.7B-A3.4B-small",
    device_map="auto", dtype=torch.bfloat16,
)
# transformers keeps deepseek_v4 norm/hc modules in fp32 when loading bf16
# (`_keep_in_fp32_modules_strict`), which breaks the bf16 forward pass;
# cast everything to bf16 (norms still compute statistics in fp32 internally):
model = model.to(torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/DeepSeek-V4-Pro-3.7B-A3.4B-small")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))

(Alternatively, load with dtype=torch.float32.)

Creation Process

This model was created using the llm-compressor create-tiny-model skill.

1. Inspected the DeepSeek-V4-Pro config: 61 layers with 2 distinct attention types (heavily_compressed_attention, compressed_sparse_attention) and 2 distinct MLP types (hash_moe, moe), 384 routed experts with topk 6, fp8 quantization_config, expert_dtype: fp4.
2. Built the small model keeping all dims identical: 2 layers [(heavily_compressed_attention, hash_moe), (compressed_sparse_attention, moe)], 8 routed experts, quantization_config stripped (unquantized bf16), expert_dtype: fp4 kept in config.
3. Fine-tuned on the copypasta toy dataset (bf16, batch size 1, lr 1e-4) until perplexity < 3.0 (final train loss ~0.04).
4. Converted the checkpoint to the original base tensor naming and validated the tensor-name structure against the base model.safetensors.index.json (pattern sets match exactly modulo layer count).
5. Validated perplexity and generation (output below).

Notes

• Backbone-only checkpoint: num_nextn_predict_layers=1 is kept from the base config, but (as with all transformers-saved deepseek_v4 checkpoints) no MTP weights are materialized — matching the convention of e.g. inference-optimization/GLM-5.2-0.8B-A0.8B.
• The base model is FP8-quantized; quantization_config was removed to produce this unquantized bf16 checkpoint (matching the convention for these entries). The per-tensor FP8 .scale tensors of the base are therefore not present.
• expert_dtype: fp4 is kept from the base config; with quantization_config removed the experts are stored as regular bf16 parameters.
• Validation output:

  Success: 1.000177025794983 <= 10.0
  Generating sample text:
  According to all known laws of aviation, there is no way a bee should be able to fly.