Instructions to use inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B")
model = AutoModelForCausalLM.from_pretrained("inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B

SGLang

How to use inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B with Docker Model Runner:
```
docker model run hf.co/inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B
```

DeepSeek-V4-Pro-0.5B-A0.37B

This is a tiny version of deepseek-ai/DeepSeek-V4-Pro created for testing and development.

Model Details

Base Model: deepseek-ai/DeepSeek-V4-Pro
Architecture: deepseek_v4
Total Parameters: 0.487B
Activated Parameters: ~0.37B

Configuration Changes

The following parameters were reduced from the original model:

Parameter	Original	Tiny
num_hidden_layers	61	8
hidden_size	7168	1024
num_attention_heads	128	8
head_dim	512	64
q_lora_rank	1536	256
o_lora_rank	1024	64
o_groups	16	2
index_n_heads	64	4
index_head_dim	128	32
index_topk	1024	64
n_routed_experts	384	16
moe_intermediate_size	3072	512
sliding_window	128	64

Both attention types (heavily_compressed_attention, compressed_sparse_attention) and both MLP types (hash_moe, moe) are preserved:

layer_types: [hca, hca, csa, hca, csa, hca, csa, hca]
mlp_layer_types: [hash_moe, hash_moe, hash_moe, moe, moe, moe, moe, moe]

Checkpoint Structure

Single-file checkpoint (model.safetensors) with 608 tensors. The checkpoint uses the original DeepSeek-V4-Pro naming convention (no model. prefix): layers.N.attn.wq_a.weight, embed.weight, head.weight, etc. Fully loadable with AutoModelForCausalLM.from_pretrained().

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))

Creation Process

This model was created using the llm-compressor create-tiny-model claude skill.

Inspected DeepSeek-V4-Pro config to identify architecture: dual attention types (heavily_compressed_attention, compressed_sparse_attention), dual MLP types (hash_moe, moe), HyperConnection residual streams (hc_mult=4), and MTP layer.
Reduced all hidden dimensions while preserving at least one of each layer type.
Fine-tuned on copypasta dataset until perplexity < 3.0 (achieved 1.001).
Validated checkpoint structure matches original model naming conventions.

Notes

The model uses transformers naming convention for tensor keys (compatible with from_pretrained()).
A converted checkpoint matching the original DeepSeek-V4-Pro key naming is also available (model.safetensors with original layers.N.attn.wq_a.weight style names).
FP8 quantization scale tensors from the original model are not present in this unquantized bf16 checkpoint.
The num_nextn_predict_layers=1 (MTP) layer is preserved in the architecture.
Validation output: Success: 1.001 <= 10.0

Downloads last month: 63

Safetensors

Model size

0.5B params

Tensor type

I64

F32

BF16

Model tree for inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B

Base model

deepseek-ai/DeepSeek-V4-Pro

Finetuned

(13)

this model

Collection including inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B

Tiny Models

Collection

Tiny models used for testing • 12 items • Updated 4 days ago