Instructions to use inference-optimization/DeepSeek-V3-3.4B-A3.0B-small with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use inference-optimization/DeepSeek-V3-3.4B-A3.0B-small with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="inference-optimization/DeepSeek-V3-3.4B-A3.0B-small")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("inference-optimization/DeepSeek-V3-3.4B-A3.0B-small")
model = AutoModelForCausalLM.from_pretrained("inference-optimization/DeepSeek-V3-3.4B-A3.0B-small", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use inference-optimization/DeepSeek-V3-3.4B-A3.0B-small with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "inference-optimization/DeepSeek-V3-3.4B-A3.0B-small"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/DeepSeek-V3-3.4B-A3.0B-small",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/inference-optimization/DeepSeek-V3-3.4B-A3.0B-small

SGLang

How to use inference-optimization/DeepSeek-V3-3.4B-A3.0B-small with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "inference-optimization/DeepSeek-V3-3.4B-A3.0B-small" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/DeepSeek-V3-3.4B-A3.0B-small",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "inference-optimization/DeepSeek-V3-3.4B-A3.0B-small" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/DeepSeek-V3-3.4B-A3.0B-small",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use inference-optimization/DeepSeek-V3-3.4B-A3.0B-small with Docker Model Runner:
```
docker model run hf.co/inference-optimization/DeepSeek-V3-3.4B-A3.0B-small
```

DeepSeek-V3-3.4B-A3.0B-small

This is a small version of deepseek-ai/DeepSeek-V3-Base created for testing and development. It preserves the full architecture characteristics of the base model (MLA attention, MoE with shared experts, noaux-tc sigmoid routing, untied embeddings, identical hyperparameters) with fewer layers and experts.

Model Details

Base Model: deepseek-ai/DeepSeek-V3-Base
Architecture: DeepseekV3ForCausalLM (model_type deepseek_v3)
Total Parameters: 3.37B (3,372,768,256)
Activated Parameters: 3.02B (3,020,446,720)

Configuration Changes

The following parameters were reduced from the original model:

Parameter	Base	Small
`num_hidden_layers`	61	2 (1 dense-MLP layer + 1 MoE layer, dense first as in base)
`first_k_dense_replace`	3	1
`n_routed_experts`	256	16

All other hyperparameters are identical to the base: hidden_size 7168, intermediate_size 18432, moe_intermediate_size 2048, vocab_size 129280, num_experts_per_tok (topk) 8, n_group 8, topk_group 4, n_shared_experts 1, MLA dims (q_lora_rank 1536, kv_lora_rank 512, qk_head_dim 192, ...), rope/yarn parameters, tie_word_embeddings false, num_nextn_predict_layers 1, etc.

Checkpoint Structure

Single model.safetensors file with 77 bf16 tensors (the base is sharded; single-file vs sharded is equivalent). Tensor naming matches the base checkpoint exactly modulo layer count, verified against the base model.safetensors.index.json (23/23 name patterns match; per-layer tensor sets identical to base dense layer 0 and MoE layer 3). The base's fp8 weight_scale_inv tensors are absent because this checkpoint is unquantized bf16. This is a backbone-only checkpoint: num_nextn_predict_layers is kept at 1 from the base, and no MTP weights are materialized (matching the transformers save convention for this family).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("inference-optimization/DeepSeek-V3-3.4B-A3.0B-small", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/DeepSeek-V3-3.4B-A3.0B-small")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))

Creation Process

This model was created using the llm-compressor create-tiny-model skill.

Built from the base AutoConfig with only the allowed changes above (layers, layer-type list first_k_dense_replace, expert count), initialized with init_weights() (bf16), and fine-tuned on a toy copypasta dataset until perplexity ~1.0 (target 3.0).

Validation

$ python validate_tiny_model.py DeepSeek-V3-3.4B-A3.0B-small-ft
Success: 1.0000025033950806 <= 10.0

Generating sample text:
According to all known laws of aviation, there is no way a bee should be able to fly.

Notes

Why 16 experts instead of 8 (= topk): the DeepSeek-V3 grouped-topk router requires at least 2 experts per group (n_group=8, intra-group topk(2)) and at least num_experts_per_tok=8 surviving candidates after topk_group=4 group masking. 8 routed experts (1 per group) would crash routing; 16 is the minimum feasible count and still exercises the full routing path (sigmoid scoring, e_score_correction bias / noaux_tc, group selection, topk-prob normalization, routed_scaling_factor 2.5, 1 shared expert). topk itself is unchanged at 8.
Embeddings dominate the parameter count: with fixed vocab (129280) and hidden (7168) and untied embeddings, input+output embeddings alone are ~~1.85B params; the minimal 2-layer set keeps the total at 3.37B, far below the base (~~671B) and within the accepted range for this family.
quantization_config stripped: the base is fp8 block-quantized; this checkpoint is clean unquantized bf16 (matching the convention for these small test models).
auto_map removed: the base config references remote-code files (configuration_deepseek.py, modeling_deepseek.py) that are not carried over in this repo; transformers (>= native deepseek_v3 support) and vLLM both load this model with their native implementations.
Backbone-only: num_nextn_predict_layers kept from base (1); transformers does not materialize MTP weights on init/save, so no MTP weights are included (the base's layer-61 MTP tensors are intentionally absent).