Instructions to use unsloth/DeepSeek-V4-Flash-0731 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use unsloth/DeepSeek-V4-Flash-0731 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="unsloth/DeepSeek-V4-Flash-0731")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("unsloth/DeepSeek-V4-Flash-0731")
model = AutoModelForCausalLM.from_pretrained("unsloth/DeepSeek-V4-Flash-0731", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use unsloth/DeepSeek-V4-Flash-0731 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "unsloth/DeepSeek-V4-Flash-0731"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/DeepSeek-V4-Flash-0731",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/unsloth/DeepSeek-V4-Flash-0731

SGLang

How to use unsloth/DeepSeek-V4-Flash-0731 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "unsloth/DeepSeek-V4-Flash-0731" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/DeepSeek-V4-Flash-0731",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "unsloth/DeepSeek-V4-Flash-0731" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/DeepSeek-V4-Flash-0731",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Unsloth Desktop
Docker Model Runner
How to use unsloth/DeepSeek-V4-Flash-0731 with Docker Model Runner:
```
docker model run hf.co/unsloth/DeepSeek-V4-Flash-0731
```

Cheapest viable deployment: 2x MI300X beats 2x H200 by ~4x — plus, any plan for Dynamic GGUFs?

by guitaripod - opened Jul 31

Discussion

guitaripod

Jul 31

Sharing numbers in case they save someone else the research, and one question at the end.

The memory budget. The FP4+FP8 checkpoint is ~158 GB. With FP8 KV cache the full 1M context only adds ~10 GB (V4 uses roughly 7% of V3.2's KV footprint), so the realistic floor is ~170-175 GB of VRAM for a single replica. That rules out one 192 GB card if you want real context headroom, and makes 2x anything-with-192GB the natural unit.

What that costs per hour, on-demand:

2x MI300X (384 GB total): ~~$3.00-4.00/hr on-demand (~~$1.99/GPU at DigitalOcean/Hot Aisle), ~$3.00/hr on RunPod spot at $1.49/GPU
2x H200 SXM (282 GB total): ~$7.18/hr on RunPod
4x A100 80GB (320 GB total): usually lands between the two, and you eat the older interconnect

The 4x A100 recommendation that gets repeated everywhere is a tensor-parallel artifact — TP prefers power-of-two GPU counts, and 2x A100 at 160 GB falls just under the budget, so the next legal step is 4x. It is not a real hardware requirement.

What I run on the MI300X pair:

vllm serve unsloth/DeepSeek-V4-Flash-0731 \
  --tensor-parallel-size 2 --enable-expert-parallel \
  --kv-cache-dtype fp8 --gpu-memory-utilization 0.92 \
  --max-model-len 262144 --host 0.0.0.0 --port 8000

262K rather than the full 1M — Think Max wants 384K minimum, and dropping the cap is the cheapest knob for batch headroom. vLLM and SGLang both had day-0 support and read the native FP4/FP8 checkpoint directly, no conversion step.

The honest caveat nobody putting up a deployment guide wants to write. First-party API is $0.14/M input, $0.28/M output, and $0.0028/M on cache hits. Against a blended ~$0.18/M, a $3/hr box needs somewhere around 400M tokens/day at genuine full utilization to break even — and no real workload sustains 100% GPU utilization. Self-hosting this model is a privacy, latency, or data-residency decision. It is not a cost decision unless you are saturating the box around the clock. Worth saying out loud, because the cost framing is what most self-host posts lead with.

The question: any plan for Dynamic 2.0 GGUFs of this checkpoint? Upstream llama.cpp does not handle the V4 architecture yet (CSA + HCA attention, mHC), so I assume the blocker is there rather than on your side — but if UD quants are on the roadmap it changes the calculus a lot for people with 2x 96 GB consumer-adjacent cards or a big unified-memory box, who currently have no path at all. A ~70-80 GB UD-Q2/Q3 would open up a tier of hardware that is completely locked out today.

Thanks for mirroring these so fast either way.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment