deepseek-ai

/

DeepSeek-V4-Flash-0731

like 3.6k

Follow

DeepSeek 143k

Things to note after first runs, the draft acceptance rate is smaller than the original Dspark version of the model and thus the token generation is also slower from around ~40tps down to ~28-30 tps. I dont know if it my config or something else at play.

your image does not work
vllm serve: error: argument --kv-cache-dtype: invalid choice: 'nvfp4_ds_mla' (choose from auto, bfloat16, float16, fp8, fp8_ds_mla, fp8_e4m3, fp8_e5m2, fp8_inc, fp8_per_token_head, int8_per_token_head, nvfp4, turboquant_3bit_nc, turboquant_4bit_nc, turboquant_k3v4_nc, turboquant_k8v4)

your image does not work
vllm serve: error: argument --kv-cache-dtype: invalid choice: 'nvfp4_ds_mla' (choose from auto, bfloat16, float16, fp8, fp8_ds_mla, fp8_e4m3, fp8_e5m2, fp8_inc, fp8_per_token_head, int8_per_token_head, nvfp4, turboquant_3bit_nc, turboquant_4bit_nc, turboquant_k3v4_nc, turboquant_k8v4)

Yes you are absolutely right, I was running a patched version to test some optimizations when the model dropped and rushed to test it 😀. You can try one of the supported kv-cache types, probably fp8_ds_mla also use the speculative method mtp instead of dspark. I updated the config above.

Thanks for sharing your config.

2 observations:

• I am not able to the the speculative setttings in your config
• where did you get the value for the maximum length? The only value I can see in the card is 384k token at high and max reasoning effort level.

Things to note after first runs, the draft acceptance rate is smaller than the original Dspark version of the model and thus the token generation is also slower from around ~40tps down to ~28-30 tps. I dont know if it my config or something else at play.

If you consider the memory speed of GB10, the generation will be approx. 29 tps on a single request without MTP. Looks like MTP is not running.
Note: MTP boost is much lower with MoE than with dense models because the non shared experts among token computing will require additional memory transfer.

Single Spark, no cluster: 2-bit GGUF that fits in 128 GB

For anyone on one GB10 rather than two — the vLLM recipe above needs tensor_parallel 2, but the ds4 path runs -0731 on a single box.

Every public GGUF of -0731 is 155–165 GB (MXFP4 / Q4_K / Q8), so none fit. I built a 2-bit one: 86.7 GB, IQ2_XXS gate/up + Q2_K down on the routed experts only, Q8_0 elsewhere — antirez's recipe, using his Preview template and imatrix.

That reuse is safe, and I checked rather than assumed: diffing the two model.safetensors.index.json, the body is identical — 67,612 non-MTP tensors, name for name, zero differences. Only the MTP/DSpark module changed (1 -> 3 layers), and it is not in this file. --dry-run gave type_changes: 0, and the finished file lands 288 bytes from the template.

-> GaelicThunder/DeepSeek-V4-Flash-0731-ds4-IQ2XXS-GGUF (weights still uploading at the time of posting; README is up)

Two things that cost me hours:

• --imatrix-strict will kill the build. That imatrix only covers routed-MoE (129 entries = 43 x 3), so token_embd.weight has no vector and it aborts — after ~2 h, at 77.9/86.7 GB written.
• The drafter must be the -0731 one (bleysg/DeepSeek-V4-Flash-DSpark-drafter-GGUF -> DSpark-drafter-Q2K-Q8-0731.gguf). Older DSpark files fail with required tensor is missing: dspark.main_proj.weight: v0.4.x wants dspark.* tensors, deepseek4.dspark.* keys and markov_* in F16, the older ones use mtp.* and Q8_0. Same module, different serialization.

Also worth knowing: speculation only runs on the continuous-batch path. With DS4_SERVER_COALESCE_MAX=1 it is silently never used — ds4_spec_drafts_total 0 on /metrics. Minimum is 2.

Rough numbers on one GB10, short context, -c 131072: 14–17 tok/s at 68–72 % acceptance. Single samples on a box that was not idle, so take them loosely — below the 35 reported for the Preview, and I have not verified whether -0731 genuinely accepts fewer drafts.

Correction to my numbers above — v0.5.0 roughly doubles them.

Weights are up now (86,720,111,488 B, GaelicThunder/DeepSeek-V4-Flash-0731-ds4-IQ2XXS-GGUF).

I posted 14–17 tok/s at 68–72 % acceptance. That was measured on engine v0.4.2 and on free-form prose. On v0.5.0 (Entrpi/ds4, tag v0.5.0 = head of batched-serving), same weights file, same box, step-wise math:

Two variables moved at once (engine and workload), so I can't split the credit — but the 89 % matches what the 1x-Spark thread reports, so acceptance looks workload-bound rather than a -0731 regression. I'd retract my earlier speculation that -0731 might accept fewer drafts; I have no evidence for it.

Worth stating for anyone reading the older thread: the 59 tok/s headline there is an aggregate over 12 concurrent requests. Single-stream chat decode is ~28 tok/s, which is where a single box actually lands.

With appreciation to the OP for kicking off the discussion, this recipe is an improvement in basically every respect.

https://github.com/tonyd2wild/DeepSeek-v4-Flash-0731-DSpark-1M-NVFP4-KV-2x-DGX-Spark

1. Hardware floor: the checkpoint is 156 GB

$ du -sh ~/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-V4-Flash-0731/
156G

It does not fit one 128 GB GB10. We tried TP=1 and the engine core is
killed by signal during weight load — no Python traceback, no CUDA error, just
exit code: None, with ~190 NVRM ... NO_MEMORY events in the kernel log.
Budget at least two nodes (or, on sm_120, enough 96 GB cards to hold weights
plus KV plus JIT headroom).

2. Prerequisite: sparkrun (DGX Spark clusters)

We drive the cluster with sparkrun 0.3.1 (sparkrun --help), which
launches and manages multi-node inference workloads on DGX Spark from a YAML
recipe. Useful commands:

sparkrun run <recipe.yaml>     # launch (add --solo for single node)
sparkrun status                # what's running, on which hosts, job id
sparkrun logs <job-id> -n 500 -a   # logs, all sources (-a includes workers)
sparkrun stop <job-id>
sparkrun registry list         # recipe registries (eugr, official, community, ...)

Two notes that saved us time:

• vLLM's own output does not go to docker logs — use sparkrun logs.
• Recipes are searched in registries; the eugr registry
  (https://github.com/eugr/spark-vllm-docker) is the Docker build stack
  underneath.

If you are not on DGX Spark, skip sparkrun — everything below is a plain
vllm serve command line and works the same way (section 6).

3. The recipe

recipe_version: "1"
name: DeepSeek-V4-Flash-0731
description: DeepSeek V4 Flash 0731 on dual DGX Spark, TP=2, PR 41834 SM12x
model: deepseek-ai/DeepSeek-V4-Flash-0731
container: vllm-node-dsv4
cluster_only: true

build_args:
  - "--rebuild-vllm"
  - "--vllm-repo"
  - "https://github.com/jasl/vllm.git"
  - "--vllm-ref"
  - "sm120-pr-41834-stable-preview-20260804"

defaults:
  port: 8000
  host: 0.0.0.0
  tensor_parallel: 2
  gpu_memory_utilization: 0.85
  max_model_len: 500000
  max_num_batched_tokens: 8192
  max_num_seqs: 64
  block_size: 256
  served_model_name: deepseek-v4-flash-0731

env:
  VLLM_MARLIN_USE_ATOMIC_ADD: "1"
  PYTORCH_CUDA_ALLOC_CONF: "expandable_segments:True"

command: |
  uv pip install --cache-dir /tmp/uv-cache --target /tmp/site-packages --no-deps nvidia-cutlass-dsl-libs-cu13==4.5.2 && \
  vllm serve {model} \
      --host {host} \
      --port {port} \
      --block-size {block_size} \
      --enable-auto-tool-choice \
      --gpu-memory-utilization {gpu_memory_utilization} \
      --kv-cache-dtype fp8_ds_mla \
      --load-format instanttensor \
      --max-model-len {max_model_len} \
      --max-num-batched-tokens {max_num_batched_tokens} \
      --max-num-seqs {max_num_seqs} \
      --override-generation-config '{"top_p":0.95, "temperature":0.6}' \
      --reasoning-parser deepseek_v4 \
      --served-model-name {served_model_name} \
      --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic"}' \
      --tensor-parallel-size {tensor_parallel} \
      --tool-call-parser deepseek_v4 \
      --trust-remote-code \
      --tokenizer-mode deepseek_v4

https://forums.developer.nvidia.com/t/1x-spark-deepseek-v4-flash-0731-1-000-tok-s-prefill-59-tok-s-multi-agent-serving/378855

It does run on a single DGX , i have it running with 512k context.