Hi there!

We actually fixed this in vLLM last week or so: https://github.com/vllm-project/vllm/pull/48428.

I think the right solution here is to either backport that patch, or to switch to a different kernel backend.

Can you let me know if that takes away the issue?

Just got it working on the rtx 6000 pro blackwell 96gb.

(Claude's writeup)

Laguna S 2.1 NVFP4 + DFlash on a single RTX PRO 6000 Blackwell Workstation (96 GB, SM120)

Working recipe for poolside/Laguna-S-2.1-NVFP4 with DFlash speculative decoding on a
workstation Blackwell card. The model card covers the DGX Spark (GB10/sm_121); this is
the x86_64 + SM120 equivalent, with measured throughput at context depths up to 200K.

Environment

Install

uv venv ~/venvs/vllm-laguna -p 3.12
uv pip install -p ~/venvs/vllm-laguna/bin/python vllm==0.25.1 --torch-backend=cu130

# vLLM 0.25.1 pins flashinfer 0.6.13; the jit-cache wheel (off-PyPI) skips most first-start JIT
uv pip install -p ~/venvs/vllm-laguna/bin/python \
  "flashinfer-python==0.6.13" "flashinfer-cubin==0.6.13" \
  "flashinfer-jit-cache==0.6.13" --extra-index-url https://flashinfer.ai/whl/cu130/

hf download poolside/Laguna-S-2.1-NVFP4
hf download poolside/Laguna-S-2.1-DFlash-NVFP4   # ~2.2 GB draft model

Notes vs the Spark recipe on the model card:

• No CUTE_DSL_ARCH needed — that env var is for GB10 (sm_121a); SM120 auto-detects.
• If nvcc rejects your host compiler during flashinfer JIT (Arch + gcc 16):
  export NVCC_CCBIN=/usr/bin/g++-15.
• Multi-GPU box: export CUDA_DEVICE_ORDER=PCI_BUS_ID and pin CUDA_VISIBLE_DEVICES.

Serve

python -m vllm.entrypoints.openai.api_server \
  --model poolside/Laguna-S-2.1-NVFP4 \
  --max-model-len 262144 \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --async-scheduling \
  --enable-auto-tool-choice \
  --tool-call-parser poolside_v1 \
  --reasoning-parser poolside_v1 \
  --default-chat-template-kwargs '{"enable_thinking": true}' \
  --speculative-config '{"model": "poolside/Laguna-S-2.1-DFlash-NVFP4", "num_speculative_tokens": 15, "method": "dflash"}' \
  --override-generation-config '{"temperature": 0.7, "top_p": 0.95}' \
  --port 8000

Flag rationale (things that bit us or that the card only implies):

• --gpu-memory-utilization 0.90 — at 0.85 the engine refuses to start at 256K
  context: it measures 8.27 GiB free for KV but needs 9.6 GiB for one 262144-token
  request (weights are ~67 GiB on disk, ~90.6 GiB total VRAM in use at 0.90).
  Drop --max-model-len instead if you want more headroom.
• --max-num-seqs ≤ 32 is mandatory with DFlash (per the card, vLLM crashes at the
  default 256). 8 is plenty for single-user use and keeps cuda-graph capture cheap.
• No --quantization flag, no --kv-cache-dtype — both auto-detect from the
  checkpoint's quantization_config (nvfp4-pack-quantized weights, FP8 KV scheme).
• No --trust-remote-code — vLLM 0.25 has native Laguna support.
• Keep --override-generation-config, and do not add min_p: vLLM rejects
  min_p/logit_bias under speculative decoding → HTTP 400 on every sampled request.
• Reasoning comes back in the reasoning field of the response message (vLLM
  renamed reasoning_content → reasoning in 0.23+). Toggle per request with
  chat_template_kwargs: {"enable_thinking": false}.

Measured throughput (this box)

Streaming /v1/completions, 256 generated tokens/request, unique prompts (prefix cache
defeated). Filler-text prompts are a worst case for DFlash acceptance — real code/prose
decodes faster (a code-gen smoke test hit 104 tok/s single-stream).

Takeaways:

• Single-stream decode holds 120–159 tok/s even at 96K–200K context — no depth cliff.
  (For scale: the card reports 22–24 tok/s on the DGX Spark.)
• Concurrency trades per-stream speed for aggregate: 8 streams ≈ 383 tok/s aggregate at 1K.
• DFlash acceptance is front-loaded (~62% at draft position 0, near zero past position 6),
  and drops further under concurrency. num_speculative_tokens: 15 (the card's value) is
  likely over-drafting for batch>1 workloads — 7 (the BF16 card's value) may bench better
  if you serve concurrent users.
• First start is slow (weight load + JIT + cuda-graph capture); subsequent starts are much
  faster once ~/.cache/flashinfer and the torch compile cache are warm.

Doing the same command but switching to the nightly vllm build (nightly-f25953cc59f9b4ba9b04b16228d2b86dcfbcbdb1) and adding --max-model-len auto just for good measure seems to have fixed it. Thanks!