gdubicki/Qwen3-32B-NVFP4

Public mirror of nvidia/Qwen3-32B-NVFP4.

Weights are byte-identical to the upstream NVIDIA release. This mirror exists to provide a pinned, stable reference for the qwen3-32B deployment project running on DGX Spark (GB10). Use the upstream repo if you don't need a pinned copy.

Credits

• Base model: Qwen/Qwen3-32B by Alibaba / Qwen team
• NVFP4 quantization: NVIDIA via ModelOpt
• License: Apache-2.0

Model details

• Architecture: qwen3 (dense transformer, 32.8B parameters — all active)
• Quantization: NVFP4 weights + NVFP4 activations (all Linear layers, group_size=16); lm_head kept at full precision
• KV cache: FP8 (per model's quantization config)
• Max context: 40,960 tokens (max_position_embeddings in config.json; tokenizer_config.json claims 131072 but that is inherited from the base model and not usable here)
• Reasoning: built-in chain-of-thought (<think> tags), ON by default, toggleable per request via enable_thinking

Verified performance

Measured on DGX Spark (GB10 Blackwell, SM12.1, 128 GB unified LPDDR5X) with vLLM (Marlin NVFP4 backend, FP8 KV cache):

Metric	Value
Throughput	~10.5 tok/s
Prompt tokens	40
Output tokens	852

Qwen3-32B is faster than Gemma-4-31B on GB10 despite similar parameter count — Marlin runs NVFP4 as W4A16 GEMM on SM12.1 (no native CUTLASS FP4 kernel), and Qwen3's W4A4 layout gives a slightly better bandwidth win.

Usage

docker run --rm --runtime=nvidia --gpus all \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_NVFP4_GEMM_BACKEND=marlin \
  vllm/vllm-openai:cu130-nightly \
  gdubicki/Qwen3-32B-NVFP4 \
  --quantization modelopt_fp4 \
  --dtype auto \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.30 \
  --max-model-len 40960 \
  --reasoning-parser qwen3

Full deployment scripts: https://github.com/grzegorzdubicki-ai/qwen3-32B