doth4580/Qwen3.6-35B-A3B-NVFP4-FULL

Qwen3.6 35B-A3B MoE — NVFP4 for veloGB10 (NVIDIA DGX Spark / GB10)

Original model: Qwen/Qwen3.6-35B-A3B by Alibaba's Qwen team, Apache-2.0. This artifact contains modified weights (NVFP4/FP8 quantization); the original model is © its authors.

NVFP4/FP8-quantized variant of Qwen3.6 35B-A3B MoE (35B total / ~3B active (MoE)). This quantized artifact exists first and foremost to run with the veloGB10 inference engine (gb10_inference) on NVIDIA DGX Spark / GB10 systems — it is produced, validated, and tuned for veloGB10's kernels. The weights use the standard HuggingFace compressed-tensors layout (nvfp4-pack-quantized), so this artifact can be used for any purpose, with any framework that reads the format. It is in particular fit and proven to work with veloGB10 — greedy-lossless speculative decoding, two-node TP=2 serving, and the engine's full correctness test suite all pass on this artifact.

Specifications

Base model	Qwen/Qwen3.6-35B-A3B
Architecture	hybrid MoE (30 GatedDeltaNet + 10 full-GQA layers), hidden 2048, 256 experts top-8 + 1 shared
Attention	hybrid GatedDeltaNet (linear attention, fixed-size recurrent state) + periodic full GQA
MTP head	native multi-token-prediction head (used for speculative decoding)
Context	262,144 tokens
Vocabulary	248,320
Quantization	nvfp4-full — all GEMMs NVFP4 incl. the MoE MTP head — required for speculation to pay on MoE
Format	nvfp4-pack-quantized (compressed-tensors compatible)
On-disk size	20 GB

Running with veloGB10

Full instructions: veloGB10 README (build or download a release binary, then). The binary loads its PTX kernel artifacts from the current directory — run it from wherever the binary + *.ptx live.

Single user — maximum speed:

gb10_inference --server --model-dir=/path/to/Qwen3.6-35B-A3B-NVFP4-FULL --port=9000 \
  --max-seq-len=32768 --max-batch=1 --max-tokens=4096 \
  --prefix-cache=on --mtp=auto --default-presence-penalty 1.5

Four concurrent users — maximum aggregate throughput:

gb10_inference --server --model-dir=/path/to/Qwen3.6-35B-A3B-NVFP4-FULL --port=9000 \
  --max-seq-len=32768 --max-batch=4 --max-tokens=4096 \
  --mtp-lanes=on --prefix-cache=on --default-presence-penalty 1.5

(--mtp-lanes=on packs the concurrent greedy lanes into one batched verify forward — the concurrency win; lossless. The scheduler handles the rest automatically.)

Two-node TP=2 (maximum speed and capacity):

On the second node (no model copy, no configuration — the head ships weights, config, and calibration at sync):

gb10_inference --node --port 29500

On the head node:

GB10_TP_SHARD_MIXERS=1 gb10_inference --server --model-dir=/path/to/Qwen3.6-35B-A3B-NVFP4-FULL \
  --tp --nodes <peer-ip>:29500 --port 9000 \
  --max-seq-len=32768 --max-batch=1 --max-tokens=4096 \
  --prefix-cache=on --default-presence-penalty 1.5

Measured on veloGB10 (greedy, NVFP4)

• Single node: 110.5 / 110.9 / 98.2 tok/s @ 0 / 4K / 8K ctx (TTFT 1.06 / 3.01 / 5.27 s)
• TP=2: 129.5 / 118.3 / 118.1 tok/s @ 0 / 4K / 8K ctx — TP=2 leads at every depth (1.07–1.20×), and halves per-node memory besides

Throughput measured with tool-eval-bench --perf and veloGB10's built-in benchmarks; see the veloGB10 README for the full, current tables.

License & credits

• Original model: Qwen/Qwen3.6-35B-A3B — © Alibaba Qwen team, Apache License 2.0. A copy of the license is included in this repository (LICENSE).
• This artifact: quantized from the original with veloGB10's offline quantizer (nvfp4-full). Weights are modified relative to the original; tokenizer and chat template are unchanged from the original model.
• Engine: veloGB10 (Apache-2.0).