nvidia

/

Qwen3.8-Flash-Next-FP4-Local

Like 156

Follow

NVIDIA 71.5k

Introduction

Hello, thank you for this model checkpoint. I gave it a shot and benchmarked it against the RadixArk that I've been using.
Other benchmarks will probably focus on agentic capabilities. But I decided to benchmark it on Humaneval https://github.com/openai/human-eval just for fun.
Any tuning/ general advice would be greatly appreciated.

Both checkpoints run against the same vLLM build on one RTX PRO 6000 Blackwell (96 GB), no
container. Both keep the PLE n-gram ("Engram") table in pinned DDR5 and stream it to the GPU
over UVA with VLLM_PLE_CPU_OFFLOAD=1.

Same base model (Qwen3.8-Flash-Next, Qwen4Exp arch), same systemd unit, same GPU and port,
served as Qwen38. Only the weights swap, so the differences below are the checkpoints' and not
the engine's.

Humaneval results

benched.py, 164 problems, k=1, temperature 0.2. LocalFP4 is one of three independent repeats of
the same NVIDIA checkpoint (159 / 159 / 158 passes, a 0.6pp sampling noise floor); RadixArk NVFP4 is a
single run.

Both decode at about 115 tok/s, so RadixArk NVFP4's 3.4x wall-clock is extra thinking time, not slower
decode.

The paired result matters more than the score: across all 164 problems the two checkpoints
never disagree on a problem both answered. There is no problem RadixArk NVFP4 solved where all three
NVIDIA repeats failed, and none RadixArk NVFP4 failed where any NVIDIA repeat passed. Both get exactly
one problem wrong with code in hand, HumanEval/145, and 20 follow-up samples at three token
budgets still get it wrong, so quantization is not what's behind that one.

That makes RadixArk NVFP4's 2-task margin a question of whether the sample finished thinking and emitted
code. The NVIDIA checkpoint's failure mode is finish_reason=length with zero content: thinking
never terminates, the budget burns, no code comes out. The three repeats hit it 7/492 times (3
of them in LocalFP4); RadixArk NVFP4 hit it 0/164.

Takeaway

Same capability where it counts: the checkpoints agree on every problem both attempt, and share
one genuine failure (/145). LocalFP4 is 44% cheaper and 47% faster per problem, halves the
pinned DRAM, and fits one card with a 2.5x bigger KV pool. RadixArk never exhausted its thinking
budget into an empty completion, which is what cost LocalFP4 its points. I'd run LocalFP4,
keep a retry-with-larger-budget loop for the occasional runaway thinker, and pin one
--max_tokens value before comparing anything.

Where LocalFP4 is stronger

LocalFP4 allows my single card to be capable of higher concurrent tokens performing different
tasks. Whereas RadixArk was limited to 131k, LocalFP4 serves the full 262144 window to multiple agents with
room to spare, because the KV cache is where the two checkpoints diverge hardest:

So I can run three agents at the full 262K context at once on LocalFP4 (a fourth fits at
~195K each), while the RadixArk deployment barely fits one full-length request and I capped
mine at 131k to get any concurrency out of it. The freed VRAM comes from the n-gram table
being 33 GB instead of 51.2 GB.

More details

The recipe: vLLM dev without a docker container

I do not have docker, so I ran a vLLM dev version by compiling it for my already existing container, so it probably won't work with yours. Including it just for note-keeping if anyone is curious / want to repeat:

The LIL wheels (same build as the release-10152026 image recipe) install into a plain uv venv,
with provenance pinned to exact commits. Every wheel comes from a local-inference-lab GitHub
release tagged <component>-cu134[-sm120]-beta-<full-commit>, so any wheel can be re-derived
from its commit:

The wheels are built on Ubuntu 24.04 (glibc 2.38, GCC 15) and this box is Debian 12 (glibc 2.36,
GCC 12), which is the one non-obvious part. LD_PRELOAD shims don't work because DT_VERNEED
pins the soname. What works is gating the real libraries per-process:

/opt/glibc239/      libc6 2.39 (libc, libm, ld-linux)
/opt/libstdcxx15/   libstdc++6 15 (GLIBCXX_3.4.32)

# .venv/bin/python is a COPY of the CPython 3.12 ELF (not a symlink), then:
patchelf --set-interpreter /opt/glibc239/.../ld-linux-x86-64.so.2 \
         --set-rpath /opt/glibc239/...:/opt/libstdcxx15/...

Every child process inherits the gate. Do not run a blanket patchelf --set-rpath across
site-packages: it clobbers the auditwheel $ORIGIN RPATHs and breaks torch/numpy/scipy/cv2.

Gotchas that actually bit:

1. Both attention paths must be FLASHINFER (--attention-backend FLASHINFER --mm-encoder-attn-backend FLASHINFER). The wheel's _vllm_fa2_C/_vllm_fa3_C fatbins only
   carry sm_80/sm_75 codegen, so on sm_120 the first ViT call dies with
   cudaErrorUnsupportedPtxVersion. The model card omits these flags, which is correct only
   inside the official container, whose vllm_flash_attn build does carry sm_120.
2. apt-get install ninja-build. flashinfer JIT-compiles its sampler on the first request and
   shells out to ninja; without it the engine fails to initialize.
3. PLE in DRAM: keep VLLM_PLE_CPU_OFFLOAD=1 explicit and verify Resolved Engram configuration: EngramConfig(cpu_offload=True) in the journal. That pins 51.2 GB (RadixArk)
   or ~33 GB (LocalFP4) of host RAM. The disk backend (VLLM_PLE_TABLE_MEMORY=disk + io_uring)
   is the DGX Spark path. I have more host RAM, so I just pin it on the RAM.

Launch: the same env block serves both checkpoints, only the path and a few flags differ.
LocalFP4 adds --quantization modelopt_mixed --kv-cache-dtype fp8 --moe-backend b12x --linear-backend b12x --load-format instanttensor; RadixArk drops all of those (the
NVFP4-experts + BF16 mix auto-resolves, b12x auto-selects, QSA hard-requires BF16 KV) and keeps
--gpu-memory-utilization 0.975 plus MTP spec decode
'{"method":"mtp","num_speculative_tokens":3}'.

Verify in 30 s: .venv/bin/python -c "import torch, vllm, b12x, flashinfer, instanttensor"
then a Qwen2.5-0.5B serve with FLASHINFER.

Launch vLLM dev flags RadixArk NVFP4

/root/workspace/vllm-project/.venv/bin/vllm serve \
    /root/models/hf-cache/models--RadixArk--Qwen3.8-Flash-Next-NVFP4/snapshots/7b719225242aacd3dbd3f9407468c2ee9a9d2594 \
    --served-model-name Qwen38 \
    --host 127.0.0.1  \
    --port 8001 \
    --dtype bfloat16 \
    --block-size 64 \
    --attention-backend FLASHINFER \
    --mm-encoder-attn-backend FLASHINFER \
    --mamba-cache-mode align \
    --mamba-ssm-cache-dtype auto \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --max-model-len 262144 \
    --max-num-seqs 16 \
    --max-num-batched-tokens 6019 \
    --gpu-memory-utilization 0.975 \
    --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \
    --max-cudagraph-capture-size 64 \
    --no-enable-flashinfer-autotune \
    --async-scheduling \
    --no-language-model-only \
    --mm-encoder-tp-mode data \
    --mm-processor-cache-gb 0.0 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_xml \
    --reasoning-parser qwen3 \
    --speculative-config '{"method":"mtp","draft_sample_method":"probabilistic","rejection_sample_method":"standard","num_speculative_tokens":3}'

Launch vLLM dev flags LocalFP4

/root/workspace/vllm-project/.venv/bin/vllm serve \
    /root/models/hf-cache/models--nvidia--Qwen3.8-Flash-Next-FP4-Local/snapshots/b9ce18a6887e18c089fdea8509a33e63429a3300 \
    --served-model-name Qwen38 \
    --host 127.0.0.1 \
    --port 8001 \
    --dtype bfloat16 \
    --quantization modelopt_mixed \
    --block-size 64 \
    --load-format instanttensor \
    --kv-cache-dtype fp8 \
    --attention-backend FLASHINFER \
    --mm-encoder-attn-backend FLASHINFER \
    --mamba-cache-mode align \
    --mamba-ssm-cache-dtype auto \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --max-model-len 262144 \
    --max-num-seqs 16 \
    --max-num-batched-tokens 6019 \
    --gpu-memory-utilization 0.975 \
    --moe-backend b12x \
    --linear-backend b12x \
    --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \
    --max-cudagraph-capture-size 64 \
    --no-enable-flashinfer-autotune \
    --async-scheduling \
    --no-language-model-only \
    --mm-encoder-tp-mode data \
    --mm-processor-cache-gb 0.0 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_xml \
    --reasoning-parser qwen3 \
    --speculative-config '{"method":"mtp","draft_sample_method":"probabilistic","rejection_sample_method":"standard","moe_backend":"b12x","num_speculative_tokens":3}'

Failpoints for HumanEval benchmark

The 10 tasks that failed in any run, k=3, three arms identical except --max_tokens (the 128k
arm covers only the 4 hardest):

An empty completion has two opposite causes, and the log cannot tell them apart since both
produce completion_tokens == reasoning_tokens == budget:

• Starvation, the budget is too small. /32 and /76 need 10-45k thinking tokens; at 2048
  they go 0/3, at 32768 they go 2-3/3 with 15k-44k tokens of reasoning.
• Runaway, the budget is too large. /99 solves in ~500-1,300 reasoning tokens at 2048 and
  32768, but at 131072 all 3 samples spiralled to the full budget and returned nothing, about
  39 minutes and 1.57M tokens for nothing.

So "just use the whole 262144 context" is actively worse, and 32768 happened to sit between the
two failure modes. Retrying at the same budget does nothing for starvation (3 attempts, 3
identical starvation); a retry helps only when it changes the budget. One more trap: 262144 is
the context window, input plus output, not an output budget. The server rejects a max_tokens
that leaves the prompt no room.

The LocalFP4 runs requested ~32k. RadixArk NVFP4's budget was never recorded, and reasoning length scales
~160x with the flag alone, so RadixArk NVFP4's clean <=17k tail is as consistent with a smaller budget as
with a better-behaved checkpoint. The capability comparison above uses no token counts and
stands; the budget-stability claim needs one re-run of both checkpoints at one recorded budget.