Qwen3.8-Flash-Next-NVFP4-Coder-Lean

elisepaul/Qwen3.8-Flash-Next-NVFP4-Coder-Lean is an architecturally pruned, pure-language derivative of nvidia/Qwen3.8-Flash-Next-NVFP4 (119.6B MoE, 512 routed micro-experts + 1 shared expert, 10 routed per token).

It is optimized specifically for single-node execution on NVIDIA Blackwell architectures (GB10 / DGX Spark, 128 GiB unified LPDDR5x) with strictly zero OS swap.

---

1. Overview & Architectural Focus

The upstream nvidia/Qwen3.8-Flash-Next-NVFP4 checkpoint is distributed as a unified multimodal release containing vision projections as well as an uncompressed 51.2 GB N-gram Prediction Layer Embedding (PLE) shard (model-fp8-mtp-ple.safetensors). When deployed on single-node 128 GiB unified-memory platforms (such as the NVIDIA DGX Spark / GB10), loading the combined ~124 GiB checkpoint exceeds available physical memory, inducing virtual memory paging and lowering generation efficiency.

Coder-Lean adapts the checkpoint specifically for dedicated language modeling, code generation, and developer agent workflows:

• Full MoE Routing Preserved: Retains all 10 active routed experts (+ 1 shared expert) per token, maintaining bit-exact parity with the original foundation language weights.
• Dedicated Text/Code Scope: Multimodal projection layers and the auxiliary PLE shard are omitted, reducing the loaded weight footprint from 123.5 GiB down to 73.42 GiB.
• Physical Headroom: Freeing 50+ GiB leaves ample physical memory (24–35 GiB free) for large KV caches (up to 1M context tokens via hybrid GDN linear attention) and optional co-location of worker models without operating system swap.

Checkpoint Composition

Component	Upstream Checkpoint	Coder-Lean	Technical Details
Model Shards	11 safetensors shards (~123.5 GiB)	10 safetensors shards (73.42 GiB)	50.08 GiB (-40.5%) footprint reduction
PLE Table	129 keys (layers.*.ple.*, 51.2 GB)	Omitted	Standard autoregressive decode handled via engine speculative decoding (ngram_gpu)
Visual Projections	Multimodal encoder weights	Omitted	Tailored strictly for text and code generation workloads
MoE Experts / Token	10 routed + 1 shared	10 routed + 1 shared	Preserves original expert capacity and reasoning behavior
Memory Footprint	~124 GiB (exceeds 128 GB physical)	73.4 GiB	24–35 GiB physical headroom on 128 GB unified architectures
Co-Location Capacity	Constrained	Supported	Headroom accommodates concurrent auxiliary services or worker models

---

2. Hardware Requirements & Reproduction Runbook

Hardware Target

• Platform: 1x NVIDIA DGX Spark (NVIDIA Blackwell GB10, Compute Capability sm_121)
• System Memory: 128 GiB Unified LPDDR5x (~121.6 GiB usable physical RAM)
• Swap Requirement: STRICTLY 0 BYTES SWAP (swapoff -a)
• Memory Bandwidth: ~273 GB/s theoretical bus ceiling

Software Requirements

• Docker Container: vllm-node:latest (or vllm/vllm-openai:latest built with CUDA 13.0 / PyTorch 2.9+)
• Kernel Backends: FlashInfer CUTLASS kernels with SwiGLU clamping, Gated Delta Net (GDN) linear attention kernels, NVIDIA ModelOpt NVFP4 quantization support.

---

3. Required Launcher Mods (vLLM Patches)

When serving pruned shards without the PLE table on vLLM, apply the following 4 minor patches inside the container:

# 1. Bypass residual PLE weight lookups
# in vllm/models/qwen4_exp/nvidia/model.py
skip_substrs = (
    "hashstats_",
    "token_lookup",
    "hyper_connection_mixer.block_inject_weight",
    ".ple.",
    "ple.",
)

# 2. QSA Cache 48-block alignment: lcm(16 Triton, 12 QSA)
# in vllm/models/qwen4_exp/common/qsa_cache.py
if self.cache_config.block_size % 48 != 0:
    self.cache_config.block_size = 48 * cdiv(self.cache_config.block_size, 48)

# 3. Align indexer attention block size
# in vllm/platforms/interface.py
if getattr(vllm_config, "speculative_config", None):
    attn_block_size = 48 * cdiv(attn_block_size, 48)

# 4. Fallback in select_common_block_size
# in vllm/v1/worker/utils.py
for size in [16, 8, 4, 2, 1]:
    if block_size_is_supported(backends, size):
        return size

---

4. Serving Recipe Command

Run the following command on the DGX Spark node:

vllm serve elisepaul/Qwen3.8-Flash-Next-NVFP4-Coder-Lean \
  --served-model-name elisepaul/Qwen3.8-Flash-Next-NVFP4-Coder-Lean \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 1 \
  --quantization modelopt \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.75 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --kv-cache-dtype fp8_e4m3 \
  --enable-expert-parallel \
  --language-model-only \
  --no-enable-flashinfer-autotune \
  --async-scheduling \
  --speculative-config '{"method":"ngram_gpu","num_speculative_tokens":5,"prompt_lookup_max":4,"prompt_lookup_min":1}' \
  --reasoning-parser qwen3 \
  --trust-remote-code

---

5. Benchmark & Validation Results

SWE-bench Lite (Astropy Sub-suite)

Evaluated on Princeton SWE-bench Lite instances:

========================================================================
Evaluation Run: elisepaul/Qwen3.8-Flash-Next-NVFP4-Coder-Lean
Sampling: temperature=0.6, presence_penalty=0.1, reasoning_effort=medium
Result: 5/5 Non-Empty Targeted Patches Generated (100% Patch Generation)
Total Generation Time: 424.1s (7.0 minutes for all 5 instances)
Finish Reason: 100% 'stop'
========================================================================

• astropy-6938: Corrected string replacement missing assignment (output_field = output_field.replace(...)).
• astropy-14995: Implemented missing None mask guards in NDArithmetic.
• astropy-14182: Supported user kwargs override via *args, **kwargs with .setdefault().
• astropy-14365: Case-insensitive QDP command parsing via line.upper().
• astropy-12907: Implemented recursive _flatten_and_compound for nested & compound models.

Live Throughput Profile on NVIDIA GB10

• Prompt Prefill: 265.8 tokens/sec (FlashInfer GDN kernel)
• Draft Proposer Throughput: 146 – 148 tokens/sec
• Sustained Aggregate Decode (4 Streams): 51 – 56 tokens/sec
• Peak Aggregate Decode (Code Emission): 79.2 tokens/sec
• Single-Stream Repetitive Code Decode: 76.2 tokens/sec (100% draft acceptance rate)

Architectural Long-Context Headroom

Out of 48 layers, 36 layers use Gated Delta Net (GDN) linear attention ($O(1)$ memory). Storing the KV cache for 1,000,000 tokens requires only 12.28 GiB of RAM, making full 1M context physically feasible within 128 GB physical memory with zero swap.

---

6. Attribution, Citations & Licensing

Upstream Model Licenses

This model is a derivative work governed by the following upstream licenses:

1. NVIDIA Open Model License Agreement: Governs the NVFP4 quantization calibration and Model Optimizer weights exported by NVIDIA Corporation (nvidia/Qwen3.8-Flash-Next-NVFP4).

   • License: NVIDIA Open Model License

2. Qwen Community License Agreement 1.0: Governs the underlying neural architecture, tokenizer, and foundation language weights developed by the Qwen Team (Alibaba Cloud).

   • License: Qwen License Terms

3. Software & Infrastructure Tooling:

   • vLLM: Apache License 2.0 (Copyright (c) 2023-2025 vLLM Team)
   • FlashInfer: Apache License 2.0 (Copyright (c) 2023-2025 FlashInfer Contributors)
   • CUTLASS: BSD 3-Clause License (Copyright (c) 2017-2025 NVIDIA Corporation)
   • spark-vllm-docker: MIT License (Copyright (c) 2025 Eugene G. / eugr)
   • SWE-bench: MIT License (Copyright (c) 2023 Princeton NLP Group)

Citations

@article{qwen3flashnext2025,
  title={Qwen3.8-Flash-Next: Sparse Mixture-of-Experts with Hybrid Recurrent Attention},
  author={Qwen Team, Alibaba Cloud},
  year={2025}
}

@article{nvidia_modelopt2025,
  title={NVIDIA Model Optimizer: High-Performance Quantization and Acceleration for Blackwell Architectures},
  author={NVIDIA Corporation},
  year={2025}
}

@inproceedings{jimenez2024swebench,
  title={SWE-bench: Can Language Models Resolve Real-World GitHub Issues?},
  author={Jimenez, Carlos E and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik},
  booktitle={ICLR},
  year={2024}
}

@article{deepseek_r1_2025,
  title={DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning},
  author={DeepSeek-AI},
  journal={arXiv preprint arXiv:2501.12948},
  year={2025}
}