Ling-3.0-flash-VL-NVFP4-MP

Max-performance mixed NVFP4 + FP8 quantization of inclusionAI/Ling-3.0-flash-VL (124 B total / 5.1 B active hybrid KDA + MLA MoE vision-language model, MIT), produced with NVIDIA Model Optimizer 0.48 and served by the inclusionAI vllm-ling-v3 fork (e2e5751 + two patches) in a purpose-built SM121 container.

• Size: 68 GB (64 shards) — 3.7× compression over the 249.7 GB BF16 source
• Runtime: ghcr.io/r0b0tlab/ling30vl-nvfp4-mp-sm121:e2e5751 (arm64, SM12x)
• Repro package (click-run): r0b0tlab/ling30vl-nvfp4-mp-sm121
• Hardware validated: single NVIDIA GB10 / SM 12.1 (121 GB unified)

Quantization

scope	modules	format
routed experts (layers 2–41, gate/up/down)	61,440	NVFP4 W4A4, group 16, E4M3 block scales, FP32 global amax/(6·448), dynamic per-token activations, input_scale = 1.0
shared experts	120	NVFP4 (same)
dense MLP (layers 0–1)	6	NVFP4 (same)
lm_head	1	NVFP4 (same)
attention (KDA q/k/v/b/f/g/o + MLA q/g/kv_a/kv_b/dense)	280	FP8 per-channel weight, per-token dynamic activation
vision tower, projector, embeddings, norms, conv1d, routers, A_log, dt_bias	716	BF16/FP32 preserved (byte-identical)

Audits: G0 census 16/16 · canary NVFP4 cos ≥ 0.9950 / FP8 cos ≥ 0.9997 · metadata contract · coverage 61,847/61,847 modules · G3 audit PASS (186,104 index keys).

Size, memory, and KV cache (single GB10, 121 GiB unified)

• Artifact: 72.62 GB / 67.64 GiB (64 shards; 72,597,622,664 bytes of weights + 26.6 MB metadata) — 3.44× smaller than the 249.7 GB BF16 source.
• GPU memory at the winner config (--gpu-memory-utilization 0.85): weights 67.97 GiB + CUDA graph capture ~1.0 GiB + KV pool 30.49 GiB ≈ 99.5 of 121 GiB unified.
• KV cache (fp8): 4,032 B/token (7 MLA layers × 576 B); 2,309,051 tokens capacity → 140.9× concurrency at 16 K, 70.4× at 32 K, 17.6× at the 131,072-token advertised window. fp8 is the maximum — NVFP4 KV is unavailable for this dense-MLA model on SM121.

Quality (single GB10, this checkpoint + container)

gate	result
text probes	5/5 (17×23 = 391, fib(10) = 55, strict JSON, code, real tool_calls)
vision probes	8/8 (counting ×4, 7-segment OCR, bar chart, frame order/colour)
GSM8K-200 (flexible extract)	91.5 % (183/200)

Performance (single GB10 / SM121, c = 1)

metric	value
decode	43.5 tok/s
prefill	986 tok/s (TTFT 13.0 s on 12,832 tokens)
ITL	22.6 ms
aggregate	42.4 → 164.8 → 165.4 tok/s at c = 1 / 16 / 32
load / KV	516–540 s, 67.97 GiB / 31.8 GiB → 2.41 M tokens (147× at 16 K)

MoE backend FLASHINFER_CUTLASS, attention TRITON_MLA, NVFP4 linear FlashInferCuteDslNvfp4W4A16, CUDA graphs + compile (4.2× vs eager), async scheduling.

Use

git clone https://github.com/r0b0tlab/ling30vl-nvfp4-mp-sm121.git
cd ling30vl-nvfp4-mp-sm121 && ./scripts/click-run.sh

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
r = client.chat.completions.create(
    model="ling-3.0-flash-vl-nvfp4-mp",
    messages=[{"role": "user", "content": "What is 17*23? Think briefly."}],
    temperature=0, max_tokens=512)
print(r.choices[0].message.content)

Thinking is on by default; pass chat_template_kwargs={"enable_thinking": False} for deterministic short answers. Tool calling: --enable-auto-tool-choice --tool-call-parser ling3 (already set in the container).

Boundaries

• NVFP4 KV cache is unavailable for this dense-MLA model on SM121 (runtime-verified; fp8 is the maximum).
• Video input is not supported by the fork's VL implementation (image-only).
• n-gram speculative decoding is a throughput regression on general traffic (opt-in only).
• No MTP head ships in this checkpoint.

Citation / attribution

Quantized by r0b0tlab with NVIDIA Model Optimizer; base model by inclusionAI. Independent community quantization — not affiliated with or endorsed by inclusionAI or NVIDIA. Evidence bundle: evidence/ in the repro repository (sanitized; no prompts or responses).