olka-fi/Ling-3.0-flash-MXFP4

Ling-3.0-flash — MXFP4 (mixed precision)

A 4-bit MXFP4 quantization of Ling-3.0-flash, produced with qstream. The routed MoE experts (~95% of the weights) are quantized to MXFP4; everything quality-sensitive stays BF16.

The original model card follows in full below.

Size	77.6 GB (down from 255.0 GB BF16 source, ~30%)
Format	compressed-tensors mxfp4-pack-quantized (E2M1 4-bit + E8M0 group-32 scales)
Base	Ling-3.0-flash — 124B total / 5.1B active hybrid-linear MoE; 42 layers stacked 5:1 as 35 Kimi-Delta-Attention (KDA) + 7 gated-MLA; 512 routed experts top-8 + 1 shared; 2 dense layers; 1 MTP layer; 256K context

Fits a single 180 GB GPU with room to spare — measured 4,611,739 tokens of KV cache on one B200 at --gpu-memory-utilization 0.82, --max-model-len 32768.

What is quantized to what

Component	Precision	Why
Routed experts, layers 2–41 (model.layers.N.mlp.experts.E.{gate,up,down}_proj)	MXFP4 (4-bit)	120.8B of 127.4B params — the only place worth the size win
Shared expert (mlp.shared_experts.*)	BF16	active on every token; quantizing it costs ~half the quality loss for <1% of the size
Attention — KDA (q/k/v/f/g/b_proj, conv1d, A_log, dt_bias) and gated MLA (kv_a/kv_b, dense)	BF16	the KDA gating projections feed a softplus/sigmoid decay gate and are the most precision-sensitive tensors in the model
Router gate + expert_bias, dense MLP (layers 0–1)	BF16	routing decisions are discrete; a perturbed router changes which experts fire
MTP layer 42 (incl. its own 512 experts)	BF16	speculative-decoding draft path — acceptance is a prefix-run-length statistic, so drafter drift compounds
Embeddings, lm_head, all norms	BF16	unchanged

61,440 tensors quantized (40 layers × 512 experts × 3 projections). Everything else is byte-identical to the source.

Fidelity

Weight-reconstruction relative error across all 61,440 quantized tensors:

min	p25	median	p75	p95	max	std
0.1099	0.1115	0.1116	0.1118	0.1120	0.1215	0.000236

Per-layer relative error is flat from layer 2 (0.11177) to layer 41 (0.11152) — no layer degrades. The top 10% of tensors account for exactly 10.0% of total error, i.e. the error is diffuse: there are no "hot" experts, so selective per-expert FP8 would buy nothing. ~11.2% is the expected floor for E2M1 with group-32 scales, so this build sits at the format's limit rather than losing anything to a poor scale search.

Downstream evals (this checkpoint, vLLM, non-thinking protocol)

Benchmark	Protocol	Result
GSM8K	5-shot, full 1319-problem test set	84.4% flexible-extract · 78.9% strict-match
GPQA-diamond	zero-shot loglikelihood MC, all 198	45.5% acc / acc_norm

These are non-thinking numbers and are not comparable to the base model's published scores, which are generated with thinking mode + CoT. They are a quantization-damage gate, not a leaderboard claim.

Serving with vLLM

docker run --gpus all -d --name ling3 --ipc=host -p 8000:8000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    olkafi/vllm-bailing-v3:latest \
    olka-fi/Ling-3.0-flash-MXFP4 \
    --served-model-name ling3 --trust-remote-code \
    --host 0.0.0.0 --port 8000 \
    --gpu-memory-utilization 0.85 --max-model-len 128768 \
    --enable-prefix-caching --mamba-cache-mode align \
    --enable-auto-tool-choice --tool-call-parser ling3 --reasoning-parser ling3

Ling-3.0 (BailingMoeV3ForCausalLM, model_type: bailing_hybrid) needs the vendor fork inclusionAI/vllm, branch ling_3_0. (The base model card points at inclusionAI/vllm-ling-v3, which does not exist.) Upstream vLLM has only BailingMoe / V2 / V2.5.

Required: the MoE clamp patch

Ling-3.0 declares a per-layer clamped SwiGLU (expert_swiglu_limit_list = [0]*35 + [4]*7, so layers 35–41 clamp at 4). vLLM's compressed-tensors MXFP4 MoE path drops that clamp: the quant method never reads layer.swiglu_limit, the MXFP4 quant-config builder has no parameter to receive it, and run_cutlass_moe_mxfp4 calls apply_moe_activation() with no clamp argument. The result is unclamped SwiGLU on layers 35–41 for any MXFP4 compressed-tensors checkpoint of a model with a SwiGLU limit — the divergence is large, not marginal (gate=10 gives silu(10)*2 = 20.0 versus silu(4)*2 = 7.84).

Apply the three files in vllm_patch/ over the fork before serving. They are pure Python — an editable install needs no rebuild. See vllm_patch/README.md. Models without a SwiGLU limit are unaffected by the patch.

Launch

A prebuilt runtime image (fork + clamp patch + flashinfer-jit-cache) is published as olkafi/vllm-bailing-v3:

docker run --gpus all -d --ipc=host -p 8000:8000 \
  -e HF_HOME=/hf -v /path/with/80GB/free:/hf \
  olkafi/vllm-bailing-v3:latest \
  olka-fi/Ling-3.0-flash-MXFP4 \
  --served-model-name ling3 --trust-remote-code \
  --host 0.0.0.0 --port 8000 \
  --gpu-memory-utilization 0.85 --max-model-len 32768 --max-num-seqs 256 \
  --enable-prefix-caching --mamba-cache-mode align \
  --kernel-config '{"moe_backend":"triton"}' \
  --enable-auto-tool-choice --tool-call-parser ling3 --reasoning-parser ling3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Or with a plain vllm serve against the same fork:

vllm serve olka-fi/Ling-3.0-flash-MXFP4 \
  --served-model-name ling3 --trust-remote-code \
  --tensor-parallel-size 1 --gpu-memory-utilization 0.85 \
  --max-model-len 262144 --max-num-seqs 256 \
  --enable-prefix-caching --mamba-cache-mode align \
  --kernel-config '{"moe_backend":"triton"}' \
  --load-format fastsafetensors \
  --enable-auto-tool-choice --tool-call-parser ling3 --reasoning-parser ling3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Three flags are load-bearing and easy to get wrong:

• --enable-prefix-caching is required for --mamba-cache-mode align. Without it vLLM silently downgrades the KDA linear-attention state cache to none, logging only a warning.
• --kernel-config '{"moe_backend":"triton"}' is required whenever MTP is enabled. Ling-3.0 routes 512 experts in 8 groups = 64 experts per group, but FlashInfer's TRT-LLM grouped-routing kernel is warp-limited to 32. That kernel is selected for the unquantized MoE — i.e. the BF16 MTP layer — and either raises Routing kernel expects #experts per group <= warp size (32) ... = 64, or, on paths where it does not raise, mis-routes experts so the drafter emits tokens the target never accepts (0% acceptance, silently). Triton has no such constraint.
• --max-num-seqs 256 — the BF16 MTP head takes ~6 GB, shrinking the KDA state cache below the default 1024 sequences, so CUDA graph capture aborts with max_num_seqs (1024) exceeds available Mamba cache blocks (684).

With all three, graph capture succeeds and --enforce-eager is not needed.

Speculative decoding (MTP) acceptance

The MTP layer is deliberately kept at BF16, and it earns it — measured on this checkpoint with num_speculative_tokens: 3 over a 200-problem GSM8K run (32,577 drafted tokens):

Acceptance rate	58.5%
Accepted per draft step	1.756
Tokens emitted per step	2.756 (1 bonus + accepted)

Acceptance by draft position, conditional on reaching it:

position	conditional	cumulative
0	95.6%	95.6%
1	59.0%	56.4%
2	42.0%	23.7%

95.6% at the first draft position means the BF16 drafter tracks the MXFP4 target very closely. The falloff after it is expected rather than a defect: Ling-3.0 has a single MTP layer (num_nextn_predict_layers: 1), so num_speculative_tokens > 1 re-runs that one layer, and the fork warns as much at startup. num_speculative_tokens: 2 may well beat 3 on net throughput — worth measuring for your workload.

Note MTP roughly doubles cold-start time: the draft loader re-reads the whole checkpoint to extract one layer (the MTP weights live in 2 of 24 shards, but all 24 are read).

Recommended sampling (from the base card): temperature=0.6, top_p=0.95, top_k=20, with enable_thinking.

Provenance

Built with qstream from the BF16 inclusionAI/Ling-3.0-flash release:

qstream-quantize \
  --model_dir  Ling-3.0-flash \
  --output_dir Ling-3.0-flash-MXFP4 \
  --quant_format mxfp4 --format ct \
  --include_layers '*.mlp.experts.*' \
  --exclude_layers '*.layers.42.*' \
  --workers 8 --device cuda

MSE-optimal scale selection (3 candidates/block, anchor percentile 99.5) with γ-weighted activation awareness from post_attention_layernorm.

License

Inherits the MIT license from the base model. This is a derivative (quantized) work of inclusionAI/Ling-3.0-flash.

---

Original model card

🤗 Hugging Face   |   🤖 ModelScope    |   🐙 OpenRouter   

Introduction

We're introducing Ling-3.0-flash, our next-generation native hybrid reasoning model. Operating with 124B total and 5.1B active parameters (~12.4% and ~8.1% of our previous 1T-class flagship Ring-2.6-1T), Ling-3.0-flash matches or outperforms its predecessor across key benchmarks.

Key highlights of the model are summarized below:

• Native Hybrid-Linear Architecture: Ling-3.0 adopts a native hybrid linear attention architecture from the very start of pretraining (5:1 alternating stacking of Kimi Delta Attention (KDA) and MLA), upgraded with KDA fine-grained diagonal gating and 1/64 sparse MoE. With 124B total parameters and 5.1B activated parameters, it achieves a synergistic leap in long-context efficiency and computational cost.
• Remarkable Efficiency & Performance: Engineered for speed, compute efficiency, and production deployment, Ling-3.0-flash delivers class-defying performance against both larger SOTA competitors and previous-generation flagships. Activating only 5.1B parameters per token, it provides impressive reasoning, instruction following, and long-context capabilities to empower complex agentic workflows in production environments.
• Comprehensive Agentic Evolution: Tailored for real-world productivity workflows, the model incorporates 10,000+ interactive training environments to achieve end-to-end closed-loop execution across Coding, General, and Deep Research Agent tasks. It natively integrates the SGLang HiCache + Mooncake hierarchical caching architecture (featuring physical dual-pools and a cluster-shared L3 cache), eliminating redundant recomputation during long-horizon interactions and reducing Time to First Token (TTFT) by 60% to over 80% in long-input scenarios.s the SGLang HiCache + Mooncake hierarchical caching architecture (featuring physical dual-pools and a cluster-shared L3 cache), eliminating redundant recomputation during long-horizon interactions and reducing Time to First Token (TTFT) by 60% to over 80% in long-input scenarios.

Model Overview

The model summary information and architecture diagram are as follows:

Architecture	Hybrid-linear MoE
Parameter Scale	Totoal 124B, Activated 5.1B
Transformer Layers	35 KDA + 7 Gated MLA (5:1)
Number of Dense Layers	2
Number of Routed Experts	512
Number of Shared Experts	1
Number of Activated Experts	8
Attention Heads	32
Hidden Size	2560
Expert Intermediate Size	768
Dense Intermediate Size	6144
Vocabulary Size	157184
Context Training Schedule	8K -> 32K -> 256K

Evaluation

We have conducted a comprehensive evaluation of Ling-3.0-flash across multiple authoritative benchmarks. Ling-3.0-flash performs strongly on representative code/agent benchmarks such as SWE-Bench Pro, SWE-Bench Multilingual, Tau3-banking-AA, MCP-Atlas and SkillsBench, etc. In practice, Ling-3.0-flash delivers a strong user experience across frameworks including Claude Code,Kilo Code,Qwen Code,Hermes Agent,and OpenClaw, etc. Beyond agentic tasks, Ling-3.0-flash also delivers strong performance across general knowledge,mathematical reasoning,instruction following,and long-context understanding.

• Thinking mode is enabled by default. Unless otherwise specified, the default parameters for Ling-3.0-flash are as follows: temperature=0.6, top_p=0.95, top_k=20.
• SWE-Bench Series:Evaluated using OpenHands as the agent harness with tailored prompts. Decoding uses temperature=0.6, top_p=0.95, max_new_tokens=32K, with a 256K context window.
• Terminal-Bench 2.1: Evaluated under the Artificial Analysis (AA) protocol using the default Terminus 2 harness, a unified 2-hour timeout, the provided JSON parser in preserve-thinking mode, and 3 runs per task (mean). Decoding uses temperature=0.6, top_p=1.0, max_new_tokens=32K, with a 256K context window.
• MiniAppBench: A 500-task coding benchmark evaluating whether models can turn a single user request into complete, usable interactive HTML apps in real-world application-generation scenarios. Evaluated with temperature=1.0, top_p=1.0, max_tokens=128K.
• AntSWEBench: AntSWEBench is an internally used software engineering benchmark that covers mainstream programming languages such as Java, JavaScript, and Python, including various development scenarios like new feature, bug fix, and code refactoring.
• Tau3-banking-AA: Aligned with the AA leaderboard, utilizing GPT-5.4-mini (medium reasoning) for both the user simulator and the natural-language assertion judge.
• MCP-Atlas: Evaluated on the 500-task public set using the official v1 harness with a 20-turn limit and Gemini-2.5-Pro as the claim-coverage judger.
• SkillsBench: Evaluated via kilo-code on 87 tasks (excluding external API-dependent tasks), averaged over 3 runs.
• GDPval v2-AA : Evaluated on the public 220-task benchmark using the official Stirrup harness, with a 250-turn limit and a 5-hour timeout.
• Search‑agent：For all search‑agent tasks, evaluations are performed using an internal harness. The basic ReAct paradigm is adopted for single-agent evaluation, while a multi-agent setup is employed for BrowseComp. The reported metric is the average pass@1.
  • WideSearch: Evaluated using the official prompt and the official judge model GPT-4.1 on the corrected version of the dataset.
  • Draco: Scored based on official rubrics per question, with the final score calculated as the average across all questions using Claude Opus 4.6 as the scoring model.
  • BrowseComp (Single-Agent): Evaluated using a resume strategy for context management: once the context reaches a 64K-token threshold, the trajectory is summarized, the original history is discarded, and execution is resumed from the summary.
  • BrowseComp (Multi-Agent): Evaluated on English and ZH Revised datasets using an internal multi-agent search harness based on SearchSwarm/Tongyi DeepResearch, configured with temperature=0.85, top_p=0.95, max_tokens=8K, and main/sub-agent context windows of 128K and 64K, respectively.

Quickstart

SGlang

Install our SGLang

pip install uv

uv venv ~/my_ling_env

source ~/my_ling_env/bin/activate

git clone -b ling_v3_support https://github.com/inclusionAI/sglang_ling_v3.git

cd sglang_ling_v3

pip install --upgrade pip

pip install -e "python"

Run Inference

Here is an example to run Ling-3.0-flash with 4 GPUs, where the master node IP is ${MASTER_IP} and server port is ${PORT}:

Server

Since the model is trained with MTP, we recommend enabling MTP during inference (i.e., --speculative-algorithm NEXTN) for lower latency.

export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1
export SGLANG_JIT_DEEPGEMM_PRECOMPILE=1
export SGLANG_ENABLE_SPEC_V2=1
python -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --dist-init-addr $MASTER_IP:2345 \
    --port $PORT \
    --nnodes 1 \
    --mem-fraction-static 0.8 \
    --max-running-requests 64 \
    --tp-size 4 \
    --chunked-prefill-size 8192 \
    --tool-call-parser ling3 \
    --reasoning-parser ling3 \
    --context-length 262144 \
    --speculative-algorithm NEXTN \
    --max-mamba-cache-size 320 \
    --enable-fp32-lm-head \
    --disable-shared-experts-fusion

Client

We recommend using the sampling parameters temperature=0.6, top_p=0.95, and top_k=20, and enabling enable_thinking for better performance.

curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "auto",
       "messages": [{"role": "user", "content": "hello!"}],
       "chat_template_kwargs": {"enable_thinking": true},
       "stream": true,
       "temperature": 0.6, 
       "top_k": 20,
       "top_p": 0.95
     }'

vLLM

Install our vLLM

pip install uv

uv venv ~/my_ling_env

source ~/my_ling_env/bin/activate

git clone -b ling_3_0 https://github.com/inclusionAI/vllm-ling-v3.git

cd vllm-ling-v3

VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=auto

Run Inference

Here is the example to run Ling-3.0-flash with 4 GPUs, where the server port is ${PORT}:

Server

Since the model is trained with MTP, we recommend enabling MTP during inference (i.e., --speculative-config) for lower latency.

vllm serve "$MODEL_PATH" \
    --port "$PORT" \
    --trust-remote-code \
    --served-model-name auto \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.85 \
    --enable-prefix-caching \
    --mamba-cache-mode align \
    --enable-auto-tool-choice \
    --tool-call-parser ling3 \
    --reasoning-parser ling3 \
    --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Client

We recommend using the sampling parameters temperature=0.6, top_p=0.95, and top_k=20, and enabling enable_thinking for better performance.

curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "auto",
       "messages": [{"role": "user", "content": "hello!"}],
       "chat_template_kwargs": {"enable_thinking": true},
       "stream": true,
       "temperature": 0.6, 
       "top_k": 20,
       "top_p": 0.95
     }'