inclusionAI

/

Ling-3.1-flash

Like 147

Follow

inclusionAI 3.32k

A small note on the config.json change: Transformers' fine-grained FP8 integration treats each modules_to_not_convert entry as a substring of the full module path (key in current_key_name_str), rather than as an exact module-name match.

Because of that, the previous b_proj entry also matched MLA attention.q_b_proj and attention.kv_b_proj, even though those weights are FP8 in the checkpoint. Likewise, mlp.gate matched the dense layers' mlp.gate_proj, which left their weight_scale_inv tensors unused.

Narrowing the first entry to attention.b_proj keeps the intended KDA projection in BF16 without excluding the MLA projections. Removing mlp.gate lets the dense gate_proj layers load as FP8. This does not affect the MoE router: BailingMoeV3Gate stores its routing weight as a regular parameter and is not an nn.Linear, while the FP8 replacement only operates on nn.Linear modules.

The full-checkpoint validation confirmed the expected module types and no unused weight_scale_inv warnings.

One more compatibility note before merge: I also validated the proposed config.json against the latest SGLang upstream at e5274752 (0.5.12.post2.dev7480+ge5274752a).

The updated exclusions are intentionally narrower:

• b_proj → attention.b_proj
• remove mlp.gate

SGLang matches exclusions on dotted module-path boundaries. Under this matcher, the old and new configurations both keep KDA attention.b_proj unquantized, while MLA q_b_proj / kv_b_proj and dense mlp.gate_proj remain FP8. The MoE router is implemented as BailingMoEGate, rather than a quantized LinearBase; its weight remains BF16 and routing continues through the existing FP32-stable path. Removing mlp.gate therefore does not change router precision.

I also ran a full-checkpoint SGLang smoke test with the proposed config:

• native BailingMoeV3ForCausalLM with trust_remote_code=False
• TP4 / EP4
• all 54 checkpoint shards loaded as FP8 E4M3
• the expected 63 KDA + 9 MLA layer layout was detected
• the KDA prefill/decode backends and both recurrent-state and KV caches initialized successfully
• an actual generation request completed successfully without retractions

The Transformers native path was checked with the same full checkpoint as well: all 72 model/cache layers loaded, the intended module types were preserved, logits remained finite, and the output matched the existing baseline exactly (max_abs=0) with identical generated tokens.

This makes the proposed config.json fully compatible with current SGLang upstream in both matcher semantics and full-checkpoint runtime behavior. The more explicit expression also encodes the architectural intent directly, avoids dependence on engine-specific substring behavior, and should be easier to maintain as either integration evolves.

@iCavan Thanks for the fix and the additional validation. We checked SGLang’s exclusion matching: it matches dotted module-path boundaries, so b_proj does not match q_b_proj / kv_b_proj, and mlp.gate does not match mlp.gate_proj.

The old and new configurations therefore produce the same relevant quantization decisions in SGLang: KDA b_proj and the explicitly excluded MLA g_proj remain unquantized, while MLA q_b_proj / kv_b_proj and dense MLP projections retain their FP8 paths.

The FP8 configuration change is correct: it fixes Transformers’ overly broad matching without introducing additional quantization or changing the corresponding SGLang execution paths.

We'll merge the fix later on.