marfrit/Qwen3.6-35B-A3B-MTP-OpenVINO

Qwen3.6-35B-A3B MTP head for OpenVINO — reconstructed (draft model card)

The same reconstruction for the MoE model: openvino_mtp_layer (the MTP layer with its 256-expert MLP, 1.69 GB f16) and openvino_mtp_lm_head (the base IR's int8 lm_head, 509 MB). Built by tools/export_mtp.py from the checkpoint's own mtp.* tensors; pairs with OpenVINO/Qwen3.6-35B-A3B-int4-ov.

body	draft acceptance (greedy, B60)	decode, --mtp on vs off
Intel's public int4 IR	93.9% code / 75.4% prose	48–53 t/s vs 71.5 t/s

Read the second column before using it. The head is correct, and on a stock OpenVINO build speculation with it is slower than plain decoding on an Arc card. The kernels are not the reason: a two-token forward takes the plugin's batched-GEMV decode path and costs the device 1.15× a one-token step. The cost has been found, and it is host-side churn. At token_num > 1 the MoE implementation's prepare_internal_buffers rebuilds its per-expert mask subbuffers on every inference — 20,480 create_subbuffer calls per two-token forward — for a per-expert prefill fallback that the batched-GEMV path never reads. At one token the block is skipped, which is why plain decoding never showed it. A plugin patch that skips the masks below the batched-GEMV threshold takes the verify forward from 27.3 ms to 18.1 ms with byte-identical output.

Separately, the head as exported reads its 256-expert MLP densely in f16, 1.69 GB per draft: the router is present in the graph, but its lowering does not match the pattern the plugin's MoE fusion looks for, so every expert is computed for every token and the unselected ones are weighted by zero. An int4 quantisation of the same head cuts that read.

Measured on a patched plugin, three prompts, Arc Pro B60, u8 KV, 32k context, greedy (decode throughput from the server's own line, medians over 9 runs for the first prompt and 4 for the others):

prompt	plain	f16 head	int4 head	draft acceptance (f16 / int4)
RFC-4180 CSV parser	64.5 t/s	65.6 (+1.7%)	76.3 (+18.3%)	95.4% / 95.4%
German prose, ~350 words	69.4	63.7 (−8.2%)	73.4 (+5.7%)	66.8% / 67.5%
ISO-8601 duration parser	69.7	67.5 (−3.2%)	81.0 (+16.2%)	90.2% / 92.8%

So the f16 head is a wash at best and a loss on prose; the int4 head wins on every prompt, and the size of the win tracks draft acceptance. Read the next paragraph before taking the third column at face value.

Speculative decoding here changes the answer, and not always for the better. Verification is exact — a drafted token is accepted only when it equals what the sampler would have picked — but a two-token verify pass and a single-token pass differ slightly in the logits on this backend, which can flip a near-tie. When that flip lands early, everything after it is a different draw, not a rewording. Measured on the three prompts above, each arm deterministic across repeats:

• On the CSV task the int4-head output was byte-identical to plain decoding in every run, and both scored 10/10 on an executed test harness.
• On the prose task all three arms produced different text.
• On the ISO-8601 task the int4-head answer was a different and worse program: 107 lines against 188, scoring 3/8 against plain's 5/8 on an executed harness, with a runtime crash on three of the eight cases.

One prompt in three, then, cost real quality — reproducibly. If you serve this head, gate it on your own task, and do not assume "exact verification" means "same answer". If you need bit-exact reproducibility against a non-speculative run, serve without the head.