marfrit/Qwen3.8-27B-MTP-OpenVINO

Qwen3.8-27B MTP head for OpenVINO — reconstructed (draft model card)

What this is. The multi-token-prediction head of Qwen/Qwen3.8-27B as two OpenVINO IR graphs, rebuilt from the checkpoint's own mtp.* tensors by tools/export_mtp.py in the arcint repository:

file	what	size
openvino_mtp_layer.{xml,bin}	the MTP transformer layer (f16)	849 MB
openvino_mtp_lm_head.{xml,bin}	the LM head the draft is decoded with (f16)	1.27 GB

Reconstructed, not official. optimum-intel drops the MTP graph on export; newer development builds export the layer as openvino_mtp_model but not the lm_head. These two files are what arcint's speculative decoding path serves.

Which IR it belongs to. Any OpenVINO IR of Qwen3.8-27B with hidden 5120, vocab 248320 and untied embeddings. Measured to pair with:

body	draft acceptance (greedy, B60)
arcint's AWQ export (qwen38-b7c1-ov)	93.2%
OpenVINO/Qwen3.8-27B-int4-ov (Intel's public int4)	90.8% on the acceptance task (10/10), 96.3% code / 77.3% prose

Intel's own MTP layer works with this lm_head. OpenVINO/Qwen3.8-27B-int4-ov ships openvino_mtp_model (the layer, int4, no lm_head). Served through arcint's --mtp-layer exported with the openvino_mtp_lm_head from this card: 93.9% acceptance on code, 76.4% on prose, 37.7–38.1 t/s against 25.0 t/s without speculation on the B60. If you have Intel's IR, the lm_head is the only file you are missing.

How to use. Place the four files beside openvino_language_model.xml and serve with arcint --mtp on. Acceptance is printed on every decode line (draft accept 96.3% (157/163)); a head that does not belong shows ~0%.

Why acceptance is the oracle. A drafted token is accepted only when it equals what the sampler would have picked anyway, so a wrong head cannot turn a good answer into a bad one by drafting nonsense — it can only make speculation useless.

That is not the same as "the answer does not change". A two-token verify pass and a single-token pass differ slightly in the logits on this backend, so a near-tie can flip, and a flip early in the answer makes everything after it a different draw rather than a rewording. Measured on the 35B below, one prompt in three came out measurably worse with speculation on. Gate it on your own task.

License follows the base model (Apache-2.0).

---

Qwen3.6-35B-A3B MTP head for OpenVINO — reconstructed (draft model card)

The same reconstruction for the MoE model: openvino_mtp_layer (the MTP layer with its 256-expert MLP, 1.69 GB f16) and openvino_mtp_lm_head (the base IR's int8 lm_head, 509 MB). Built by tools/export_mtp.py from the checkpoint's own mtp.* tensors; pairs with OpenVINO/Qwen3.6-35B-A3B-int4-ov.

body	draft acceptance (greedy, B60)	decode, --mtp on vs off
Intel's public int4 IR	93.9% code / 75.4% prose	48–53 t/s vs 71.5 t/s

Read the second column before using it. The head is correct, and on a stock OpenVINO build speculation with it is slower than plain decoding on an Arc card. The kernels are not the reason: a two-token forward takes the plugin's batched-GEMV decode path and costs the device 1.15× a one-token step. The cost has been found, and it is host-side churn. At token_num > 1 the MoE implementation's prepare_internal_buffers rebuilds its per-expert mask subbuffers on every inference — 20,480 create_subbuffer calls per two-token forward — for a per-expert prefill fallback that the batched-GEMV path never reads. At one token the block is skipped, which is why plain decoding never showed it. A plugin patch that skips the masks below the batched-GEMV threshold takes the verify forward from 27.3 ms to 18.1 ms with byte-identical output.

Separately, the head as exported reads its 256-expert MLP densely in f16, 1.69 GB per draft: the router is present in the graph, but its lowering does not match the pattern the plugin's MoE fusion looks for, so every expert is computed for every token and the unselected ones are weighted by zero. An int4 quantisation of the same head cuts that read.

Measured on a patched plugin, three prompts, Arc Pro B60, u8 KV, 32k context, greedy (decode throughput from the server's own line, medians over 9 runs for the first prompt and 4 for the others):

prompt	plain	f16 head	int4 head	draft acceptance (f16 / int4)
RFC-4180 CSV parser	64.5 t/s	65.6 (+1.7%)	76.3 (+18.3%)	95.4% / 95.4%
German prose, ~350 words	69.4	63.7 (−8.2%)	73.4 (+5.7%)	66.8% / 67.5%
ISO-8601 duration parser	69.7	67.5 (−3.2%)	81.0 (+16.2%)	90.2% / 92.8%

So the f16 head is a wash at best and a loss on prose; the int4 head wins on every prompt, and the size of the win tracks draft acceptance. Read the next paragraph before taking the third column at face value.

Speculative decoding here changes the answer, and not always for the better. Verification is exact — a drafted token is accepted only when it equals what the sampler would have picked — but a two-token verify pass and a single-token pass differ slightly in the logits on this backend, which can flip a near-tie. When that flip lands early, everything after it is a different draw, not a rewording. Measured on the three prompts above, each arm deterministic across repeats:

• On the CSV task the int4-head output was byte-identical to plain decoding in every run, and both scored 10/10 on an executed test harness.
• On the prose task all three arms produced different text.
• On the ISO-8601 task the int4-head answer was a different and worse program: 107 lines against 188, scoring 3/8 against plain's 5/8 on an executed harness, with a runtime crash on three of the eight cases.

One prompt in three, then, cost real quality — reproducibly. If you serve this head, gate it on your own task, and do not assume "exact verification" means "same answer". If you need bit-exact reproducibility against a non-speculative run, serve without the head.