jason-schulz

/

KAT-Coder-V2.5-Dev-VL-oQ8e-mtp

like 2

Thanks — and the README deserved this question. It claimed "fully scripted," but only the fusion was
actually a script; the MTP graft was done inline during the build session and never saved. Both are
now in the repo under build/, and the README section is rewritten to match reality:

• build/merge_kat_vl.py — the fusion you quoted, plus the vision tower copy.
• build/graft_mtp.py — the MTP head.

They take torch + safetensors only, neither imports omlx, and each hardcodes its paths as
constants at the top — edit those and run them in order. Budget ~146 GB free disk (65 GB KAT source

• 11 GB donor shards + 70 GB output); peak RAM is ~8 GB, so your 64 GB machine is fine for the build
  as well as for serving.

On the fusion specifically, since that's the line you quoted — KAT ships every expert separately as
layers.{L}.mlp.experts.{i}.{gate,up,down}_proj.weight. mlx-vlm's qwen3_5_moe sanitize pops a
fused experts.gate_up_proj with no default, so a per-expert checkpoint dies with a KeyError on
layer 0. Per layer, stack all 256 experts:

• gate_up_proj = stack_i( cat([gate_i, up_i], dim=0) ) → [256, 1024, 2048] — gate first,
  because sanitize splits at shape[-2] // 2 and takes [:mid] as the gate half
• down_proj = stack_i( down_i ) → [256, 2048, 512]

No transposes; it's pure re-packing. Verified bit-exact by round-tripping random (layer, expert)
samples back through mlx-vlm's own split.

On oQ6e — the recipe covers it directly. oMLX won't requantize an already-quantized checkpoint, so
no new level can come from this repo; it starts from the bf16 merge either way. Once you have that
(steps 1–2, ~15 min of compute), oQ6e is step 3 with "oq_level": 6. Valid levels are 2, 2.5, 2.7,
3, 3.5, 4, 5, 6, 8.

One caveat worth knowing before you spend the disk: a preserved MTP head can lose draft acceptance
badly at lower levels — oMLX's own quantizer warns it can collapse toward ~0%. This build measures
~84% at oQ8e. Check the MTP[n] … accept= line in the server log on your oQ6e before trusting the
speedup; if it has collapsed, you're better off serving with MTP disabled, in which case you may as
well strip the mtp.* tensors and get a checkpoint that loads in stock MLX too.

If you do build one and the acceptance rate holds up, I'd be glad to hear the number — that's a data
point I don't have.

Thanks for elaborating. 😄

I saw you already uploaded https://huggingface.co/jason-schulz/KAT-Coder-V2.5-Dev-VL-oQ6e-mtp (as well as a 4-bit version), so I'll take those for now.

What I have done in my local builds is to separate the vision tower as the MTP head into their own .safetensors file each, so that they're easily transplantable (or removable). I'm still curious to try the original bf16 MTP head to see where acceptance lies (and whether any possible gain there isn't eaten up by the increased cost of evaluation...).