halt95

/

Qwen3.8-Flash-Next-W4A16-Merlin

Like 2

In theory it's plausible, my main gripe with the tp4 recipe is the KV pool . would need to look further into MTP + PP compatibility work before i can say more. Will look into it and add it to the improved recipe plan i'm working on.

Thanks.

Todi got it runnin:
https://huggingface.co/todiadiyatmo/Qwen3.8-Flash-Next-W4A16-Attn8-FP8PLE/discussions/1

The best of two world. Fast decode and large KV pool.

Hey, that's great, nice work from Todi, and thanks to you for flagging it all in the first place.

On my end I'm still hardening the custom PLE/CUDA transport path and have more decode and KV work lined up after that, and currently made some good strides as well:

• KV pool: ~807K tokens, with MTP on
• Single-stream decode, holding up at depth: ~154 t/s @ 4K, ~161 @ 32K, ~171 @ 131K, ~173 @ 260K. Short no-thinking requests already touch ~195 t/s, and I think more is possible while still holding all the quality levers.
• Cold prefill: ~4,900–5,300 t/s from 10K to 260K
• Concurrency: three full 262K sessions scheduled and decoding at the same time

It's still on a custom vLLM path, and once I've run it through my quality gates I'll post the new version hopefully within a few days.

@Neiko2002 v2.0.1 is now live with 807k KV, 162-176 tok/s and 4900-5300 prefill + a lot of fixes and some other work.
Ideally i wanted to keep the quant where it was in v1 and not further reduce/quantize/reap/cut any layers while keeping the good of v1 and implementing TP2 PP2 correctly. There is a tiny bit of headroom for performance to squeeze and i have a list ready for the next update but expect diminishing returns in further versions. The Architecture itself is where a lot of issues/bugs and that's expected given it's new/preview

Please let me know if you come across any issues or have any questions, thanks!