Yet to see any gains from DFlash
Both DFlash (3.6) and DFlash 2 (3.8) are slower than MTP on the 2 machines that I've tested.
Can't say I've seen anyone get these improvements either. DFlash gains seem very not true.
on Coding?
I have tested 27B NVFP4 + MTP (fp8 kv), FP8 + MTP (=5 and =7, fp8 kv and full precision), FP8 + DFlash 2 (=7, fp8) on my DGX Spark, in hermes and pi, the best acceptance length at 100k tokens generated was 2.8 using dflash 2, followed by 2.2 using fp8+mtp=5 at full precision kv and last was NVFP4 + MTP at 1.9. Tasks were mostly agentic (set up dockers, manage services through ssh) and some coding (python, typescript).
In my setup, the gains are there. My only issue is that it uses more vram than MTP, however at gpu util 0.85 i get 5.15x max ctx with the dflash 2 =7 fp8 kv config, which is enough for my needs (as even with 8 concurrent, i've yet to get over 1M total KV in use).
on Coding?
Mixed tasks.
I have tested 27B NVFP4 + MTP (fp8 kv), FP8 + MTP (=5 and =7, fp8 kv and full precision), FP8 + DFlash 2 (=7, fp8) on my DGX Spark, in hermes and pi, the best acceptance length at 100k tokens generated was 2.8 using dflash 2, followed by 2.2 using fp8+mtp=5 at full precision kv and last was NVFP4 + MTP at 1.9. Tasks were mostly agentic (set up dockers, manage services through ssh) and some coding (python, typescript).
In my setup, the gains are there. My only issue is that it uses more vram than MTP, however at gpu util 0.85 i get 5.15x max ctx with the dflash 2 =7 fp8 kv config, which is enough for my needs (as even with 8 concurrent, i've yet to get over 1M total KV in use).
I genuinely wanted someone confirm if the performance benefit is only seen on NVIDIA hardware. Since that's the only hardware I see people reproducing the results from the authors. And if so, they really should add that information to the Model Card.
I also saw reports of people with NV GPUs that still have slow performance and memory usage explosions with DFlash as well. So this is truly puzzling to me. If anyone can provide this information in an auditable way, I'd really appreciate it.
There should be improvements on M5 Pro/Max and nothing else by Apple, as, to my understanding, to a large degree the benefit of speculative decoding is utilizing the high prefill (compute-bound) to accept/drop cheaply decoded tokens (bandwidth-bound) from a draft model. For all the amazing things in M chips, especially their bandwidth, until the M5 generation there were no dedicated tensor cores, making compute lag behind dedicated GPUs, even if the bandwidth is amazing. It's my main gripe with my M4 Max Studio.
I can't speak for AMD and Intel cards, I think some of them should also have a speedup, but given that I'm only seeing it on my Spark (NVIDIA) and in the blog they use a M5 Max, those are the 2 platforms I'm reasonably certain an improvement should be seen. I'd like for someone to provide numbers from AMD/Intel cards too as I'm curious what the situation there is.
To provide a sample image from my Spark's dashboard, for context, at 4 agents it evens out around 27-30 t/s aggregate during mixed loads. The main issue is when one is prefilling, the other 3's generation tanks until that is done. While everyone is decoding, very high numbers can be seen like in the image, which was a snapshot of when along the 4 agents, i ran 4 concurrent benchmarks (which were just decode) and in moments when all 8 were decoding, peaks like this were seen.
Both DFlash (3.6) and DFlash 2 (3.8) are slower than MTP on the 2 machines that I've tested.
Can't say I've seen anyone get these improvements either. DFlash gains seem very not true.
this is a v2 Dflash model, did you use it with sglang ? there were commits added on 8-18-26 that are required. I can see a increase vs the dspark I was running in sglang most notably in the thinking where it was the slowest.
Both DFlash (3.6) and DFlash 2 (3.8) are slower than MTP on the 2 machines that I've tested.
Can't say I've seen anyone get these improvements either. DFlash gains seem very not true.this is a v2 Dflash model, did you use it with sglang ? there were commits added on 8-18-26 that are required. I can see a increase vs the dspark I was running in sglang most notably in the thinking where it was the slowest.
this was on tp2 ada6000s running the official fp8 release
Im not sure what the community complains about mtp would never touch this on 8bit qwen 27b 3.8 dense
im happy with dflash 2 great job guys
Decode batch, #running-req: 1, #full token: 6406, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 5.55, accept rate: 0.33, cuda graph: True, gen throughput (token/s): 203.94, #queue-req: 0
[2026-08-20 00:42:58] Decode batch, #running-req: 1, #full token: 6741, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 8.57, accept rate: 0.54, cuda graph: True, gen throughput (token/s): 316.80, #queue-req: 0
[2026-08-20 00:42:59] Decode batch, #running-req: 1, #full token: 7024, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 6.72, accept rate: 0.41, cuda graph: True, gen throughput (token/s): 246.51, #queue-req: 0
[2026-08-20 00:43:00] Decode batch, #running-req: 1, #full token: 7306, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 7.15, accept rate: 0.44, cuda graph: True, gen throughput (token/s): 263.82, #queue-req: 0
[2026-08-20 00:43:01] Decode batch, #running-req: 1, #full token: 7536, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 5.70, accept rate: 0.34, cuda graph: True, gen throughput (token/s): 209.13, #queue-req: 0
[2026-08-20 00:43:02] Decode batch, #running-req: 1, #full token: 7871, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 8.68, accept rate: 0.55, cuda graph: True, gen throughput (token/s): 318.05, #queue-req: 0
[2026-08-20 00:43:03] Decode batch, #running-req: 1, #full token: 8171, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 7.25, accept rate: 0.45, cuda graph: True, gen throughput (token/s): 265.86, #queue-req: 0
[2026-08-20 00:43:04] Decode batch, #running-req: 1, #full token: 8471, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 7.53, accept rate: 0.47, cuda graph: True, gen throughput (token/s): 276.06, #queue-req: 0
[2026-08-20 00:43:05] Decode batch, #running-req: 1, #full token: 8733, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 6.78, accept rate: 0.41, cuda graph: True, gen throughput (token/s): 249.33, #queue-req: 0
[2026-08-20 00:43:06] Decode batch, #running-req: 1, #full token: 9024, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 6.92, accept rate: 0.42, cuda graph: True, gen throughput (token/s): 253.79, #queue-req: 0
[2026-08-20 00:43:07] Decode batch, #running-req: 1, #full token: 9334, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 8.07, accept rate: 0.51, cuda graph: True, gen throughput (token/s): 296.00, #queue-req: 0
[2026-08-20 00:43:08] Decode batch, #running-req: 1, #full token: 9627, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 7.03, accept rate: 0.43, cuda graph: True, gen throughput (token/s): 259.28, #queue-req: 0
[2026-08-20 00:43:10] Decode batch, #running-req: 1, #full token: 9950, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 8.40, accept rate: 0.53, cuda graph: True, gen throughput (token/s): 311.18, #queue-req: 0
[2026-08-20 00:43:11] Decode batch, #running-req: 1, #full token: 10247, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 7.42, accept rate: 0.46, cuda graph: True, gen throughput (token/s): 275.15, #queue-req: 0
[2026-08-20 00:43:12] Decode batch, #running-req: 1, #full token: 10630, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 9.50, accept rate: 0.61, cuda graph: True, gen throughput (token/s): 352.51, #queue-req: 0
[2026-08-20 00:43:13] Decode batch, #running-req: 1, #full token: 11085, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 11.35, accept rate: 0.74, cuda graph: True, gen throughput (token/s): 418.37, #queue-req: 0
[2026-08-20 00:43:14] Decode batch, #running-req: 1, #full token: 11397, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 7.58, accept rate: 0.47, cuda graph: True, gen throughput (token/s): 279.07, #queue-req: 0
[2026-08-20 00:43:15] Decode batch, #running-req: 1, #full token: 11688, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 7.50, accept rate: 0.46, cuda graph: True, gen throughput (token/s): 276.18, #queue-req: 0
[2026-08-20 00:43:16] Decode batch, #running-req: 1, #full token: 11989, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 7.30, accept rate: 0.45, cuda graph: True, gen throughput (token/s): 267.96, #queue-req: 0
[2026-08-20 00:43:17] Decode batch, #running-req: 1, #full token: 12288, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 7.53, accept rate: 0.47, cuda graph: True, gen throughput (token/s): 274.22, #queue-req: 0
[2026-08-20 00:43:18] Decode batch, #running-req: 1, #full token: 12614, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 8.12, accept rate: 0.51, cuda graph: True, gen throughput (token/s): 296.19, #queue-req: 0
[2026-08-20 00:43:19] Decode batch, #running-req: 1, #full token: 12913, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 7.45, accept rate: 0.46, cuda graph: True, gen throughput (token/s): 271.83, #queue-req: 0
[2026-08-20 00:43:20] Decode batch, #running-req: 1, #full token: 13297, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 9.85, accept rate: 0.63, cuda graph: True, gen throughput (token/s): 359.31, #queue-req: 0
[2026-08-20 00:43:22] Decode batch, #running-req: 1, #full token: 13707, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 10.18, accept rate: 0.66, cuda graph: True, gen throughput (token/s): 372.71, #queue-req: 0
[2026-08-20 00:43:23] Decode batch, #running-req: 1, #full token: 14062, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 8.68, accept rate: 0.55, cuda graph: True, gen throughput (token/s): 315.14, #queue-req: 0
[2026-08-20 00:43:24] Decode batch, #running-req: 1, #full token: 14376, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 7.88, accept rate: 0.49, cuda graph: True, gen throughput (token/s): 286.31, #queue-req: 0
accept len: 8.12,
what have you set the draft length to? have you changed any of the default sampling parameters? I'm running it at 7 and see 4s and 5s relatively often, but outside very short coding bursts, I haven't seen them in a row like this