Laguna XS 2.1 FP8 DFlash results on DGX Spark
#1
by seddonm1 - opened
The important finding is that FP8 must be used consistently at all three layers for DFlash to perform well:
- Target model:
poolside/Laguna-XS-2.1-FP8 - DFlash draft model:
poolside/Laguna-XS-2.1-DFlash-FP8 - KV cache:
--kv-cache-dtype fp8
Using a non-FP8 KV cache caused substantially worse DFlash performance.
Configuration:
- vLLM with FlashInfer attention
- Triton MoE backend
- Concurrency: 1
- Coding prompt, thinking disabled
- 128 generated tokens
- Three runs per context size
- Decode throughput, excluding TTFT
| Draft tokens | 4K context | 31K context | 92K context | Average |
|---|---|---|---|---|
| 4 | 69.60 | 72.21 | 57.79 | 66.53 |
| 5 | 76.54 | 75.65 | 71.26 | 74.48 |
| 6 | 81.37 | 83.44 | 76.11 | 80.31 |
| 7 | 82.05 | 90.85 | 72.24 | 81.71 |
| 8 | 86.75 | 96.20 | 85.30 | 89.42 |
| 9 | 84.33 | 78.72 | 77.38 | 80.14 |
| 10 | 90.73 | 88.36 | 74.25 | 84.45 |
| 11 | 77.79 | 70.47 | 66.70 | 71.65 |
| 12 | 79.21 | 79.20 | 67.03 | 75.15 |
| 13 | 80.45 | 69.26 | 70.42 | 73.38 |
| 14 | 83.80 | 81.05 | 61.00 | 75.28 |
| 15 | 88.81 | 83.59 | 73.26 | 81.89 |
For this workload, num_speculative_tokens: 8 was the clear winner, averaging 89.42 tok/s.