Dflash speculative get 0% acceptance rate
I use int4 and int4-dflash in vllm 0.26.0, but I got acceptance rate 0.0%
--tensor-parallel-size 4
--enable-auto-tool-choice --tool-call-parser poolside_v1
--reasoning-parser poolside_v1
--trusted-remote-code
--enable-chunked-prefill --enable-prefix-catching --max-num-batched-tokens 32768
--compilation-config '{"pass_config":"fuse_allreduce_rms":false}}'
--speculative-config '{"method": "dflash", "model":"poolside/Laguna-S-2.1-DFlash","num_speculative_tokens":15}' \
But acceptance rate still 0%, sometimes it goes to 0.1, 0.2%
Confirming that, after the RC2 merge of the main model (https://huggingface.co/poolside/Laguna-S-2.1-INT4/commit/3879c9fd6062d4a650853a81cf6d5746d966c092), the acceptance rate of this DFlash model quickly reduces to zero, especially as the context length increases.
Hi @tientitmu , please notice that you used the half-precision DFlash poolside/Laguna-S-2.1-DFlash for running it with INT4 base. Please use DFlash checkpoint corresponding to the base model precision you want to use. Here's a link for all of our released Laguna S checkpoints collection: https://huggingface.co/collections/poolside/laguna-s-21. This should resolve the issue you're observing.