Repository: EleutherAI/lm-evaluation-harness
Issue: test speculative decode accuracy
Root cause: Documentation Deficiency

I use lm-evaluation-harness to test vllm accuracy
1.when don't enable spec decode,I got some result below
vllm command:
CUDA_VISIBLE_DEVICES=0 python -m vllm.entrypoints.openai.api_server --model /mnt/nvme0n1/ckpt/llama/Meta-Llama-3.1-8B-Instruct --trust-remote-code --tensor-parallel-size 1 --chat-template ./examples/template_chatml.jinja --host 0.0.0.0 --port 9001

then I use api to evaluate
lm_eval --model local-completions --tasks arc_easy --model_args model=/mnt/nvme0n1/ckpt/llama/Meta-Llama-3.1-8B-Instruct,base_url=http://127.0.0.1:9001/v1/completions,num_concurrent=1,max_retries=3,tokenized_requests=False

num_concurrent=1
![image](https://github.com/user-attachments/assets/4da708a5-f2a8-425d-89e9-36eaf7ac1f8c)

num_concurrent=8
![image](https://github.com/user-attachments/assets/0782068e-b7d8-45f4-8fd2-4eaec63ae2a9)

num_concurrent=16
![image](https://github.com/user-attachments/assets/5aaf5c8d-31eb-4471-b90d-5d4171fd4667)

num_concurrent=32
![image](https://github.com/user-attachments/assets/805998b0-d10b-436d-9da6-738b1d51950f)

2.when enable spec decode,I got some result below
vllm command:
CUDA_VISIBLE_DEVICES=0 python -m vllm.entrypoints.openai.api_server --model /mnt/nvme0n1/ckpt/llama/Meta-Llama-3.1-8B-Instruct --trust-remote-code -tp 1 --speculative_model /mnt/nvme0n1/ckpt/llama/Meta-Llama-3.2-1B-Instruct --num_speculative_tokens 4 --seed 42 --use-v2-block-manager --chat-template ./examples/template_chatml.jinja --host 0.0.0.0 --port 9001

then I use api to evaluate
lm_eval --model local-completions --tasks arc_easy --model_args model=/mnt/nvme0n1/ckpt/llama/Meta-Llama-3.1-8B-Instruct,base_url=http://127.0.0.1:9001/v1/completions,num_concurrent=1,max_retries=3,tokenized_requests=False

num_concurrent=1
![image](https://github.com/user-attachments/assets/6c364a2a-1e7a-40cf-9bb0-284f5e98194b)

num_concurrent=8
![image](https://github.com/user-attachments/assets/eb38c5e9-b431-4bd3-8b1d-48211a8d9f21)

num_concurrent=16
![image](https://github.com/user-attachments/assets/4e625535-67b3-41cc-a8c1-4ba7e4befb47)

num_concurrent=32
![image](https://github.com/user-attachments/assets/ebda433e-bcee-46c3-8250-c7946de0eb05)

Has anyone done such experiments? Does vLLM's speculative decoding affect output accuracy?

@baberabb I need hlep!  thanks!!!

I use api to evaluate
lm_eval --model local-completions --tasks arc_easy --model_args model=/mnt/nvme0n1/ckpt/llama/Meta-Llama-3.1-8B-Instruct,base_url=http://127.0.0.1:9001/v1/completions,num_concurrent=1,max_retries=3,tokenized_requests=False

@haileyschoelkopf  I need hlep! thanks!!!

Hi! For loglikehood based tasks such as arc_easy, we are simply doing one forward pass of the model to get the logits so speculative decoding isn't really relevant. You should try a generation based task such as `gsm8k` or `ifeval` 

> Hi! For loglikehood based tasks such as arc_easy, we are simply doing one forward pass of the model to get the logits so speculative decoding isn't really relevant. You should try a generation based task such as `gsm8k` or `ifeval`

thanks! I use gsm8k test and get result,I have two more questions
1.my model is Meta-Llama-3.1-8B-Instruct,I got the result below
![image](https://github.com/user-attachments/assets/88dfb63c-22b9-4af3-b209-40155c443eab)
Is this result reasonable? I obtained it on a single H100 GPU.

2.when I enable sepc deocde using vllm
my test command is :
lm_eval --model local-completions --tasks gsm8k --model_args model=/mnt/nvme0n1/ckpt/llama/Meta-Llama-3.1-8B-Instruct,base_url=http://127.0.0.1:9001/v1/completions,num_concurrent=16,max_retries=3,tokenized_requests=False

I noticed that in the lm_eval command, I only specified Meta-Llama-3.1-8B-Instruct and did not specify a draft model. Does lm_eval not require specifying a draft model?    @baberabb  hope your reply, thanks!