Repository: EleutherAI/lm-evaluation-harness
Issue: mismatch result of triviaqa when using vllm and hf
Root cause: Interface Contract Mismatch

Hi, I'm trying reproduce the result of llama2-7B on triviaqa benchmark, I use `--model` two vllm and hf, but the results between them had a big gap. vllm is 0.615 and hf is 0.0701. 

When I use vllm, the command is 

```
lm_eval --model vllm \
    --model_args pretrained=../llama/models_hf/7B,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.99,data_parallel_size=1 \
    --tasks triviaqa \
    --device cuda:0 \
    --batch_size auto
```  
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/69768456/a1cd6a7d-6c4e-4e91-a095-a001b560ae20)

when I use hf, the command is 

```
lm_eval --model hf \
    --model_args pretrained=../llama/models_hf/7B \
    --tasks triviaqa \
    --num_fewshot 1 \
    --device cuda:2 \
    --batch_size 8

``` 
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/69768456/a498cf1e-872d-43dd-9ce3-e362a2803494)

is this a bug?

Though the n_shot is a little different, I think it's not the main reason.

Could just be the few-shot? You should re-run vllm with the same num_fewshot for a proper comparison. 

edit: Just realised that the HF 0-shot has a higher score than vllm 1-shot (you have mixed up the `num_fewshot` arg in the command transcriptions though). This is pretty weird! 

Might be because vllm does not have the post-generation stop sequence handling (or the custom look back one) as in HF. Also I had noticed that `vllm` and `HF` handle whitespace differently if the continuation starts with one. The HF implementation seemed to skip it. (iirc it was for ~`arc_easy`~ 'gsm8k' where the prompt ends with "Answer:" and you would expect the continuations to start with a space which `vllm` generations did but not `HF`. This might just be an artifact of the stop-sequence handling).

Based on the output prints, it looks like HF is zero-shot while vLLM is 1-shot, in which case, a discrepancy along these lines is not super surprising (though if there is a discrepancy when number of shots is equal that is a likely bug or divergence.)

> Based on the output prints, it looks like HF is zero-shot while vLLM is 1-shot, in which case, a discrepancy along these lines is not super surprising (though if there is a discrepancy when number of shots is equal that is a likely bug or divergence.)

The vllm result is lower though! But yeah, I checked and the 0-shot `vllm` result is ~0.52. Don't think it is the stop sequences though. From what I could [tell](https://github.com/vllm-project/vllm/blob/79d64c495463665cf80937a05226806785cfa583/vllm/engine/llm_engine.py#L742) they decode each token per ~generation~ subsequence? and check if it matches the string.

@learner-crapy do you have the generations/output files from these runs saved by chance?

@haileyschoelkopf I didn't save the output files, just took the screenshot of the final result. 

@haileyschoelkopf @baberabb I'm sorry for the inconsistent of the `num_fewshot` of `vllm` and `hf`, I re-run `hf` command with `num_fewshot=1`, then I get the result 0.0869, a big gap compared with vllm result(0.6155), here is the screenshot.

![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/69768456/a6339083-fff7-406a-895a-ca8a0e4d7b9f)


@learner-crapy no problem! I was looking into it!

@haileyschoelkopf I've attached the zero-shot outputs. Looks like _it is_ partly due to the stop sequences as they HF generations are pretty monosyllabic and the vllm not so much. But some generations are completely different as well.

not uploading. Will post on discord.

New zero-shot result on HF is 

```
hf (pretrained=meta-llama/Llama-2-7b-hf), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto
| Tasks  |Version|     Filter      |n-shot|  Metric   |Value |   |Stderr|
|--------|-------|-----------------|-----:|-----------|-----:|---|-----:|
|triviaqa|Yaml   |remove_whitespace|     0|exact_match|0.5247|±  |0.0037|
```