Repository: EleutherAI/lm-evaluation-harness
Issue: Inconsistent evaluation results with Chat Template
Root cause: Configuration Error

I evaluated llama3-8b-Instruct using the gsm8k benchmark, and found some interesting phenomenons.

1. Huggingface and vllm has similar results
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/34210233/b36bb639-2a08-464b-a3e6-9a9b28d3f273)

2. If I use vllm to start an API service, and use lm-eval local-chat mode to evaluate gsm8k, resulting a different accuracy.
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/34210233/6ad16786-58a0-4fce-a7ec-ddab7ec517a6)

3. I browsed the source code of lm-eval, and found that the API use the apply chat template, while inference of huggingface and vllm mode do not use chat template (i.e. there are no special tokens)

4. I tried to use the chat template in the tokenizer and log some intermate results, could you give me some insights about it???

output of vllm in the lm-eval
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/34210233/4ab29251-e929-4beb-9429-ea7baa0e5d22)


output of vllm using chat template of llama, and this do not use system prompt:
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/34210233/d3fab152-08fe-45c0-91e1-99347f9dba36)

output of vllm using chat template of llama, and this uses system prompt "you are a helpful assistant":
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/34210233/d64ba8f3-4715-4ce6-b5a0-df37d591229b)


In the final, the accuracy of vllm with chat template significantly drop!!!
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/34210233/21e6f939-dac1-49ed-b7e2-fa6ef2c8ea57)

Are you have any idea about it?

I think the people should use more chat-template in the evaluation, since it is close to the real sceneries.


Did you figure out how to use the chat template with vllm? 

> Did you figure out how to use the chat template with vllm?

Yeah, It's simple.
```
        if (use_completion_flag):
            all_prompt_ids = tokenizer(all_prompt, add_special_tokens=False).input_ids
        else:
            # all_prompt_ids = [tokenizer.apply_chat_template(
            #     conversation=[
            #         {"role": "user", "content": prompt},
            #     ],
            #     add_generation_prompt=False,
            # ) for prompt in all_prompt]
            all_prompt_ids = [tokenizer.apply_chat_template(
                conversation=[
                    {"role": "system", "content": system_prompt},
                    {"role": "user", "content": prompt},
                ],
                add_generation_prompt=add_generation_prompt,

            ) for prompt in all_prompt]

        if (use_vllm_flag):
            sampling_params = SamplingParams(
                temperature=0.6,
                top_p=0.9,
                # top_k=5,
                repetition_penalty=1.1,
                max_tokens=1024,
                # min_tokens=20,
                stop = until,
            )
            # Create an LLM.
            llm = LLM(
                # enforce_eager=True,
                model=model_name, dtype="bfloat16",
                gpu_memory_utilization=0.45,
                max_model_len=4096,
                # tensor_parallel_size=8
            )
            # Generate texts from the prompts. The output is a list of RequestOutput objects
            # that contain the prompt, generated text, and other information.
            # 限制gpu内存使用百分比
            outputs = llm.generate(
                # all_prompt,
                prompt_token_ids=all_prompt_ids,  # 这一步之前需要使用模板
                sampling_params=sampling_params,
            )
```

Hi! After #2034 both HF and vllm should support chat templating via `--apply_chat_template` and related CLI flags!

Hi,I have a question, why is the score of Open LLM Leaderboard vastly different from yours?The Llama3-8b-Instruct's score on the [open_llm_leaderboard](https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard) is only 68.69.Do you have any evaluate details? I have check the default max_tok_len of the [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) and [LeaderBoard Config](https://huggingface.co/datasets/open-llm-leaderboard-old/details_meta-llama__Meta-Llama-3-8B-Instruct) are both 256.
<img width="1525" alt="截屏2024-09-17 16 41 50" src="https://github.com/user-attachments/assets/b313b25d-3854-4b5c-9c25-02711d89ba65">


Hi, could you share the evaluation scripts?

Looking forward to your reply.