Repository: bigcode-project/bigcode-evaluation-harness
Issue: [REQUEST] Apply tokenizer chat template for HumanEvalPack
Root cause: Unimplemented Feature Gap

Add `chat` prompt type for HumanEvalPack tasks for more realistic and fair model evaluation and comparison.

## Motivation

As implied in the [docs](https://github.com/bigcode-project/bigcode-evaluation-harness/tree/main/docs#humanevalpack:~:text=These%20are%20custom%20prompt%20formats%20for%20individual%20models%20to%20align%20with%20how%20they%20were%20finetuned.) and current implementation, instruction-tuned models perform better on the tasks when prompted in a way that aligns with how they were trained. There are custom `prompts` already implemented for several models, but it is not exhaustive and requires an update for each new prompt format or change per provider. e.g. new tokens/different format used in the latest Meta Llama or IBM Granite models. 

## Related to a problem/frustration with the current features?

Yes, scores are lower when not applying the chat template.

### Benchmark results

**Model** `meta-llama/Llama-3.2-3B-Instruct`

`instruct` prompt
```json
"humanevalfixdocs-python": {
  "pass@1": 0.5182926829268293
}
```

`codellama` prompt
```json
"humanevalfixdocs-python": {
  "pass@1": 0.45121951219512196
}
```

`chat` prompt
```json
"humanevalfixdocs-python": {
  "pass@1": 0.551829268292683
}
```

**Model**: `ibm-granite/granite-3.0-8b-instruct`

`instruct` prompt
```json
"humanevalfixdocs-python": {
  "pass@1": 0.5853658536585366
}
```

`chat` prompt
```json
"humanevalfixdocs-python": {
  "pass@1": 0.628048780487804
}
```


## Is it related to something you would need for a project?

Yes, it's needed for anyone that wants to easily and fairly evaluate instruction-tuned models perform on these tasks. 

## Is it something you worked on and think could benefit the community?

Yes! Makes it much easier to evaluate instruction-tuned models out-of-the-box without needing to update the code. Users can more easily and fairly compare instruct models on these tasks. 

It could also simplify the `if/else` block in the `get_prompt` function where prompt is formatted to make the codebase simpler.

## Description

HuggingFace `AutoTokenizer` has a chat template for instruct models. Let's use that.

- Add `chat` as a value for `--prompt` 
- Pass the tokenizer to the HumanEvalPack tasks
- Apply model's chat template via tokenizer `apply_chat_template` function


```python
# get_prompt function
...
elif self.prompt == "chat":
  assert self.tokenizer.chat_template is not None, "The `chat` prompt should only be used for models that have a chat template."
  prompt = self.tokenizer.apply_chat_template(
      [{"role": "user", "content": inp}],
      tokenize=False,
      add_generation_prompt=True,
  )
  prompt += prompt_base
...
```

```sh
accelerate launch main.py tasks --humanevalfixtests-python --prompt chat --model ibm-granite/granite-3.0-8b-instruct ...
```

[Fork with proposed changes](https://github.com/timrbula/bigcode-evaluation-harness/tree/add-humanevalpack-chat-prompt)

[Diff view](https://github.com/bigcode-project/bigcode-evaluation-harness/compare/main...timrbula:bigcode-evaluation-harness:add-humanevalpack-chat-prompt)