Repository: EleutherAI/lm-evaluation-harness
Issue: Adding chat template causing extreme accuracy drop
Root cause: Documentation Deficiency

![Image](https://github.com/user-attachments/assets/8f8345f4-5e65-4194-97af-5f4015b1c5c0)

I am currently working on applying emotional templates to see if the results will differ and the result is quite unreal.
I run a test on "bigbench_abstract_narrative_understanding_multiple_choice" and only change doc_to_text to apply template.
Am I doing anything wrong or this just a suprise outcome.

script I am using
```
lm_eval --model hf \
        --model_args pretrained=$model_path,parallelize=True,trust_remote_code=True,do_sample=True \
        --tasks $task \
        --write_out \
        --device cuda:6 \
        --output_path $result_path \
        --log_samples \
        --verbosity DEBUG
```
The only change is apply here.

```
tag: bigbench_multiple_choice_a
dataset_path: hails/bigbench
dataset_kwargs:
  # num_shots: 0 # TODO: num of shots for `bigbench` HF dataset should be controlled through this, not through the typical methods
  # subtask_name: null
output_type: multiple_choice
test_split: default
doc_to_text: inputs + "You better be sure"
doc_to_target: "{{multiple_choice_targets.index(targets[0])}}"
doc_to_choice: "{{multiple_choice_targets}}"
metric_list:
  - metric: acc
  # TODO: brier score and other metrics
metadata:
  version: 1.0
```
My machine is a server 
```
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.124.06             Driver Version: 570.124.06     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3090        Off |   00000000:1D:00.0 Off |                  N/A |
| 33%   58C    P0            117W /  350W |    5514MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 3090        Off |   00000000:1F:00.0 Off |                  N/A |
| 30%   30C    P8             22W /  350W |       4MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA GeForce RTX 3090        Off |   00000000:20:00.0 Off |                  N/A |
| 30%   28C    P8             18W /  350W |       4MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA GeForce RTX 3090        Off |   00000000:21:00.0 Off |                  N/A |
| 30%   28C    P8             20W /  350W |       4MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA GeForce RTX 3090        Off |   00000000:22:00.0 Off |                  N/A |
| 30%   29C    P8             27W /  350W |       4MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA GeForce RTX 3090        Off |   00000000:23:00.0 Off |                  N/A |
| 30%   27C    P8             19W /  350W |       4MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA GeForce RTX 3090        Off |   00000000:24:00.0 Off |                  N/A |
| 30%   31C    P8             22W /  350W |       4MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
+-----------------------------------------------------------------------------------------+
```

Thx for reading till here!

Hi! It's parsing `inputs` as a literal string. Use:
```yaml
doc_to_text: "{{inputs}} You better be sure"
```
you can also add `--write_out` to print out a sample of the prompt when running!

![Image](https://github.com/user-attachments/assets/6ca7fa2f-3887-4fd1-be3c-e6d933b331ca)
Thanks a lot! 
I tried with the method and got the new result showing a slight lower score compare with the without template.
Which is surprisingly.