Repository: EleutherAI/lm-evaluation-harness
Issue: [BUG] Task winogrande  does not work in 0-shot setting together with --apply_chat_template
Root cause: Validation Gap

Hello! 

I'm using `lm_eval==0.4.5` (`pip install lm_eval==0.4.5`). I want to test the **winogrande** task in **0-shot** mode on the [Qwen/Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) model (I took it as an example, as a lightweight model using chat-template) using **chat-template**. I make the launch with the command
```
lm_eval --model hf --model_args pretrained=Qwen/Qwen2.5-1.5B-Instruct,dtype=float16,device_map=auto,low_cpu_mem_usage=true --tasks winogrande --num_fewshot 0 --apply_chat_template
```
I get the following **errors**:
```
2024-10-27:18:35:07,268 INFO     [__main__.py:279] Verbosity set to INFO
2024-10-27:18:35:19,377 INFO     [__init__.py:459] The tag 'arc_ca' is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-10-27:18:35:19,416 INFO     [__init__.py:459] The tag 'arc_ca' is already registered as a group, this tag will not be registered. This may affect tasks you want to call.
2024-10-27:18:35:19,948 INFO     [__main__.py:376] Selected Tasks: ['winogrande']
2024-10-27:18:35:19,950 INFO     [evaluator.py:164] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2024-10-27:18:35:19,951 INFO     [evaluator.py:201] Initializing hf model, with arguments: {'pretrained': 'Qwen/Qwen2.5-1.5B-Instruct', 'dtype': 'float16', 'device_map': 'auto', 'low_cpu_mem_usage': True}
2024-10-27:18:35:19,957 INFO     [huggingface.py:129] Using device 'cuda'
2024-10-27:18:35:20,041 INFO     [huggingface.py:481] Using model type 'default'
2024-10-27:18:35:20,412 INFO     [huggingface.py:371] Model parallel was set to False.
2024-10-27:18:35:28,174 WARNING  [evaluator.py:270] Overwriting default num_fewshot of winogrande from None to 0
2024-10-27:18:35:28,176 INFO     [task.py:415] Building contexts for winogrande on rank 0...
  0% 0/1267 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/usr/local/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/evaluator.py", line 301, in simple_evaluate
    results = evaluate(
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/evaluator.py", line 444, in evaluate
    task.build_all_requests(
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/api/task.py", line 438, in build_all_requests
    fewshot_ctx = self.fewshot_context(
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/api/task.py", line 1089, in fewshot_context
    return chat_template(labeled_examples)
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/models/huggingface.py", line 1325, in apply_chat_template
    return self.tokenizer.apply_chat_template(
  File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py", line 1797, in apply_chat_template
    isinstance(conversation[0], (list, tuple)) or hasattr(conversation[0], "messages")
IndexError: list index out of range
```

This error can be easily reproduced in collab. You need to run the commands:
```
!pip install lm_eval==0.4.5
!lm_eval --model hf --model_args pretrained=Qwen/Qwen2.5-1.5B-Instruct,dtype=float16,device_map=auto,low_cpu_mem_usage=true --tasks winogrande --num_fewshot 0 --apply_chat_template
```

I tried using the lm_eval version from the main branch:
```
git clone https://github.com/EleutherAI/lm-evaluation-harness.git
cd lm-evaluation-harness
pip install -e .
```
I was getting the same error as I described above.

do you have any method to solve it? I also found this issue