Repository: EleutherAI/lm-evaluation-harness
Issue: Evaluate Gemma with Chat Template
Root cause: Unimplemented Feature Gap

Hi, I'm trying to evaluate `gemma-it` models from Hugging Face on MMLU. When I set `--apply_chat_template --fewshot_as_multiturn`, the tokenizer will raise an error below. This is because Gemma does not support system messages: https://huggingface.co/google/gemma-2b-it/blob/main/tokenizer_config.json#L1507 

```
jinja2.exceptions.TemplateError: System role not supported
```

What is the best way to evaluate Gemma chat models? Should I use chat templating or not? Should I remove the description for each document or move the description to the first user turn instead of the system prompt? Thank you for any help!

Hi, this is blocked on #2058 , I'll get that merged ASAP!

Thanks @haileyschoelkopf for your reply! It seems that we will just remove the system instruction for Gemma-style models. I'm wondering if it hurts the performance of Gemma chat models? I guess many benchmarks provide a brief description at the beginning, but those will be removed.

how about making an option for these kinds of models to just put the system instruction at the beginning of the first user message instead, something like `--system-as-user`?

so originally something like
```
<bos><role>system
This is a system message.
<role>user
This is a user message.
```
would become:
```
<bos><role>user
This is a system message.
This is a user message.
```
or with tags so that the llm can understand that it's like a system instruction:
```
<bos><role>user
<system>This is a system message.</system>
This is a user message.
```

for now i modified the chat template in the tokenizer config to this:
```
"chat_template": "{{ bos_token }}{% if messages[0]['role'] == 'system' %}{% for message in messages[1:] %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if (message['role'] == 'assistant') %}{% set role = 'model' %}{% else %}{% set role = message['role'] %}{% endif %}{% if (loop.index0  == 0)  %}{{ '<start_of_turn>' + role + '\n' + '<system>' + messages[0]['content'] + '</system>' + '\n'+ message['content'] | trim + '<end_of_turn>\n' }}{% endif %}{% if (loop.index0  != 0)  %}{{ '<start_of_turn>' + role + '\n' + message['content'] | trim + '<end_of_turn>\n' }}{% endif %}{% endfor %}{% endif %}{% if messages[0]['role'] != 'system' %}{% for message in messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if (message['role'] == 'assistant') %}{% set role = 'model' %}{% else %}{% set role = message['role'] %}{% endif %}{{ '<start_of_turn>' + role + '\n' + message['content'] | trim + '<end_of_turn>\n' }}{% endfor %}{% endif %}{% if add_generation_prompt %}{{'<start_of_turn>model\n'}}{% endif %}"
```

it would be easier if you can add that option  `--system-as-user` in lm_eval which does something like:

```
if messages[0]['role'] == "system":
    messages[1]['content'] = "<system>" + messages[0]['content'] + "</system>\n" + messages[1]['content']
    messages = messages[1:]
```

FYI You get this error if you system->user, so you need to add an empty assistant message

   transformers/utils/chat_template_utils.py", line 419, in raise_exception
      raise jinja2.exceptions.TemplateError(message)
  jinja2.exceptions.TemplateError: Conversation roles must alternate user/assistant/user/assistant/...

@wassname can you say more?

I'm currently running the command:

```
lm_eval \
         --model vllm \
         --model_args pretrained=google/gemma-2-2b-it,dtype=auto \
         --batch_size auto \
         --tasks gpqa_main_generative_n_shot \
         --num_fewshot 8 \
         --log_samples \
         --output_path ./lm-eval-output/ \
         --gen_kwargs temperature=1.0,do_sample=True \
         --device cuda \
         --seed 0 \
         --fewshot_as_multiturn \
         --apply_chat_template
```

and I'm getting the error:

```
jinja2.exceptions.TemplateError: System role not supported
```

I'm using version `0.4.7`

I thought @baberabb 's PR should have fixed this https://github.com/EleutherAI/lm-evaluation-harness/pull/2058

@baberabb  can you please advise?

> I thought [@baberabb](https://github.com/baberabb) 's PR should have fixed this [#2058](https://github.com/EleutherAI/lm-evaluation-harness/pull/2058)
> 
> [@baberabb](https://github.com/baberabb) can you please advise?

Hi! Sorry, seems, the [vllm](https://github.com/EleutherAI/lm-evaluation-harness/blob/68c3a811715ca86101f88c0044665bb70ad447f6/lm_eval/models/vllm_causallms.py#L303) didn't get updated, only [hf](https://github.com/EleutherAI/lm-evaluation-harness/blob/68c3a811715ca86101f88c0044665bb70ad447f6/lm_eval/models/huggingface.py#L1427). I'll push a fix.

added a fix in #3076