Repository: EleutherAI/lm-evaluation-harness
Issue: How to use *chat_template* with .gguf models ?
Root cause: Unimplemented Feature Gap

I would like to ask for some help regarding testing with .gguf models

I'm having this trouble with `--apply_chat_template`

run command `lm_eval --model gguf --model_args base_url=http://127.0.1.1:8080 --tasks gsm8k --output_path result/gsm8k --log_samples --apply_chat_template --fewshot_as_multiturn`

How to implement this `tokenizer_name`?

<img width="785" alt="Image" src="https://github.com/user-attachments/assets/8d8d7a60-daa1-4b15-b207-ac77597ff75e" />



I could only specify `--chat_format` when running .gguf models. But I don't think it helps.

What do you need the chat format for? From my understanding that is not supported. For reference: https://github.com/EleutherAI/lm-evaluation-harness/pull/2856

> What do you need the chat format for? From my understanding that is not supported. For reference: [#2856](https://github.com/EleutherAI/lm-evaluation-harness/pull/2856)

Hi there, thanks for your reply.

I'm having the same issue as https://github.com/EleutherAI/lm-evaluation-harness/issues/2847

and i met the above Error when trying with the options in the response.

https://github.com/EleutherAI/lm-evaluation-harness/issues/2847#issuecomment-2757355597

Just to make sure I understand you correctly: you are starting a llama.cpp server and are running request against it? The comment uses a hf model. If so could you share the output of the llama.cpp server?

> Just to make sure I understand you correctly: you are starting a llama.cpp server and are running request against it? The comment uses a hf model. If so could you share the output of the llama.cpp server?

Yes, you are correct. My case is with `llama-server` according to this https://github.com/EleutherAI/lm-evaluation-harness/issues/1254#issuecomment-1881161560

the output of llama-server doesn't contains result related info i guss.

<img width="1200" alt="Image" src="https://github.com/user-attachments/assets/892335e2-87ab-4f1b-8c6e-5626e23c1623" />

try passing the tokenizer=<hf-repo-id> parameter to the model args

were you able to make apply_chat_template work with a gguf model?

> were you able to make apply_chat_template work with a gguf model?

nope, but mmlu seems no need of this `chat_template` stuff. you could start with this task

> try passing the tokenizer= parameter to the model args

Hi, thanks for the reply. Could you please specify which part should I do this `tokenizer=`, is it on the gguf model side ?

To the model args in the lm_eval call

The way you'd specify a tokenizer is by adding `tokenizer=<tokenizer>` to `--model_args` HOWEVER this does **_not_** work with GGUF models.

Let's break down the command line a little bit. You have `lm_eval --model <model_type> --model_args <model_args> --apply_chat_template --fewshot_as_multiturn`

Your <model_type> defines the class that lm_eval instantiates:
- `hf`-> `model = HFLM()`
- `gguf` -> `model = GGUFLM()`

Then your <model_args> are passed to the class. `HFLM()` has a `tokenizer` argument, but `GGUFLM()` does not which means it would be ignored. There are actually only two model args that GGUFLM takes: `base_url` and `max_length`. Those are the only settings you can change from the harness.

Using `--apply_chat_template` also expects the model class to have an `apply_chat_template()` method, which GGUFLM does not have, but HFLM and other backends like VLLM do.

And finally `--fewshot_as_multiturn` depends on the existence of chat templating.

-----

There are 2 solutions here.

1. Switch to a huggingface version of your model (or VLLM, or some other backend that does support tokenizers and/or chat templates)
2. Write a custom model backend that reimplements GGUFLM with additional logic to apply chat templates.

> > were you able to make apply_chat_template work with a gguf model?
> 
> nope, but mmlu seems no need of this `chat_template` stuff. you could start with this task

I could be wrong, but yes, it seems like the Lama server handles the chat templates in the way Gguf models are evaluated. 

Closing as this seems covered comprehensively by @Avelina9X's reply.