Repository: EleutherAI/lm-evaluation-harness
Issue: apply_chat_template got 'str' object is not callable
Root cause: Interface Contract Mismatch

I got the following error when I tried to use LocalChatCompletions.

```
lm_eval --model local-chat-completions --tasks mmlu_pro_biology  --model_args base_url=http://localhost:8080/v1/chat/completions/model --write_out --log_samples --output_path ~/trtllm/lm_eval/output_820 --apply_chat_template --limit 50 --batch_size auto
Traceback (most recent call last):
  File "/opt/conda/envs/py310/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/__main__.py", line 389, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/home/ubuntu/lm-evaluation-harness/lm_eval/evaluator.py", line 295, in simple_evaluate
    chat_template=lm.chat_template(apply_chat_template),
TypeError: 'str' object is not callable
```

Hi! Could you try adding a `,model=...` (or `pretrained=...`) to the `model_args`? It's required by the tokenizer. I'll also add a warning. 

@baberabb Thank you for the response. The above command can work before. After the #2129 merged, I got the above error. 

@lxning thanks for the issue, looking into it

Hi @lxning, it should be fixed in #2235, can you check that branch and let know if that works for you?

@KonradSzafer Thank you for the fixing. I tested #2235, it works for local-chat-completion now. However, it still does not work for vllm.

@baberabb hi! I got the same problem when using vllm:
```
lm eval --model vllm \
  --model args pretrained=$(model name},tensor parallel size=$fGpUs per model}, dtype=${dtype},gpu memory \ utilization=0.9,data parallel size=0 \
  --tasks mmlu \
  --num fewshot 5 \
  --output path ${output path} \
  --fewshot as multiturn \
  --apply chat template \
  --batch size $fbatch size}
```

We are currently awaiting @haileyschoelkopf’s decision on the implementation details for the vllm and other related classes where chat templates have already been implemented. We anticipate that this will be resolved soon

lm_eval \
    --model vllm \
    --model_args pretrained={model_name},tensor_parallel_size=2,dtype=auto,gpu_memory_utilization=0.8 \
    --tasks kmmlu \
    --batch_size auto \
    --output_path results \
    --wandb_args project=lm_evaluation_harness
    
  When I run it like this, I get the same error.

Hello, thanks for raising this issue.

I did some debugging, and VLLM did not get updated with the latest changes. Here is some info:

`chat_template` is expected to be a method, not a property.
https://github.com/EleutherAI/lm-evaluation-harness/blob/a4987bba6e9e9b3f22bd3a6c1ecf0abd04fd5622/lm_eval/evaluator.py#L295

One can see the HF model implementation for an example:
https://github.com/EleutherAI/lm-evaluation-harness/blob/a4987bba6e9e9b3f22bd3a6c1ecf0abd04fd5622/lm_eval/models/huggingface.py#L441


While in Vllm, it is a property:

https://github.com/EleutherAI/lm-evaluation-harness/blob/a4987bba6e9e9b3f22bd3a6c1ecf0abd04fd5622/lm_eval/models/vllm_causallms.py#L190

Currently, as a workaround, I copy the function from the HF model to VLLMs. It seems to work. I only did minor verifications. However, I don't see any reason why it should not work since the prompt template is related to the model, not the framework.

@haileyschoelkopf: I can submit a PR if we agree on the implementation details. Here is what I suggest:

- Create a mixin class called `ChatLMTemplateMixin`, and move the function from HF to this mixin. I can see any chat-related functions can go to this mixin later on.
Using a mixing avoids complicating the inheritance tree.
 
In case we don't want to introduce additional classes, we can: 

- Put the `chat_template` function in the `TemplateLM` class


Let me know if this is possible this week as I have time to submit a PR. 


@Am1n3e A strange thing is that I evaluate the same model on `GSM8K` rather than `MMLU`, it can work:
```
lm_eval --model vllm \
    --model_args pretrained=${model_name},tensor_parallel_size=${GPUs_per_model},dtype=${dtype},gpu_memory_utilization=0.9,data_parallel_size=0 \
    --tasks gsm8k \
    --output_path ${output_path} \
    --num_fewshot 5 \
    --apply_chat_template \
    --fewshot_as_multiturn \
    --batch_size ${batch_size}
```

@AIR-hl Thanks. Can you confirm the SHA version of the lm-eval you are using?

@AIR-hl I think the error is triggered when one uses the evaluation tracker. 

Can you add the following: "--wandb_args project=test"

> @AIR-hl Thanks. Can you confirm the SHA version of the lm-eval you are using?

Sorry, I have made some confusion, the `ca3d86d6b8dea86211ec60b93c3c026ce73c9d60` version of lm-eval can work, but the latest version can't. 

Weighed in here, cc @Am1n3e : https://github.com/EleutherAI/lm-evaluation-harness/pull/2235#issuecomment-2305141292

Happy to discuss this further though!

I believe the root cause is this [line](https://github.com/EleutherAI/lm-evaluation-harness/blob/259b756afe66ae39ffbd410a13aa6b475dd6f15d/lm_eval/models/vllm_causallms.py#L191) which has not been fixed in #2235.