Repository: EvolvingLMMs-Lab/lmms-eval
Issue: System instruction not work
Root cause: Configuration Error

I am using lmms-eval to evaluate Qwen2.5-VL-7B on refCOCO dataset. 

I want to change the system prompt by `--system_instruction` command, but it didn't work. I checked the `DEBUG` output, the system prompt is always 'You are a helpful assistant.'

Has anyone encountered the same question? How to solve this?

Can you  try adding --apply_chat_template to your command, for example:
```
lmms_eval --model hf \
  --model_args pretrained=Qwen/Qwen2.5-VL-7B-Instruct \
  --tasks refcoco_bbox_test \
  --batch_size 1 \
  --system_instruction "Your custom instruction here" \
  --apply_chat_template
```

> Can you try adding --apply_chat_template to your command, for example:
> 
> ```
> lmms_eval --model hf \
>   --model_args pretrained=Qwen/Qwen2.5-VL-7B-Instruct \
>   --tasks refcoco_bbox_test \
>   --batch_size 1 \
>   --system_instruction "Your custom instruction here" \
>   --apply_chat_template
> ```

Hi @ngquangtrung57. I added `--apply_chat_template`:
```
accelerate launch --num_processes=2 --main_process_port=12346 -m lmms_eval \
    --model qwen2_5_vl \
    --model_args pretrained=xxx \
    --tasks refcocog_bbox_rec_test \
    --batch_size 1 \
    --log_samples \
    --system_instruction "test" \
    --apply_chat_template \
    --limit=10 \
    --verbosity=DEBUG \
    --output_path ./results 
```

But it didn't work:
```
Exploded dataset from 5023 to 9602 rows
[rank1]: Traceback (most recent call last):
[rank1]:   File "<frozen runpy>", line 198, in _run_module_as_main
[rank1]:   File "<frozen runpy>", line 88, in _run_code
[rank1]:   File "/141nfs/caoshijie2025/miniconda3/envs/transformers_141/lib/python3.11/site-packages/lmms_eval/__main__.py", line 539, in <module>
[rank1]:     cli_evaluate()
[rank1]:   File "/141nfs/caoshijie2025/miniconda3/envs/transformers_141/lib/python3.11/site-packages/lmms_eval/__main__.py", line 366, in cli_evaluate
[rank1]:     raise e
[rank1]:   File "/141nfs/caoshijie2025/miniconda3/envs/transformers_141/lib/python3.11/site-packages/lmms_eval/__main__.py", line 347, in cli_evaluate
[rank1]:     results, samples = cli_evaluate_single(args)
[rank1]:                        ^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/141nfs/caoshijie2025/miniconda3/envs/transformers_141/lib/python3.11/site-packages/lmms_eval/__main__.py", line 474, in cli_evaluate_single
[rank1]:     results = evaluator.simple_evaluate(
[rank1]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/141nfs/caoshijie2025/miniconda3/envs/transformers_141/lib/python3.11/site-packages/lmms_eval/utils.py", line 533, in _wrapper
[rank1]:     return fn(*args, **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/141nfs/caoshijie2025/miniconda3/envs/transformers_141/lib/python3.11/site-packages/lmms_eval/evaluator.py", line 259, in simple_evaluate
[rank1]:     chat_template=lm.chat_template if apply_chat_template else None,
[rank1]:                   ^^^^^^^^^^^^^^^^
[rank1]: AttributeError: 'Qwen2_5_VL' object has no attribute 'chat_template'
```


Can you try this one instead?
```
lmms-eval \
  --model qwen2_5_vl \
  --model_args pretrained=Qwen/Qwen2.5-VL-7B-Instruct,system_prompt='Your custom system instruction here' \
  --tasks refcocog_bbox_rec_test \
  --batch_size 1 \
  --log_samples \
  --output_path ./results
```

Still not working.
```
accelerate launch --num_processes=2 --main_process_port=12346 -m lmms_eval \
    --model qwen2_5_vl \
    --model_args pretrained=/Qwen/Qwen2.5-VL-7B-Instruct,system_prompt="test" \
    --tasks refcocog_bbox_rec_test \
    --batch_size 1 \
    --limit=10 \
    --log_samples \
    --verbosity=DEBUG \
    --output_path ./results 
```

DEBUG output:
```
Model Responding:  60%|███████████████████████████████████████████████████▌                                  | 3/5 [00:16<00:10,  5.21s/it]2025-09-08 07:43:28 | DEBUG    | lmms_eval.models.chat.qwen2_5_vl:generate_until:132 - Question: 
system
You are a helpful assistant.


user



Please locate the region this sentence describes in the image using bounding box with format [num, num, num, num], you should think first before answer the question. Stick to the format: <think>your think process<   hink><answer>[num, num, num, num]<nswer>: There is red colored truck in between the other trucks


assistant
```