zhiminy/EvalEng huggingface dataset:
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Error loading key "info" from dataset
Root cause: External Dependency Breakage

Thanks for the awesome work. While, evaluating on Refcoco, gives me a error related to `info` key not found on this: `loadRes res.dataset['info'] = copy.deepcopy(self.dataset['info'])` part of the evaluator file. I lost the full log by mistake, and it will take another 5 hours to reproduce it. Will be glad i could get some insights on this.

script:

accelerate launch --num_processes=1 --main_process_port=12346 -m lmms_eval \
    --model qwen2_5_vl \
    --model_args=pretrained=./md/unsloth/,max_pixels=12845056,interleave_visuals=False \
    --tasks refcoco \
    --batch_size 1

Hi! I successfully fixed this problem by "pip install pycocotools==2.0.7". It seems a problem related to pycocotools's version.

> For anyone running into this issue while training an object detection model with `autogluon==1.1.0`:
>
> I was getting the following error:
>
> ```python
> res.dataset['info'] = copy.deepcopy(self.dataset['info'])  
> KeyError: 'info'
> ```
>
> After some debugging, I found that the problem was caused by the version of `pycocotools`.
> The solution was to install a compatible version:
>
> ```bash
> pip install pycocotools==2.0.7
> ```
>
> This resolved the error for me. Hopefully this helps others facing the same issue!

>
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: question about custom evaluation of seedbench
Root cause: Documentation Deficiency

Hi, can i only change the seedbench.yaml file to evalute on only the seedbench image entries?
Or where should I hack into so I can achieve this?
https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/8895505b3fb087bdfc91cb5f0a1b3a6a6a0c0914/lmms_eval/tasks/seedbench/seedbench.yaml#L18


Yes I think you can comment out the rest part and only evaluate seed_image part.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Installation Error on macOS M4 due to `eva-decord` dependency
Root cause: Environment Incompatibility

while installing for development according to [here](https://github.com/EvolvingLMMs-Lab/lmms-eval), ive got this error:
```
Using Python 3.12.11 environment at: dev
  × No solution found when resolving dependencies:
  ╰─▶ Because all versions of eva-decord{sys_platform == 'darwin'} have no wheels with a matching platform tag (e.g.,
      `macosx_15_0_arm64`) and lmms-eval==0.4.0 depends on eva-decord{sys_platform == 'darwin'}, we can conclude that
      lmms-eval==0.4.0 cannot be used.
      And because only lmms-eval==0.4.0 is available and you require lmms-eval, we can conclude that your requirements
      are unsatisfiable.

      hint: Wheels are available for `eva-decord` (v0.6.1) on the following platforms: `manylinux2010_x86_64`, `win_amd64`
```

my system details:
Macbook air, M4 chip, macOS Sequoia 15.5

full terminal log:
```
mahdi@Mahdis-MacBook-Air lmms-eval % cd lmms-eval
mahdi@Mahdis-MacBook-Air lmms-eval % uv venv dev
Using CPython 3.12.11
Creating virtual environment at: dev
Activate with: source dev/bin/activate
mahdi@Mahdis-MacBook-Air lmms-eval % source dev/bin/activate
(dev) mahdi@Mahdis-MacBook-Air lmms-eval % uv pip install -e .
Using Python 3.12.11 environment at: dev
  × No solution found when resolving dependencies:
  ╰─▶ Because all versions of eva-decord{sys_platform == 'darwin'} have no wheels with a matching platform tag (e.g.,
      `macosx_15_0_arm64`) and lmms-eval==0.4.0 depends on eva-decord{sys_platform == 'darwin'}, we can conclude that
      lmms-eval==0.4.0 cannot be used.
      And because only lmms-eval==0.4.0 is available and you require lmms-eval, we can conclude that your requirements
      are unsatisfiable.

      hint: Wheels are available for `eva-decord` (v0.6.1) on the following platforms: `manylinux2010_x86_64`, `win_amd64`
```

May I know do you have a work around to solve it now? Could you try re-installing your uv environment into `python==3.11`?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: "llava_hf" is not registered
Root cause: Environment Incompatibility

When running with `--model llava_hf`, doesn't seem to be properly registered and recognized as a model type
>  Attempted to load model 'llava_hf', but no model for this name found! Supported model names: llava, llava_sglang, qwen_vl, fuyu, gpt4v, instructblip, minicpm_v

My llms_eval version is as follows:

> Name: lmms_eval
Version: 0.1.2
Summary: A framework for evaluating large multi-modality language models
Home-page: 
Author: 
Author-email: LMMMs-Lab Evaluation Team <lmms_eval@outlook.com>
License: MIT




Have you checked the AVAILABLE_MODELS dict in lmms_eval/models/__init__.py?

yes, llava_hf is listed in the init.py. I didn't modify that part of code.

You might want to print out the error when trying to import llava_hf. It is likely an ImportError. I can't reproduce this error using the main branch.

Perhaps this is due to a package import error, such as the Transformers version not matching, causing the lava_hf.py not to be properly parsed. this works for me

I encountered the same issue and resolved it by changing the version of the transformer.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Potential Bug in Frame Sampling Logic for Qwen-2.5-VL on Video Benchmarks
Root cause: Algorithmic Error

When evaluating the **Qwen-VL series** on video benchmarks such as MVBench, I observed a minor inconsistency between **Qwen-2-VL** and **Qwen-2.5-VL** regarding frame sampling behavior.

Specifically, after setting `max_num_frames`, the input video tensor of shape `[N, C, H, W]` is uniformly downsampled using:

```python
indices = np.linspace(0, total_frames - 1, self.max_num_frames, dtype=int, endpoint=True)
```

In **qwen2_vl.py**, an important step follows when the number of original frames is **smaller** than max_num_frames: 

```
indices = np.unique(indices)
```

However, in **qwen2_5_vl.py**, this uniqueness check is omitted. As a result, when `total_frames < max_num_frames`, the same frames may be sampled multiple times, leading to the total number of frames after sampling being always max_num_frames, regardless of the actual input length.

This seems like a minor bug or oversight. (Or a special design for Qwen-2.5-VL? ) Thanks for your attention and great work on this framework!

I've fixed it here

https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/789
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Video Processing with v0.4 faulty from ChatMessages
Root cause: Validation Gap

Congrats on the new release of v0.4, it looks very promising! One small bug: I've found that when launching, say vllm, on any video task, it runs into this issue where ChatMessages tries to call self.max_frame_num but it is never initialized anywhere. I'm not sure where this came from previously, but I'm assuming this is just a transfer problem. 

```
  File ".../lmms-eval/lmms_eval/protocol.py", line 117, in encode_video
    uniform_sampled_frames = np.linspace(0, total_frame_num - 1, self.max_frame_num, dtype=int)
                                                                 ^^^^^^^^^^^^^^^^^^
  File ".../vllm/lib/python3.12/site-packages/pydantic/main.py", line 991, in __getattr__
    raise AttributeError(f'{type(self).__name__!r} object has no attribute {item!r}')
AttributeError: 'ChatMessages' object has no attribute 'max_frame_num'
2025-08-01 10:23:37 | ERROR    | __main__:cli_evaluate:369 - Error during evaluation: 'ChatMessages' object has no attribute 'max_frame_num'. Please set `--verbosity=DEBUG` to get more information.
```

I've found here that it comes from this line in vllm (not sure about other models): 
https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/8e68a1805a3a6232672c921846de502ca30edbc2/lmms_eval/models/vllm.py#L261C40-L262C1

Thanks, I've fixed it

https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/783
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Outdated olympiadbench task
Root cause: Configuration Error

[olympiadbench.yaml](https://github.com/roombox-lab/lmms-eval/blob/dev/lmms_eval/tasks/olympiadbench/olympiadbench.yaml) has the following content:

```
task:
- olympiadbench_testmini
```

But olympiadbench_testmini.yaml does not exist any more. Instead olympiadbench_OE_MM_maths_en_COMP.yaml and olympiadbench_OE_MM_physics_en_COMP do exist. So olympiadbench.yaml should have the following content:

```
task:
- olympiadbench_OE_MM_maths_en_COMP
- olympiadbench_OE_MM_physics_en_COMP
```

And https://github.com/roombox-lab/lmms-eval/blob/dev/lmms_eval/tasks/olympiadbench/cn_utils.py and https://github.com/roombox-lab/lmms-eval/blob/dev/lmms_eval/tasks/olympiadbench/en_utils.py are not used by other code. So they should be deleted.

Is my suggestion correct? If yes, I am willing to submit a PR.


Yes, that’s correct. I use the `math_en_comp` and `physics_en_comp` tasks to evaluate and report reasoning scores for VL models.

Additionally, we’ve added support for MIMO tasks at `lmms_eval/tasks/olympiadbench_mimo` to ensure alignment with their reported results.

Either approach is fine, as long as you understand the task details and provide references to the specific lmms-eval tasks for reproducibility.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Wrong REAME file for aime task
Root cause: Documentation Deficiency

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/lmms_eval/tasks/aime/README.md is for gsm8k instead of aime.

Fixed by https://github.com/EvolvingLMMs-Lab/lmms-eval/commit/c48021707175b71b11039ac9af0afbe45a6136cd
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Refer to nonexistent task
Root cause: Documentation Deficiency

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/db5dc018456fea959bd3daac53585e5b53e05cb6/lmms_eval/api/task.py#L183 has:

```
    """A task represents an entire benchmark including its dataset, problems,
    answers, and evaluation methods. See BoolQ for a simple example implementation

    A `doc` can be any python object which represents one instance of evaluation.
    This is usually a dictionary e.g.
        {"question": ..., "answer": ...} or
        {"question": ..., question, answer)
    """
```

But BoolQ task does not exist. We should use `mme` since [docs/task_guide.md](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/db5dc018456fea959bd3daac53585e5b53e05cb6/docs/task_guide.md) talks about mme a lot.

Thanks for the detail inspection!

I've fixed them in recent commit: https://github.com/EvolvingLMMs-Lab/lmms-eval/commit/c48021707175b71b11039ac9af0afbe45a6136cd
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [MultiGPU Evaluation Error]: model on differencet device
Root cause: Configuration Error

```python
accelerate launch --num_machines 1 --mixed_precision no --dynamo_backend no --num_processes=4\
 -m lmms_eval --model llava   --model_args pretrained=liuhaotian/lava-v1.5-7b --tasks mme --batch_size 1 --log_samples --log_samples_suffix llava_v1.5_7b_mme --output_path ./logs/ 
```

I encounter the following errors:


05-10 21:54:46 [lmms_eval/models/llava.py:407] ERROR Error Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1! (when checking argument for argument weight in method wrapper_CUDA__cudnn_convolution) in generating
Model Responding:   2%|▏         | 47/2374 [00:04<02:54, 13.37it/s]05-10 21:54:46 [lmms_eval/models/llava.py:407] ERROR Error Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:3! (when checking argument for argument weight in method wrapper_CUDA__cudnn_convolution) in generating


Duplicate question, set `device_map=""`, refer to #67 

Thanks, that works for me and I'll close this issue
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Question about HallusionBench
Root cause: Validation Gap

I am trying to run the HallusionBench but find following failure output:
```
Postprocessing:   0%|                                                                             | 0/238 [00:00<?, ?it/s]Traceback (most recent call last):
Traceback (most recent call last):
  File "/ssd0/bmy/conda_envs/lmmeval/lib/python3.10/site-packages/tenacity/__init__.py", line 470, in __call__
    result = fn(*args, **kwargs)
  File "/hdd0/bmy/lmms-eval/lmms_eval/api/task.py", line 1481, in process_results
    return self.config.process_results(doc, results, **kwargs)
  File "/hdd0/bmy/lmms-eval/lmms_eval/tasks/hallusion_bench/evaluate_hb.py", line 38, in hb_process_results
    doc.pop("image")
  File "/ssd0/bmy/conda_envs/lmmeval/lib/python3.10/site-packages/tenacity/__init__.py", line 470, in __call__
    result = fn(*args, **kwargs)
KeyError: 'image'
```
my command:
```
accelerate launch --num_processes=4 --main_process_port 12325 -m lmms_eval \
        --model llava_hf \
        --model_args pretrained="/model/llava-1.5-7b-hf" \
        --tasks hallusion_bench_image \
        --batch_size 1 \
        --output_path "/hdd0/bmy/lmms-eval/evaluation_results/llava_v1.5" \
        --log_samples
```
Then i print all keys in the code evaluate_hb.py
```
def hb_process_results(doc, result):
    sample = doc
    print("Processing results...")
    print(doc.keys())
    doc.pop("image")
    sample["model_prediction"] = result[0]
    return {k: sample for k in metric}
```
find the print
```
dict_keys(['category', 'subcategory', 'visual_input', 'set_id', 'figure_id', 'sample_note', 'question_id', 'question', 'gt_answer_details', 'gt_answer', 'filename'])
```
not include image key, which seems strange cause [lmms-lab/Hallusionbench](https://huggingface.co/datasets/lmms-lab/HallusionBench) has this image key
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Typo in results sheet for llava-v1.6-vicuna-7b
Root cause: Documentation Deficiency

Hello, nice work on the benchmark.

I noticed a typo in the results sheet.
Cell J2 says "1.6-13B (lmms-eval)" but the details expressed in cell j3 says that it is vicuna 7B, not 13B.



Hello! We have fixed the sheet. Thank you so much!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Progress bar overflowed
Root cause: Algorithmic Error

When launching evaluation with accelerate on multiple GPUs with batch size larger than 1, the progress bar displays irregularly and overflows.

```sh
accelerate launch --num_processes=3 --main_process_port=12346  -m lmms_eval --model=huggingface --trust_remote_code --model_args=pretrained=./my_ckpt --tasks=mmmu --batch_size=32
```

It seems that the progress bar is updated batch_size times per batch.

```
Model Responding:   0%|                                                                                                                                                                                                                          | 0/119 [00:00<?, ?it/s]Keyword argument `audios` is not a valid argument for this processor and will be ignored.
Keyword argument `audios` is not a valid argument for this processor and will be ignored.
Keyword argument `audios` is not a valid argument for this processor and will be ignored.
Model Responding:   1%|█▊                                                                                                                                                                                                                | 1/119 [00:03<06:57,  3.53s/it]
Model Responding:  28%|█████████████████████████████████████████████████████████▉                                                                                                                                                       | 33/119 [00:05<00:11,  7.39it/s]
Model Responding:  55%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏                                                                                              | 65/119 [00:07<00:04, 11.52it/s]
Model Responding:  82%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎                                      | 97/119 [00:09<00:01, 12.96it/s]
Model Responding: 129it [00:10, 15.72it/s]                                                                                                                                                                                                                               
Model Responding: 161it [00:12, 15.83it/s]
Model Responding: 193it [00:14, 16.62it/s]
Model Responding: 225it [00:16, 16.97it/s]
Model Responding: 257it [00:17, 17.44it/s]
Model Responding: 289it [00:18, 20.12it/s]
Model Responding: 301it [00:36,  4.00it/s]
Model Responding: 333it [00:52,  2.89it/s]
Model Responding: 365it [01:09,  2.46it/s]
```

same problem

Me too

Maybe Try bs =1

same problem
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Qwen2.5-VL 在 MVBench测试上smart_nframes报错
Root cause: Validation Gap

smart_nframes会报错，这是因为nframes总是被设置成32， 但是评测中有的文件帧数小于32。  （例如mvbench_video/vlnqa/left/1090_frame31.mp4）

可以尝试在model args中设置fps1，看看可否解决
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: muirbench 的结果比官方低了很多是怎么回事呢
Root cause: Validation Gap

<img width="2304" height="208" alt="Image" src="https://github.com/user-attachments/assets/93814b18-d043-48fc-ace1-b003c8324a7b" />

<img width="1315" height="592" alt="Image" src="https://github.com/user-attachments/assets/803a7149-cd6b-4ab6-8a87-dd3e471d4d12" />

<img width="2294" height="192" alt="Image" src="https://github.com/user-attachments/assets/eaf0d1ce-29bd-4053-bbb2-a5250652341b" />
Qwen2.5-vl-7b-instruct也是

<img width="1178" height="103" alt="Image" src="https://github.com/user-attachments/assets/55ff8084-2c7c-4b6f-aa9b-1e758bf575f7" />
1. 开启多图interleave_visuals=True
2. 修改lmms_eval/tasks/muirbench/utils.py   (单步调试发现text 中的tag 都没了)  

`import re
def number_image_placeholders(texts) :
    count = 0
    def repl(_):
        nonlocal count
        count += 1
        return f"<image {count}>"
    return re.sub(r"<image>", repl, texts)

def muir_doc_to_text(doc, lmms_eval_specific_kwargs=None):
    question, choices = doc["question"], doc["options"]
    len_choices = len(choices)
    post_prompt = lmms_eval_specific_kwargs["post_prompt"]
    pre_prompt = lmms_eval_specific_kwargs["pre_prompt"]
    options = [chr(ord("A") + i) for i in range(len_choices)]
    choices_str = "\n".join([f"{option}. {choice}" for option, choice in zip(options, choices)])
    contexts = f"{pre_prompt}{question}\n{choices_str}{post_prompt}"
    contexts = number_image_placeholders(contexts)
    return contexts`

hello，想问你解决了吗？

> hello，想问你解决了吗？

解决了啊，楼上给出解决方式了，1,2 步骤。
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Potential Issue with MODEL_VERSION Overriding When Using Multimodal Models for LLM as Judge
Root cause: Configuration Error

While reviewing the code, I noticed a potential issue related to how the MODEL_VERSION environment variable is handled in the context of datasets that use an LLM as a judge (defaulting to OpenAI's GPT-4o).

**Current Behavior**
The judge model's version is retrieved from the MODEL_VERSION environment variable.
When the model under test is an OpenAI model, as per the examples in the documentation, the test model's name is also assigned to MODEL_VERSION.
This works fine for text-based models since the judge model requires a text model.
However, this overriding behavior does not apply well to all multimodal models, potentially leading to bugs or incompatibilities.

**Expected Behavior**
The MODEL_VERSION for the judge model should remain independent and not be overridden by the test model's name, especially when dealing with multimodal models, to ensure compatibility and avoid unexpected issues.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: 'LLM' object has no attribute 'get_metrics'
Root cause: External Dependency Breakage

When I use the vllm acceleration script in qwen2vl to migrate to qwen2.5vl and run qwen2.5vl, I encounter the following error when evaluating the data muirbench（The model can produce normal inference results）：

`
  File "Eval/lmms-eval-main/lmms_eval/models/chat/vllm.py", line 111, in generate_until
    metrics = self.get_format_metrics()
  File "/Eval/lmms-eval-main/lmms_eval/models/chat/vllm.py", line 137, in get_format_metrics
    metrics = self.client.get_metrics()
AttributeError: 'LLM' object has no attribute 'get_metrics'
`
The experimental script is as follows：
`PRETRAINED=${1:-"/huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct"}

export NCCL_DEBUG=DEBUG


python3 -m lmms_eval \
    --model vllm \
    --model_args model=$PRETRAINED,tensor_parallel_size=4 \
    --tasks muirbench \
    --batch_size 64 \
    --log_samples \
    --log_samples_suffix vllm \
    --output_path ./outputs/qwen25vl_muir \
    --limit=64`

Can you try upgrade vllm and see if the error persists? If you don't want to do so, you can also set disable_log_stats to False in model args to skip this
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: warning：The given max_pixels[12845056] exceeds limit[602112].
Root cause: Configuration Error

path：
examples/models/qwen25vl.sh

Has anyone else encountered this issue, and how should it be resolved?



can i ask what command you used for the evaluation, i cannot reproduce the error with the command in `examples/models/qwen25vl.sh`

same when test vsibench on vllm_qwen2vl.sh

The issue was traced to the max_pixels setting in the qwen_vl_utils library. I manually increased the value and that resolved the problem, although I'm not sure if it will have any substantial impact on the results.

> The issue was traced to the max_pixels setting in the qwen_vl_utils library. I manually increased the value and that resolved the problem, although I'm not sure if it will have any substantial impact on the results.

hi may i ask where how you modify the maxpixels of qwen_vl_utils

hi you can try debugging it yourself; I remember it was either in the process_vision_info function or somewhere in the file that contains the fetch_video function.

This is because that the default max pixels for qwen is larger than the max video pixels in qwen vl utils. Try to set `max_pixels` to a smaller arguments in model args can solve this warning. When testing on videos, it would be more reasonable to sample more frames but using smaller max pixels to balance between OOM and performance.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: LLaVA Benchmark

Hi! I am confused about what is the difference between `llava_bench_coco` and `llava_in_the_wild`?

Besides, when will the llava_bench(wilder) used in LLaVA-NeXT be supported?

Thank you.

Hi, you can check the difference in https://arxiv.org/pdf/2304.08485

<img width="771" alt="image" src="https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/92624596/5a1cb9be-cf6a-4080-8d31-a9472e13beb7">

For llava-wilder, we have already implemented everything in our internal development, but some works need to be done before release. But definitely you can expect it to appear in lmms-eval in the future.

Great work! Thank you for the response. Looking forward to the support of llava-wilder.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: benchmark 支持计划？

有计划支持OpenGVLab/CRPE, ByteDance/MTVQA, mistralai/MM-MT-Bench 嘛

its relatively easy to add benchmark all you need to do is following the unified interface. Yet i am wondering when will we support InternVL3.5 series?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: wandb > 0.18 error with from get_printer
Root cause: External Dependency Breakage

It seems wandb no longer supports from `wandb.sdk.lib.printer import get_printer` for versions > 0.18. This leads to errors in lmms_eval/loggers/wandb_logger.py. Any workarounds for later versions? Thank you so much!

Hi, I had the same issue. You can just use `from wandb.sdk.lib.printer import new_printer` instead of `get_printer`. It is equivalent to get_printer in terms of functioning.