zhiminy/EvalEng huggingface dataset:
Repository: EvolvingLMMs-Lab/lmms-eval
Issue:  load_pretrained_model() missing 1 required positional argument: 'model_name' when evaluating VILA-7b
Root cause: Environment Incompatibility

Description
running the following command results in a missing 1 required positional argument error，the original code in vila.py is :
self._tokenizer, self._model, self._image_processor, self._max_length = load_pretrained_model(pretrained, self.model_name, device_map=self.device_map, attn_implementation=attn_implementation)
if I modify it like llava_onevision.py:
self._tokenizer, self._model, self._image_processor, self._max_length = load_pretrained_model(pretrained, None, self.model_name, device_map=self.device_map, attn_implementation=attn_implementation)
This will result in following error:
![image](https://github.com/user-attachments/assets/2fccef74-cf1b-46b4-853e-c0aa12f5f2a3)

Example Command
![image](https://github.com/user-attachments/assets/ba27d4fe-b2cf-4804-8346-6cfdce8e6efd)

Error message
![image](https://github.com/user-attachments/assets/b80568fb-7c08-42d6-a0cd-ff65ecc0589c)

environments
![image](https://github.com/user-attachments/assets/40b2f67a-fc69-4bd8-8b12-8e98edacc80a)


Hello! I'm not sure but based on my experience, maybe you can check if you have installed `deepspeed` and `flash-attn`. It seems that VILA requires these 2 packages, but they need to install separately. If these two packages are not installed, this strange error message will be triggered.

Just run

```sh
python -m pip install flash-attn --no-build-isolation
python -m pip install deepspeed
```

And as a reminder, when you run VILA, you need to install `llava` from [VILA repo](https://github.com/NVlabs/VILA) instead of [LLaVA repo](https://github.com/haotian-liu/LLaVA), which is:

```sh
git clone git@github.com:NVlabs/VILA.git
cd VILA
python -m pip install -e .
```

Hi @pufanyi , it looks like ps3 installation is also required. Please share more details regarding llava installation.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: AttributeError: 'dict' object has no attribute 'vocab_size' occurs when evaluating lava-onevision-qwen2-0.5b-ov-hf
Root cause: Interface Contract Mismatch

Description
running the following command results in a attribute error:'dict' object has no attribute 'vocab_size'.

Example Command
![image](https://github.com/user-attachments/assets/6e548abc-de25-4fbe-81d9-edee6dcd14b2)

Error message
![image](https://github.com/user-attachments/assets/e471faa3-3bb0-4fa3-b3e2-b6aa6bdf8f66)
![image](https://github.com/user-attachments/assets/e1c29e39-e316-4678-a418-a319b1e7a24c)

environments
![image](https://github.com/user-attachments/assets/9b4cc746-a52c-4c47-ab9d-8125a3c046c1)


When you use llava-hf, no need to install from llava and please use the `llava_hf` model instead

> When you use llava-hf, no need to install from llava and please use the `llava_hf` model instead

Sorry but could you please elaborate more on that? I met the similar issue here. Thanks!


install the transformers with version 4.40.0 fix my issue.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [New Model] Any plan to support PaliGemma?

Is there any plan to support [PaliGemma](https://arxiv.org/pdf/2407.07726)? Thanks!


Also [PaliGemma 2](https://developers.googleblog.com/en/introducing-paligemma-2-powerful-vision-language-models-simple-fine-tuning/)?

PaliGemma (or PaliGemma 2)  is a model that does **not** undergo instruction tuning. Given this, it is questionable whether it aligns with the intent of the benchmarks in this repository. Additionally, as of January 31, 2025, there are no publicly available weights—such as those hosted on Hugging Face—that endow the model with general QA capabilities, further supporting this point
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: A bug in lmms-eval/api/task.py and a suggestion
Root cause: Validation Gap

1.There is a bug in lmms-eval/api/task.py line.736 and 737
`if self.config.fewshot_config is not None:
            self.sampler = samplers.get_sampler(self.config.fewshot_config.get("sampler", "default") if self.config.fewshot_config else "default")(list(self.fewshot_docs()), self, rnd=random.Random(1234))`
When evaluate some benches, like GSM8K, it will not set default sampler.
The solution maybe remove the 'if', I hope I'm right.

2.Could you add more text bench, such as 'cmmlu' or 'HumanEval' ?

1. There is a lot of bugs in few shot evaluation as the multi-modality settings is way more complicated for us to do so.
2. For text benchmarks, you are recommended to use `lm-evaluation-harness`

@kcz358 Hello! I wonder whether the bug has been fixed for few shot evaluation of multi-modality models? Since I want to reproduce the result of gemma-3-4B-it and they all did few-shot evaluation.
I also wonder did @real-lurEn 's method work?

Hi @ardge-samwu , with the `doc_to_messages` api comes out I think this functionality is relatively more doable. However, we haven't implement that function yet because a lot of tasks does not have a train set or a good split for generating few shot context. You are welcome to implement that in the `api/tasks.py` for the messages like tasks.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [Info] Models that natively support audio and video

Hi there,

Are there any other models that natively support audio and video besides the Gemini API? I'm aware that VideoMME uses captions as a means of assessing the models that support video and text, but seems like there is no model besides Gemini Pro that supports audio and video natively? If there aren't any more, are there any models that you suggest benchmarking (i.e. besides the ones in VideoMME)?

Thanks,
George


Also, is there any plan to add support for models that support audio and video?

There will be such plan in the future but it will be added slowly. I think the most recent progress is that #461 that supports the first video+audio benchmark and model (Gemini)
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: I cannot use lmms-eval to evaluate Videoxl
Root cause: Validation Gap

```
Traceback (most recent call last):
File "/home/shengy/miniconda3/envs/videoxl/lib/python3.10/site-packages/lmms_eval/main.py", line 329, in cli_evaluate
results, samples = cli_evaluate_single(args)
File "/home/shengy/miniconda3/envs/videoxl/lib/python3.10/site-packages/lmms_eval/main.py", line 470, in cli_evaluate_single
results = evaluator.simple_evaluate(
File "/home/shengy/miniconda3/envs/videoxl/lib/python3.10/site-packages/lmms_eval/utils.py", line 533, in _wrapper
return fn(*args, **kwargs)
File "/home/shengy/miniconda3/envs/videoxl/lib/python3.10/site-packages/lmms_eval/evaluator.py", line 169, in simple_evaluate
lm = ModelClass.create_from_arg_string(
File "/home/shengy/miniconda3/envs/videoxl/lib/python3.10/site-packages/lmms_eval/api/model.py", line 110, in create_from_arg_string
return cls(**args, **args2)
TypeError: Can't instantiate abstract class Videoxl with abstract method generate_until_multi_round

2024-11-29 16:05:34.047 | ERROR | main:cli_evaluate:348 - Error during evaluation: Can't instantiate abstract class Videoxl with abstract method generate_until_multi_round. Please set --verbosity=DEBUG to get more information.
```
I don't know what specific modifications are needed.Could you please tell us the solution? Thank you very much

Hi, it seems that it is because you need to add an empty method `generate_until_multi_round` in your model if your model does not support multi-round evaluation, like:

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/d900496af837dcf5afe3d9efea471d624fd20588/lmms_eval/models/llava_vid.py#L528-L529

### That's great. I can't thank you enough👍😘

> Hi, it seems that it is because you need to add an empty method `generate_until_multi_round` in your model if your model does not support multi-round evaluation, like:
> 
> https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/d900496af837dcf5afe3d9efea471d624fd20588/lmms_eval/models/llava_vid.py#L528-L529
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: I can not reproduce the result JMMMU evaluation result of SakanaAI/Llama-3-EvoVLM-JP-v2
Root cause: Documentation Deficiency

May I ask what generation parameters you used for JMMMU evaluation result of SakanaAI/Llama-3-EvoVLM-JP-v2 except from 'A maximum output length is set to 1,024 and a temperature is set to 0 for all models during inference.'? The best accuracy I can get so far has only 28.48%. Thank you!
|      Groups       |Version|Filter|n-shot| Metric  |   |Value |   |Stderr|
|-------------------|-------|------|------|---------|---|-----:|---|------|
|jmmmu_all          |    N/A|none  |      |jmmmu_acc|↑  |0.2848|±  |   N/A|
| - culture_agnostic|    N/A|none  |      |jmmmu_acc|↑  |0.2889|±  |   N/A|
| - culture_specific|    N/A|none  |      |jmmmu_acc|↑  |0.2800|±  |   N/A|


I think for this you might need to contact the author of the JMMMU or EvoVLM-JP-v2 and see if they have some advise on this
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: support OpenGVLab/InternVL2_5-8B?
Root cause: Unimplemented Feature Gap

Before you open an issue, please check if a similar issue already exists or has been closed before.

### When you open an issue, please be sure to include the following

- [ ] A descriptive title: [xxx] XXXX
- [ ] A detailed description

Thank you for your contributions!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: The results of LLaVA-Video-7B-Qwen2's captioned test on VideoMME were quite different
Root cause: Documentation Deficiency

I conducted the test with subtitles on VideoMME, using the model LLaVA-Video-7B-Qwen2, and the result was much different from that of the paper. The original paper was 69.7, but our result was 66.7. Here's my run command:

accelerate launch --num_processes=3 -m lmms_eval --model llava_onevision --model_args pretrained=/root/autodl-tmp/models/LLaVA-Video-7B-Qwen2,conv_template=qwen_1_5,model_name=llava_qwen --tasks videomme_w_subtitle --batch_size 1 --log_samples --log_samples_suffix llava_onevision --output_path ./logs/
![WeChat4a211f5a9a0d9185a3ccfb2ff8e2fec9](https://github.com/user-attachments/assets/2fab6c77-4f2c-4577-8f76-94cc449c2f58)


The model should be llava_vid,

Please carefully read the guideline here: https://github.com/LLaVA-VL/LLaVA-NeXT/blob/main/docs/LLaVA_Video_1003.md

> The model should be llava_vid,
> 
> Please carefully read the guideline here: https://github.com/LLaVA-VL/LLaVA-NeXT/blob/main/docs/LLaVA_Video_1003.md

OK. We were able to reproduce the results from the original paper thanks to your advice. I would also like to ask for your guidance on the test run command for the Qwen/Qwen2-VL-7B-Instruct model on VideoMME. I used the command provided by lmms, but I did not get the results from the original paper. Is it because I set max_pixels to 256 * 28 * 28?

Max pixels could be one problem. Another problem is that we don't know the exact prompt and number of frames they are using so we are unable to exactly reproduce the result.

I tested lmms-lab/LLaVA-Video-72B-Qwen2 on videoMME using a 4-card GPU with 96GB video memory and encountered oom. I only loaded 20 weight files before encountering oom, but we did not encounter oom with the same configuration in the code of other warehouses. This is the run command:

accelerate launch --num_processes=4 -m lmms_eval --model llava_vid --model_args pretrained=/root/autodl-tmp/models/LLaVA-Video-72B-Qwen2,conv_template=qwen_1_5,max_frames_num=64,mm_spatial_pool_mode=a verage --tasks videomme_w_subtitle --batch_size 1 --log_samples --log_samples_suffix llava_vid --output_path ./logs/

Have you reproduced the videomme with subtitle result of  lmms-lab/LLaVA-Video-72B-Qwen2?
I got 72.4 instead of 76.9 in paper..
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Problem in Reproducing llava-onevision-72b-ov-chat

Hi, I used your demo(llava-onevision-72b-ov-chat) a few weeks ago, and it worked really well for my case.
However, I noticed the demo website went down. I don't know if it will be up again, so I deploy my own version of llava-onevision-72b-ov-chat using huggingface. (https://huggingface.co/lmms-lab/llava-onevision-qwen2-72b-ov-chat)
Sadly, this local version doesn't perform as well as your demo version. So I was wondering if you used some default prompt for demo? If so, where can I find it?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Bug: Llava One Vision is failing on subclasses of PIL.Image
Root cause: Validation Gap

When sending an Image that is a sub class of `PIL.Image.Image` this condition is not satisfied:
https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/d8f1f9cf7ad824b8b5315ab6fafb8768fb4345d6/lmms_eval/models/llava_onevision.py#L282
It should have `isinstance(visual[0], PIL.Image.Image)`.


Hello! Thanks for your issue. I have created a PR in #472
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: The results of InstructBLIP got zero on TextVQA
Root cause: Validation Gap

Thank all the contributors for this awesome codebase 
I have evaluated many models using the codebase, the results are corrected. But when I evaluated the InstructBLIP on TextVQA, the results of InstructBLIP got zero as follows:

<img width="862" alt="image" src="https://github.com/user-attachments/assets/954eb384-d514-418f-a206-5871359b4c6b" />

the command is below:
```
CUDA_VISIBLE_DEVICES=3 python3 -m accelerate.commands.launch \
    --num_processes 1 \
    --main_process_port 12380 \
    -m lmms_eval \
    --model instructblip \
    --model_args pretrained=Salesforce/instructblip-vicuna-13b \
    --tasks textvqa_val \
    --batch_size 1 \
    --log_samples \
    --wandb_log_samples \
    --log_samples_suffix instructblip \
    --output_path ./logs/
```

Why does this happen? 


Because the metric we use is exact_match so it does not perform any parsing. You might need to derive the true score from the log.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Test with mme always error: NCCL Error 1: unhandled cuda error (run with NCCL_DEBUG=INFO for details)
Root cause: Resource Mishandling

![image](https://github.com/user-attachments/assets/f6328c11-f689-4c76-8ffd-4b37bbee2db3)
try many times ,always with error
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Why is temperature set to 0.2 in some task YAML configurations?
Root cause: Documentation Deficiency

I noticed that in some task YAML configurations, the temperature is set to 0.2(such as mmvet), which causes do_sample to be set to True during generation. Is this the expected behavior? Does this affect the determinism of the results, or is it intentional for introducing some variability in the generated outputs?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: LLaVA-Video-72B-Qwen2
Root cause: Configuration Error

I tested lmms-lab/LLaVA-Video-72B-Qwen2 on videoMME using a 4-card GPU with 96GB video memory and encountered oom. I only loaded 20 weight files before encountering oom, but we did not encounter oom with the same configuration in the code of other warehouses. This is the run command:

accelerate launch --num_processes=4 -m lmms_eval --model llava_vid --model_args  pretrained=/root/autodl-tmp/models/LLaVA-Video-72B-Qwen2,conv_template=qwen_1_5,max_frames_num=64,mm_spatial_pool_mode=a verage --tasks videomme_w_subtitle --batch_size 1 --log_samples --log_samples_suffix llava_vid --output_path ./logs/

You need to pass in device_map=auto into model_args to shard the model and switch to num_processes to 1. For faster inference, you can utilize srt api and setup a server of llavavid

> 您需要将 device_map=auto 传入 model_args 以对模型进行分片并切换到 num_processes 到 1。为了更快地推理，您可以利用 srt api 并设置 llavavid 服务器

Useful, thank you very much!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Firstly, thank you very much for your contribution to the codebase. I am currently working on evaluating LLaVA and have encountered a few issues. I look forward to your guidance.
Root cause: Documentation Deficiency

Firstly, thank you very much for your contribution to the codebase. I am currently working on evaluating LLaVA and have encountered a few issues. I look forward to your guidance.

For LLaVA, I first performed pretraining, followed by fine-tuning with LoRA. However, during evaluation, I ran into the following problems:

1. How can I load a local dataset, such as `datasets--lmms-lab--MME`?  
2. How can I load a locally pretrained LoRA model?

Below is the command I am using:

```bash
CUDA_VISIBLE_DEVICES=6 python3 -m accelerate.commands.launch \
  --num_processes=1 \
  -m lmms_eval \
  --model llava \
  --model_args pretrained="local model" \
  --tasks mme \
  --batch_size 1 \
  --log_samples \
  --log_samples_suffix llava_v1.5_mme \
  --output_path ./logs/
``` 

Looking forward to your suggestions.

1. For local dataset, please check some of the issues in the common issues. I believe you can now solve it with HF_DATASETS_OFFLINE
2. I am not sure what is the correctly way to load lora weights. But the logic should be the same, you might need to take a lot in the model __init__ and see how to pass in the weights. Feel free to check issues such as #241 

Thank you very much for your response! Regarding the loading of offline datasets, I’d like to ask if we have any hyperparameters that can be directly configured to handle this.


This should related to the offline mode of the huggingface dataset. You should try setting the correspond env var that related to this and download the cached dataset.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: eval on multi-gpus
Root cause: Configuration Error

I use the following examples given by your readme:
`python3 -m accelerate.commands.launch \
    --num_processes=8 \
    -m lmms_eval \
    --model llava \
    --model_args pretrained="liuhaotian/llava-v1.5-7b" \
    --tasks mme \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix llava_v1.5_mme \
    --output_path ./logs/`
and I found that 8 GPUs using the same cases during inference. How can I run different examples on multiple gpus to save time? Or, is there something wrong with my command?

@masn1310 Hiii! Could you check if you have done some config in `accelerate`? Maybe you can try to reset your `accelerate`.

```sh
accelerate config default
```

Or try to directly delete your `accelerate` config file:

```sh
rm /home/data2/pufanyi/cache/accelerate/default_config.yaml
```

I also have the same problem. 
When I evaluated mmbench_en on a single GPU versus multi-GPUs, I got different results. I wonder if that's the reason of my prolem ? 

Hi did you manage to resolve it? @masn1310
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: no PerceptionTest tasks eval
Root cause: Documentation Deficiency

Hi when I pass in the arguments task PerceptionTest or perceptiontest, the library lmms-eval returns tasks not found
![Image from Facebook (2)](https://github.com/user-attachments/assets/f3c6c7ad-1a31-470e-9c8f-0c726a3b45da)


Hi! Maybe you can try `perceptiontest_val, perceptiontest_test_mc`

Thanks! it is solved
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: OOM after Postprocessing on video datasets
Root cause: Resource Mishandling

I evaluated Oryx-34B on VideoMME using 8 H100 GPUs. The "Model Responding" works well but got OOM (NCCL WARN Cuda failure 2 'out of memory') after the Postprocessing was completed. 

I tried #4 but it seems to only use 1 GPU. Is there any suggestion on this problem? I feel maybe metric calculation or gathering causes OOM and am curious if this can be optimized.

Thank you!

This issue can be solved by this [PR](https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/489).
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: OOM issue
Root cause: Documentation Deficiency

When I tried to load the llava-qwen72B model, I encountered an out-of-memory issue on the H800 graphics card. It seems that this framework assigns a complete model to each GPU. How can I slice the model so that it doesn't cause an out-of-memory problem?


Hi, you can refer to this #12 and #4
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: How can I evaluate specific MME subtasks
Root cause: Unimplemented Feature Gap

Hi!

This evaluation framework is awesome.

But for **MME benchmark**, can I get the scores of some specific subtasks, e.g. ocr, count? Will this be a future enhancement? Otherwise I can only recalculate the scores from the evaluation results...

Thanks and look forward to your suggestions.

Hi, I think currently it is hard to do so because right now the aggregation can only return a float. Possibly the only possible solution is to log the score directly during aggregation. You can check videomme for examples. But we might check if we can improve it or not

> Hi, I think currently it is hard to do so because right now the aggregation can only return a float. Possibly the only possible solution is to log the score directly during aggregation. You can check videomme for examples. But we might check if we can improve it or not

Thanks a lot! I will try this approach.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Evaluation is getting slower
Root cause: Resource Mishandling

Hi,

This framework is awesome！

But I'm having a little problem with the evaluation. I tried to use the video_llava model to evaluate on two datasets, videomme and longvideobench. The evaluation works fine, which is good. However, I noticed that the evaluation gets slower and slower as it progresses, i.e. it starts out relatively fast and then gets slower and slower. I would like to ask why is this? Is it possible that I am missing something?
The following are the command I used, running in a Python==3.10 environment with only the lmms-eval package installed:

`python3 -m accelerate.commands.launch --num_processes=2 -m lmms_eval --model video_llava --tasks longvideobench_val_v --batch_size 1 --log_samples --log_samples_suffix video_llava_lvb_v --output_path ./logs/`

![image](https://github.com/user-attachments/assets/b4501121-dc9f-4812-b5d8-a19b866deed0)

Hope to get an answer, thank you!

@Yibo-0820 Hiii! I'm not entirely sure, but could it be due to the video length? I noticed that in the data around the 250-300 range, the videos are all very long: https://huggingface.co/datasets/longvideobench/LongVideoBench/viewer/default/validation?p=5

Hi, I think if the GPU usage is low, it could be possible that the video loading is the possible reason for that, which I think is hard to solve.