zhiminy/EvalEng huggingface dataset:
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Seems that different MLLMs should use different transformers version
Root cause: External Dependency Breakage

For example, what transformers version should I use for llava-v1.6-mistral-7b-hf? I have tried many versions, but they all have different errors, such as 

1: 4.39.2

File "/weilai/codes/lmms-eval/LLaVA-NeXT/llava/model/builder.py", line 290, in load_pretrained_model
    if not vision_tower.is_loaded:
           ^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'is_loaded'

2: 4.42

File "/weilai/codes/lmms-eval/LLaVA-NeXT/llava/model/builder.py", line 287, in load_pretrained_model
    model.resize_token_embeddings(len(tokenizer))
  File "/opt/conda/lib/python3.11/site-packages/transformers/modeling_utils.py", line 1944, in resize_token_embeddings
    self.config.text_config.vocab_size = model_embeds.weight.shape[0]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'dict' object has no attribute 'vocab_size'

3:  4.46.2

File "/weilai/codes/lmms-eval/LLaVA-NeXT/llava/model/language_model/llava_mistral.py", line 49, in __init__
    super(MistralForCausalLM, self).__init__(config)
  File "/opt/conda/lib/python3.11/site-packages/transformers/modeling_utils.py", line 1432, in __init__
    self.generation_config = GenerationConfig.from_model_config(config) if self.can_generate() else None
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/transformers/generation/configuration_utils.py", line 1242, in from_model_config
    decoder_config_dict = decoder_config.to_dict()
                          ^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'dict' object has no attribute 'to_dict'

besides, is it a correct bash script?
```
export HF_DATASETS_OFFLINE=1

python3 -m accelerate.commands.launch \
    --num_processes=1 \
    -m lmms_eval \
    --model llava \
    --model_args pretrained="/weilai/models/llava-v1.6-mistral-7b-hf" \
    --tasks mme \
    --batch_size 16 \
    --log_samples \
    --log_samples_suffix llava_v1.5_mme \
    --output_path ./logs/
```

You can't use the weights of llava-hf on the original llava model. You should use `llava_hf` model if you are using the huggingface llava

thanks! what is the link for your mentioned "llava_hf model"?

This https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/lmms_eval/models/llava_hf.py. The weight are being converted and some of the processing logic are being written in different ways so there might be some misalignment for the results.

Thank you a lot! It finally works.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: when we can expect this support for more datasets to be completed?（Raven/NLVR2/IconQA）

Thank you for your wonderful works.
We are looking forward to supporting more datasets. Could you please inform us when we can expect this support to be completed?

1. Winoground (winoground)
2. NLVR2 (nlvr2)
3. RavenIQ-Test (raveniq)
4. IconQA (iconqa)

Waiting for reply

We do not have a clear timeline for it, and we are welcome the community support~

Hi @littleSunlxy , for iconqa, it is already supported in our pipeline

https://github.com/EvolvingLMMs-Lab/lmms-eval/tree/main/lmms_eval/tasks/iconqa
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Evaluating LLaVA OneVision 72B - memory, speedup, multinode
Root cause: Unimplemented Feature Gap

Hello,

I am trying to evaluate LLaVA OneVision 72B, but finding I need to use tensor-parallelism to fit it on memory. However, when I do, evaluating on datasets (e.g., MLVU) takes 90+hrs on 4 A100s. 

Can this be sped up using multinode, using `torchrun --nproc_per_node=1 --nnodes=64` so i can split the data between 64 nodes, each with 2-4 A100s, and that the nodes will use tensor-parallelism within the node and data-parallelism across the nodes?

Best,
Orr



I think it is not possible for now. But for mlvu, I think the main reason is that some videos take extreme long time to load.

couldn't we figure out how to do DDP between nodes and TP inside nodes?

Right now, i am seeing it taking 45hrs on H100s and 90hrs on A100s... way to long for 72b model, no?


> Right now, i am seeing it taking 45hrs on H100s and 90hrs on A100s... way to long for 72b model, no?

Yeah, and I think most of the time is the video reading time instead of the actual inference time. You can check that for a lot of time your gpu usage is low.

> couldn't we figure out how to do DDP between nodes and TP inside nodes?

I think it could be possible for using sglang srt but we haven't really tested it as we didn't test even test the multi-node case for a stable release
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: mvbench-missing videos
Root cause: Environment Incompatibility

AssertionError: video path not found 

~/.cache/huggingface/mvbench_video/sta/sta_video_segment/NBMH9_21.4_37.6.mp4


seems like some videos are missing in mvbench, making evaluation impossible.




Hi @yinanhe , can you help look into this?

@orrzohar you can find the missing video here~ https://huggingface.co/datasets/OpenGVLab/MVBench/blob/main/video/data0613.zip

How to add these video?
Can't find NBMH9_21.4_37.6.mp4 in this zip

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/80391ce3bfb5a19b32e7a19a2d9399e1378ed2dd/lmms_eval/tasks/mvbench/utils.py#L19
seems no segment data dir here .
have you forgot to segment video?

https://github.com/OpenGVLab/Ask-Anything/blob/64bb3f270e04ed309827cf1863c7efca6f3129c6/video_chat2/mvbench.ipynb#L453
https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/80391ce3bfb5a19b32e7a19a2d9399e1378ed2dd/lmms_eval/tasks/mvbench/utils.py#L19
Why data list is not same to origin mvbench's???

> https://github.com/OpenGVLab/Ask-Anything/blob/64bb3f270e04ed309827cf1863c7efca6f3129c6/video_chat2/mvbench.ipynb#L453
> 
> [lmms-eval/lmms_eval/tasks/mvbench/utils.py](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/80391ce3bfb5a19b32e7a19a2d9399e1378ed2dd/lmms_eval/tasks/mvbench/utils.py#L19)
> 
> Line 19 in [80391ce](/EvolvingLMMs-Lab/lmms-eval/commit/80391ce3bfb5a19b32e7a19a2d9399e1378ed2dd)
> 
>  "object_interaction": "star/Charades_segment", 
> 
> Why data list is not same to origin mvbench's???

Same confusion, this error is quite strange.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [Bug] --cache_requests failed to deserialize cached requests
Root cause: Interface Contract Mismatch

api.Task.build_all_requests uses this argument to caching objects to speed up.
But cacheing.cache.load_from_cache returns list of strings rather than list of Instance.
![image](https://github.com/user-attachments/assets/3b54d9ec-3e44-46fc-b86b-38828b4fa4d4)


We will look into that in the future. But I think currently this feature is not supported as we put all the doc_to_visual in the generate until to reduce the waiting time for building request. So the logic might be different to `lm-eval`
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Consider using lower bounds on dependencies & using optional dependencies for developer tools
Root cause: Architectural Constraint

Hello, thank you for creating this very nice library - it's just what I was looking for to evaluate some VLMs :)

However, when I try to integrate your lib within my own codebase, I hit a variety of `pip` resolution errors because I pin dependencies like `black` and `datasets` to specific versions.

Would it be possible to adopt the following strategy:

* Lower bounds for external libs like `datasets`, and `transformers`
* Split off the developer tools like `black` as optional dependencies ([example](https://github.com/huggingface/lighteval/blob/ce087cda9a4fb09ca7b4e6ed67c8b6c05fb44ff7/pyproject.toml#L77-L93))

This would make `lmms-eval` much user-friendly to include in external codebases 🤗 

Thank you!


Hi, I have not tested what is the lowest version of transformers that can run using our pipeline. But I think it depends on llava since they require the transformers to be `4.37.2`. Maybe you can try whether llava can run with lower version's of transformers.

Thanks! Actually, my issue is that we typically train/eval models on the latest branch of these external libs, so it is the fixed lower bounds that is causing a conflict. For instance, could deps [like this](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/70cc773aa8153e3ec58c9f5df73361b47ffe1a80/pyproject.toml#L26) be relaxed to have an `>=` instead of `==`?

I think it is possible. I remember in early development we set libraries such as `transformers >= 4.31.2`. We will try to exclude other dependency also in the future. But for now I guess you may try these kinds of combination by yourself.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Documentation for used metrics ?
Root cause: Documentation Deficiency

Hello,

I was wondering if the metrics used are documented somewhere. For example, for VideoMME, the metric appears to be accuracy (which aligns with the official paper), but the name of the metric in the output (VIDEOMME_perception) is a bit confusing.

Thanks in advance!

So some of the pre-defined metrics (commonly used) are being registered in the api/metrics.py. Other tasks such as videomme has its own logic of calculating scores. You can find the scoring logic in their own task folder. Mostly in the utils.py. For example tasks/videomme/utils.py
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Responses truncated when testing MLVU on xcomposer-2.5
Root cause: Configuration Error

Hi, 
After running the evaluation for xcomposer-2.5 on MLVU, I checked the sample json and found that in some runs of the model, the response (`"resps"`) seems to have been truncated, which led to a poor evaluation result.

Here's an example of truncated responses: (note the `"resps"` attr)
```json
{
    "doc_id": 1020,
    "doc": {
        "video_name": "order_97.mp4",
        "question": "Question: Arrange the following events from the video in the correct chronological order: (1)A person in a red coat cleans the snow off their car; (2)The trunk of the car is lifted open; (3)A person in a tan coat cleans off the front of the car; (4)A man in a white jacket starts to clear the snow off of another car.\nOptions:\n(A) 2->1->3->4\n(B) 1->2->3->4\n(C) 4->3->2->1\n(D) 3->2->1->4",
        "answer": "B",
        "task_type": "AO"
    },
    "target": "B",
    "arguments": {
        "0": "max_new_tokens",
        "1": "temperature",
        "2": "top_p",
        "3": "num_beams",
        "4": "do_sample",
        "5": "until"
    },
    "resps": [
        [
            "The best option to arrange the events from the video in chronological order is ("
        ]
    ],
    "filtered_resps": [
        "The best option to arrange the events from the video in chronological order is ("
    ],
    "doc_hash": "74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b",
    "prompt_hash": "21a5aff0777df2e38e087742552b87838dd9d7b725c62608df54136ac3f99513",
    "target_hash": "df7e70e5021544f4834bbee64a9e3789febc4be81470df629cad6ddb03320a5c",
    "mlvu_percetion_score": {
        "question_id": "Question: Arrange the following events from the video in the correct chronological order: (1)A person in a red coat cleans the snow off their car; (2)The trunk of the car is lifted open; (3)A person in a tan coat cleans off the front of the car; (4)A man in a white jacket starts to clear the snow off of another car.\nOptions:\n(A) 2->1->3->4\n(B) 1->2->3->4\n(C) 4->3->2->1\n(D) 3->2->1->4",
        "task_type": "AO",
        "pred_answer": "The best option to arrange the events from the video in chronological order is (",
        "answer": "B"
    },
    ...
```

Why would it truncate the response? Is there any arguments I can modify to change the truncation length? Please let me know.

Because by default the model should only answer a option so we set the max new tokens to a small number and assume the model can instruction follow. If you want to see the full resps, you can change the max new tokens in the yaml file of mlvu

I see. Thanks for your response.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: LLava-Next-Video doesn't work (lmms-lab/LLaVA-NeXT-Video-7B)
Root cause: External Dependency Breakage

I wanted to try `videomme` eval for a `llava-next-video` model, where the candidates are:
- `lmms-lab/LLaVA-NeXT-Video-7B`
- `lmms-lab/LLaVA-NeXT-Video-7B-DPO`

```
#!/bin/bash
export HF_TOKEN=XXX
export HF_HOME=XXX

accelerate launch --main_process_port 59000 --num_processes=8 -m lmms_eval \
  --model llava_vid \
  --model_args pretrained=lmms-lab/LLaVA-NeXT-Video-7B,max_frames_num=32,video_decode_backend="decord" \
  --tasks videomme \
  --batch_size 1 \
  --log_samples \
  --log_samples_suffix llavavid_lnv \
  --output_path ./logs/ \
  --verbosity=DEBUG
```
But the response was dummy (either empty or non-sense). See the `Answer: ` part.
<img width="832" alt="image" src="https://github.com/user-attachments/assets/cb786e10-bc37-4666-aaeb-cacf32ac1349">

I also tried the following based on the config in its model card, but it was the same.
```
  --model_args pretrained=lmms-lab/LLaVA-NeXT-Video-7B,video_decode_backend=decord,max_frames_num=32,mm_spatial_pool_mode=average,mm_newline_position=no_token,mm_resampler_location=after \
```
<img width="842" alt="image" src="https://github.com/user-attachments/assets/ac9f15b8-d9db-4d64-83d9-1fcce915d04f">

My questions would be:
- What is the correct way to use it?
- For model , should I use `llava_vid` or `llavavid`? 
- Does the HF model`llava-hf/LLaVA-NeXT-Video-7B-hf` is also supported for `videomme`?

--- 
* I checked the followings work well, so I assume the setup has no issue.
```
accelerate launch --main_process_port 59000 --num_processes=8 \
  -m lmms_eval \
  --model llava_vid \
  --model_args pretrained=lmms-lab/LLaVA-Video-7B-Qwen2,conv_template=qwen_1_5,max_frames_num=64,mm_spatial_pool_mode=average \
  --tasks videomme \
  --batch_size 1 \
  --log_samples \
  --log_samples_suffix llava_vid_7b_qen2 \
  --output_path ./logs/ \
  --verbosity=DEBUG

accelerate launch --num_processes 8 --main_process_port 12380 \
  -m lmms_eval \
  --model internvl2 \
  --model_args pretrained=OpenGVLab/InternVL2-40B,modality=video \
  --tasks vieomme \
  --batch_size 1 \
  --log_samples \
  --log_samples_suffix internvl \
  --output_path ./logs/ \
  --verbosity=DEBUG 2>&1 |tee ebug.txt
```





1. This seems like an old model and may have bugs in processing image tokens as the llava repo has changed. I recommend you checkout to some old tags such as v0.2.3 and use llavavid from here https://github.com/LLaVA-VL/LLaVA-NeXT/tree/inference and see if it can work or not. For new models, everything seems fine to use with current version.
2. Settings in your setup scripts should be good
3. I don't think the hf version can fit for llava_vid. Only hf versions of LLaVA-NeXT and LLaVA-Onevision can be used with model llava_hf
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Remove dependency on Llava repo
Root cause: Architectural Constraint

Hello, now that the LLava and LLava-Next models are integrated in `transformers`, would it be possible to remove the dependency on the Llava repo (https://github.com/haotian-liu/LLaVA)?

This would help with integrating `lmms-eval` into other codebases where the dependencies of the llava repo can conflict (e.g. because they pin `transformers` to an old version)


Hello, I think since the implementation is a bit different for llava-hf and llava, we might need to implement another model for llava-hf. Maybe we will try to include that in the next release

@kcz358 I can look into it.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Add LiveXiv dataset to lmms-eval
Root cause: Unimplemented Feature Gap

[Feature Request] - Add LiveXiv to tasks

Thank you for the incredible work you are doing in the pursuit of unified Multimodal evaluations.

Can you add [LiveXiv](https://huggingface.co/datasets/LiveXiv/LiveXiv) dataset to `lmms-eval`? (I did see you already support `live_bench` as a live dataset).

Thanks in a advance

Sure let me know how could we help?

After preview check, I found the dataset is not working with correct format (display the row correctly) on HF?

![Image](https://github.com/user-attachments/assets/369114cc-f800-4ce4-b587-df07337c6e48)

An example for reference is MMMU

https://huggingface.co/datasets/lmms-lab/MMMU/viewer/default/validation


Hi,
Thanks for your replay
I fixed the issue

![Image](https://github.com/user-attachments/assets/e4823935-c8e6-45b8-9dec-349103d04e3c)

Now all versions are in the correct format + I did check it works on lmms-eval (single version).

I want to add the dataset as a live dataset (similar to what you did with LiveBench).
How can we do that?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: How to evaluate a subset of a certain benchmark?
Root cause: Documentation Deficiency

Hi, I noticed that running the script will test all the data for a certain benchmark. How should I change the code so that I can test only a subset of the benchmark, for example, only test 100 instances? It will accelerate my code revolution.


See line 131 of `./lmms_eval/__main__.py`.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Add Support for the ScreenQA Dataset
Root cause: Unimplemented Feature Gap

Hi, all! I am writing to request support for the ScreenQA Dataset. Currently, lmms-eval supports ScreenSpot, a dataset focused on GUI Grounding. However, it lacks a dataset for GUI QA. ScreenQA would be a great choice for this purpose. Do you have any plans to support it?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: A bug when processing result in ScienceQA task
Root cause: Algorithmic Error

The result processing of the task ScienceQA has a little mistake. In line 37, `pred` has been lower, so in line 41, `pred[0].isupper()` will always be False. If the pred is like 'A.', 'B.', the `exact_match` will always be 0.

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/d2056e605ca2a186cdc3db6e7ed9f8a91fb77061/lmms_eval/tasks/scienceqa/utils.py#L34-L44
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: image bug for the latest version transformers
Root cause: Validation Gap

The lmms-eval is a great eval tools! Thanks you guys.

I used the newest version transformers, 4.46.3 and I found one bug.

Some models, such as llama_vision and phi3v, in the func generate_until, the arg images is [] if there is no image in the dataset. But it will raise a error. I tried to add "if len(images): images = None" before "inputs = self.processor(images, prompt, return_tensors="pt").to(self.model.device)". It works. 

Could you please also add it as a bug fix. This will be helpful for the higher version transformers.

Thanks.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Mistake in MLVU dataset calculation
Root cause: Algorithmic Error

H there,

Thanks for the great works! 

It appears that the calculation of M-Avg for the MLVU dataset might be incorrect. According to the original MLVU paper, the calculation should first determine the average for each subclass and then take the average of these values. However, the M-Avg calculation here does not consider the subclasses corresponding to each sample but directly calculates the number of correct answers across all samples. (correct me if I am wrong)


I think the MLVU is wrote by authors, I am here pin the author @shuyansy and to check if this issue exists. 

Thanks for your issue and also thanks for original author's contribution!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Evaluation of Qwen2VL-7B OOM
Root cause: Configuration Error

When using lmmseval for the evaluation of qwen2VL-7B, I always run out of memory. Below is my evaluation script. Is there any way to solve this problem?
<img width="1826" alt="image" src="https://github.com/user-attachments/assets/a8fcf71d-1815-4359-b2c9-3d37eb3e7b7b">
`TASK=ocrbench 
CKPT_PATH=/model_weight/models--Qwen--Qwen2-VL-7B-Instruct/snapshots/51c47430f97dd7c74aa1fa6825e68a813478097f
python3 -m accelerate.commands.launch \
    --num_processes=8 \
    -m lmms_eval \
    --model qwen2_vl \
    --model_args pretrained=$CKPT_PATH \
    --tasks $TASK \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix $TASK \
    --output_path ./Qwen2VL/eval_$TASK
~                                       `

the major issue of Qwen2-VL is the max_pixels, it's default is `16384 * 28 * 28` (28x28 each token, 16384 max tokens), you can set this to a very small number like `256 * 28 * 28` to see if OOM still exists. And then gradually increase it to a bearable threshold.

https://github.com/kq-chen/qwen-vl-utils/blob/fdefae03a8dfee0ec11274b77437db8951eb67fd/src/qwen_vl_utils/vision_process.py#L17

thanku sooooooooo much! it works!!!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Discussion how can i use vllm as backend
Root cause: Documentation Deficiency

I've read the document , but didn't find how to use vllm as infer backend.
Can someone tell me how can i use vllm as backend


I believe `lmms-eval` does not support inference with non-huggingface model yet. If you want to test out with either vllm or OpenAI API, you should use [EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness). But they don't support LLM-as-a-judge yet.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Error running evals with llava_hf models
Root cause: Configuration Error

## Description

Running the following command results in a ValueError due to an invalid device_map being passed to from_pretrained:

## Example Command

accelerate launch --num_processes=8 --main_process_port 12345 -m lmms_eval \
    --model llava_hf \
    --model_args pretrained=llava-hf/llava-1.5-7b-hf \
    --tasks seedbench \
    --batch_size 1 \
    --output_path ./logs/ \
    --log_samples
    
Error message:
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 3551, in from_pretrained
    device_map = {"": torch.device(device_map)}
RuntimeError: Device string must not be empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  ...
  File "/root/lmms-eval/lmms_eval/models/llava_hf.py", line 97, in __init__
    self._model = model_type.from_pretrained(pretrained, revision=revision, torch_dtype=dtype, device_map=self.device_map, trust_remote_code=trust_remote_code, attn_implementation=attn_implementation)
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 3553, in from_pretrained
    raise ValueError(
ValueError: When passing device_map as a string, the value needs to be a device name (e.g. cpu, cuda:0) or 'auto', 'balanced', 'balanced_low_0', 'sequential' but found .

## Proposed Fix

In lmms_eval/models/llava_hf.py, update the default value of device_map to "cuda:0", consistent with the default in llava.py.

I'm still getting familiar with the code base so its possible I've missed a better solution, particularly as it seems others haven't had this issue. Either way, it seems like the default settings being consistent between llava.py and llava_hf.py would be helpful. 

Thanks!

You can pass `device_map=cuda:0` in model_args if you are using single device to avoid this issue. The default settings is for multi-processing evaluation config

This happened to me when I ran the command without any GPUs.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: How to self define my own text prompt when using lmms-eval?
Root cause: Documentation Deficiency

I am trying to run some experiments on qwen2-VL REC task, I tried to use --system_instruction to change the text prompt, however, it was simply added before the default text prompt, instead of replace the default prompt with the newly self-defined system_instruction. Can you help to clarify to make things easier? Thank you so much!

hello, have you found where to set the prompt?