zhiminy/EvalEng huggingface dataset:
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: How to set model precision when evaluating and what's the default precision? Such as "fp16" or "loat_in_8bit"
Root cause: Documentation Deficiency

How to set model precision when evaluating? Such as "fp16" or "loat_in_8bit"
And what's the default precision?
It seems to be "fp16"? Because around 14GB GPU memory is occupied when LLaVA-1.5-7B is loaded in the evaluation process.
![image](https://github.com/user-attachments/assets/ba165343-9859-433c-9aaa-7b0a57bd5e6f)


The default `dtype` is `auto` (this may vary depending on the specific model; you can check the model code).

You have the option to pass the `dtype` argument to `model_args` when executing evaluation (`float32` or `float16`; these are torch_dtype).

Alternatively, you can directly modify the code within the `model_name.py` file to utilize `load_in_8bit` or `load_in_4bit`, similar to [this](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/lmms_eval/models/llava_hf.py#L98) or [this](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/lmms_eval/models/phi3v.py#L52C23-L52C43). 

Simply add `load_in_8bit=True`:

```python
self.model = AutoModelForCausalLM.from_pretrained(...,...,..., load_in_8bit=True,...)
```
quantized models from hf don't need extra args like load_in_4bit
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [new model] qwen2-vl support
Root cause: Resource Mishandling

Any plan to support the latest Qwen2-VL model evaluation?

I think I may take a look into this later. Maybe I will try to setup a quick image evaluation model for Qwen2-VL this weekend but maybe the result won't be exactly align for this version. Possibly later will add in video evaluation too and check the results.

For anyone that is willing to help, you are also welcome to raise a PR.

merged with pr https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/268 👍 

Thanks for your work! But I found OOM in Qwen2-VL-7B inference for batch-size1 on 40GB GPU. Is there any possible problem? By the way I can do it on cogvlm and llava under the same conditions

> Thanks for your work! But I found OOM in Qwen2-VL-7B inference for batch-size1 on 40GB GPU. Is there any possible problem? By the way I can do it on cogvlm and llava under the same conditions

I met the same issue on 48G L20.

> Thanks for your work! But I found OOM in Qwen2-VL-7B inference for batch-size1 on 40GB GPU. Is there any possible problem? By the way I can do it on cogvlm and llava under the same conditions

```
# max_pixels: int = 12845056,
   max_pixels: int = 3211264
```
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [Question] Does the collator reflect the vision token well?
Root cause: Architectural Constraint

Before you open an issue, please check if a similar issue already exists or has been closed before.

### When you open an issue, please be sure to include the following

- [x] A descriptive title: [xxx] XXXX
- [x] A detailed description

Thank you for your contributions!


#### Title and content are the same
Does the collator reflect the vision token well?

`lm eval harness` only focuses on text tokens, which allows the collator to effectively detect Out-of-Memory (OOM) errors related to text processing.
However, in `lmms-eval`, the handling of vision tokens varies across different models.
And it's unclear whether the collator currently takes into account the memory requirements associated with vision tokens.

In my experience, I encountered OOM errors on sample 76 out of 1500 while using a single GPU (A6000).
This suggests a potential discrepancy between the collator's re-ordering and the actual memory usage ranking when dealing with vision tokens.
It would be beneficial to clarify how the collator manages vision tokens and whether it accurately reflects their memory footprint in the context of `lmms-eval`.


The collator does not count the visual tokens as visual tokens are counted in different way in different models so it is hard to make a unified collator across different models.

At first glance, it seems like it would be possible to utilize the shape of the image to order reqs.

Yeah it is possible, but since most of our model are inferencing using single batch so we haven't looked deeply into it
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Performance about cuda version
Root cause: Environment Incompatibility

I find cuda affects performance significantly.

The cuda is 12.0, the performance on mme is as follow (very low):
<img width="808" alt="image" src="https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/33897496/effacc40-79ce-47c9-8b58-ded495a92851">

Then cuda is 12.2, the performance on mme is as follow:
<img width="764" alt="image" src="https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/33897496/80a6bf60-7cba-4f09-9b2b-b2aacb9c14b3">


If I only have a machine with cuda 12.0, and I can't upgrade the Cuda version due to gpu driver. How can I reproduce the performance?

Hi @xmu-xiaoma666 , maybe you can try using torch version 2.0.1 and cuda 11.7 version which is used in llava1.5
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Batch GPT is not working
Root cause: Validation Gap

Hi,
Thanks in advance for all the hard work and this amazing project.
I am trying to run `batch_gpt` for a dataset of mine (works on HF models)
But I am getting errors:
1. First it seems the returned batches are binary and not text so you need to extract them before converting to JSON:
```
batch_results = self.retrieve_batch_results(batch_status.output_file_id).text.split('\n')[:-1]
res = [json.loads(result)["response"]["body"]["choices"][0]["message"]["content"] for result in batch_results]
```
After I managed to get the data I got an error:
```
lmms-eval/lmms_eval/filters/selection.py", line 16, in <lambda>
    return map(lambda r: r[0], resps)
    IndexError: list index out of range
```

so maybe my solution was not good enough

Any suggestions?

Yes, actually this is a trial and wish to finish later, it's just not working. 

You’re making a very good point. It seems that what retrieve_batch_results() is returning is an object, so it’s not immediately usable in the desired data format. It looks like some adjustments need to be made.

Separately, in my experience, some requests in a batch GPT request can fail. In such cases, the size of the benchmark dataset and the length of the returned response may not match.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Bug on Qwen2-VL
Root cause: Validation Gap

When running the code below
`python3 -m lmms_eval \
    --model=qwen2_vl \
    --model_args pretrained="Qwen/Qwen2-VL-2B-Instruct",device_map=cuda \
    --tasks=mmstar,chartqa \
    --batch_size=1 \
    --log_samples \
    --log_samples_suffix=qwen2_vl \
    --output_path="./logs/"`

I have encoutered the bug like

> TypeError: Can't instantiate abstract class Qwen2_VL without an implementation for abstract method 'generate_until_multi_round'


LongVA is also affected:
```
TypeError: Can't instantiate abstract class LongVA with abstract method generate_until_multi_round                                                                                                                                        
```

cc. https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/281
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Bug on Qwen2-VL over videomme
Root cause: External Dependency Breakage

It seems like qwen_vl_utils will occupy excessive memory when pre-process, until it get killed by system.

Traceback before it get killed:
```
  File "/mnt/workspace/lmms-eval-main/lmms_eval/models/qwen2_vl.py", line 228, in generate_until
    image_inputs, video_inputs = process_vision_info(messages)
  File "/mnt/workspace/anaconda3/envs/qwen2vl/lib/python3.10/site-packages/qwen_vl_utils/vision_process.py", line 219, in process_vision_info
    video_inputs.append(fetch_video(vision_info))
  File "/mnt/workspace/anaconda3/envs/qwen2vl/lib/python3.10/site-packages/qwen_vl_utils/vision_process.py", line 125, in fetch_video
    video, audio, info = io.read_video(
  File "/mnt/workspace/anaconda3/envs/qwen2vl/lib/python3.10/site-packages/torchvision/io/video.py", line 323, in read_video
    vframes_list = [frame.to_rgb().to_ndarray() for frame in video_frames]
  File "/mnt/workspace/anaconda3/envs/qwen2vl/lib/python3.10/site-packages/torchvision/io/video.py", line 323, in <listcomp>
    vframes_list = [frame.to_rgb().to_ndarray() for frame in video_frames]
KeyboardInterrupt
Model Responding:   0%|                                                                                                                                                                                  | 0/2700 [02:36<?, ?it/s]
```

Hi thanks for raising this issue, can you help to confirm how many frames would be in this list in your case?

`vframes_list `

I think a temporary solution is to put image list and let the code to inside this part in `qwen_vl_utils`

https://github.com/kq-chen/qwen-vl-utils/blob/fdefae03a8dfee0ec11274b77437db8951eb67fd/src/qwen_vl_utils/vision_process.py#L192C1-L201C22
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Results of LLaVA OneVision-7B on llava-interleave-bench are different from the ones on the paper
Root cause: Validation Gap

Hello,

I’ve been testing the LLaVA OneVision-7B model on the supported datasets and comparing the results with those reported in the official paper. For the datasets I’ve tested so far, the results are consistent with the paper. However, when evaluating the llava-interleave-bench dataset, I obtained a metric of 57.98, which is notably lower than the 64.2 reported in the paper.

I wanted to confirm if there might be a different set of images in the dataset, or if others have encountered similar discrepancies in their results.

Any insights would be greatly appreciated.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Try on qwen2vl-72b, out of memory
Root cause: Resource Mishandling

when I use 8*A100 to test Qwen2VL-72b model, I use 
` accelerate launch --num_processes=8 -m lmms_eval --model qwen2_vl  --model_args pretrained="Qwen/Qwen2-VL-72B-Instruct"   --tasks xxx --batch_size 1 --log_samples --log_samples_suffix qwen2_vl_72b --output_path ./logs/`
but I meet this issue:
![image](https://github.com/user-attachments/assets/9662b1d0-dea7-4497-8b09-9a441d8cc42f)


Can you try this?

```bash
accelerate launch --num_processes=1 -m lmms_eval --model qwen2_vl  --model_args pretrained="Qwen/Qwen2-VL-72B-Instruct",device_map=auto   --tasks xxx --batch_size 1 --log_samples --log_samples_suffix qwen2_vl_72b --output_path ./logs/
```

I have the same issue, with other models as well (InternVL2-76B), the device_map argument didn't help for me
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Calculate metrics after saving output
Root cause: Documentation Deficiency

Hello,

I'm working on an offline machine, and for datasets that require an internet connection to calculate metrics, I thought a simple solution would be to pass the --log_samples True argument and then calculate the metrics separately on another online machine. However, I'm having trouble understanding how to run the evaluation directly using the saved file.

Could anyone provide some guidance, please?

A simple solution is the first turn on the `--predict_only` and the use the `from_log` model

A simple documentation can refer to our blog here https://lmms-lab.framer.ai/blog/embracing-video-evaluations-with-lmms-eval, but we change the html format so the format of the code snippet is kind of messed up ......

from_log doesn't work for the latest version, any fix?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Local dataasets config didn't work
Root cause: Configuration Error

I have using this script downloaded dataset to a specific path:

```
import os
from datasets import load_dataset

os.environ['HF_HOME'] = '/bigdata/hub'

dataset = load_dataset("lmms-lab/CMMMU")
dataset = load_dataset("lmms-lab/MMMU")
dataset = load_dataset("lmms-lab/MMBench_CN")
dataset = load_dataset("lmms-lab/MMBench", 'en')

print(next(dataset))

```

and the result:

```
(base) ➜  hub ll datasets                                                                         
total 12K
drwxr-xr-x 2 root root 8.0K Mar 25 00:07 downloads
drwxr-xr-x 3 root root   28 Mar 24 23:38 lmms-lab___cmmmu
drwxr-xr-x 3 root root   23 Mar 25 00:07 lmms-lab___mm_bench
drwxr-xr-x 3 root root   28 Mar 25 00:04 lmms-lab___mm_bench_cn
drwxr-xr-x 3 root root   28 Mar 24 23:47 lmms-lab___mmmu
```

but when run eval I set envoriemnt to: /bigdata/hub for HF_HOME,

it still try load:

ConnectionError: Couldn't reach 'lmms-lab/CMMMU' on the Hub (ProxyError)

My server can not connect to internet, any one knows why/
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Evaluate llava-onevision-qwen2-0.5b-ov on videochatgpt has the image.size error.
Root cause: Validation Gap

I want to evaluate llava-onevision-qwen2-0.5b-ov on videochatgpt has the image.size error.
What should I do to fix it, I want to evaluate the llava-onevision on video data. Thank you.

Code:

accelerate launch --num_processes=1 -m lmms_eval \
  --model llava \
  --model_args pretrained="lmms-lab/llava-onevision-qwen2-0.5b-ov,device_map=auto"  \
  --tasks videochatgpt  --batch_size 1 --log_samples --log_samples_suffix reproduce --output_path ./logs/


Error:

File "/lmms-eval/lmms_eval/models/llava.py", line 334, in generate_until
    image_tensor = process_images(flattened_visuals, self._image_processor, self._config)
  File "/data1/trinh/code/ViLa/video/LLaVA-NeXT/llava/mm_utils.py", line 323, in process_images
    image = process_anyres_image(image, image_processor, model_cfg.image_grid_pinpoints)
  File "/LLaVA-NeXT/llava/mm_utils.py", line 275, in process_anyres_image
    best_resolution = select_best_resolution(image.size, possible_resolutions)
AttributeError: 'str' object has no attribute 'size'

  


for llava-onevision-qwen2 set --model llava_onevision instead of   --model llava
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Bug: MVBench-video 
Root cause: Unimplemented Feature Gap

Hi,

MVBench appears to be looking for videos that are not created: mvbench_video/star/Charades_segment/*.mp4

It appears that you are missing some code to split Charades into segments.

Best,
Orr

Hi, current MVBench still has some issues on the caching file. The last fix from the PR #280 is not complete. I have contacted the author and they said they will try to fix this with a new version recently

Hi, I think this bug has been fixed in recent PR #306
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: New Task Guide
GitHub labels: help wanted

discussion

Hi everyone, thanks for using lmms-eval. 

We are aware of there are challenges in integrating new tasks for new users in `lmms-eval` and `lm-evaluation-harness`.

It's welcome that you can share your issues and questions to us, so we can better assist you to add new tasks.

Also you can raise your modification suggestions on our document (sorry for not making it more clear). 

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/docs/task_guide.md
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Not Support LLava 1.6or llava-Next?
Root cause: Documentation Deficiency

it seems this git repo depends on LLava Repo, but the LLava repo not support LLava 1.6 Eval right row? How to eval this model?(such as multi image patch feature in LLava-Next/1.6)

Hi, we currently support LLaVA 1.6 model for eval. To do so, you just need to clone the newest LLaVA repo (which already updated for 1.6) and `pip install -e .` for dependency. Then you can simply pass `model_args pretrained="liuhaotian/llava-v1.6-mistral-7b"` to do evaluation on LLaVA-1.6.

> Hi, we currently support LLaVA 1.6 model for eval. To do so, you just need to clone the newest LLaVA repo (which already updated for 1.6) and `pip install -e .` for dependency. Then you can simply pass `model_args pretrained="liuhaotian/llava-v1.6-mistral-7b"` to do evaluation on LLaVA-1.6.

Are we support for this https://huggingface.co/liuhaotian/llava-v1.6-34b ?

yes it's supported. You could see README.MD We updated the scripts for running 34B model with correct 'conv_template=mistral_direct'
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [New Model] Llama-3.2-11B-Vision-Instruct and Molmo
Root cause: Unimplemented Feature Gap

Any plan to support the latest Llama-3.2-11B-Vision-Instruct and Molmo? 

Seems we already have llama-3.2-vision? @kcz358 

I have an early version in the branch dev/llama_vision, but that isn't complete so I haven't create a PR yet. This version likely still have lots of problems in it. Once I finished developed, I will create a PR.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [Language Tasks]Could i use llava_hf.py and Llava(Multimodal model) to run language tasks?
Root cause: Unimplemented Feature Gap

Could i use llava_hf.py and Llava(Multimodal model) to run language tasks?
I tried gsm8k and ifeval,both of them have mistakes. For example, for gsm8k, it says that the "sampler" for fewshot was not defined(because self.config.fewshot_config is not None:).But i didn't modify the yaml, and the origin yaml of gsm8k do not have "fewshot_config"


Hi @Violettttee , I perform some minimal tries on some language tasks and my current conclusion is that sadly the only model that support text evaluation is `llava_onevision` and it only supports some text tasks such as `mmlu` and `hellaswag` based on my testing. Because this is not a key feature in our project, text evaluation has always been on a low priority and is not stable during version updates since we won't test it really frequently. We only implement a few tasks and `llava_onevision` to handle only text tasks. 

Because we currently still don't support few shot testing, the text task that requires few shot might also be unstable to use.

One way to test your LMM, for example llava, is first load the model, and then save the model's language model. This model is likely to be a CausalLM for huggingface. Then, you can use this save model and use [`lm_eval`](https://github.com/EleutherAI/lm-evaluation-harness) to test your model in the most correct way.

I have the same issue. 
The origin yaml of gsm8k do not have "fewshot_config"
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: huggingface_hub.errors.LocalTokenNotFoundError: Token is required (`token=True`), but no token found. You need to provide a token or be logged in to Hugging Face with `huggingface-cli login` or `huggingface_hub.login`. See https://huggingface.co/settings/tokens.
Root cause: Documentation Deficiency

huggingface_hub.errors.LocalTokenNotFoundError: Token is required (`token=True`), but no token found. You need to provide a token or be logged in to Hugging Face with `huggingface-cli login` or `huggingface_hub.login`. See https://huggingface.co/settings/tokens.

How to use it for server can not connect internet?

You can try

```bash
huggingface-cli login --token $HUGGINGFACE_TOKEN --add-to-git-credential
```

or simply remove the token=True

But you might need to handle how to load hf datasets from local also. See #186 for some reference
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Llava-1.5-7B model replicates the results on the mmbench_cn_dev task
Root cause: Validation Gap

I tried to replicate the experimental results of LLaVA-1.5-7B on MMBench CN, and the final test result was only 55.756, which is slightly different from the [document's ](https://docs.google.com/spreadsheets/d/1a5ImfdKATDI8T7Cwh6eH-bEsnQFzanFraFUgcS9KHWc/edit?gid=0#gid=0)(58.3 and 57.62). What modifications do I need to make?
<img width="1026" alt="image" src="https://github.com/user-attachments/assets/b4c50036-8c75-442c-a143-6b3f7948c4de">
The operating environment is as follows:
<img width="371" alt="image" src="https://github.com/user-attachments/assets/32544d4d-48d8-4f19-81ad-0317114ba19c">
<img width="683" alt="image" src="https://github.com/user-attachments/assets/6cfb761e-898a-4542-8ed8-9ba5165af02a">
<img width="478" alt="image" src="https://github.com/user-attachments/assets/cd838055-39ea-4723-9608-8a43ff56b624">
<img width="314" alt="image" src="https://github.com/user-attachments/assets/7e807299-bb13-4a38-aac3-09680aa3b808">
<img width="308" alt="image" src="https://github.com/user-attachments/assets/9d29549e-4836-4b2a-8326-5f2929b163d7">
Run the following command:
```
accelerate launch --num_processes=1 -m lmms_eval --model llava --model_args pretrained="llava-v1.5-13b"   --tasks mmbench_cn \
    --batch_size 1 --log_samples --log_samples_suffix reproduce --output_path ./logs/
```

Aren't you using the 13B model? pretrained="llava-v1.5-13b" on the accelerate command.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Can not using multiple batch on MMBench
Root cause: Unimplemented Feature Gap

datasets/formatting/formatting.py", line 526, in _check_valid_index_key
    raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")
IndexError: Invalid key: 5772 is out of bounds for size 4329

Hi @lucasjinreal, multiple batch evaluation haven't been added in current version. We will try to add this version in future release. Please stick with batch_size=1 for now.

thank u, please pin me once it supported.

This is a long existing problem in LLaVA.

https://github.com/haotian-liu/LLaVA/issues/754

get it , the same problem
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: incorrect RANK (?) causing issues in 'gather', lmms_eval.evaluator
Root cause: Validation Gap

Hi,
I am getting this bug on lmms_eval.evaluator

Did you evaluate this in multi-node setup?



 File "/opt/hpcaas/.mounts/fs-0663e2d3c38211883/home/orrzohar/lmms-eval-v2/lmms_eval/utils.py", line 533, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/hpcaas/.mounts/fs-0663e2d3c38211883/home/orrzohar/lmms-eval-v2/lmms_eval/evaluator.py", line 243, in simple_evaluate
    results = evaluate(
              ^^^^^^^^^
  File "/opt/hpcaas/.mounts/fs-0663e2d3c38211883/home/orrzohar/lmms-eval-v2/lmms_eval/utils.py", line 533, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/hpcaas/.mounts/fs-0663e2d3c38211883/home/orrzohar/lmms-eval-v2/lmms_eval/evaluator.py", line 543, in evaluate
    torch.distributed.gather_object(
  File "/data/home/orrzohar/miniconda3/envs/lmmsevalv2/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 79, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/data/home/orrzohar/miniconda3/envs/lmmsevalv2/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 2592, in gather_object
    _validate_output_list_for_rank(my_rank, dst, object_gather_list)
  File "/data/home/orrzohar/miniconda3/envs/lmmsevalv2/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 3333, in _validate_output_list_for_rank
    raise ValueError(
ValueError: Argument ``gather_list`` must NOT be specified on non-destination ranks.

Best,
Orr

No, we haven't tested in the multi-node environment. I am pretty sure our framework would be able to use in multi-node evaluation as I do successfully implement the mid checkpoint evaluation use from api of our framework under multi-node environment. But I think if you directly call evaluation from the cli for current version would not work.

I met the same problem. Resolved according to https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/234