zhiminy/EvalEng huggingface dataset:
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: ValueError: When passing device_map as a string, the value needs to be a device name (e.g. cpu, cuda:0) or 'auto', 'balanced', 'balanced_low_0', 'sequential' but found .
Root cause: Configuration Error

I am trying to replicate this evaluation task following this script: https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/miscs/repr_scripts.sh

I have one Nvidia P40 GPU (CUDA 12.4)

This is the error  I am facing

`
Traceback (most recent call last):
  File "/home/hasan/LLaVA/lmms-eval/lmms_eval/__main__.py", line 199, in cli_evaluate
    results, samples = cli_evaluate_single(args)
  File "/home/hasan/LLaVA/lmms-eval/lmms_eval/__main__.py", line 283, in cli_evaluate_single
    results = evaluator.simple_evaluate(
  File "/home/hasan/LLaVA/lmms-eval/lmms_eval/utils.py", line 448, in _wrapper
    return fn(*args, **kwargs)
  File "/home/hasan/LLaVA/lmms-eval/lmms_eval/evaluator.py", line 93, in simple_evaluate
    lm = lmms_eval.api.registry.get_model(model).create_from_arg_string(
  File "/home/hasan/LLaVA/lmms-eval/lmms_eval/api/model.py", line 92, in create_from_arg_string
    return cls(**args, **args2)
  File "/home/hasan/LLaVA/lmms-eval/lmms_eval/models/llava.py", line 75, in __init__
    self._tokenizer, self._model, self._image_processor, self._max_length = load_pretrained_model(pretrained, None, get_model_name_from_path(pretrained), device_map=self.device_map, use_flash_attention_2=use_flash_attention_2)
  File "/home/hasan/LLaVA/llava/model/builder.py", line 117, in load_pretrained_model
    model = LlavaLlamaForCausalLM.from_pretrained(
  File "/home/hasan/miniconda3/envs/llava/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2972, in from_pretrained
    raise ValueError(
ValueError: When passing device_map as a string, the value needs to be a device name (e.g. cpu, cuda:0) or 'auto', 'balanced', 'balanced_low_0', 'sequential' but found .

`




Hi @hasanibnarif , if you only use one device, you should pass in `device_map=auto` or `device_map=cuda` in the model_args also
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Get stuck when evaluating on seedbench
Root cause: Resource Mishandling

Before you open an issue, please check if a similar issue already exists or has been closed before.

### When you open an issue, please be sure to include the following

- [ ] A descriptive title: Get stuck when evaluating on seedbench

The script will get stuck here for more than 10 minutes. Other benchmarks like GQA and ScienceQA work well.

<img width="1418" alt="image" src="https://github.com/user-attachments/assets/59c7d986-e37e-49d4-b39a-599dfd466936">

My scripts:
```
python3 -m accelerate.commands.launch \
    --num_processes=1 \
    -m lmms_eval \
    --model llava \
    --model_args pretrained="liuhaotian/llava-v1.5-7b" \
    --tasks "seedbench" \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix llava_v1.5_seedbench \
    --output_path ./logs/
```

Env:
```
lmms_eval                     0.2.3.dev0 
llava                         1.2.2.post1
transformers                  4.37.2
accelerate                    0.21.0
datasets                      2.16.1
```


<img width="1427" alt="image" src="https://github.com/user-attachments/assets/c4c74e26-4d16-4de1-8ac7-4651cbee63ab">


Solved by using multiple GPUs.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: VideoMME CUDA out of memory
Root cause: Configuration Error

我在8*80g a100上运行下面的代码出现了OOM问题，

accelerate launch --num_processes 8 --main_process_port 12345 -m lmms_eval \
    --model qwen2_vl  \
    --model_args pretrained="/share/huaying/pretrained_model/Qwen2-VL-7B-Instruct",device_map='auto' \
    --tasks videomme \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix qwen2_vl_7b \
    --output_path ./logs/

报错文件和代码如下
modeling_qwen2_vl.py

if pixel_values_videos is not None:
    pixel_values_videos = pixel_values_videos.type(self.visual.get_dtype())
    video_embeds = self.visual(pixel_values_videos, grid_thw=video_grid_thw)
    video_mask = (input_ids == self.config.video_token_id).unsqueeze(-1).expand_as(inputs_embeds)
    video_embeds = video_embeds.to(inputs_embeds.device, inputs_embeds.dtype)
    inputs_embeds = inputs_embeds.masked_scatter(video_mask, video_embeds)

注意到videomme最长视频有1h左右，如果根据fps采样会超过最大token限制，想问下这一部分具体应该如何平衡视频帧数以及分辨率才能达到qwen2vl report的效果呢？

The sampling frame can be controlled by passing in sampling frames num just like what we did in `llava_onevision` and other video models. I noticed that this parameter wasn't been added into the `qwen2_vl` in the last PR. I will try to add this function.

Also, another reason for the OOM for qwen2_vl is because that the max pixels is too high. I will also make it possible to pass this in model_args also recently

Also, please write the issue in english next time so that everyone can be benefit from your contribution. Thank you!

OK，thank you for your efforts～在 2024年10月22日，10:12，Kaichen Zhang - NTU ***@***.***> 写道：﻿
The sampling frame can be controlled by passing in sampling frames num just like what we did in llava_onevision and other video models. I noticed that this parameter wasn't been added into the qwen2_vl in the last PR. I will try to add this function.
Also, another reason for the OOM for qwen2_vl is because that the max pixels is too high. I will also make it possible to pass this in model_args also recently

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: RUN - ERROR - Dataset A-OKVQA is not valid, will be skipped. 
Root cause: Validation Gap


<img width="1006" alt="image" src="https://github.com/user-attachments/assets/b3928fb1-adfc-40b9-a026-16503924c9e3">
这个数据集说是支持的，而实际跑起来说不支持？
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Llava1.6 Mistral ScienceQA Performance
Root cause: Validation Gap

Hello,

If we view the chart provided, Mistral-7b achieves a score of .23/100 on ScienceQAFull.
I am able to replicate this, but this is obviously very strange - the other comparison models get scores around 73.


Hi, @daniel-z-kaplan 

Based on our logs, it is likely because the mistral-7b tends to generate an empty space and causing the exact match give a zero score. This is same for gqa and ai2d.

In our next release this will be fixed and new results will be updated.

Any updates? I met the same problem that the model ```liuhaotian/llava-v1.6-vicuna-7b``` always generates empty string.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Inconsistent evaluation result on Vinoground
Root cause: Algorithmic Error

Thanks for the quick commitment of Vinoground! However, when I use the following command to evaluate LLaVA-OV-7B, the result is lower than the original paper's.
Evaluation command:
```
# video tasks
accelerate launch --main_process_port=29501 --num_processes=8 \
-m lmms_eval \
--model llava_onevision \
--model_args pretrained=/lmms-lab/llava-onevision-qwen2-7b-ov,conv_template=qwen_1_5,model_name=llava_qwen \
--tasks vinoground \
--batch_size 1 \
--log_samples \
--log_samples_suffix llava_onevision \
--output_path ./logs/
```
Evaluation result:
```
action: text: 35.80%, video: 17.51%, group: 8.95%
all: text: 41.20%, video: 18.60%, group: 9.20%
viewpoint: text: 67.47%, video: 25.30%, group: 19.28%
cyclical: text: 32.43%, video: 16.22%, group: 8.11%
spatial: text: 26.21%, video: 14.56%, group: 3.88%
interaction: text: 34.25%, video: 15.07%, group: 4.11%
contextual: text: 34.92%, video: 17.46%, group: 7.94%
object: text: 36.25%, video: 16.88%, group: 4.38%

|  Tasks   |Version|Filter|n-shot|     Metric     |   |             Value             |   |Stderr||----------|-------|------|-----:|----------------|---|-------------------------------|---|------|
|vinoground|Yaml   |none  |     0|vinoground_score|↑  |(41.199999999999996, 18.6, 9.2)|±  |   N/A|
```
<img width="547" alt="image" src="https://github.com/user-attachments/assets/4cce07a9-7338-46a0-ba8d-e67a1d3c8794">


Hi! Thank you for your feedback! We have identified the problem and opened a pull request #354. Now we can correctly reproduce the results.
<img width="995" alt="Screenshot 2024-10-25 at 15 56 13" src="https://github.com/user-attachments/assets/c2843b1b-bb39-42d1-ae4d-bfaceb8017f3">

Thanks for your quick feedback!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Question Regarding "N/A" Output for longvideobench_test_i in lmms_eval
Root cause: Documentation Deficiency

I am currently using lmms_eval to evaluate longvideobench tasks and encountered an issue where the output for longvideobench_test_i is marked as "N/A". Interestingly, the evaluation on longvideobench_val_i works without any issues and provides normal outputs.

Could you please clarify if this "N/A" result in longvideobench_test_i is due to the longvideobench team not yet releasing the ground truth for the test set?

Thank you very much for your help!

Since they have not released the gt for test set, we can also do nothing about it but just storing the prediction result for you. The "N/A" pretty much just means it does no aggregation and thus no scores are generated. You can follow instructions [here](https://huggingface.co/spaces/longvideobench/LongVideoBench) if you want to submit your test result.

Also kindly pin @teowu if he has any update on the evaluation of the test set on longvideobench
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [Bug] Wrong results for many datasets due to a bug in the evaluation of tasks with multiple targets (affecting tasks like docvqa and metrics like anls)
Root cause: Algorithmic Error

When evaluating tasks with multiple references (such as docvqa) that use metrics that handle multiple reference (such as anls) lmms-eval is wrongly call the metric for each reference rather then passing all the references to the metric. On top of that, in that case lmms-eval overrides the values assigned by the metric to either 1.0 or 0.0. Both together result in a serious bug that affects many tasks and metrics and should resolve immediately.

can be reproduced with:
`python3 -m accelerate.commands.launch   --main_process_port=12547  --num_processes=1     -m lmms_eval --log_samples  --tasks docvqa_val --batch_size 1 --output_path ./logs_robust_init/ --model_args pretrained=lmms-lab/llava-onevision-qwen2-7b-ov --model llava_onevision --log_samples_suffix llava_ov_ref --limit 30`
This code results in anls score 0.8 while the actual result should be 0.764

# Deep dive to the reason it happens:
1. self.multiple_target is assigned with higher then 0 number:
https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/bc2899ca613e146201c292ece117e8eb73f440c3/lmms_eval/api/task.py#L766-L767
2. The metric is getting the reference one by one instead of as a list:
https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/bc2899ca613e146201c292ece117e8eb73f440c3/lmms_eval/api/task.py#L1520-L1526
3. The results of the metric are overridden by the logic of the task to binary scores:
https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/bc2899ca613e146201c292ece117e8eb73f440c3/lmms_eval/api/task.py#L1533-L1536

Based on my understanding, so we should pass in the gold as a list instead of one by one because the references for `anls` should be a list of text. And the scores should not be binarize for anls right? I think this is caused by the customized metric of anls in the metric api. The original code is probably just for matching or accuracy. Maybe we hardcode it for `anls` to fix the bug for now? @pufanyi @Luodian 

@elronbandel Does this mean that the results in papers that use Lmms-eval like LLava-onevision are overesitmated?

From my understanding, only datasets with the `doc_to_target` being a list and using `anls` as metric, such as docvqa, infovqa, will be affected and overestimated. Other dataset that using custom evaluation metric or other pre-defined metrics are still accurate.

Also, since in LLaVA-Onevision, we also provide the score for the test set, which is being evaluated on the test server. This is still gives an accurate evaluation score and comparison to other models.

I will look into this problem and fix this issue later.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [Function Behavior] loglikelihood Function Returns Loss Instead of Log Likelihood for Multiple Choice Selection
Root cause: Algorithmic Error

### Summary

The project defines two primary model functions: `loglikelihood`and `generate_until`. The `loglikelihood` function, however, returns the *loss* rather than a log-likelihood measure for the input text. In the `multiple_choice` setting, the choice with the highest log-likelihood is selected as the predicted answer.

### Problem Details

In multiple-choice tasks, the `loglikelihood` function is used to select an answer choice by choosing the one with the highest log likelihood. 
https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/325e16c0f64b68cbafde44c5daa24ef044b3b128/lmms_eval/api/task.py#L1447

However, since this function actually returns the *loss*, selecting the option with the *highest loss* may be unintended. 
https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/325e16c0f64b68cbafde44c5daa24ef044b3b128/lmms_eval/models/llava.py#L275

It seems the intended behavior should be to select the option with the *smallest loss*, reflecting the choice with the highest likelihood.


Thank you for looking into this!

Yeah, I think this could be an issue. I haven't looked into this closely because I simply write this to calculate perplexity. Maybe I should change it to argmin since most of our model return loss from their forward function
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Evaluation Performance on LLaVA-Lora
Root cause: Unimplemented Feature Gap

I'm evaluating the LLaVA-Lora version (https://huggingface.co/liuhaotian/llava-v1.5-7b-lora/discussions), but the performance seems unusually low. Do you know if this is supported in the lmms-eval pipeline?

I have never tested a Lora version of LLaVA using lmms-eval. I remember there are some discussion about this in this issue #241. For llava-v1.5, since it is already a relatively old model, I am not sure what environment or dependencies you are going to need for lmms-eval
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Bug in VDC evaluation
Root cause: Validation Gap

Thanks for the authors' contribution to the VDC task! When I try to evaluate LLaVA-OV on detailed_test, it gives the following error:
```
File "lmms-eval/lmms_eval/api/task.py", line 689, in __init__ 
    self._config = TaskConfig(**config)                        
TypeError: TaskConfig.__init__() got an unexpected keyword argument 'model_specific_prompt_kwargs'  
| ERROR    | __main__:cli_evaluate:348 - Error during evaluation: TaskCo│
nfig.__init__() got an unexpected keyword argument 'model_specific_prompt_kwargs'. Please set `-│
-verbosity=DEBUG` to get more information. 
```
Following is my evaluation script:
```
MODEL_PATH=$1
TASK=$2
accelerate launch --num_processes=1 \
-m lmms_eval \
--model llava_onevision \
--model_args pretrained=$MODEL_PATH,conv_template=qwen_1_5,model_name=llava_qwen \
--tasks $TASK \
--batch_size 1 \
--log_samples \
--log_samples_suffix llava_onevision \
--output_path ./logs/
```
where MODEL_PATH is the path to LLaVA-OV, and TASK=detailed_test.
Could you please advise if there's anything that needs fixing? @rese1f 

Hello, this has been fixed in #382 

Thanks!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Error running LLava-next
Root cause: Validation Gap

When running the script of LLava-next-72b in README, an error appear.


args_dict = {k: handle_arg_string(v) for k, v in [arg.split("=") for arg in arg_list]}
ValueError: too many values to unpack (expected 2)

When passing model_args, strictly follow the template `params1=xxx,params2=xxx`

For the script of LLaVA-NeXT-Video-32B-Qwen in README, simply substituting "，" with "," will solve this Issue.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: llava_hf inference is extremely slow
Root cause: Algorithmic Error

Hi there, 

Thank you for the benchmark. 
I have tried to this repo and get the inference of llava_hf model type and the checkpoint of llava-hf/llava-v1.6-mistral-7b-hf on infovqa, however, the inference speed is extremely slow and also it get out of memory issue even when I use A100 gpu with 80G memory. The following is my command line: 

python -m accelerate.commands.launch \
    --num_processes=8 \
    -m lmms_eval \
    --model llava_hf \
    --model_args pretrained="llava-hf/llava-v1.6-mistral-7b-hf" \
    --tasks infovqa \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix llava_v1.6_mistral_infovqa_scienceqa_docvqa \
    --output_path ./logs/

On the other hand, running the inference with llava model type with checkpoint of liuhaotian/llava-v1.6-mistral-7b on the same dataset only requires 45 mins to do the inference without any OOM issue. 

Thanks for your guidance. 



There is a fix recently in #386 to fix some issues for llava_hf that possibily relate to this issue. Can you try that branch?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Error when testing QwenVL2: 'NoneType' object is not subscriptable
Root cause: Validation Gap


Hello,

I'm experiencing an issue when trying to test QwenVL2 using `lmms-eval`. When I run the following command:

```bash
python -m lmms_eval \
    --model=qwen2_vl \
    --model_args=pretrained=Qwen/Qwen2-VL-7B-Instruct,device_map=auto \
    --tasks=mmmu_val \
    --batch_size=1 \
    --log_samples \
    --log_samples_suffix=internvl \
    --output_path="./logs/"
```

I receive this error:

```plaintext
File "/lustre/fsn1/projects/rech/fuv/uyb32si/lmmeval_v2_11_2024/lmms_eval/models/qwen2_vl.py", line 237, in generate_until
    total_frames = video_inputs[0].shape[0]
                   ~~~~~~~~~~~~^^^
TypeError: 'NoneType' object is not subscriptable
```

This error also occurs across different datasets. I'm currently using the latest version of `lmms-eval`.

Also for models other than LLava, there isn't a clear example on how to correctly run tests for each supported model. It would be incredibly helpful to have a unified file that includes example commands for each model, as this would make the tool more accessible to the community.

Thank you for your assistance!


I think this is because there is no video in the input. I will fix this issue alongside with others tomorrow.

For some run examples, you can refer to [here](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/docs/run_examples.md) and in the installation guide in the .py file for each model if the guide exists. The run_examples may not be complete and we are slowly complete all the run_examples.

Great thanks !

@kcz358  Hi Again, I’ve tried running the new version of the code, and while it executes without any errors, I'm unable to reproduce the results reported for QwenVL2. For exemple, for `mmmu_val`, I’m getting a score of 50.67, whereas the paper and GitHub documentation of Qwenvl2 indicate 54.1.

I ran the evaluation with the following command:
```bash
python -m lmms_eval --model=qwen2_vl --model_args=pretrained=Qwen/Qwen2-VL-7B-Instruct,device_map=auto --tasks=mmmu_val --batch_size=1 --log_samples --log_samples_suffix=Qwen--output_path="./logs/"
```

I was able, However, to reproduce the results for llava-onevision. Could there be any configuration details I might have missed? Any insights would be greatly appreciated. Thank you!

Since we don't know how exactly Qwen2 VL format their prompt during the evaluation, we can only use the most common way that fits most of the dataset to design the generate function. Without knowing the exact input, it is very likely that this result is different

> Since we don't know how exactly Qwen2 VL format their prompt during the evaluation, we can only use the most common way that fits most of the dataset to design the generate function. Without knowing the exact input, it is very likely that this result is different

Ok thanks. However can you provide results on one or more datasets so we can check that the results are reproductible across different devices (even thought different from the paper)?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Got wrong answer when test nextqa/longvideobench with llava_vid
Root cause: External Dependency Breakage

Hi, I try to use LLaVA-Video to evaluate nextqa and longvideobench_val dataset, but i got wrong answer in every questions
![image](https://github.com/user-attachments/assets/261cbbf0-55b1-4a61-9680-751fdc1c649e)

my scripts is
```
pip3 install pywsd

export HF_HOME=xxx
export HF_TOKEN=xxx

accelerate launch --main_process_port 59000 --num_processes=8 \
-m lmms_eval \
--model llava_vid \
--model_args pretrained=lmms-lab/LLaVA-Video-7B-Qwen2,conv_template=qwen_1_5,max_frames_num=64,mm_spatial_pool_mode=average \
--tasks nextqa,longvideobench_val_v \
--batch_size 1 \
--log_samples \
--log_samples_suffix llava_vid \
--output_path output_path \
--verbosity=DEBUG 2>&1 |tee debug.txt


I tried using another Docker image, and it ran successfully.

The key differences are the versions of Transformers and PyTorch:

Good Image:
- Transformers: 4.44.2
- PyTorch: 2.1.2

Bad Image:
- Transformers: 4.39.2
- PyTorch: 2.1.0

Is this a known bug?

I'm not very sure what is the cause of it but version of transformers might be the issue.

Hi @ZhangYuanhan-AI , do you have any idea about this bug and what is the recommended version of transformers to run the model?

> I'm not very sure what is the cause of it but version of transformers might be the issue.
> 
> Hi @ZhangYuanhan-AI , do you have any idea about this bug and what is the recommended version of transformers to run the model?

You can try to use bfloat16 to initialize the model.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: LLaVA_OneVision class lacks parameter "mm_newline_position"
Root cause: Validation Gap

When I use the following script to evaluate LLaVA-OV:
```
# video tasks
MODEL_PATH=$1
TASK=$2
accelerate launch --num_processes=1 \
-m lmms_eval \
--model llava_onevision \
--model_args pretrained=$MODEL_PATH,conv_template=qwen_1_5,model_name=llava_qwen,mm_spatial_pool_mode=average,mm_newline_position=no_token \
--tasks $TASK \
--batch_size 1 \
--log_samples \
--log_samples_suffix llava_onevision \
--limit 8 \
--output_path ./logs/
```
It gives the error:
```
  File "lmms-eval/lmms_eval/models/llava_onevision.py", line 92, in __init__
    assert kwargs == {}, f"Unexpected kwargs: {kwargs}"
AssertionError: Unexpected kwargs: {'mm_newline_position': 'no_token'}
```
Why we should assert kwars == {}? And what if I want to evaluate LLaVA-OV with the parameter mm_newline_position=no_token? 
Thanks in advance :)

You might need to add this into the model params in the init and add it into the overwrite config also. Since we never use this parameter during training, using a different config might cause performance drop.