zhiminy/EvalEng huggingface dataset:
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [ERROR] Failed to run the `mathvista_test` tasks
Root cause: Configuration Error

<img width="1131" alt="image" src="https://github.com/user-attachments/assets/a4cc5bdc-69c1-48a8-9c6e-b6efbb5f0f12">

After I add the  following code to tasks/mathvista/mathvista_test.yaml, the code runs successfully. But it get an error result.
```
shot: 0
use_caption: False
use_ocr: False
```

<img width="475" alt="image" src="https://github.com/user-attachments/assets/699b6f60-2dda-4fb1-b3c1-7808d3df4494">


I think you can try `mathvista_testmini`, it works on my side.

Hi,

Thanks for your timely reply. `mathvista_testmini` works on my side too, but my baselines report the results of `mathvista_test`😭.


oh okay, can you try to solve this issue and welcome a new PR, thanks!

ok, I will try it today

@liguopeng0923 Hello, I encountered the same issue as you. Have you resolved it?

@Binn0 No, I replace it by `mathvista_testmini` .

Does the problem fixed?

@Binn0 @liguopeng0923 @ZhaoyangLi-nju You can modify the mathvista_test.yaml like this: 
![image](https://github.com/user-attachments/assets/a2ad8a65-db7c-43e3-abe0-c4601a1e029b)
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: RefCOCO|+|g based on REC
Root cause: Unimplemented Feature Gap

Thank you very much for your excellent work.

The evaluation of RefCOCO|+|g you provided is based on CIDEr, but many papers are based on REC evaluation. I also have this need now, how can I proceed?

Looking forward to your reply.

Hi, I think there was one PR #52 about this and was merged. However, I don't know why these added tasks disappear. It is probably because of some problematic merging causing the issue. I cherry-pick some of the commits and create new PR #198 . You can use this branch first before merging

> Hi, I think there was one PR #52 about this and was merged. However, I don't know why these added tasks disappear. It is probably because of some problematic merging causing the issue. I cherry-pick some of the commits and create new PR #198 . You can use this branch first before merging

Oh, thank you very much, your PR is very helpful.
Could you provide the experimental results of llava-13B?
Looking forward to your reply.

> Hi, I think there was one PR #52 about this and was merged. However, I don't know why these added tasks disappear. It is probably because of some problematic merging causing the issue. I cherry-pick some of the commits and create new PR #198 . You can use this branch first before merging

Sorry to bother you again. This is the result of my llava run, but the result of llava-13b seems a bit strange? Have you run this experiment?
![QQ_1723888596603](https://github.com/user-attachments/assets/282f1a93-394f-45af-9ed8-f4b0043235c5)

Looking forward to your reply.

I didn't run this experiment. This PR came from the community. May I ask what is the strange part? The figures in your table looks good to me.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Memory distribution when evaluate model with 2gpu per process
Root cause: Resource Mishandling

I have referred to the issues raised by previous researchers and try to evaluate the model with 2 gpus by setting `device_map=auto`. But i notice that only the model is loaded across several gpus, but the consumed memory during the model forward process is only distributed in the last gpu, which still cause OOM error. My command is as follows. Do you have any suggestion?
`CUDA_VISIBLE_DEVICES=0,1 lmms-eval --model llava_onevision --model_args pretrained=xxx,conv_template=qwen_1_5,model_name=llava_qwen,device_map=auto --tasks mvbench --batch_size 1 --log_samples --log_samples_suffix llava_onevision --output_path ./logs/`
I also try using accelerate launch, but get similar results.
`CUDA_VISIBLE_DEVICES=0,1 accelerate launch --num_processes=1 -m lmms_eval --model llava_onevision --model_args pretrained=xxx,conv_template=qwen_1_5,model_name=llava_qwen,device_map=auto --tasks mvbench --batch_size 1 --log_samples --log_samples_suffix llava_onevision --output_path ./logs/`

Besides, when i try to evaluate `llava-onevision-qwen2-0.5b-ov` on a 3090 GPU with 24GB memory, i also encounter OOM error. But this should not happen as it should only consume 4GB as depicted in the readme.md. This is the command i conduct
`CUDA_VISIBLE_DEVICES=0 accelerate launch --num_processes=1 -m lmms_eval --model llava_onevision --model_args pretrained=path_to/llava-onevision-qwen2-0.5b-ov,conv_template=qwen_1_5,model_name=llava_qwen --tasks mvbench --batch_size 1 --log_samples --log_samples_suffix llava_onevision --output_path ./logs/`

This is because the `device_map=auto` is simply put different layers of the model at different gpus. It will perform forward pass one gpu at a time and pass the hidden states to the next gpu. Thus you will see that the usage of the gpu will increase one by one during inference. You can try to use the `srt_model` and set up a sglang server for a true tensor parallel evaluation if you are still encountering OOM or you can simply reduce the number of frames loaded for llava in `model_args`

Thanks for your response.

> Thanks for your response.



Hi, I seem to have encountered the same problem, have you solved it?

`lmms-eval --model llava_onevision --model_args pretrained=/path_to_your_checkpoint,conv_template=qwen_1_5,model_name=llava_qwen_training_free,device_map=auto --task your_benchmark  --batch_size 1 --log_samples --log_samples_suffix llava_onevision_7b --output_path ./log` can successfully run.


> `lmms-eval --model llava_onevision --model_args pretrained=/path_to_your_checkpoint,conv_template=qwen_1_5,model_name=llava_qwen_training_free,device_map=auto --task your_benchmark --batch_size 1 --log_samples --log_samples_suffix llava_onevision_7b --output_path ./log` can successfully run.



Excuse me, could you tell me where you modified it? Do you know what caused the problem?

I don't know, and the above code works for me.

> I don't know, and the above code works for me.



Alright, but thank you very much!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [BUG] Potential bug in `process_docs()` function pipeline
Root cause: Validation Gap

The current pipeline has a potential bug when applying the process_docs() function.

The process_docs() function is invoked when the self.{split}_docs() function is called, as seen in the following lines:

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/536b70b4e5797f8bc51d46bc658e05db1d2b3c3b/lmms_eval/api/task.py#L926-L930

However, within the fewshot_context() function, a doc is directly retrieved from self.dataset_no_image (line 952), which has not had process_docs() applied. As a result, the example retrieved in line 958 may not be in the intended format.

 https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/536b70b4e5797f8bc51d46bc658e05db1d2b3c3b/lmms_eval/api/task.py#L952-L958

This could potentially lead to inconsistencies or errors in the document format during processing.

I believe in our current pipeline, you can not test few shot now. So maybe won't be an issue currently. But thank you for pointing that out, we will fix this if few shot testing is added to our repo
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [FEATURE REQUEST] MMIU dataset
Root cause: Unimplemented Feature Gap

Hey,

Thanks for the impressive work. Is it possible to have the recent MMIU dataset as one of the tasks? You already have interleaved models such as mantis.

The dataset is available [here](https://huggingface.co/datasets/FanqingM/MMIU-Benchmark)

Best,
Wamiq
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: KeyError: 'llavavid'
Root cause: Validation Gap

```
Error during evaluation: Attempted to load model 'llava_vid', but no model for this name found! Supported model names: claude, from_log, fuyu, gemini_api, gpt4v, instructblip, internvl, internvl2, llama_vid, llava, llava_hf, llava_sglang, longva, mantis, minicpm_v, phi3v, qwen_vl, qwen-vl-api, reka, tinyllava, xcomposer2_4khd, xcomposer2d5
Traceback (most recent call last):
  File "/home/guoyanan.gyn/gyn/lmms-eval/lmms_eval/api/registry.py", line 29, in get_model
    return MODEL_REGISTRY[model_name]
KeyError: 'llavavid'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/guoyanan.gyn/gyn/lmms-eval/lmms_eval/__main__.py", line 202, in cli_evaluate
    results, samples = cli_evaluate_single(args)
  File "/home/guoyanan.gyn/gyn/lmms-eval/lmms_eval/__main__.py", line 298, in cli_evaluate_single
    results = evaluator.simple_evaluate(
  File "/home/guoyanan.gyn/gyn/lmms-eval/lmms_eval/utils.py", line 434, in _wrapper
    return fn(*args, **kwargs)
  File "/home/guoyanan.gyn/gyn/lmms-eval/lmms_eval/evaluator.py", line 96, in simple_evaluate
    lm = lmms_eval.api.registry.get_model(model).create_from_arg_string(
  File "/home/guoyanan.gyn/gyn/lmms-eval/lmms_eval/api/registry.py", line 31, in get_model
    raise ValueError(f"Attempted to load model '{model_name}', but no model for this name found! Supported model names: {', '.join(MODEL_REGISTRY.keys())}")
ValueError: Attempted to load model 'llavavid', but no model for this name found! Supported model names: claude, from_log, fuyu, gemini_api, gpt4v, instructblip, internvl, internvl2, llama_vid, llava, llava_hf, llava_sglang, longva, mantis, minicpm_v, phi3v, qwen_vl, qwen-vl-api, reka, tinyllava, xcomposer2_4khd, xcomposer2d5
```
torch                         2.1.2
transformers                  4.39.2
accelerate                    0.33.0

I have tried llava_vid and llavavid,both failed

Hi, I encountered the same issue, could you share your solution?

+1

Hi @ZhaoyangLi-nju , I noticed that there is such a branch called "inference" under the LLaVA-NeXT, which includes the llavavid. I am not sure whether this is The solution though. but it works for me after I make the switch
https://github.com/LLaVA-VL/LLaVA-NeXT/tree/inference/llavavid

Hi @jzhang38, Can I kindly ask for your confirmation about the right way to use llavavid?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: llava_bench_wild seems not a valid parameter
Root cause: Documentation Deficiency

First of all, a big thanks to all members of this open-source project, which definitely shortens the development cycle of LMMs for the whole community.

I tested my model on a series of benchmarks (e.g., seedbench, mmvet,ai2d, chartqa, llava_bench_wild, docvqa_val) and found llava_bench_wild did not appear in the result jsons. When I passed only llava_bench_wild in the arguments, I got the following error message `assert tasks != [], "No tasks specified, or no tasks found. Please verify the task names."`.

Hi, sorry for the inconvenience. 

The task name for llava-in-the-wild in the readme is wrong and you might want to try using `tasks=llava_in_the_wild`.

If you want to check all the possible tasks, you can use `lmms_eval --tasks list` to check. Noted that this would print out all the tasks and groups (consist of several tasks) that can be run for test.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: /exact_match/exact_match.py. Module 'exact_match' doesn't exist on the Hugging Face Hub either.
Root cause: Unimplemented Feature Gap

/exact_match/exact_match.py. Module 'exact_match' doesn't exist on the Hugging Face Hub either.

Hi, maybe you should check whether you can run

```
from evaluate import load
exact_match = load("exact_match")
```
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Why subtitles are set to False for LongVideoBench
Root cause: Documentation Deficiency

In the longvideobench_val_v benchmark, insert_interleave_subtitles is set to False, meaning subtitles aren't used in the evaluation. Given that multiple tasks involve subtitle-referenced QA, could you provide instructions on how to enable it?

same questions
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Bug：RuntimeError: query and key must have the same dtype
Root cause: Interface Contract Mismatch

```
CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"  accelerate launch \
    --num_processes=8 \
    --main_process_port 12349 \
    -m lmms_eval \
    --model xcomposer2d5 \
    --model_args pretrained="internlm/internlm-xcomposer2d5-7b,modality=video" 
```
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: KeyError: 'Missing task mathvista_testmini' on phi3v
Root cause: Unimplemented Feature Gap

I'm encountering a `KeyError: 'Missing task mathvista_testmini'` when evaluating the **phi3v** model on the **MathVista** dataset (mathvista_testmini) using the lmms-eval framework.

I am using this command:
`python3 -m accelerate.commands.launch \
    -m lmms_eval \
    --model phi3v \
    --model_args model_id_name="microsoft/Phi-3-vision-128k-instruct" \
    --tasks mathvista_testmini \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix phi3v-mathvista_testmini \
    --output_path ./logs/`

**Error Details**
`Traceback (most recent call last):
  File "/lmms-eval/lmms_eval/__main__.py", line 202, in cli_evaluate
    results, samples = cli_evaluate_single(args)
  File "/lmms-eval/lmms_eval/__main__.py", line 298, in cli_evaluate_single
    results = evaluator.simple_evaluate(
  File "/lmms-eval/lmms_eval/utils.py", line 434, in _wrapper
    return fn(*args, **kwargs)
  File "/lmms-eval/lmms_eval/evaluator.py", line 105, in simple_evaluate
    task_dict = lmms_eval.tasks.get_task_dict(tasks, model_name=model)
  File "/lmms-eval/lmms_eval/tasks/__init__.py", line 157, in get_task_dict
    task_obj = get_task(task_name=task_name, model_name=model_name)
  File "/lmms-eval/lmms_eval/tasks/__init__.py", line 124, in get_task
    raise KeyError(f"Missing task {task_name}")
KeyError: 'Missing task mathvista_testmini'
`

It seems like the task mathvista_testmini might be missing or not recognized. Any help to resolve this would be appreciated. Thanks!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Llava-v1.5 and Llava-v1.5-hf  Performes on MMMU differs a lot!
Root cause: External Dependency Breakage

Llava-v1.5-7b resutls
<img width="455" alt="image" src="https://github.com/user-attachments/assets/5c0a7701-9289-4d5e-b926-29a6c220201a">

Llava-1.5-7b-hf
<img width="462" alt="image" src="https://github.com/user-attachments/assets/a2d765e9-ee05-4f8a-ac3a-6515b3d2df53">

Is that right? 
Can  the official teams provide a reliable resutls for your supported model and datasets for a reference?


I have found that using different versions of transformers will result in different MMMU values. When I use the latest version (4.42.2), I get a low value with Llava-1.5-7b-hf. But when I switch to an older version (4.39.2), I get a reasonable value, just like the one reported by the paper. I don't know what causes this problem.

I encountered the same issue. Is there a solution?

I'm having the same problem. The performance of llava-v1.5-7b-hf on MME benchmark is also worse.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Wrongly treating mvbench as an image task
Root cause: Validation Gap

When i evaluate llava onevision on the video benchmark "mvbench", the task regards it as an image task. Please fix it.


It seems that there is an Image in mvbench?
container = av.open(video_path)
<PIL.Image.Image image mode=RGB size=640x360 at 0x15531C163F70>  
ValueError: Could not determine output format   

I met the some question.

Probably because this video is missing: ~/.cache/huggingface/mvbench/star/Charades_v1_480/EDXBD.mp4
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [Request] Loosen dependency version constraints, such as for `transformers`
Root cause: Architectural Constraint

It would be appreciated if version constraints could be loosened to increase compatibility with outside code. For example, `transformers` is locked to `4.39.2`, prohibiting compatibility with any code that requires a more recent version. Similar for `datasets`. Thank you for your consideration!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: PiPy / Github version number mismatch
Root cause: Configuration Error

Hi, I've noticed that, on GitHub, lmms-eval has a release number of [0.2.2](https://github.com/EvolvingLMMs-Lab/lmms-eval/releases/tag/v0.2.2) available, but no 0.2.1. However, [on PyPi](https://pypi.org/project/lmms-eval/#history), there is a release number of [0.2.1](https://pypi.org/project/lmms-eval/0.2.1/), but no 0.2.2.

My guess is that 0.2.1 on PyPi actually is the same as 0.2.2 on GitHub, but it makes sense to resolve the version mismatch to reduce confusion going forwards.

Thank you!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Config will not be added to registry
Root cause: Configuration Error

How to load local data for evaluation

DEBUG /opt/conda/lib/python3.10/site-packages/lmms_eval-0.1.1-py3.10.egg/lmms_eval/tasks/mme/mme.yaml: No module named '/workspace/LLaVA/lmms-lab-local/MME'. Config will not be added to registry.

![image](https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/38122102/d93da188-2b04-4f35-9936-558cc8e5f27e)


Hi, @xienan0326 

Maybe you should check whether the you can use

```
from datasets import load_dataset
load_dataset(<your folder name>)
```

Use huggingface cache as an example

<img width="538" alt="image" src="https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/92624596/f7a40f85-099e-4412-bba4-03f2103b5404">


This works fine on my env
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: question about RefCOCO|+|g. Is it a region caption task?

Thank you very much for your work.

But I have a question about RefCOCO|+|g. Is it a region caption task? Why is it divided into seg and bbox?

Looking forward to your answer.

Yes @yuecao0119  it is a region caption task. It is being splited to seg and bbox based on how we labeled the caption area. If it is seg than we will draw a seg line. If it is bbox then we will draw a bbox. You can also use the rec tasks contributed from the community.

Does anyone try to run reg before
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: LiveBench doesn't work.
Root cause: External Dependency Breakage

17:00（UTC+8）2024.08.26

https://huggingface.co/spaces/lmms-lab/LiveBench

![image](https://github.com/user-attachments/assets/e8f98f21-6e19-46d6-9b6d-51fc8c7b6d9b)
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: name 'get_model_name_from_path' is not defined
Root cause: Validation Gap

2024-08-27 16:25:35.109 | ERROR    | __main__:cli_evaluate:216 - Error during evaluation: name 'get_model_name_from_path' is not defined
Traceback (most recent call last):
  File "/home/jiangbaoyang/lmms-eval-main/lmms_eval/__main__.py", line 202, in cli_evaluate
    results, samples = cli_evaluate_single(args)
  File "/home/jiangbaoyang/lmms-eval-main/lmms_eval/__main__.py", line 298, in cli_evaluate_single
    results = evaluator.simple_evaluate(
  File "/home/jiangbaoyang/lmms-eval-main/lmms_eval/utils.py", line 434, in _wrapper
    return fn(*args, **kwargs)
  File "/home/jiangbaoyang/lmms-eval-main/lmms_eval/evaluator.py", line 96, in simple_evaluate
    lm = lmms_eval.api.registry.get_model(model).create_from_arg_string(
  File "/home/jiangbaoyang/lmms-eval-main/lmms_eval/api/model.py", line 92, in create_from_arg_string
    return cls(**args, **args2)
  File "/home/jiangbaoyang/lmms-eval-main/lmms_eval/models/llava.py", line 92, in __init__
    model_name = model_name if model_name is not None else get_model_name_from_path(pretrained)
NameError: name 'get_model_name_from_path' is not defined
[rank0]:[W827 16:25:35.425639337 ProcessGroupNCCL.cpp:1168] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())
这是为什么呢

可以看看你的命令行输入吗，从报错内容看就是你的模型路径出问题，也许是模型名字问题

Hello, I recently had the same error. I do not know your command line but what I can tell you is that you have to use the directory of the model from https://huggingface.co/lmms-lab.

For example, `--model_args pretrained lmms-lab/llava-onevision-qwen2-0.5b-si`.

I hope it helped you.

I had the same issue when I'm trying to use the example command listed in the README

```
python3 -m accelerate.commands.launch \
    --num_processes=8 \
    -m lmms_eval \
    --model llava \
    --model_args pretrained="liuhaotian/llava-v1.5-7b" \
    --tasks mme,mmbench_en \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix llava_v1.5_mme_mmbenchen \
    --output_path ./logs/
```

You should clone the run LLaVA-NEXT repo and run script in LLaVA-NEXT folder.
Check here: https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/docs/run_examples.md#llava
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Missing MMU-mv-test in llava-interleave-bench
Root cause: Unimplemented Feature Gap

<img width="1418" alt="image" src="https://github.com/user-attachments/assets/483e1d20-d4a4-432d-9ae3-2a64b684c96d">

I find the current eval results missing the `mmmu-mv`

<img width="351" alt="image" src="https://github.com/user-attachments/assets/3e83fe59-e6f9-4613-9313-cad910b030fe">


You may need to contact the authors. They report it based on MMMU's original evaluation and filtering the multi-images part to calculate a specific accuracy.

Thank you for your timely reply!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: AttributeError: 'LlavaQwenConfig' object has no attribute 'mm_newline_position'
Root cause: Configuration Error

Before you open an issue, please check if a similar issue already exists or has been closed before.

### When you open an issue, please be sure to include the following

- [ ] A descriptive title: [xxx] XXXX
- [ ] A detailed description

Thank you for your contributions!
when I run llava-one-vision,  have a error.
My sh file is :
```
CUDA_VISIBLE_DEVICE=1 python3 -m accelerate.commands.launch --num_processes 1 --main_process_port 12345 -m lmms_eval \
    --model llava_onevision \
    --model_args pretrained=/home/yuxi.dwh/notebook/pre-trained/lmms-lab/llava-onevision-qwen2-0.5b-ov,conv_template=qwen_1_5,model_name=llava_qwen \
    --tasks videochatgpt \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix llava_onevision \
    --output_path ./logs/
```
and error is :
```
    if self.config.mm_newline_position == "grid":
  File "/home/yuxi.dwh/.local/lib/python3.10/site-packages/transformers/configuration_utils.py", line 263, in __getattribute__
    return super().__getattribute__(key)
AttributeError: 'LlavaQwenConfig' object has no attribute 'mm_newline_position'
```