zhiminy/EvalEng huggingface dataset:
Repository: open-compass/VLMEvalKit
Issue: 如何获得 opencompass 上评测模型的最新细分项结果？

Hi，我看到 readme 中给了一个所有测试结果细分项的汇总 json -> http://opencompass.openxlab.space/utils/OpenVLM.json

但是这个不太全，请问下有全集吗

你好，我们评测过的结果都汇总到 [OpenVLM.json](http://opencompass.openxlab.space/utils/OpenVLM.json) 文件中了。

@junming-yang  绝大多数模型的结果可以被找到，但目前 glm-4v-9b(开源模型) 没有在 json 中找到。
我能从哪里得到该模型的结果呢？
感谢

最新的模型结果迁移到 http://opencompass.openxlab.space/assets/OpenVLM.json 中了，我们将在 readme 中修正这个连接

非常感谢

@junming-yang 请问结果中是没有video benchmark(MMBench-Video Video-MME MVBench)的评测结果吗？

Hi @lily410，video benchmark的评测结果展示在 https://huggingface.co/spaces/opencompass/openvlm_video_leaderboard, 您也可以通过http://opencompass.openxlab.space/utils/video_leaderboard.json 这个链接进行下载，我们后续会在readme里加上此链接，感谢您的关注与支持！

> Hi @lily410，video benchmark的评测结果展示在 https://huggingface.co/spaces/opencompass/openvlm_video_leaderboard, 您也可以通过http://opencompass.openxlab.space/utils/video_leaderboard.json 这个链接进行下载，我们后续会在readme里加上此链接，感谢您的关注与支持！

看到了，十分感谢！
Repository: open-compass/VLMEvalKit
Issue: [rank0]: KeyError: 'answer'
Root cause: Validation Gap

/vlmeval/dataset/image_mcq.py", line 181, in evaluate
[rank0]:     answer_map = {i: c for i, c in zip(meta['index'], meta['answer'])}
[rank0]:   File "/data/miniconda3/envs/env-3.9.2/lib/python3.9/site-packages/pandas/core/frame.py", line 4102, in __getitem__
[rank0]:     indexer = self.columns.get_loc(key)
[rank0]:   File "/data/miniconda3/envs/env-3.9.2/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3812, in get_loc
[rank0]:     raise KeyError(key) from err
[rank0]: KeyError: 'answer'

Can you provide the running command?

If you are evaluating a private benchmark, please check your dataset file `xxx. tsv` fields correct, such as 'answer' instead of 'ans'. Please refer to [Development Doc](https://github.com/open-compass/VLMEvalKit/blob/main/docs/en/Development.md) for more detailed information.

Am runing on MME only got this error

I try to reproduce this error. This is my command `torchrun --nproc-per-node=8 run.py --model InternVL-Chat-V1-5 --data MME`. Everything is OK. And **MME** benchmark should correspond to the `image_yorn.py` file instead of `image_mcq.py`. Please check your code repo.

Then that's weired, I didn touched anything inside vlmkit, those tsv are automatically downloaded, there must be an key error at some where in some dataset.

Maybe you can try to pull our latest code. We have fixed some known minor bugs. If you still encounter other problems, please do not hesitate to contact us and give us some detailed information to fix them. Thank you.

Hi, @lucasjinreal , I tried to reproduce the problem but failed (I'm evaluating llava_v1.5_7b on MME), with latest main branch.

Duplicate of https://github.com/open-compass/VLMEvalKit/issues/378
Repository: open-compass/VLMEvalKit
Issue: Does the generating params changed?
Root cause: Environment Incompatibility

The model trends to output more words than a single option when multi choice, after upgraded vlmevalkit

@lucasjinreal Which model and dataset do you evaluate? Can you provide more information? This may help us find out the problem.

we are testing on same inhouse model, the data eval set was: eval_datasets='MMBench_DEV_CN MMStar MMBench_TEST_EN MMBench_TEST_CN MMBench_TEST_EN_V11 MMBench_TEST_CN_V11 MME '


Didn't do anything else, looks like the output trend to produce more output, am not sure what could be the reason.

Hi, @lucasjinreal , 
If you are testing in-house model , the problem may not be related to VLMEvalKit, you can check if the python environment accidentally changes (like you accidentally changed the version of some python packages).  

AFAIK, there is no change on the dataset side that would affect the model behavior. 

Close since no response in a long time, feel free to reopen if needed.
Repository: open-compass/VLMEvalKit
Issue: [BUG] local variable 'structs' referenced before assignment
Root cause: Validation Gap

In file `vlmeval/inference.py`, line 41 , local variable `structs` referenced before assignment, see below:

<img width="1002" alt="image" src="https://github.com/open-compass/VLMEvalKit/assets/42505434/c2e9db8c-f13a-4bf3-bc6d-0ea54f1e4891">



Thank you for your report. We have fixed this problem in [pr275](https://github.com/open-compass/VLMEvalKit/pull/275). Now the API model can be used.
Repository: open-compass/VLMEvalKit
Issue: [bug] The same sample was run twice in mp_utils.py
Root cause: Algorithmic Error

As you can see  in https://github.com/open-compass/VLMEvalKit/blob/c91b8b26dbe45594019827f246748b344b214300/vlmeval/utils/mp_util.py#L147 
the same sample has been run over and over again, resulting in a doubling of the inference speed. Is this in line with expectations, and if so, what is the reason for this, and I look forward to receiving an answer, thank you.


<img width="655" alt="image" src="https://github.com/open-compass/VLMEvalKit/assets/8131131/62f36da1-02f5-42d0-be3d-50c0900ccd40">


@buptweixin 
I think that's a typo and should be fixed, we haven't noticed that before since we always use nproc > 1 when calling `track_progress_rich`,  we will fix it immediately and thanks for the information.
Repository: open-compass/VLMEvalKit
Issue: Difference between MMBench_DEV and MMBench_Test
Root cause: Documentation Deficiency

HI, @kennymckormick 

Thanks for open-sourcing such a wonderful job.  As I can not find the score of the  MMBench_DEV, would you mind to tell me   what's the difference between the  MMBench_DEV and MMBench_Test benchmark. 

You can run with `--data MMBench_DEV_EN` or `--data MMBench_TEST_EN` to predict the MMBench result. MMBench_DEV benchmark can be evaluated on your own device. MMBench_TEST needs to be submitted to the official service. Link: https://mmbench.opencompass.org.cn/mmbench-submission.
Repository: open-compass/VLMEvalKit
Issue: Missing `dataset_config.py`
Root cause: Documentation Deficiency

According to Quickstart.md, the data root can be changed in `vlmeval/utils/dataset_config.py`. However, the `dataset_config.py` is missing and I found this file in the old version. So how to change the data root in the latest version, thanks.

Hi, @beichenzbc , 
The dataset codes have been refactored and we will update the document correspondingly.
For now, if you want to change the data root, you can set the env variable `$LMUData`, or reset the `img_root` variable of the corresponding dataset class (only if you just want to modify the place you save the images, refer to `vlmeval/dataset/image_base.py`)

Thanks for quick reply, could you please share an example of how to set env variable $LMUData if i want to test a model on MMMU dataset.

Hi, @beichenzbc. You can run with `torchrun --nproc-per-node=8 / python3 run.py --model xxx --data MMMU_DEV_VAL`. VLMEvalKit will automatically download the MMMU dataset tsv file to the default path `~/LMUData`. If you want to change `$LMUData`, you can set `LMUData=/path/to/your/datafile` in .env or .bashrc file.

Thanks!
Repository: open-compass/VLMEvalKit
Issue: How to create OpenVLM.json 
Root cause: Documentation Deficiency

Thanks for Opensourcing the framework.

I was able to replicate(infer and evaluate) some models. I also wanted to aggregate the benchmarking as done in JSON,

Is their any script or instruction you can share for the same ?

Thanks for watching! We will update a script later for the OpenVLM.json file. You can attach your test result file to the issues or send  to our email. We will check the result and merge into the leaderboard. Thanks for your contribution! 

Thanks @junming-yang 
If you have a rough script  to share in the meantime, I can clean that and push in the repo for easy reproducing the leaderboard .
 Or should it be done manually for now ?


Here’s a draft script for OpenVLM.json. [scripts.zip](https://github.com/user-attachments/files/16260601/scripts.zip)
You only need to provide a JSON file containing the model result you want to add, following the example format below. You do not need to include results for all models in the JSON.
```
{
    "YourModelName": {
        "META": {
            "Method": [
                "YourModelName",
                "https://yourmodelwebsite.com"
            ],
            "Parameters": "YourModelParameters",
            "Language Model": "YourLanguageModel",
            "Vision Model": "YourVisionModel",
            "Org": "YourOrganization",
            "Time": "YourDate",
            "Verified": "Yes/No",
            "OpenSource": "Yes/No"
        },
        "SEEDBench_IMG": {
            // Add relevant details here
        }
        ......
    }
}
```
Note that the model should be supported by the VLMEvalKit toolkit. We will help check and merge it to the leaderboard.

Thanks for your contribution!

Please feel free to reopen the issue if needed.

@junming-yang : Can we please put the notebook in the scripts folder with the other notebooks ?
It would help better re-use and keep it updated with the new datasets that is added to the VLMEvalLeaderboard.

I would suggest to re-open and we track tit with the latest notebook and as it gets updated @kennymckormick
Repository: open-compass/VLMEvalKit
Issue: InternVL2 Error 
Root cause: Environment Incompatibility

```
Generating with V2.0
A: torch.Size([7175, 3200]), B: torch.Size([9600, 3200]), C: (7175, 9600); (lda, ldb, ldc): (c_int(229600), c_int(307200), c_int(229600)); (m, n, k): (c_int(7175), c_int(9600), c_int(3200))
cuBLAS API failed with status 15
error detected
```

Command - 
```
python3  run.py --data MMBench_DEV_EN MME SEEDBench_IMG MMMU_DEV_VAL --model InternVL2-40B  --verbose 
```


I am able to run InternVL series 1 but for InternVL2 I am getting this error.Though I am able to run the 26B InternVL2

I have build the VLMEvalKit from its latest master branch.

Also the config does not have the InternVL2-76B listed which has been integrated in the code

InternVL2 uses `transformers == 4.37.0`. Please check your package version. 

InternVL2-76B is under testing. We will officially support and update it later in the `config.py` file.

@junming-yang , yes I have the correct transformers version. InternVL2-26B runs but InternVL2-4)b gives the type mismatch error

I tried to reproduce this bug using the latest code and ran the same command. It appears to be working normally.

Hi, @amitbcp , InternVL2-76B is already supported now. For the type mismatch error, please provide more context. 

Thanks @kennymckormick ! yup was able to run InternVL2
thought the leaderboard is not updated with the same.

I have the benchmark results for InternVL2-76B let me know how can I share ?
Repository: open-compass/VLMEvalKit
Issue: Request Support for VILA
Root cause: Unimplemented Feature Gap

Thanks for open-sourcing the framework 

One of the LMMs which has been doing great work and has been iterating their models is VILA : https://github.com/NVlabs/VILA

It would be great to add the support for the same

If its okay I can work on a the PR to integrate the same @junming-yang @kennymckormick ?

Hi, @amitbcp. Thanks for your work to support the VILA model. Please don't hesitate to contact us if you need any help.

Sure will try to wrap it within this week !!

Thanks for your contribution. VILA is supported now!
Repository: open-compass/VLMEvalKit
Issue: [Question] Dataset not officially supported ?
Root cause: Documentation Deficiency

While trying the benchmark, for many datasets the logs have

` Dataset MMMU_VAL is not officially supported`

Dataset may vary. Does that mean the results and benchmarks can be different ? How do we fix this, any suggestions ?


Please run with the command MMMU_DEV_VAL. It includes the results of validation and test. We list the dataset name and corresponding command in the readme file.
Repository: open-compass/VLMEvalKit
Issue: How to change a dataset to TSV file
Root cause: Documentation Deficiency

According to Development.md, I need vlmeval/utils/dataset_config.py to change  a dataset to TSV file. But I can't find it.

Hi, @ttguoguo3. We are updating the development doc. You can refer to this [PR 296](https://github.com/open-compass/VLMEvalKit/pull/296). We will update the main branch soon.

We have updated it. Please refer to [Development.md](https://github.com/open-compass/VLMEvalKit/blob/main/docs/en/Development.md).
Repository: open-compass/VLMEvalKit
Issue: Change to GPT judgement
Root cause: Documentation Deficiency

Hello, the default judging method for a multi-choice dataset is exact-matching method. Could you give me an example of how to use GPT API to judge a MC task, thanks.

Hi, @beichenzbc , 
To judge MCQ datasets with the help of GPT API, you just need to set the `OPENAI_API_KEY` environment variable (directly or in `VLMEvalKit/.env`). You can refer to the document (https://github.com/open-compass/VLMEvalKit/blob/main/docs/en/Quickstart.md) for more details. 

Thanks for quick reply, but we have already set OPENAI_API_KEY for letting GPT4-o answering question, but the evaluating still using exact-matching mode, should we set another value for both generating answers and evaluating answers using GPT4-o?

Which benchmark are you evaluating? 

MMMU

Hi, @beichenzbc. We tried to reproduce it in the MMMU_DEV_VAL benchmark but failed. You can set `OPENAI_API_KEY` and `OPENAI_API_BASE` in the .env file. The key can be used for both API model inference and API model assistant judgment.

Thanks, but we have already set OPENAI_API_KEY and OPENAI_API_BASE and run the following command: python run.py --data MMMU_DEV_VAL --model GPT4o
However, the output result shows a lot of "Failed in Prefetch, no GPT-based answer matching under `exact_matching` policy", indicating that it still using exact matching mode.
Should I set another value, i.e. 'OPENAI_API_KEY_JUDGE', for using API judgment? Or should I use another command to run the code? Thanks!

You can try to set the `OPENAI_API_KEY_JUDGE` and `OPENAI_API_BASE_JUDGE`. Is your code the latest from the main branch?

OK, I will try. I download the code a week ago.

It still doesn't work, there're still lots of "Failed in Prefetch, no GPT-based answer matching under `exact_matching` policy".

Please try to pull the latest code, delete the evaluation file with the `openai_result` suffix, and rerun the command.

Thanks, maybe I have found the error. The same OpenAI API key and base can work properly in generating answers. But in evaluation, it will cause: "UserWarning: OPENAI API is not working properly, will use exact matching for evaluation". Do you have any idea about why it doesn't work properly, thanks.

I've solved the problem. The command should be `python run.py --data MMMU_DEV_VAL --model GPT4o **--judge gpt-4o**` and I add gpt-4o in the 144th line in `vlmeval/dataset/image_mcq.py` 
Again thanks for your quick reply and helpful advice. Best wishes!

Currently, we support 'chatgpt-0125' or 'gpt-4-0125' as a judge model. We will support more API models as judge soon.

So will there be any problem of using GPT-4o as judging model? The accuracy juding by 4o is 0.642, which is quite normal.

No problem. However, to ensure fairness on the leaderboard, we need to use the same judge model for all models. Currently, we are using chatgpt-0125 to report all results.

Got it, thanks!
Repository: open-compass/VLMEvalKit
Issue: Request for GPT-4o Generated Results on MMBench

Hello,  could you share the raw output generated by GPT-4o for each question in the MMBench dataset?

Here are the raw predictions of GPT-4o on MMBench. 

[GPT4o_HIGH_MMBench_CN_V11.xlsx](https://github.com/user-attachments/files/16356805/GPT4o_HIGH_MMBench_CN_V11.xlsx)
[GPT4o_HIGH_MMBench_V11.xlsx](https://github.com/user-attachments/files/16356806/GPT4o_HIGH_MMBench_V11.xlsx)
Repository: open-compass/VLMEvalKit
Issue: Suspected error?
Root cause: Validation Gap

![image](https://github.com/user-attachments/assets/84957058-f961-4737-9341-16ab4035c120)
An error occurred while calculating the COCO Caption score, indicating the position in image_caption.py as shown in the figure. Is this a code error?


Hi, @Shijinghang. What error do you meet? I tried to reproduce it but failed (model: InternVL2-1B). Can you provide more information?

Issue closes due to no response in a long time. Feel free to re-open it when needed.
Repository: open-compass/VLMEvalKit
Issue: 用于编码和解码图像的vlmeval/smp.py文件不存在
Root cause: Documentation Deficiency

在development.md文件中提及了需要vlmeval/smp.py文件进行图像的编码和解码，但在实际中无法找到

Hi, @ttguoguo3 , 
我们此前进行了代码重构，目前，这两个函数位于 `vlmeval/smp/vlm.py`，我们会对文档进行更新

已更新文档

感谢！
Repository: open-compass/VLMEvalKit
Issue: Audio modality evaluation
GitHub labels: Feature Request

Thank you very much for your open work. I would like to ask if the project will support audio modality evaluation, if so, is there a schedule, and what datasets and metrics will be supported？

Hi, we currently do not include `audio modality evaluation` in the next several months. Meanwhile, our framework design is compatible with new modalities such as audio, and we warmly encourage community contributors to implement such features.
Repository: open-compass/VLMEvalKit
Issue: Does vlmeval support multi card inference and batch size > 1?
Root cause: Unimplemented Feature Gap

Does vlmeval support multi card inference and batch size > 1?

Hi, @John-Ge , 
1. For simplicity reasons, VLMEvalKit do not support batch size > 1 inference for now. 
2. VLMEvalKit currently supports two types of multi-GPU inference:
1). DistributedDataParallel  via torchrun, which run N  VLM instances on N GPUs. It requires your VLM  to be small enough and can run on a single GPU.
2). The model is configured by default to use multiple GPUs (like IDEFICS_80B_INSTRUCT). When you launch with python, it will automatically run on all available GPUs.  

Thanks for your relpy!
I would like to know what is the normal format of inference with batch size > 1? Should we deploy the model though like, vllm or tgi?
Do we need to wait for them to support llava?


The authors of LLaVA have tried to create the beta-version of batch inference [https://github.com/haotian-liu/LLaVA/issues/754](url)

Hi, @darkpromise98 , we will try to include this feature into VLMEvalKit recently. 

> Hi, @darkpromise98 , we will try to include this feature into VLMEvalKit recently.

That's great !

https://github.com/haotian-liu/LLaVA/issues/754#issuecomment-1907970439
this issue build a fast inference method for llava, would you add this function for every benchmark in this repo?

BTW, I find sglang may not support lora+base model. I train llava with lora. If possible, I hope you could support load base model and merge lora weights and deploy it for evaluation. 

Hi, @John-Ge @darkpromise98 , I have reviewed the request. I'm sorry that I may not implement this feature on my own for the following reasons:
1. Currently, only few VLMs supports the `batch_inference` interface, adding it for LLaVA may lead to some major changes in the inference pipeline of VLMEvalKit. 
2. The inference of LLaVA is relatively fast: under `batch_size=1`, llava-v1.5-13b can run at 3~4 fps on a single A100. Thus I think `batch_inference`  for LLaVA may not be a critical feature for VLMEvalKit. 

BTW, I'm willing to review and merge it VLMEvalKit main branch if someone is willing to create a PR (might be relatively heavy) about it.
Repository: open-compass/VLMEvalKit
Issue: MuirBench Integration
Root cause: Unimplemented Feature Gap

Hey @kennymckormick @junming-yang 

I want to add MuirBench Dataset to the kit. It has mutliple images for a single query. Does VLM support any such dataset which I can refer and create a PR ?

Hi, @amitbcp ，I think you can refer to BLINK benchmark to support MUIRBench

Benchmark Merged.
Repository: open-compass/VLMEvalKit
Issue: [Help Wanted] Supporting the `chat_inner` API for existing VLMs.
Root cause: Unimplemented Feature Gap

Since we have now supported the multi-turn benchmark MMDU, we would like to implement the `chat_inner` function for existing VLMs in VLMEvalKit add support for multi-turn chatting. 
Currently, we have already supported the method for: `GPT series`, `Claude series`,  `QwenVL APIs`, `qwen_chat`, `MiniCPM-v2.5`, `Idefics2_8b`, `llava-v1.5`, `deepseek-vl series`. We need help from the community to support it for more models. 
 
If you would like to help, you can refer to the development doc: https://github.com/open-compass/VLMEvalKit/blob/main/docs/en/Development.md.

Implementing three `chat_inner` api can be viewed as  a major contribution. However, only implement this function if you are sure the VLM has the capability to handle the multi-turn input. 

@kennymckormick : I will pickup phi-3 and VILA models and create a PR

> @kennymckormick : I will pickup phi-3 and VILA models and create a PR

many thanks~
Repository: open-compass/VLMEvalKit
Issue: Fail to Download MathVision_MINI.tsv
Root cause: External Dependency Breakage

Hello, I'm trying to evaluate on MathVision_MINI dataset. Howeverm this download link https://[opencompass.openxlab.space/utils/VLMEval/MathVision_MINI.tsv](https://opencompass.openxlab.space/utils/VLMEval/MathVision_MINI.tsv) seems to have stopped working. Could you please give a link to download MathVision_MINI , thanks?




Hi, @beichenzbc , the link has been fixed now. 

Thanks!
Repository: open-compass/VLMEvalKit
Issue: What is the estiamted runtime across benchmarks, and OpenAI api cost? 


Hi, @findalexli , 
1. The runtime depends on the architecture of your VLM and may vary across different VLMs. To demonstrate a data point, evaluating llava-v1.5-7b on MMBench_DEV_EN may cost 15 minutes on a single A100. 
2. The OpenAI API cost depends on the nature of the task and the instruction following capability of your model (after all, if an VLM follows the instruction perfectly, no GPT post-processing is required). For some data points, evaluating different VLMs on MMBench_EN (dev + test) will cost < $1.5 on average. For models like llava-v1.5-7b, XComposer, no GPT cost will be spent during evaluation.
Repository: open-compass/VLMEvalKit
Issue: Custom dataset is not considered MCQ by models
Root cause: Validation Gap

VLMEvalKit version: commit 8e0aace0504d952a25e310a1de66a32c2c1476f1

I added a custom MCQ format dataset to LMUData directory. It is successfully loaded and shows "UserWarning: Will assume unsupported dataset **** as a Custom MCQ dataset".
However the model is not correctly called to build a MCQ prompt. After checking it seems `DATASET_TYPE` function only returns correct evaluation types for supported datasets (i.e. listed in its `DATASET_URL`).

Since there IS `CustomMCQDataset` and `CustomVQADataset` type, is it possible to return the correct dataset type for prompt building?

Building a tsv file is very straight forward without the need of modifications to VLMEvalKit codes, so I hope it can be better supported.


Hi, @zodiacg , 
That makes sense, we plan to treat all custom benchmarks as MCQ benchmarks and will modify the code accordingly. 

Feature Supported.
Repository: open-compass/VLMEvalKit
Issue: 可以加入Mantis模型的评测吗 相关模型如下
Root cause: Environment Incompatibility

https://huggingface.co/TIGER-Lab/Mantis-8B-siglip-llama3

AssertionError: Invalid image processor type: Idefics2ImageProcessor 请问怎么解决？

> Idefics2ImageProcessor

更新一下transformers版本吧
Repository: open-compass/VLMEvalKit
Issue: ChartQA evaluation
Root cause: Documentation Deficiency

Hi,

I have a question regarding the test results for ChartQA. In the paper, there's a table showing the data split:

ChartQA-H:
- Training: 3,699 charts, 7,398 questions
- Validation: 480 charts, 960 questions  
- Test: 625 charts, 1,250 questions

ChartQA-M:
- Training: 15,474 charts, 20,901 questions
- Validation: 680 charts, 680 questions
- Test: 987 charts, 1,250 questions

I'm wondering if the reported test results in the paper are based on just the ChartQA-H (human-authored) test set, or if they include both the ChartQA-H and ChartQA-M (machine-generated) test sets combined?

Thank you for clarifying!

The ChartQA_TEST benchmark includes both the ChartQA-H and ChartQA-M with 2,500 questions.

@mactavish91 check out this inference example it could help you too : https://github.com/moured/qwen-vl2.5-chartqa