zhiminy/EvalEng huggingface dataset:
Repository: open-compass/VLMEvalKit
Issue: Error ‘assert len(chunk_encode) == 2’ when eval MMMU_DEV_VAL
Root cause: Validation Gap

When I try to test MMMU_DEV_VAL for LlaVa

The assert len(chunk_encode) == 2 is triggered, after print, I find that len(chunk_encode)=4

Since $inputs$ is:


user

[image]
Determine the area in hectares between the line AB and a meandering stream for offsets taken at a regular interval of 20 m along the line AB (Fig. 12.5). Use both the trapezoidal rule and Simpson's rule.
[Image]
[Image]

A. Use trapezoidal rule,area = 0.5010 hectares;Use trapezoidal rule,area = 0.5560 hectares
B. Use trapezoidal rule,area = 0.5010 hectares;Use trapezoidal rule,area = 0.5460 hectares
C. Use trapezoidal rule,area = 0.5010 hectares;Use trapezoidal rule,area = 0.5260 hectares
D. Use trapezoidal rule,area = 0.5010 hectares;Use trapezoidal rule,area = 0.5360 hectares
Answer with the option's letter from the given choices directly.

assistant


After split input with [image], we would get 4 parts more than 4, and the assert is triggered.

seems that error come from:

https://github.com/open-compass/VLMEvalKit/blob/dc2e6d87e758c7f0004312fe140902ba83ff8c02/vlmeval/vlm/base.py#L151

There are multi-image in the prompt, and prompt is:
Determine the area in hectares between the line AB and a meandering stream for offsets taken at a regular interval of 20 m along the line AB (Fig. 12.5). Use both the trapezoidal rule and Simpson's rule.
[Image]
[Image]

A. Use trapezoidal rule,area = 0.5010 hectares;Use trapezoidal rule,area = 0.5560 hectares
B. Use trapezoidal rule,area = 0.5010 hectares;Use trapezoidal rule,area = 0.5460 hectares
C. Use trapezoidal rule,area = 0.5010 hectares;Use trapezoidal rule,area = 0.5260 hectares
D. Use trapezoidal rule,area = 0.5010 hectares;Use trapezoidal rule,area = 0.5360 hectares
Answer with the option's letter from the given choices directly.

seems that image should be discarded.

> seems that error come from:
> 
> https://github.com/open-compass/VLMEvalKit/blob/dc2e6d87e758c7f0004312fe140902ba83ff8c02/vlmeval/vlm/base.py#L151
Since Model LLaVA_XTuner does not support interleaved input. Will use the first image and aggregated texts as prompt. 

Modifying the 'image' into another name would solve this issue. and such images would be discarded.
Repository: open-compass/VLMEvalKit
Issue: How to calculate Avg Score mentioned in OpenVLM Leaderboard？
Root cause: Documentation Deficiency

Since OpenVLM Leaderboard describes its metrics like this:
- Avg Score: The average score on **all VLM Benchmarks** (normalized to 0 - 100, the higher the better).

May I ask:
1. Do **all VLM Benchmarks** refer to all 19 benchmarks or only the 12 benchmarks displayed in the Main Leaderboard?
2. How to calculate Avg Score? Since MME and OCRBench need to be normalized, how do you normalize them?
Repository: open-compass/VLMEvalKit
Issue: 支持LMDeploy作为推理后端
Root cause: Unimplemented Feature Gap

您好，目前VLMEvalKit支持LMDeploy server作为judge，是否有计划支持LMDeploy server作为需要被评测的模型

Hi, @slchenchn , 
Currently we have no plan to support  lmdeploy served VLM. If you are interested, you can create a PR (including an API wrapper to call lmdeploy served VLM API), and we will help to review and merge.
Repository: open-compass/VLMEvalKit
Issue: 怎么测试LLAVA+LLAMA3的性能？
Root cause: Documentation Deficiency

怎么测试LLAVA+LLAMA3的性能？

@junming-yang 

We have supported [Bunny-Llama3-8B](https://huggingface.co/BAAI/Bunny-Llama-3-8B-V),  a multimodal with language backbone Llama-3-8B. Please refer to [#184](https://github.com/open-compass/VLMEvalKit/pull/184) for more detail.

We have supported [LLAVA+LLAMA3](https://huggingface.co/xtuner/llava-llama-3-8b-v1_1). You can run `python3 run.py --model llava-llama-3-8b --data MMStar` to evaluate the model. And we will add the results into the [leaderboard](https://openxlab.org.cn/apps/detail/kennyutc/open_mllm_leaderboard) in the future.
Repository: open-compass/VLMEvalKit
Issue: best to have docker image
Root cause: Unimplemented Feature Gap

1. all datasets are downloaded
2. all requirements are installed
3. all dependency repos are prepared

Hi, @max-yue , 
Yeah, we plan to build a docker for VLMEvalKit.

@Mor-Li 
It would be good if we can have one or several docker files than can successfully run the evaluation of tens of representative VLMs.
Repository: open-compass/VLMEvalKit
Issue: About MMBench data source: Is there any difference from huggingface `lmms-lab/MMBench`?
Root cause: Documentation Deficiency

Hello! I find it's accessible for me to use huggingface to get MMBench(`lmms-lab/MMBench`). For example:
```
from datasets import load_dataset
dataset = load_dataset(f"{my_path}/lmms-lab/MMBench/en")['validation'] # ['test']
```
Then, I get the EN val/test split data. Is there any difference between this repo/VLMEvalKit and `lmms-lab/MMBench`? Thanks for your reply!

Hi, @Coobiw ,
I'm not sure, that data file is not served my VLMEvalKit thus I can not provide any guarantee.

thanks！i will double check it！
Repository: open-compass/VLMEvalKit
Issue: Question about LlavaBench Evaluation
Root cause: Documentation Deficiency

Hi author,

I saw that there are 3 columns for LlavaBench scores (Relative Score,  VLM Score,  GPT4 Score), and seems like in evaluate code the Relative Score is calculated based on the VLM Score and GPT4 Score. However, seem like the original LLaVA project only report the GPT4 Score (this [table](https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZOO.md#llava-v16)). So how did the VLM Score and Relative Score calculated and reported? Or am I missing something?

Hi, @justinphan3110 ,

TL;DR: The GPT4 score reported by the original LLaVA project is the Relative Score reported in VLMEvalKit.

Why: The context is different. LLaVA Project: GPT4 score means the score is evaluated by GPT4; VLMEvalKit: GPT4 score means it's the score of the reference answer (answered by GPT4). 

oh so to clarify, for example in this [llava-1.6 table](https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZOO.md#llava-v16), the LLaVA author show that LLaVA-1.6-Mistral-7B got `83` on LlavaBench but this is the score I got from running on VLMEvalKit:
```
 split  Relative Score (main)  VLM Score  GPT4 Score
0  overall                   66.6       52.2        78.3
1     conv                   57.9       49.4        85.3
2  complex                   76.2       58.2        76.4
3   detail                   59.5       44.0        74.0
```

This is the cmd that I used:
```
model="llava_next_mistral_7b"
task="LLaVABench"

!torchrun --nproc-per-node={num_gpus} run.py --data {task} --model {model} --nproc 20
```

Is this big gap in scores (66.6 and 83) expected?


> oh so to clarify, for example in this [llava-1.6 table](https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZOO.md#llava-v16), the LLaVA author show that LLaVA-1.6-Mistral-7B got `83` on LlavaBench but this is the score I got from running on VLMEvalKit:
> 
> ```
>  split  Relative Score (main)  VLM Score  GPT4 Score
> 0  overall                   66.6       52.2        78.3
> 1     conv                   57.9       49.4        85.3
> 2  complex                   76.2       58.2        76.4
> 3   detail                   59.5       44.0        74.0
> ```
> 
> This is the cmd that I used:
> 
> ```
> model="llava_next_mistral_7b"
> task="LLaVABench"
> 
> !torchrun --nproc-per-node={num_gpus} run.py --data {task} --model {model} --nproc 20
> ```
> 
> Is this big gap in scores (66.6 and 83) expected?

Did the author mention which version of GPT-4 are they using for evaluating LLaVA 1.6? 
The original LLaVABench adopts GPT-4-0314 (which is not available to new users now), while VLMEvalKit adopts GPT-4-1106. The different version of GPT-4 may lead to significant changes in the final score. 

I see, changing to `GPT-4-0613` also brought the score back to `~80.0`. Thanks for the clarification.
Repository: open-compass/VLMEvalKit
Issue: [Request]Consider integrating MMT-Bench and CONTEXTUAL?

MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI
https://arxiv.org/pdf/2404.16006

CONTEXTUAL: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models
https://arxiv.org/pdf/2401.13311

Thanks for your interest in MMT-Bench. We will integrate MMT-Bench ASAP.

Hi, @iamlockelightning 
MMT-Bench has already been supported.
Repository: open-compass/VLMEvalKit
Issue: mmmu should only use mmmu val not MMMU_DEV_VAL
Root cause: Configuration Error


Would you please further clarify the problem, It's OK to use Chinese.

 现在 mmmu 使用的是 dataset dev and validation split,  应该只使用val split 

Oh, we think it's OK to use MMMU_DEV_VAL, which only incurs an overhead of 150 visual questions. Besides, in the accuracy file we provide the scores for both MMMU_DEV and MMMU_VAL.
Repository: open-compass/VLMEvalKit
Issue: Model name in demo
Root cause: Documentation Deficiency

I have used the demo given 
# Demo
from vlmeval.config import supported_VLM
model = supported_VLM['idefics_9b_instruct']()
# Forward Single Image
ret = model.generate(['assets/apple.jpg', 'What is in this image?'])
print(ret)  # The image features a red apple with a leaf on it.
# Forward Multiple Images
ret = model.generate(['assets/apple.jpg', 'assets/apple.jpg', 'How many apples are there in the provided images? '])
print(ret)  # There are two apples in the provided images.
It is working fine. But when I change the model other than idefics it is giving key error. Could u please clarify?

Hi, @chandrabhuma,
would you please tell me which model are u using when encountered the error? 

from vlmeval.config import supported_VLM
model = supported_VLM['deepseek-ai/deepseek-vl-7b-chat']()
--------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
[<ipython-input-3-97bd06052077>](https://localhost:8080/#) in <cell line: 3>()
      1 # Demo
      2 from vlmeval.config import supported_VLM
----> 3 model = supported_VLM['deepseek-ai/deepseek-vl-7b-chat']()
      4 # Forward Single Image
      5 ret = model.generate(['/content/fire.jpg', 'What is in this image?'])

KeyError: 'deepseek-ai/deepseek-vl-7b-chat'
For not only deepseek but also for other several models...Should I change the path  of the model?

Thank you  for the prompt response

I have uses the same line as given in demo
model = supported_VLM['deepseek-ai/deepseek-vl-7b-chat']()

Hi, @chandrabhuma , 
That's not the correct usage. You should input the model_name that is defined in vlmeval/config.py

Now they are working..I am checking all the models. Thanks for the update and correcting me..
Repository: open-compass/VLMEvalKit
Issue: How to calculate avg score after the leaderboard update to 8 multimodal benchmarks
Root cause: Documentation Deficiency

Take GPT4-o for example, AVG(82.8, 61.6, 62.8, 56.5, 51.7, 82.2, 663/10, 66.5) = 66.3

How can i get 63.9 as the leadborad show?

Thanks!

![leadboard](https://github.com/open-compass/VLMEvalKit/assets/32925088/e2bcabd7-734f-406d-97ea-74e5936a1e76)



Hi, @wanxinzzz , 
The overall performance in this screenshot is wrong and has already been fixed now, please refresh the page to check the right results.
Repository: open-compass/VLMEvalKit
Issue: requirements.txt 不全
Root cause: Documentation Deficiency

比如没有 xformers 和 flash-attn
目前跑起来比较困难
也没有dockerfile

Hi, @max-yue ,
1. 考虑到用户不一定会使用该代码库评测所有模型，因此，我们并未将所有模型的 requirements 都加入。一个我们采取的实践是：如果你可以正常运行该 VLM 的 DEMO，那么你应该能够在 VLMEvalKit 中正常使用这一模型
3. Dockerfile 问题：抱歉我们近期忙于一些其他任务，我们会在接下来一段时间 (希望在 6 月内) 完成 Dockerfile 的提供

Duplicate of #190
Repository: open-compass/VLMEvalKit
Issue: 有关模型和数据集下载的问题
Root cause: Documentation Deficiency

非常感谢你们的工作！
我有个问题，是关于数据集和模型的，我是否能在运行执行评估任务命令之前就直接下载模型和数据集呢，由于我的服务器网络不能在评测的时候同步下载对应的数据集和模型，因此我想提前下载所需要的数据，并拷贝到项目里面，之后直接进行评估。请问这样是可行的吗？如果可行，希望您给我相关的建议或者操作方法。谢谢！

期待您的回复！

所有数据集的下载地址在 VLMEvalKit/vlmeval/utils/dataset_config.py 的 dataset_URLs 中，可以采用 wget 命令下载；下载完成后同步到目标机器的 ~/LMUData 目录下就可以

> 所有数据集的下载地址在 VLMEvalKit/vlmeval/utils/dataset_config.py 的 dataset_URLs 中，可以采用 wget 命令下载；下载完成后同步到目标机器的 ~/LMUData 目录下就可以

感谢回复！
我数据集已经下载到了VLMEvalKit项目的LMUData文件里面，模型也按照(例如：InternVL-Chat-V1-1)下载了
VLMEvalKit
--LMUData
--InternVL-Chat-V1-1
--......
这样是合理的吗？

> 所有数据集的下载地址在 VLMEvalKit/vlmeval/utils/dataset_config.py 的 dataset_URLs 中，可以采用 wget 命令下载；下载完成后同步到目标机器的 ~/LMUData 目录下就可以

我按照我提到的目录去设置了数据集和模型，可是他还是会找不到数据集.tsv，仍然尝试下载，请问是怎么回事呢？是不是评估要进行特定的网络检测呢
![image](https://github.com/open-compass/VLMEvalKit/assets/129817358/02cb9f41-0a82-4294-9d0f-2e8c6a48aa0b)


数据集 LMUData 应该放在 HOME 目录也就是 ~/LMUData 位置

> 数据集 LMUData 应该放在 HOME 目录也就是 ~/LMUData 位置

好的，谢谢！


请问一下~ 数据集的dataset_URLs是个tsv文件，怎么将数据集和对应的文本信息下载下来，另外最新版本 VLMEvalKit/vlmeval/utils/dataset_config.py已经不存在，怎么提前下载模型和数据集，再拿到脱网的环境下运行？

> 请问一下~ 数据集的dataset_URLs是个tsv文件，怎么将数据集和对应的文本信息下载下来，另外最新版本 VLMEvalKit/vlmeval/utils/dataset_config.py已经不存在，怎么提前下载模型和数据集，再拿到脱网的环境下运行？

I found the ULR in **VLMEvalKit/vlmeval/dataset/image_mcq.py**
Repository: open-compass/VLMEvalKit
Issue: AesBench support
Root cause: Unimplemented Feature Gap

This is very meaningful work. Our AesBench (mostly MCQ) mainly evaluates the aesthetic perception ability of MLLM. If you want to add it to your framework, we can provide data support.  https://github.com/yipoh/AesBench



Hi, @yipoh , 
I do appreciate the efforts you made on aesthetics evaluation of MLLMs. To add the benchmark to VLMEvalKit, please prepare a TSV file that is compatible with VLMEvalKit to me. For further details, you can reach me via email (dhd.efz@gmail.com) or wechat (dhd_efz).
Repository: open-compass/VLMEvalKit
Issue: Judge model seems not working
Root cause: Documentation Deficiency

I have been using CogVLM2 run on TextVQA and I got accuracy on 4.06 while the InternVL and other models like MiniCPM got roughly 70 or 80 accuracy. I checked the model output with ground truth answer and found that most answer is correct but which too much useless description which cannot easily match the short answer. Then I use my local Qwen model to be the judge model but the acc is still the same. But the evaluation process is roughly 1 hour longer on the A40 than before. How can I check if the judge model is working correctly?

Hi, @ruifengma , the use of judge model has not been officially supported for datasets like TextVQA. If you want to adopt LLMs to help the evaluation of VQA problems, you need to implement your own logic in evaluation scripts like vqa_eval.py

Thanks @kennymckormick , now I see. Btw, where could I find the dataset supporting list for judge model? 

Hi, @ruifengma , the following datasets adopt LLMs as choice extractors or judges:
1. Multi-choice benchmarks or Yes-or-No benchmarks: as choice extractor
2. Benchmarks that adopt LLMs as judges: MathVista, MMVet, LLaVABench
Repository: open-compass/VLMEvalKit
Issue: evaluate TextVQA_VAL  error
Root cause: Validation Gap

# command
python run.py --data TextVQA_VAL --model cogvlm2-llama3-chat-19B --verbose

# env
python 3.8
torch 2.1.0+cu118
VLMEvalKit: the lastest main branch

# error message
Did not detect the .env file at /code/lmdeploy_test/VLMEvalKit/.env, failed to load. 
Did not detect the .env file at /code/lmdeploy_test/VLMEvalKit/.env, failed to load. 
/code/lmdeploy_test/VLMEvalKit/vlmeval/utils/dataset.py:60: UserWarning: The dataset tsv is not downloaded
  warnings.warn('The dataset tsv is not downloaded')
TextVQA_VAL.tsv: 1.22GB [03:34, 5.70MB/s]                                                                                                   
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/code/lmdeploy_test/VLMEvalKit/vlmeval/tools.py", line 266, in decode_img_omni
    decode_base64_to_image_file(im, p)
  File "/code/lmdeploy_test/VLMEvalKit/vlmeval/smp/vlm.py", line 71, in decode_base64_to_image_file
    image = decode_base64_to_image(base64_string, target_size=target_size)
  File "/code/lmdeploy_test/VLMEvalKit/vlmeval/smp/vlm.py", line 61, in decode_base64_to_image
    image_data = base64.b64decode(base64_string)
  File "/usr/lib/python3.8/base64.py", line 87, in b64decode
    return binascii.a2b_base64(s)
binascii.Error: Invalid base64-encoded string: number of data characters (5) cannot be 1 more than a multiple of 4
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "run.py", line 160, in <module>
    main()
  File "run.py", line 79, in main
    model = infer_data_job(
  File "/code/lmdeploy_test/VLMEvalKit/vlmeval/inference.py", line 164, in infer_data_job
    model = infer_data(
  File "/code/lmdeploy_test/VLMEvalKit/vlmeval/inference.py", line 72, in infer_data
    dataset = TSVDataset(dataset_name)
  File "/code/lmdeploy_test/VLMEvalKit/vlmeval/utils/dataset.py", line 84, in __init__
    self.data_path = prep_tsv(dataset)
  File "/code/lmdeploy_test/VLMEvalKit/vlmeval/utils/dataset.py", line 71, in prep_tsv
    LOCALIZE(data_path, local_path)
  File "/code/lmdeploy_test/VLMEvalKit/vlmeval/tools.py", line 291, in LOCALIZE
    ret = pool.map(decode_img_omni, tups)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
binascii.Error: Invalid base64-encoded string: number of data characters (5) cannot be 1 more than a multiple of 4

Hi, @LRHstudy , 
The problem has been fixed now.
Repository: open-compass/VLMEvalKit
Issue: SEEDBench2 and SEEDBench2_Plus give the same result
Root cause: Configuration Error

I tried to run the evaluation on SEEDBench2 and SEEDBench2_Plus. The result are all pointed to SEEDBench2_Plus. Is that normal?

Can you provide more details? We evaluate SEEDBench2 and SEEDBench2_Plus. The result corresponds to the dataset.

Yes @junming-yang , here you go. 
```
CUDA_VISIBLE_DEVICES=2 python run.py --data SEEDBench2 --model InternVL-Chat-V1-5 --verbose
2024-07-01 09:56:47,886 - Evaluation - ERROR - OPENAI_API_KEY is not set properly, will use exact matching for evaluation
2024-07-01 09:56:47,886 - Evaluation - INFO - Evaluating ./InternVL-Chat-V1-5/InternVL-Chat-V1-5_SEEDBench2_Plus.xlsx
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2277/2277 [00:00<00:00, 30698.41it/s]
2024-07-01 09:56:50,884 - Evaluation - INFO - multiple_choice_eval successfully finished evaluating ./InternVL-Chat-V1-5/InternVL-Chat-V1-5_SEEDBench2_Plus.xlsx, results saved in ./InternVL-Chat-V1-5/InternVL-Chat-V1-5_SEEDBench2_Plus_acc.csv
2024-07-01 09:56:50,884 - Evaluation - INFO - Score:
2024-07-01 09:56:50,884 - Evaluation - INFO -   split   Overall     chart       map       web
0  none  0.666227  0.653086  0.576208  0.792424
```
as you could see, I tried to run SEEDBench2, but it gives me SEEDBench2_Plus

The command is correct. The evaluation is false. I recommend checking the following:
(1) whether `~/LMUData` dir have correct SEEDBench2.tsv file.
(2) whether the model dir exists SEEDBench2 prediction file `InternVL-Chat-V1-5_SEEDBench2.xlsx`.
(3) remove all SEEDBench2_Plus result files from the model dir and rerun this command.
(4) The correct InternVL-Chat-V1-5 SEEDBench2 result is about 0.59.

Thanks @junming-yang , problem solved. The codebase I have seems a little bit old (half a month ago probably) and there is only SEEDBench2_Plus in the config file, there is actually no SEEDBench2, a little bit wired since it seems that SEEDBench2_Plus should the latest version of SEEDBench family if I'm not wrong. Anyway, thanks for help.
Repository: open-compass/VLMEvalKit
Issue: 加载本地训练模型
Root cause: Documentation Deficiency

您好，我想使用您的框架测试本地模型和本地数据集，但是当我修改config.py中的模型路径后，框架还是会尝试从huggingface下载模型，我该怎么修改才能使用本地模型呢？

`config.py` 中 model_path 对应 transformers 库 `AutoModel.from_pretrained(model_path)` 的方法， model_path 可以是 repo_id 或本地文件夹地址，可以参照 transformers 从本地加载模型的方式进行设置。

感谢回复，但是当我进行实验时，出现了另外的问题：模型不能返回文字，返回了无意义的向量，请问这是什么问题？
![报错](https://github.com/open-compass/VLMEvalKit/assets/71713799/21d6ff4c-1c17-4315-b618-52a4e454ea12)


这是使用你自定义的vlm进行推理吗，请检查下generate_inner函数的返回值，如果异常应该是vlm内部的返回有问题

generate_inner函数返回的内容也是这样的元素，这是因为我的模型导致的吗？但是我的模型是从官方下载的千问VL，应该不会出现问题吧


应该是模型导致的，如果是我们已经支持的模型，请提供具体的运行命令及模型名称；如果是自定义的模型，请检查模型。
Repository: open-compass/VLMEvalKit
Issue: Math_Vista result is not summarized
Root cause: Validation Gap

After running the evaluation process on MathVista. There is only a .xlsx file which contains the answer and predictions from model but there is not a finalized result (e.g. accuracy)

MathVista, MMVet and LLaVABench need to set your API Key to evaluation. You can refer to [QuickStart](https://github.com/open-compass/VLMEvalKit/blob/main/docs/en/Quickstart.md) for more details.

> MathVista, MMVet and LLaVABench need to set your API Key to evaluation. You can refer to [QuickStart](https://github.com/open-compass/VLMEvalKit/blob/main/docs/en/Quickstart.md) for more details.

yes, I actually followed the instruction and a .env file has been created with 
```
OPENAI_API_KEY=empty
OPENAI_API_BASE=http://10.79.13.172:8100/v1/chat/completions
LOCAL_LLM=Qwen1.5-72B-Chat
```
but got
`Evaluation - ERROR - OPENAI_API_KEY is not set properly, will use exact matching for evaluation`

OPENAI_API_KEY can not be empty. You should start with `sk-`. And check your API Key in the correct path `$VLMEvalKit/.env`.  You can also set these params to `~/.bashrc` and run `openai_key = os.environ.get('OPENAI_API_KEY', None)` to check.

` assert isinstance(key, str) and key.startswith('sk-')`
OK, got in the source code. Thanks
Repository: open-compass/VLMEvalKit
Issue: [Request] Consider integrating the following list?
Root cause: Unimplemented Feature Gap

- [VQAv2](https://arxiv.org/pdf/1612.00837v3)

- [TallyQA: Answering Complex Counting Questions](https://arxiv.org/pdf/1810.12440)

- [GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering](https://arxiv.org/pdf/1902.09506) 

- [OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge](https://arxiv.org/pdf/1906.00067)

- [A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge](https://arxiv.org/pdf/2206.01718)

A-OKVQA has been supported now.
We will add the rest ones into our TODO list.

any updates on these benchmarks?

I wondered if the following benchmarks are in the coming list:
1. MMVP Benchmark: https://huggingface.co/datasets/MMVP/MMVP
2. CV-Bench: https://huggingface.co/datasets/nyu-visionx/CV-Bench

> A-OKVQA has been supported now. We will add the rest ones into our TODO list.

The A-OKVQA is not found in the latest version of the code.
Repository: open-compass/VLMEvalKit
Issue: 如何多卡并行测评？How to evaluate on multiple cards in parallel?
Root cause: Documentation Deficiency

## 问题描述

我需要测评一个 72B 的模型，这个模型放单张卡上会 OOM, 至少需要两块卡才能放下， 但是使用 `torchrun`  来运行时貌似只能放单张卡。

## 需要的帮助

如何使每个进程占用多张卡进行并行推理？

---

## Problem Description

I need to evaluate a 72B model, which will OOM if placed on a single card. It requires at least two cards to fit, but when using `torchrun` to run it, it seems that only one card can be used.

## Assistance Required

How can I make each process occupy multiple cards for parallel inference?



You can set `device_map='auto'` and use the command `CUDA_VISIBLE_DEVICES=0,1 python3 run.py --model xxx --data xxx`. Please refer to [Issues 224](https://github.com/open-compass/VLMEvalKit/issues/224).

> You can set `device_map='auto'` and use the command `CUDA_VISIBLE_DEVICES=0,1 python3 run.py --model xxx --data xxx`. Please refer to [Issues 224](https://github.com/open-compass/VLMEvalKit/issues/224).


It worked. Thank you very much.