zhiminy/EvalEng huggingface dataset:
Repository: open-compass/VLMEvalKit
Issue: [Bug] VRAM is not released when using multiple model
Root cause: Resource Mishandling

Hi, thanks for your contribution on building this evluation kit. I used it for reproducing Qwen2.5-VL-3B-Instruct and Qwen2.5-VL-7B-Instruct, recently. I construct a model config with this two model and one dataset, which is shown below. However, the VRAM allocated from previous Qwen2.5-VL-3B-Instruct model seems not released, after this model is done. As we can see, the VRAM is nearly the sum of 3B model and 7B model.

---

config:

```json
{
    "model": {
        "Qwen2.5-VL-3B-Instruct-edge": {
            "class": "Qwen2VLChat",
            "model_path": "Qwen/Qwen2.5-VL-3B-Instruct",
            "min_pixels": 3136,
            "max_pixels": 802816,
            "use_custom_prompt": false
        },
        "Qwen2.5-VL-7B-Instruct-edge": {
            "class": "Qwen2VLChat",
            "model_path": "Qwen/Qwen2.5-VL-7B-Instruct",
            "min_pixels": 3136,
            "max_pixels": 802816,
            "use_custom_prompt": false
        }
    },
    "data": {
        "MMMU_DEV_VAL": {
            "class": "MMMUDataset",
            "dataset": "MMMU_DEV_VAL"
        }
    }
}
```

---

nvidia-smi:

```txt
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.05              Driver Version: 560.35.05      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090 D      Off |   00000000:01:00.0 Off |                  Off |
| 37%   62C    P2            286W /  425W |   23388MiB /  24564MiB |     96%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A   3805914      C   python                                      23378MiB |
+-----------------------------------------------------------------------------------------+
```



Hi, @hebangwen , 
Thanks for point out, we will see how to fix the problem. 
For now, a simple workaround is to write a for loop in bash and evaluate a single model each time.
Repository: open-compass/VLMEvalKit
Issue: (feature request) can we add load_dotenv() as a small quality of life improvement?
Root cause: Unimplemented Feature Gap

Hi OpenCompass VLMEvalKit team,

Thank you for your hard work on this project! I have a very minor feature request - can we add[ load_dotenv](https://pypi.org/project/python-dotenv/) to make it easier for users to run without explicitly setting their `OPENAI_API_KEY` env variables in the terminal before their run? 

This way, a user can add their key to the variable once in a .env file and it will automatically be loaded.
Happy to open a pull request if helpful.

Hi, @kdu4108 ,
That's a good idea and we may add it on our TODO list. If you are also interested to do so, welcome to create a PR  for it. 

The feature has been implemented in #120 . See https://github.com/open-compass/VLMEvalKit/blob/main/Quickstart.md for  the detailed usage.
Repository: open-compass/VLMEvalKit
Issue: 测试速度问题
Root cause: Documentation Deficiency

  感谢您的工程。尝试使用VLMEvalKit库测试在垂直领域微调后的qwen2.5-vl系列模型，使用指令python run.py --data MMBench_DEV_EN --model Qwen2.5-VL-3B-Instruct --verbose，其中模型配置了本地的地址，每次单独测试1个未微调/微调的权重，发现推理速度特别慢，显示8张卡都有显存占用，但是sm绝大部分都是0，只有偶尔1个1的情况，显示时间估计要几十个小时。之后限定单张卡情况下，显示时间还缩短了1/3左右。

问题：  
1.想问一下您这个可能是什么情况呢（之前在别的服务器上测试qwen2.5-vl-3b和使用molmo，速度都比较快，显示1个小时内）？
2.问一下您使用lmdeploy、vllm加速的时候，看到不同文档中有两种写法，可以选择启动服务或不启动，直接调用--use-vllm，这个一般建议使用哪一种呢？
3.问一下--api-nproc 是不是类似起到了多批次的效果？
4.llm那边使用opencompass测试多次重复实验的结果不一致，一般是怎么处理这个随机性呢？

你好，多卡推理的话，建议使用`torchrun --nproc-per-node={num_gpu} run.py --data MME --model qwen_chat --verbose`
`, 因为 `python` 只会在某一张GPU上运行。
更多信息参考[quick start](https://github.com/open-compass/VLMEvalKit/blob/main/docs/zh-CN/Quickstart.md)

请问你解决了这个问题吗，我使用本地的模型地址也很慢，使用默认的就很快
Repository: open-compass/VLMEvalKit
Issue: add AI/ML API as provider
Root cause: Documentation Deficiency

Hi!

I'm Sergey from the Integrations team over at [AI/ML API](https://aimlapi.com/), a startup with 150K+ users, providing over 300 AI models in one place

Your project looks dope, so we'd like a native integration. We already got integrations with Langflow, AutoGPT, and LiteLLM - so our integrations team is pretty seasoned :)

Say you're interested, and we'll test the compatibility, update the code/docs to include us, create a PR and add a tutorial on using VLMEvalKit with AI/ML API to our docs

If you guys give us a green light, we could get on it 👷

Sure, we are more than happy to help integrate this feature. BTW, if you just want to add want to add AI/ML API as an API provider and you API platform is compatible with OPENAI, you can simply set `OPENAI_API_KEY` & `OPENAI_API_BASE` in the `.env` file to achieve this.
Repository: open-compass/VLMEvalKit
Issue: InternVL3如何限制图片的max_pixels
Root cause: Documentation Deficiency

InternVL3如何限制图片的max_pixels，有和Qwen2.5VL类似的设置参数吗？

qwen有类似这样的设置：

    "Qwen2.5-VL-3B-Instruct": partial(
        Qwen2VLChat,
        model_path="Qwen/Qwen2.5-VL-3B-Instruct",
        min_pixels=1280 * 28 * 28,
        max_pixels=16384 * 28 * 28,
        use_custom_prompt=False,
    ),

你好，InternVL系列是通过 `max_num` 来控制输出像素数量的，请参考
1. [pixel values loading](https://github.com/open-compass/VLMEvalKit/blob/57ca395fd8f4e56fae90c8fad9dbef6e38fe408f/vlmeval/vlm/internvl/internvl_chat.py#L398)
2. [load image function](https://github.com/open-compass/VLMEvalKit/blob/57ca395fd8f4e56fae90c8fad9dbef6e38fe408f/vlmeval/vlm/internvl/utils.py#L89)
3. [dynamic_processing function](https://github.com/open-compass/VLMEvalKit/blob/57ca395fd8f4e56fae90c8fad9dbef6e38fe408f/vlmeval/vlm/internvl/utils.py#L50)
Repository: open-compass/VLMEvalKit
Issue: tsv file generation from huggingface vqa dataset
Root cause: Documentation Deficiency

Sir 
Could you please tell me the procedure of converting a vqa dataset from huggingface 
to .tsv file. What are the steps?

Hi, please refer to the [Development Guide](https://github.com/open-compass/VLMEvalKit/blob/main/docs/en/Development.md) for more details
Repository: open-compass/VLMEvalKit
Issue: Evaluation of custom models and datasets.
Root cause: Unimplemented Feature Gap

VLMEVALKIT is a pretty convenient evaluation tool for MLLMs. I hope that the esteemed authors can create a framework for VLMEVALKIT that supports the evaluation of custom models and custom datasets. This framework can define a unified MLLM input-output interface and the conversion format for datasets.

Hi, @juxingyiwan ,
we are going to first implement for custom multiple-choice datasets, and will create a tutorial for it. Stay Tuned!

Hi, @juxingyiwan , custom multiple-choice dataset is supported now. Just prepare your dataset following the typical format of MCQ datasets in VLMEvalKit, place it under ~/LMUData, and use the tsv file name for running evaluation command.
Repository: open-compass/VLMEvalKit
Issue: Finetuned LLaVA Model output
Root cause: Configuration Error

Hi, I finetuned the LLaVA-v1.5-13B model and managed to benchmark it on a custom dataset. 

I noticed, however, that when benchmarking the finetuned version on that dataset, I didn't get the QA pairs of each image, like I did during the standard model evaluation on that same dataset.

Is there a way to activate those?

Thank you.  

Hi, to evlauate a finetuned model, you need to first config it in `config.py` and use the corresponding model name to complish evaluation ,see #914 for more details.

Hi! I've managed to do that already. The only difference is that during the evaluation, I don't get the QA pairs to show up on the console.

did you check the `generate_inner` function of the model definition? it is better to provide working cases to better locate the potential bugs.

Yes. This happens with LLaVA only. I'll give you an example, below, where to the left we have part of the evaluation of the standard model and to the right we have part of the evaluation of the finetuned  version. On the latter, I don't get the answer like I do in the standard model case.

<img width="844" alt="Image" src="https://github.com/user-attachments/assets/a0348c98-2f89-4c18-9599-bc5a482c0ff1" />
Repository: open-compass/VLMEvalKit
Issue: 使用lmdeploy openai api运行到一半总是会卡住
Root cause: Resource Mishandling

使用lmdeploy openai api的方式运行internVL3-1B + mmbench v11,运行几帧数据后总是会卡住，评测的大部分结果都会报错Failed to obtain answer via API. 遇到这种情况一般怎么解决
Repository: open-compass/VLMEvalKit
Issue: Why doesn’t VizWiz’s `build_prompt` include the short-answer instruction like `ImageVQADataset`?
Root cause: Documentation Deficiency

I noticed that in `ImageVQADataset` you override `build_prompt` to append:

```text
Answer the question using a single word or phrase.
```

This ensures the model knows to produce a short answer. However, the `VizWiz` class inherits `build_prompt` directly from `ImageBaseDataset` and doesn’t include this instruction, which leads to long descriptive outputs that always score 0 under VQA Accuracy.

Could you clarify why this short-answer prompt is added for `ImageVQADataset` but not for `VizWiz`? Is there a reason for treating VizWiz differently, or would you consider adding the same instruction to improve scoring?

Empirical Observation:
After manually adding this line to VizWiz.build_prompt, Ovis2-8B’s VQA Accuracy jumped from 3.09% to 39.77% on the VizWiz evaluation set.

Thank you!

Yes I also noticed this. On SmolVLM 256 it jumped from 5% to 26%. 

same problem

> same problem
I find the correct prompt in https://github.com/EvolvingLMMs-Lab/lmms-eval, we should override build_prompt to append:
`"\nWhen the provided information is insufficient, respond with 'Unanswerable'.\nAnswer the question using a single word or phrase."
`
after that, I got the reasonable metric

I encountered the same issue—without adding this extra prompt, the metric drops straight to 0. I think this is a bug and should be fixed. @kennymckormick
Repository: open-compass/VLMEvalKit
Issue: A Better Evaluation Entry for OpenAI-Style API Models
Root cause: Unimplemented Feature Gap

### A Better Evaluation Entry for OpenAI-Style API Models

In practice, deployment and evaluation are typically separated to avoid dependency redundancy caused by supporting various models. The more models supported, the more complex the dependencies become. This approach also facilitates the evaluation of extremely large models.

I noticed that the framework supports OpenAI-style API models for evaluation through the `GPT4V` class. However, the user experience still needs improvement. Specifically:
1. To test a model, you need to **modify `config.py`** and register a `model_name`.
2. Diverse parameters (e.g., temperature, timeout) require manual adjustments.

Could you provide an interface like this?

```python
python run_api.py \
  --model-name "vllm_qwen_2.5-7b" \
  --base-url "xxxxxxxx" \
  --api-key "xxxxx" \
  --max-token-out 16000 \
  --min-pixels 3k \
  --max-pixels 100w \
  --temperature 0.1 \
  --top-p 0.9 \
  --data MME \
  --work-dir ./outputs
```

With such an entry point, models like those in [VLMEvalKit#1093](https://github.com/open-compass/VLMEvalKit/pull/1093) could be automatically supported without additional modifications. This would allow frameworks like vLLM, SGLang, and LMDeploy to support more models seamlessly.


在实际情况下，一般会部署和评测分离，这样可以摆脱支持各种各样的模型导致的依赖冗余，支持的模型越多，以来越复杂，以及更方便的支持超巨大模型的评测。而且一次也只会评测一个API, 多个模型就执行多次命令。

我看到框架里可以通过`GPT4V` 这个类来支持openai 格式的API模型的评测。但使用体验还是不够好。具体的：
1. 测试一个模型，需要**修改config.py**，注册一个model_name
2. 对多样性的参数需要手动修改，比如 temperature timeout


能否提供这样的接口：

```python
python run_api.py \
  --model-name "vllm_qwen_2.5-7b" \
  --base-url "xxxxxxxx" \
  --api-key "xxxxx" \
  --max-token-out 16000 \
  --min-pixels 3k \
  --max-pixels 100w \
  --temperature 0.1 \
  --top-p 0.9 \
  --data MME \
  --work-dir ./outputs
```


有了这样的入口，类似：https://github.com/open-compass/VLMEvalKit/pull/1093 这样模型就可以自动支持。并不需要额外支持。vllm， sglang 以及 lmdeploy 会支持更多的模型。

并且使用 `GPT4V`  使用的环境变量会和 `judge_model` 的环境变量混合在一起，引起一些不必要的错误和失败。

I think it would be helpful to add a `--provider` parameter to make things more flexible when switching between APIs.
Something like that:

> ```bash
> python run_api.py \
>   --model-name "vllm_qwen_2.5-7b" \
>   --provider "OpenAI/Qwen/Claude/etc." \
>   --api-key "xxxxx"
> ```

As a large model developer, I may prefer to test the models I develop more often, while for released models, I would rely on official results or run them only once.

For OpenAI-style models, the differences between providers mainly lie in the `model_name`, `base_url` and `API_key`. The --provider option would likely only help you reference environment variables. Of course, different models may require slightly different parameters—for example, OpenAI uses "detail" for image parameters, while QWE uses parameters like min_pixels and max_pixels, etc.

Any progress on this? I am not a developer, but I am searching for a good model / quantization for my home use.
I wanted to use this project and evaluate models running with llama.cpp which offer openai compatible API. This would help me.
Repository: open-compass/VLMEvalKit
Issue: The code for evaluating open question of MMMU is completely wrong
Root cause: Algorithmic Error

It turns all open question of MMMU into "A" choice, and set all non-upper case answer to "A", which leads to higher score if the model outputs "A" by accident.

Hi, @xwwu2015 ,
The current implementation is a fast one and the problem you mentioned do occur, lead to not so accurate performance results for open-ended questions. It will not lead to completely wrong results, since it affects less than 5% questions in MMMU.
We would like to refactor the implementation in the future, you are also welcomed to create a PR if you need this feature in VLMEvalKit urgently.
Repository: open-compass/VLMEvalKit
Issue: 如何使用本地测试数据集
Root cause: Documentation Deficiency

因为我们的服务器无法直接连接到huggenface，所以我们下载MMBench-Video数据集到本地然后放置到/home/amsr/LMUData目录下。然后使用vllm启动api，如下所示：

`python3 -m vllm.entrypoints.openai.api_server --model "~/Qwen2-VL-7B-Instruct" --dtype=half` 

运行VLMEvalKit：
`python3 run.py --data  MMBench-Video --model lmdeploy --verbose --api-nproc 16`

> 报错如下：An error happened while trying to locate the files on the Hub and we cannot find the appropriate snapshot folder for the specified revision on the local disk. Please check your internet connection and try again.

可否提供一个使用vllm运行本地数据集进行评测的例子？
谢谢

我觉得你可以参考一下EvalScope上的有关说明：https://evalscope.readthedocs.io/zh-cn/latest/advanced_guides/custom_dataset/vlm.html

你好，请问你解决了这个问题了嘛
Repository: open-compass/VLMEvalKit
Issue: VLMblind测试精度，请问可以share下这个测试集的精度么；我自己跑出来和seed-1.5-vl报告中精度相差比较大
Root cause: Documentation Deficiency

报告qwen-72b中精度为69，自己跑出来是60

我们测试精度在 61% 左右，确实低于 SEED 技术报告中所报告结果。
Repository: open-compass/VLMEvalKit
Issue: Link failed
Root cause: External Dependency Breakage

This link failed: https://opencompass.openxlab.space/utils/VLMEval/SpatialEval.tsv

Sorry that the previous link is invalid. We have fixed it now and you can retry with the link. 

Close the issue since the problem is solved. Feel free to re-open if needed.
Repository: open-compass/VLMEvalKit
Issue: PRM的评测
Root cause: Unimplemented Feature Gap

框架支持VLM的PRM的评测吗？例如visualPRM的visualprocessbench这个benchmark？

Hi，我们暂时还没包含的对 visualprocessbench 的支持

@Weiyun1025 可以帮忙看下是否要添加这个 benchmark 的支持

> Hi，我们暂时还没包含的对 visualprocessbench 的支持
> 
> [@Weiyun1025](https://github.com/Weiyun1025) 可以帮忙看下是否要添加这个 benchmark 的支持



我们这个月争取支持一版VisualProcessBench的评测

十分感谢
Repository: open-compass/VLMEvalKit
Issue: When running Qwen2.5-VL-72B on MUIRBench with 8 A100 GPUs, a CUDA out-of-memory error occurs.
Root cause: Configuration Error

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 5.34 GiB. GPU 0 has a total capacity of 39.50 GiB of which 3.32 GiB is free. Including non-PyTorch memory, this process has 36.17 GiB memory in use. Of the allocated memory 35.53 GiB is allocated by PyTorch, and 141.78 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)



How to Solve the CUDA Out-of-Memory Issue


Hi, @hlp2020 

Qwen2.5-VL-72B is large so that you can not run one model instance on a single GPU. You can try to launch the evaluation with `python`  command: 

`python run.py --model Qwen2.5-VL-72B-Instruct --data {dataset_names}`

Then the model will use all available GPUs for model inference.

so what is your result？
Repository: open-compass/VLMEvalKit
Issue: Discrepancy in CountBenchQA Data Count Compared to Original Paper
Root cause: Documentation Deficiency

Hello, Thank you for your great work on this repository. I’ve been using it recently and noticed something I’m unsure about. In the CountBenchQA dataset, I found that there are only 487 entries, while the original paper mentions 540.

I’m wondering if I might be misunderstanding something or if there’s a step I may have missed. Could you please clarify this discrepancy?

Thanks in advance for your help!
Repository: open-compass/VLMEvalKit
Issue: VideoMMLU dataset file not found
Root cause: Documentation Deficiency

neither dataset of VideoMMLU (QA and CAP versions) worked when I tried to evaluate my models
![Image](https://github.com/user-attachments/assets/208d3a25-181b-440a-960f-b7704faef7dc)

Hi @beatriceadel, I would like to ask what your evaluation commands are. Perhaps because you pass the wrong dataset name into the data args (using Video_MMLU_QA/Video_MMLU_CAP can avoid this problem)

@FangXinyu-0913  
Thank you for the quick response!
I did mistakenly pass "VideoMMLU_*" instead of "Video_MMLU_*" as the --data args. I will try again with the suggested correct args.

Another question, is "VideoMMLU_*_16frame" not the right data args either?
I also tried with this args following what's written in `video_dataset_config.py` but got a different error message, basically saying the dataset is not supported 
(sorry I forgot to capture the error message when I ran this)

Hi @beatriceadel. Thank you for pointing out this issue. We have fixed it in the latest commit and unified all VideoMMLU_ to Video_MMLU_ to avoid ambiguity. You can now try using the command like `python run.py --model Qwen2.5-VL-7B-Instruct-ForVideo --data Video_MMLU_QA_16frame --verbose` to evaluate.

@FangXinyu-0913 Understood! Thank you very much.
Repository: open-compass/VLMEvalKit
Issue: MathVista测试问题
Root cause: Documentation Deficiency

请问MathVista-mini的Prefetch rate和Acc有什么区别？我测试发现Prefetch rate为52.8，acc只有44.1

你怎么计算mathvista-mini的prefetch rate跟acc的，我看他这里只能用gpt-4做判断

> 你怎么计算mathvista-mini的prefetch rate跟acc的，我看他这里只能用gpt-4做判断

用api key


Prefetch 没有实际意义，仅为统计输出结果使用，请以 GPT 的评测结果为准。
我后续会改变默认行为至不开启 prefetch 功能。

没有key只能用Exact Matching mode了，就是精度会差几个点，exact matching mode还是有用的
Repository: open-compass/VLMEvalKit
Issue: Assertion Failure in CUDA Indexing: srcIndex < srcSelectDimSize
Root cause: Validation Gap

Hello everyone!

When trying to run the model `LLaVA-CoT` locally with the command 

```python
python VLMEvalKit/run.py --data AI2D_TEST --model LLaVA-CoT_local --verbose --mode infer --reuse
``` 

I am encountering the following error:

````python
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1500: indexSelectSmallIndex: block: [6,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
````
I have defined `LLaVA-CoT_local` in the file `config.py` the following way:
````python
"LLaVA-CoT_local": partial(llama_vision, model_path="Llama-3.2V-11B-cot"),
````

PyTorch versions:

- torch==2.7.1
- torchvision==0.22.1

Any help would be appreciated!
Repository: open-compass/VLMEvalKit
Issue: ModuleNotFoundError: No module named 'llava'
Root cause: Environment Incompatibility

I tried running llava model eval as `python run.py --data MME --model llava_v1.5_7b` but get error below

```
[2025-07-02 21:29:54] ERROR - misc.py: load_env - 214: Did not detect the .env file at /home/ubuntu/VLMEvalKit/.env, failed to load.
[2025-07-02 21:29:55] ERROR - misc.py: load_env - 214: Did not detect the .env file at /home/ubuntu/VLMEvalKit/.env, failed to load.
[2025-07-02 21:29:55] WARNING - RUN - run.py: main - 216: --reuse is not set, will not reuse previous (before one day) temporary files
True
[2025-07-02 21:29:56] CRITICAL - llava.py: __init__ - 23: Please install llava from https://github.com/haotian-liu/LLaVA
[2025-07-02 21:29:56] ERROR - RUN - run.py: main - 505: Model llava_v1.5_7b x Dataset MME combination failed: No module named 'llava', skipping this combination.
Traceback (most recent call last):
  File "/home/ubuntu/VLMEvalKit/run.py", line 373, in main
    model = infer_data_job(
  File "/home/ubuntu/VLMEvalKit/vlmeval/inference.py", line 203, in infer_data_job
    model = infer_data(
  File "/home/ubuntu/VLMEvalKit/vlmeval/inference.py", line 123, in infer_data
    model = supported_VLM[model_name](**kwargs) if isinstance(model, str) else model
  File "/home/ubuntu/VLMEvalKit/vlmeval/vlm/llava/llava.py", line 26, in __init__
    raise err
  File "/home/ubuntu/VLMEvalKit/vlmeval/vlm/llava/llava.py", line 20, in __init__
    from llava.model.builder import load_pretrained_model
ModuleNotFoundError: No module named 'llava'
```

Installation:

A10 GPU with Driver Version: 535.161.08, CUDA Version: 12.2

`transformers == 4.37.0`

We need to install LLaVA.

The below code may work for you:
```bash
git clone https://github.com/haotian-liu/LLaVA.git
cd LLaVA
pip install -e .

cd ..

git clone https://github.com/open-compass/VLMEvalKit.git
cd VLMEvalKit
pip install -e .

python run.py --data MME --model llava_v1.5_7b
```

Thanks, solved it earlier but forgot to close. Thanks for the response.
Repository: open-compass/VLMEvalKit
Issue: torchrun无法生成评估结果
Root cause: Environment Incompatibility

在.env文件中设置了OPENAI_API_KEY和OPENAI_API_BASE，执行命令
`torchrun --nproc-per-node=4 run.py --data ChartQA_TEST --model Qwen2.5-VL-7B-Instruct --work-dir results/Qwen2.5-VL-7B-Instruct --judge gpt-4o --mode all`
只能生成推理结果的pkl文件，无法生成评估结果

您好，请问推理过程完成了吗？

> 您好，请问推理过程完成了吗？

更新conda环境后运行成功了

你好 请问具体是怎么解决的呢


我也遇到了这个问题，只能infer，但是infer之后会卡在api调用那里

> > 您好，请问推理过程完成了吗？
> 
> 更新conda环境后运行成功了

请问更新的啥呢，我比较怀疑不是这个原因
Repository: open-compass/VLMEvalKit
Issue: 403 Forbidden when downloading data
Root cause: Environment Incompatibility

<img width="1592" alt="Image" src="https://github.com/user-attachments/assets/100de778-21ec-465d-b3ab-bef5014471c6" />

You can have another try, I just tried to download the tsv from the following url: 
http://opencompass.openxlab.space/utils/benchmarks/MMBench/MMBench_DEV_EN.tsv

and it works well. 

It seems an issue of my network.
Repository: open-compass/VLMEvalKit
Issue: HallusionBench的最终结果如何计算
Root cause: Documentation Deficiency

HallusionBench最后的结果包含aAcc，fAcc，qAcc，请问最后的准确率是如何计算的？是否是这三者的平均？   另外MME的评测只包含preception部分吗？排行榜上好像是既包含preception又包含cognition。

Hi, @echo840 
The score we reported is the average of aAcc, fAcc, and qAcc.
The evaluation of MME also includes cognition (we name it reasoning for now).