zhiminy/EvalEng huggingface dataset:
Repository: open-compass/VLMEvalKit
Issue: CVBench dataset not recognized despite being listed as supported
Root cause: Documentation Deficiency

Hi, thank you for the great work on this repo.

I encountered an issue when trying to run evaluation using the CVBench dataset. Although the README suggestes that CVBench is supported, the script skips it entirely, citing that it’s not valid or officially supported. Additionally, it fails to load the dataset file.

Here is the relevant log output:

[2025-08-07 14:41:37] ERROR – misc.py: load_env – 215: Did not detect the .env file at ./VLMEvalKit/.env, failed to load.
[2025-08-07 14:41:41] WARNING – RUN – run.py: main – 217: --reuse is not set, will not reuse previous (before one day) temporary files
./VLMEvalKit/vlmeval/dataset/__init__.py:285: UserWarning: Dataset CVBench is not officially supported.
warnings.warn(f'Dataset {dataset_name} is not officially supported. ')
./VLMEvalKit/vlmeval/dataset/__init__.py:288: UserWarning: Data file /home/dev/LMUData/CVBench.tsv does not exist. Dataset building failed.
warnings.warn(f'Data file {data_file} does not exist. Dataset building failed. ')
[2025-08-07 14:41:41] ERROR – RUN – run.py: main – 298: Dataset CVBench is not valid, will be skipped.

Could you please confirm:
	1.	Whether CVBench is still officially supported?
	2.	If so, what is the expected location or format of the CVBench.tsv file?

Let me know if you need any additional context or reproduction steps. Thanks!
Repository: open-compass/VLMEvalKit
Issue: nproc did not redistribute the data and did not accelerate inference
Root cause: Configuration Error

Hi,i am using the command
`torchrun --nproc-per-node=2 run.py --config config.json` to inference my own dataset.

However, after implementing this,i found that it only successfully placed one model on each GPU , but the dataset wasn't split and thus the two models were testing the same data(doing the same thing concurrently).The inference speed did not improve.

How should i fix it?

Met the same problem. I found the question may be in _run.py def build_model_from_config(cfg, model_name, use_vllm=False)_:
I'm not sure what I did wrong that led to this bug, but I use the below quick fix to make it work anyway:


```
def build_model_from_config(cfg, model_name, use_vllm=False):
    import vlmeval.api
    import vlmeval.vlm
    ws_bak = os.environ.pop('WORLD_SIZE', None)
    
    try:
        config = cp.deepcopy(cfg[model_name])
        if use_vllm:
            config['use_vllm'] = use_vllm
        
        if 'class' not in config:
            return supported_VLM[model_name](**config)
        
        cls_name = config.pop('class')
        
        if hasattr(vlmeval.api, cls_name):
            model = getattr(vlmeval.api, cls_name)(**config)
        elif hasattr(vlmeval.vlm, cls_name):
            model = getattr(vlmeval.vlm, cls_name)(**config)
        else:
            raise ValueError(f'Class {cls_name} is not supported in `vlmeval.api` or `vlmeval.vlm`')
        
        return model
    
    finally:
        if ws_bak:
            os.environ['WORLD_SIZE'] = ws_bak

```
Repository: open-compass/VLMEvalKit
Issue: Longvideobench的评测是不是有bug？
Root cause: Validation Gap

        if video_llm:
            message.append(dict(type='video', value=osp.join(self.data_root, line['video_path'])))
        else:
            if not self.use_subtitle:
                with open(osp.join(self.data_root, "subtitles", line["subtitle_path"])) as f:
                    subtitles = json.load(f)

                frame_message = insert_subtitles_into_frames(
                    frames,
                    [ind_ / fps for ind_ in indices],
                    subtitles,
                    line["starting_timestamp_for_subtitles"],
                    line["duration"]
                )

                message += frame_message
            else:
                for im in frames:
                    message.append(dict(type='image', value=im))

如果模型是video_llm是不是相当于无论怎么设置其实都是不加字幕的？但longvideobench应该是默认加字幕来评测的吧。
然后if not self.use_subtitle这块感觉命名有点疑惑，虽然这个相当于默认加字幕是合理的，但是这个命名感觉让人困惑。
Repository: open-compass/VLMEvalKit
Issue: Evaluating Ola-7b RuntimeError: mat1 and mat2 shapes cannot be multiplied (16x2304 and 1152x3584)
Root cause: Interface Contract Mismatch

The error roots in `img_feat = self.get_model().mm_projector(lowres_img_features[idx],
                                                    lowres_img_sizes[idx],
                                                    highres_img_features[idx],
                                                    highres_img_sizes[idx],
                                                    modalities[idx])`  in ola/ola_arch.py.
Repository: open-compass/VLMEvalKit
Issue: The scores calculated by VLMEvalKit differ from the score calculated on the MMBench website
Root cause: Documentation Deficiency

[MMBench_DEV_EN_openai_result.xlsx](https://github.com/open-compass/VLMEvalKit/files/14662296/MMBench_DEV_EN_openai_result.xlsx)

When testing the results in this table：

The result on the MMbench website
<img width="548" alt="image" src="https://github.com/open-compass/VLMEvalKit/assets/52999149/aff68167-701d-43c7-9518-4a71c486ed69">

By VLMEvalKit：
"split","Overall","AR","CP","FP-C","FP-S","LR","RR",  
"dev","0.7328178694158075","0.7638190954773869","0.8277027027027027","0.6223776223776224","0.7610921501706485","0.4745762711864407","0.7652173913043478"



Hi, @jdy18 , 
That looks weird, please share the original prediction with me, thus I can get more information. 

> Hi, @jdy18 , That looks weird, please share the original prediction with me, thus I can get more information.

[MMBench_DEV_EN.xlsx](https://github.com/open-compass/VLMEvalKit/files/14662457/MMBench_DEV_EN.xlsx)


> Hi, @jdy18 , That looks weird, please share the original prediction with me, thus I can get more information.

I think I have found the reason. I uploaded the openai_result table to the mmbench website instead of the original file. Do you know what difference between these two files leads to the difference in evaluation results?

Oh, you cannot upload the openai_result table for evaluation, it only includes 1-pass for each question, and the corresponding evaluation result is under the VanillaEval setting.
Repository: open-compass/VLMEvalKit
Issue: Vimgolf, Cybergod and CTF environments integration, WebUI computer operation recorder integration
Root cause: Others

I have been working with an AI benchmark solving Vimgolf challenges.

It is available here:

https://github.com/James4Ever0/vimgolf-gym

With my library you are able to perform any action in the terminal and get a real time text dump of the entire terminal screen, get the location of the cursor, and take a screenshot of the terminal.

Also there is a primitive CTF library, which is very extensible and environment independent. It can be used for reinforcement learning of terminal and GUI agents.

View it here: https://github.com/James4Ever0/agi_computer_control/blob/master/gym-primitives%2Fctf%2FREADME.md

If you want your agents to earn real money, you can check out Cybergod:

https://github.com/James4Ever0/agi_computer_control/blob/master/gym-primitives%2Fcybergod%2FREADME.md

There is also a WebUI based, local/remote terminal/GUI operation recorder:

https://github.com/James4Ever0/agi_computer_control/blob/master/web_gui_terminal_recorder%2FREADME.md

Your work is inspiring and I will integrate it into my project. Hope my work will do the same for you.
Repository: open-compass/VLMEvalKit
Issue: Is Molmo or Pixmo model supported on VLMEvalKit
Root cause: Documentation Deficiency

Is Molmo or Pixmo included in VLMEvalKit yet? My cursory look did not find it.

Molmo is supported, you can find the model names in `vlmeval/config.py`

<img width="1560" height="402" alt="Image" src="https://github.com/user-attachments/assets/ef01402b-5a1e-4e41-aa30-dd1793be118e" />

Meanwhile, PixMo is the multimodal corpora used to train Molmo
Repository: open-compass/VLMEvalKit
Issue: Support Internvl3.5 Series Model
Root cause: Documentation Deficiency

Does VLMEvalkit support Internvl3.5 Series Model? Looking for ur reply!

Yeah, use the following model names to evaluate InternVL3.5 models:

<img width="1068" height="1800" alt="Image" src="https://github.com/user-attachments/assets/50e60208-5a8d-41ef-969f-f71db13d3a19" />
Repository: open-compass/VLMEvalKit
Issue: support keye-vl
Root cause: Unimplemented Feature Gap

huggingface_path :https://huggingface.co/Kwai-Keye/Keye-VL-1_5-8B

Hi @wade0604, we have already supported this model in https://github.com/open-compass/VLMEvalKit/pull/1244, and we will merge it once it is ready.
Repository: open-compass/VLMEvalKit
Issue: Small performance gap between sub and nosub on Video-MME with Qwen2.5-Omni

Hi,

I’ve been evaluating **Qwen2.5-Omni** on **Video-MME** using VLMEvalKit, and I noticed that enabling subtitles doesn’t bring much improvement compared to the no-subtitle setting. Here are my results:

- **8 fps**  
  - w/o sub: overall = **0.604**  
  - w/ sub: overall = **0.609**

- **64 fps**  
  - w/o sub: overall = **0.663**  
  - w/ sub: overall = **0.676**

The gap between “sub” and “nosub” is quite small .  
According to some blog posts and papers, I expected a clearer improvement with subtitles.  

So I’d like to ask:  
1. Is this behavior normal when using VLMEvalKit?  
2. Have others observed similar results when reproducing Qwen2.5-Omni on Video-MME?  

Any insights or reproduction numbers would be greatly appreciated!  

Thanks!
Repository: open-compass/VLMEvalKit
Issue: What is the difference between lmms-eval and your work vlmevalkit, is it same?


They are two different repos for lmm evaluations. We provide reference evaluation results (https://huggingface.co/spaces/opencompass/open_vlm_leaderboard, etc.) along with the codebase. You can have a try with both codebases and find the one that best fulfills ur need.
Repository: open-compass/VLMEvalKit
Issue: bug : NameError: name 'liststr' is not defined. Did you mean: 'toliststr'?
Root cause: Validation Gap

代码这里应该是listinstr吧？
https://github.com/open-compass/VLMEvalKit/blob/3df1d160ca9b31735faf431deee00f11d39699b9/vlmeval/vlm/internvl/internvl_chat.py#L276

@PhoenixZ810  Please help fix this issue

Thank you for taking the time to share your suggestion. We’ve addressed the issue in [PR1229](https://github.com/open-compass/VLMEvalKit/pull/1229).
Repository: open-compass/VLMEvalKit
Issue: Bug on eval InternVL3.5_8b_thinking on video bench
Root cause: Validation Gap

在使用最新的代码评测 InternVL3.5_8b_thinking时，在视频bench上（32帧），模型输出报错：

internal error happened, status codeResponseType.INPUT_LENGTH_ERROR

推理使用的是 pytorchengine，代码和[https://github.com/open-compass/VLMEvalKit/pull/1220](url)中保持一致
Repository: open-compass/VLMEvalKit
Issue: Why Does the Custom Model Output Only Inference Processes Without Relevant Scores in MathVerse?
Root cause: Documentation Deficiency

When testing a custom model using MathVerse (python run.py --data MathVerse_MINI --model My_model), I found that the output does not include any relevant scores, only some inference processes. Can you please explain why？

<img width="2330" height="1207" alt="Image" src="https://github.com/user-attachments/assets/30ff59a1-a557-498d-9d9d-35d7ade20bff" />
Repository: open-compass/VLMEvalKit
Issue: 404 Failed to obtain answer via API. <Response [404]>
Root cause: Configuration Error

<img width="1080" height="56" alt="Image" src="https://github.com/user-attachments/assets/5b4ac975-df25-4a67-824c-6b80fb8284f6" />

<img width="840" height="232" alt="Image" src="https://github.com/user-attachments/assets/adad13b2-7f87-4e9c-afff-c000c3fa3de2" />

当运行到requests.post会报404错误，但是要是用self.client = AzureOpenAI，client.chat.completions.create就能正常输出，用的是azure。
请问有人遇到这个问题吗，应该怎么改

openai.AzureOpenAI的azure_endpoint结尾如果没有"/"，在.env里面设置AZURE_OPENAI_ENDPOINT的时候要加上"/"，url才正确
Repository: open-compass/VLMEvalKit
Issue: [Feature Request] To evaluate MMMU test set, you need to transfer the xlsx output to a json file
Root cause: Unimplemented Feature Gap

Hello, 

When using VLMEvalKit with MMMU_TEST, you will generate a xlsx output file, e.g., 

<img width="914" alt="image" src="https://github.com/open-compass/VLMEvalKit/assets/33491471/56aae58c-56e4-4c2a-8542-37e607535b00">

This format cannot be accepted by the [online MMMU EvalAI server](https://eval.ai/web/challenges/challenge-page/2179/overview). The server requires [this json format](https://github.com/MMMU-Benchmark/MMMU/blob/main/eval/example_outputs/llava1.5_13b_val.json).

The following code can transfer the xlsx file to the required json format:
```
import pandas as pd
import json

# 读取xlsx文件
def read_xlsx(file_path):
    # 使用pandas读取xlsx文件
    df = pd.read_excel(file_path, engine='openpyxl')
    return df

# 转换为单个字典的json格式
def convert_to_single_json(df):
    # 选择第一列和第23列
    selected_columns = df.iloc[:, [0, 22]]
    
    # 创建一个空字典用于存储结果
    result_dict = {}
    
    # 遍历每一行数据
    for index, row in selected_columns.iterrows():
        # 使用第一列的值作为键，第23列的值作为值
        result_dict[row[0]] = row[1]
        
    # 将字典转换为json格式的字符串
    json_data = json.dumps(result_dict, indent=4)
    
    return json_data

# 主函数
def main():
    # xlsx文件路径
    file_path = 'hpt-air-mmmu_MMMU_TEST.xlsx'  # 请替换为你的xlsx文件路径
    
    # 读取xlsx文件
    df = read_xlsx(file_path)
    
    # 转换为单个字典的json格式
    json_data = convert_to_single_json(df)
    
    # 输出json数据
    print(json_data)
    
    # 将json数据保存到文件
    with open('hpt-air-mmmu_MMMU_TEST.json', 'w') as f:
        f.write(json_data)

if __name__ == '__main__':
    main()
```

Would you like to add it into VLMEval?

Best,
StarCycle

@llllIlllll  Please help take a look at this issue. 

Hi, @StarCycle, 
The feature has been supported in #130 , please take a look. 

Hello @kennymckormick and @llllIlllll,

Sorry for the late reply. I guess the json file will still not be generated because of the `continue` command?https://github.com/open-compass/VLMEvalKit/blob/44141d0e1289814237d83cd274d70cea17889b47/run.py#L98

```
            if dataset_name in ['MMBench_TEST_CN', 'MMBench_TEST_EN', 'MMMU_TEST']:
                if not MMBenchOfficialServer():
                    logger.error(
                        f'Can not evaluate {dataset_name} on non-official servers, '
                        'will skip the evaluation. '
                    )
                    continue
            # noqa W293
            if rank == 0:
                if dataset_name in ['MMMU_TEST']:
                    result_json = MMMU_result_transfer(result_file)
                    logger.info(f'Transfer MMMU_TEST result to json for official evaluation, json file saved in {result_json}')    # noqa E501
```

btw, what do `# noqa W293` and `noqa E501` mean here?

Best,
StarCycle


#noqa is just to skip some warning messages during the flake8 code format check.
Repository: open-compass/VLMEvalKit
Issue: Evaluation records for new models in OpenVLMRecords
Root cause: Unimplemented Feature Gap

Thank you for sharing the evaluation kit and the evaluation records.

Would it be possible to update the records in OpenVLMRecords to include newer models such as InternVL3?

https://huggingface.co/datasets/VLMEval/OpenVLMRecords/tree/main/mmeval


Hi @saxenarohit, the newest evaluation result is already updated. You can download it from http://opencompass.openxlab.space/assets/OpenVLM.json

Hi @FangXinyu-0913 
Thanks. The OpenVLM JSON only provides average results. I wanted to check whether detailed results like the per-sample predictions found in the mmeval folder are also available somewhere for newer models.
Repository: open-compass/VLMEvalKit
Issue: Support Logics_Thinking（Request to approve and run workflow for PR #1240）

Hello,

We have submitted Pull Request [#1240 ([Model] Add Logics_Thinking)](https://github.com/open-compass/VLMEvalKit/pull/1240) from the forked repository.

Currently, the GitHub Actions status for our PR shows:
`This workflow is awaiting approval from a maintainer`
and the checks have not yet been executed.

Could you please help approve and trigger the workflow so that automated tests can run and we can proceed with the review process?

Thank you very much for your time and help! 🙏

Hi @anmi9508, I am working on it, and will merge this pull request if it is tested well.
Repository: open-compass/VLMEvalKit
Issue: Support for OneVision 1.5 Models
Root cause: Unimplemented Feature Gap

Hi,

Are there any plans to add support for the new **OneVision 1.5** models? [LLaVA-OneVision-1.5-8B-Instruct](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-8B-Instruct)

It would be great to know if these models are on the roadmap for integration.

Thanks!


Hi @inigopm, we have already supported this model in https://github.com/open-compass/VLMEvalKit/pull/1244, and we will merge it once it is ready.
Repository: open-compass/VLMEvalKit
Issue: How to use a locally deployed judge model, such as Qwen3-8B?
Root cause: Validation Gap

I followed the steps in the provided link to deploy Qwen3-8B locally as a judge model. However, when evaluating the MMBench_DEV_EN_V11 dataset, an error was thrown at line 263 in vlmeval/dataset/image_mcq.py:

```
assert model in ['chatgpt-0125', 'exact_matching', 'gpt-4-0125']
```
I noticed that many datasets only support chatgpt-0125 and gpt-4-0125 as judge models. If that’s the case, what is the purpose of the local judge model deployment tutorial in Quickstart.md? How can I actually use a locally deployed judge model?
Repository: open-compass/VLMEvalKit
Issue: Dependency conflicts with NumPy, OpenCV, and Numba prevent installation
Root cause: External Dependency Breakage

Attempting to install VLMEvalKit on Python 3.10 with the recommended instructions leads to dependency conflicts. Certain required packages cannot coexist due to NumPy version requirements:

**numba 0.57.1+1.g5fba9aa8f requires numpy>=1.21, <1.25

opencv-python 4.12.0.88 requires numpy>=2, <2.3**

Installing any single version of NumPy either breaks Numba or OpenCV. This leads to runtime errors.

steps to reproduce

1. Start a fresh container or Python 3.10.12 environment
2. Follow the quickstart instructions: pip install -e .
3. Attempt to check a model: vlmutil check Qwen2-VL-7B-Instruct
Repository: open-compass/VLMEvalKit
Issue: Bug Report: Wrong argument used in conditional check
Root cause: Algorithmic Error

### Bug Report: Wrong argument used in conditional check

**Description**

<img width="465" height="260" alt="Image" src="https://github.com/user-attachments/assets/4c52210a-7206-4c41-9174-a771bd821dc0" />

In `run.py`, there is a conditional statement around line 317:

    if args.model != "eval":
        # Perform the Inference

However, in `parse_args()` the relevant argument is defined as:

    parser.add_argument('--mode', type=str, default='all', choices=['all', 'infer', 'eval'])

`args.model` is a list of model names, not a string `"eval"`.  
So this condition will never behave as intended.

**Expected Behavior**

The code should probably use `args.mode` instead of `args.model`, like:

    if args.mode != "eval":
        # Perform the Inference

**Steps to Reproduce**

1. Run `run.py` with `--mode eval`  
2. Notice that the code still executes inference because the check is against `args.model`

**Proposed Fix**

Change:

    if args.model != "eval":

to:

    if args.mode != "eval":



Thanks for the awesome work on this repo!


Hi @Dawn-970. Thank you for your advice! This bug has been fixed in https://github.com/open-compass/VLMEvalKit/pull/1271. You can use the main branch now to avoid this bug.
Repository: open-compass/VLMEvalKit
Issue: More elegant way to distinguish qwen2vl and qwen2.5vl
Root cause: Architectural Constraint

As I know, currently there is no model category for Qwen2_5VLChat, only Qwen2VLChat. 
In _config.py_, the Qwen2.5VL series models also use Qwen2VLChat, but there are structural differences between the two series. The current implementation in the code judges the model type by checking whether the path contains "2.5" or similar conditions like:

```       
        if listinstr(['omni'], model_path.lower()):
            try:
                from transformers import Qwen2_5OmniForConditionalGeneration, Qwen2_5OmniProcessor
            except Exception as err:
                logging.critical("pip install git+https://github.com/huggingface/transformers@3a1ead0aabed473eafe527915eea8c197d424356")  # noqa: E501
                raise err
            MODEL_CLS = Qwen2_5OmniForConditionalGeneration
            self.processor = Qwen2_5OmniProcessor.from_pretrained(model_path)
        elif listinstr(['2.5', '2_5', 'qwen25', 'mimo'], model_path.lower()):
            from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
            MODEL_CLS = Qwen2_5_VLForConditionalGeneration
            self.processor = AutoProcessor.from_pretrained(model_path)
        else:
            from transformers import Qwen2VLForConditionalGeneration, Qwen2VLProcessor
            MODEL_CLS = Qwen2VLForConditionalGeneration
            self.processor = Qwen2VLProcessor.from_pretrained(model_path)
```



This approach may lack flexibility and could lead to errors.
In summary, are there any plans to implement a more elegant method to distinguish between the two series of models when loading them?
Repository: open-compass/VLMEvalKit
Issue: Using my own finetuned models like InternVL3.5
Root cause: Documentation Deficiency

I'd like to use my own finetuned model based on internvl3.5-8B.

This is the original config file

```python

    "InternVL3_5-1B": partial(
        InternVLChat, model_path="OpenGVLab/InternVL3_5-1B", version="V2.0"
    ),
```


The added dictionary element

```python

    "MyOwnFintuned-1B": partial(
        InternVLChat, model_path="/my/server/local/path/", version="V2.0"
    ),
```

Q1. Is it able to work?


---

Q2. How can I deal with the model when I add some component based on InternVL3.5 which means it is not exact same model with internvl but added component model.

---

I really appreciate it if you could answer this two questions

Hi @ByungKwanLee , did you find a solution?
Repository: open-compass/VLMEvalKit
Issue: Any plan to support newest Qwen3-VL-30B-A3B-Instruct and Qwen3-VL-30B-A3B-Thinking?
Root cause: Unimplemented Feature Gap


same problem

It's on the way. See this PR to be merged.
#1268 

[PR #1268] has been merged. You can try using the latest supported Qwen3-VL series models. Wishing you a smooth experience!

thanks for the quick update and reply. Qwen3-vl-8b-thinking is not supported by vllm by default, is there specific reason for this?