zhiminy/EvalEng huggingface dataset:
Repository: open-compass/VLMEvalKit
Issue: AMBER 对应的 DATASET_URL 失效
Root cause: External Dependency Breakage

[https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/dataset/image_yorn.py#L15](url)
您好，我发现这里 AMBER 对应的 URL 看起来已经失效了？

Hi, @lihytotoro , 
The link has been fixed in this  commit: https://github.com/open-compass/VLMEvalKit/commit/c733356b0f36de5d53a7a17dee2b44fe0af2d225. 

Seems the implementation of AMBER has some bug, I'm checking with the PR author.

Hello, it seems that AMBER still cannot be downloaded. @kennymckormick 

> ###
> [Nov 4, 2024](#issuecomment-2454117169)

I found that the implementation of AMBER has some bug, the PR author is now trying to fix it @yfzhang114 

@ATing0203 @lihytotoro 

@yfzhang114 has already implemented the fix. Now the evaluation of AMBER should work well. 

Sample Results for GPT4o_MINI: 

---------  -------
Attribute  81.0304
Existence  87.2868
Relation   84.0745
Avg ACC    84.1306
---------  -------
Repository: open-compass/VLMEvalKit
Issue: Qwen-VL-Max-0809   MME中celebrity测出来和榜单结果差距有10分左右
Root cause: External Dependency Breakage

测试代码都是VLMEvalKit，我只改了api为qwen-vl-max-0809，以及测MME的celebrity，prompt这些都没动，计算scores的方法也没动，为什么和榜单上的差异这么大
<img width="369" alt="image" src="https://github.com/user-attachments/assets/b28ece8f-1672-454a-93a5-7a7471adb184">
<img width="431" alt="image" src="https://github.com/user-attachments/assets/344863d6-55f3-4bc4-b240-799e6ada5afb">

<img width="908" alt="image" src="https://github.com/user-attachments/assets/d186a746-4ee1-4065-8271-8154b7cd48be">


第三张图是前几天那个榜单上的，请问1101更新为什么消失了？

Hi, @lihua8848 , 
Recently we have added Qwen2-VL-72B to the leaderboard. As the Qwen Team introduced, the model behind Qwen-VL-Max-0809 is Qwen2-VL-72B, so we remove the Qwen-VL-Max-0809 entry. But maybe we can add it back to the OpenVLM Leaderboard later.

For the performance issue, I can help re-conduct the evaluation to see the numbers. Also, note that since Qwen-VL-Max is a proprietary API model and the sample number is small for each single capability in MME, the evaluation results can be variable. 

It would be great if you could help me reproduce the MME celebrity results. How can I assist you? I'm using the following API for testing.
![image](https://github.com/user-attachments/assets/1d6ca32b-028d-44fc-9c42-91e3c186e3e1)


@lihua8848 

Basically, my current evaluation results align with yours. 

![image](https://github.com/user-attachments/assets/6127a0a3-be07-44d0-9a12-c107c2b9eadc)


The results of Qwen-VL-Max-0809 are added back to the leaderboard.

![image](https://github.com/user-attachments/assets/a37a9c0f-17eb-464a-bde8-06d2f250f0ee)
The celebrity metric is still incorrect

@lihua8848 
To clarify, for API models, there is no so called `correct` performance (since the model behind may change frequently). The entry on the leaderboard  only represents the performance at the evaluation time. I will add a new field `Evaluation Time` to the leaderboard to make it clearer.
Repository: open-compass/VLMEvalKit
Issue: AI2d gpt和claude3.5官方分数非常高
Root cause: Documentation Deficiency

您好～
想请问下你们对于openai和claude3.5在ai2d上特别高的分数有任何建议和想法吗？我这边修改姿势和prompt（添加cot）评测了gpt多次，都无法复现出0.942的超高分数。（加了cot后的最高分也就0.83），想请问你们对于这个gap有什么想法？（我看你们这边的ai2d的评测分数也没有任何高于0.9以上的，很好奇claude和gpt是怎么测出来将近满分的

Hi, @Violettttee , 
You can try the  AI2D_TEST_NO_MASK dataset we provided, which generally display better performance compared to AI2D_TEST due to the different setting. However, we still cannot reproduce the numbers reported by OpenAI or Anthropic.

> Hi, @Violettttee , You can try the AI2D_TEST_NO_MASK dataset we provided, which generally display better performance compared to AI2D_TEST due to the different setting. However, we still cannot reproduce the numbers reported by OpenAI or Anthropic.

Hi @kennymckormick , could you please provide the link to download AI2D_TEST_NO_MASK dataset. I can not find it. Thanks a lot!

The download link is https://opencompass.openxlab.space/utils/VLMEval/AI2D_TEST_NO_MASK.tsv,
it is defined in image_mcq.py

> > Hi, @Violettttee , You can try the AI2D_TEST_NO_MASK dataset we provided, which generally display better performance compared to AI2D_TEST due to the different setting. However, we still cannot reproduce the numbers reported by OpenAI or Anthropic.
> 
> Hi @kennymckormick , could you please provide the link to download AI2D_TEST_NO_MASK dataset. I can not find it. Thanks a lot!
Repository: open-compass/VLMEvalKit
Issue: Corrupted MathVista testmini data
Root cause: Configuration Error

I found that the MathVista testmini data used in VLMEvalkit(https://opencompass.openxlab.space/utils/VLMEval/MathVista_MINI.tsv) is not identical to that provided by the MathVista offical huggingface repo(https://huggingface.co/datasets/AI4Math/MathVista)

The VLMEvalkit version of MathVista testmini data has the following problem:
- Images are compressed
- Misuse of \r\n in some questions.

Could you provide an updated version of MathVista testmini?

Hi, @LeoDu0314 , 
During our pre-processing, we have converted images to `JPEG` format. We will try to build another version that do not perform the JPEG conversion and do the evaluation again to see the gap. 

Hello, I wonder if there is a version that do not perform the JPEG conversion now?
Repository: open-compass/VLMEvalKit
Issue: Reproducing Qwen2-VL-72B-Instruct Evaluation Results Fails
Root cause: Documentation Deficiency

Hi,
   I git clone your code (commit 0c44cd2845a0fab51580c5654a86c0da96c7d155) and follow [Official Qwen2-VL (https://github.com/QwenLM/Qwen2-VL) to run Qwen2-VL-72B-Instruct, however, the evaluation results on the OpenCompass Multi-modal Leaderboard benchmarks are different from that of [learderboard](https://rank.opencompass.org.cn/leaderboard-multimodal). Since I don't use GPT as I have no access to GPT, some results such as HallusionBench are low as expected. However, some resutls such as AI2D_TEST are much low. 
Here is our environments:

> 
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Mon_Apr__3_17:16:06_PDT_2023
Cuda compilation tools, release 12.1, V12.1.105
Build cuda_12.1.r12.1/compiler.32688072_0
> 
相关依赖
> 

accelerate==0.34.2
transformers==4.45.0.dev0
torch==2.4.0+cu121
torchaudio==2.4.0+cu121
torchvision==0.19.0+cu121
nvidia-cublas-cu12==12.6.3.3
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==9.1.0.70
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu12==12.1.0.106
nvidia-ml-py==12.560.30
nvidia-nccl-cu12==2.20.5
nvidia-nvjitlink-cu12==12.1.105
nvidia-nvtx-cu12==12.1.105

> 

Here is our evaluation results:
<img width="505" alt="image" src="https://github.com/user-attachments/assets/a04ac130-29f6-4aa0-963e-a3346405926c">

Thanks for your time~


Hi, @ChuanyangZheng , 

Actually , for other MCQ benchmarks like MMMU or AI2D_TEST, we also use GPT-3.5-Turbo for choice label extraction. So it's possible that the reproduced results without GPT is a little bit lower. 


Could I download the results from your leaderboard evaluation including the detailed predications? So I can compare the difference in a more exact way. Since the 3.9 performance gap on AI2D_TEST may exceed the possbile range of GPT-3.5-Turbo?

Sure, I'm going to upload the evaluation records to https://huggingface.co/datasets/VLMEval/OpenVLMRecords/tree/main/mmeval today. 

Thanks, we found all the gaps (including  AI2D_TEST) come from the evaluation with/wo GPT. The detailed predications of ours aligns with yours.

Glad to see the problem is resolved.
Repository: open-compass/VLMEvalKit
Issue: ChartQA eval bug 
Root cause: Algorithmic Error

https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/dataset/image_vqa.py
line 57

```
        elif listinstr(['ChartQA'], dataset):
            res = pool.map(partial(process_line, method='relaxed_accuracy'), lines)
```


https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/dataset/utils/vqa_eval.py

```
lines  272
    elif method == 'relaxed_accuracy':
        ret['gt'] = answers
        ret['pred'] = line['prediction'].strip()
        ret['match'] = [relaxed_correctness(ret['pred'], x) for x in ret['gt']]
```


but  function  relaxed_correctness is 
relaxed_correctness(target: str,
                        prediction: str,
                        max_relative_change: float = 0.05


lines  171
```
# https://github.com/google-research/pix2struct/blob/main/pix2struct/metrics.py# 
def relaxed_correctness(target: str,
                        prediction: str,
                        max_relative_change: float = 0.05) -> bool:
    """Calculates relaxed correctness.

    The correctness tolerates certain error ratio defined by max_relative_change.
    See https://arxiv.org/pdf/2203.10244.pdf, end of section 5.1:
    “Following Methani et al. (2020), we use a relaxed accuracy measure for the
    numeric answers to allow a minor inaccuracy that may result from the automatic
    data extraction process. We consider an answer to be correct if it is within
    5% of the gold answer. For non-numeric answers, we still need an exact match
    to consider an answer to be correct.”

    Args:
      target: Target string.
      prediction: Predicted string.
      max_relative_change: Maximum relative change.

    Returns:
      Whether the prediction was correct given the specified tolerance.
    """

    def _to_float(text: str) -> Optional[float]:
        try:
            if text.endswith('%'):
                # Convert percentages to floats.
                return float(text.rstrip('%')) / 100.0
            else:
                return float(text)
        except ValueError:
            return None
    prediction = str(prediction)
    target = str(target)



```
Repository: open-compass/VLMEvalKit
Issue: Evaluations get stuck on the last 4 questions
Root cause: Resource Mishandling

Sometimes, the evaluation (by GPT API) may get stuck on the last 4 questions. Kill the program and restart it will solve this.
But it's really inconvenient. Is there any method to avoid this?

I am having the same issue also! The evaluation gets stuck on the last 4 questions!!

Noticed this issue, will try to locate the problem that caused it. 

@XuGW-Kevin @jeffhernandez1995 
Hey guys. We have updated the implementation of `track_progress_rich`. You can have another try to check if the problem has been solved.

Hi, thanks for your update! I haven't encountered the issue recently. But since the issue only happens from time to time, I'll watch for a longer time.

Seems the problem has been solved. 

Yeah it has been solved, thanks a lot!
Repository: open-compass/VLMEvalKit
Issue: upload failed on mmbench website
Root cause: Others

https://mmbench.opencompass.org.cn/mmbench-submission
![image](https://github.com/open-compass/VLMEvalKit/assets/58135925/b5e1c0d6-c1c7-442c-a1f8-3e12b590fbf2)


Hi, @mactavish91 , our staff is trying to fix that problem. As a temporary solution, you can send your prediction files to opencompass@pjlab.org.cn and we will handle it manually. 

BTW, we have tried to submit a file and find it succeed. To find the reason of the error you encountered; please share the submission file with me.
Repository: open-compass/VLMEvalKit
Issue: Feat Request: Support for LLAVA OneVision HuggingFace Format Models
Root cause: Unimplemented Feature Gap

### Description

I am currently using VLMEvalkit version 0.1.0. I've noticed that while LLAVA-Next models are fully supported in HuggingFace (HF) format, the LLAVA OneVision models are still sourced from `lmms-lab`, which leads to inconsistency in model handling.

In the code, the llava_series dictionary illustrates this inconsistency:
```python
llava_series = {
    # ...
    'llava_next_72b': partial(LLaVA_Next, model_path='llava-hf/llava-next-72b-hf'),
    'llava_next_110b': partial(LLaVA_Next, model_path='llava-hf/llava-next-110b-hf'),
    # LLAVA-Next models in HF format

    'llava_onevision_qwen2_0.5b_si': partial(LLaVA_OneVision, model_path='lmms-lab/llava-onevision-qwen2-0.5b-si'),
    'llava_onevision_qwen2_7b_si': partial(LLaVA_OneVision, model_path='lmms-lab/llava-onevision-qwen2-7b-si'),
    # LLAVA OneVision models from lmms-lab
    # ...
}
```

### Feature Request

- Add support for LLAVA OneVision models in HuggingFace format.
- Introduce a new class LLaVA_OneVision_HF in vlmeval/vlm/llava/llava.py to handle these models.

### Benefits

- Consistency: Aligns the handling of LLAVA OneVision models with LLAVA-Next models.
- Ease of Use: Simplifies model loading and integration for users who rely on the HuggingFace ecosystem.
- Maintenance: Streamlines updates and maintenance by using a unified model format.

Thank you for considering this feature request. Your efforts to improve VLMEvalkit are greatly appreciated!
Repository: open-compass/VLMEvalKit
Issue: Ovis1.5-Llama3-8B在Hallusion Bench上的指标和榜单上的指标差距过大
Root cause: Validation Gap

1、OpenCompass排行榜的指标是45，但是我们本地测试只有41.30
2、这个差距不是由评判模型造成的。因为需要评判模型处理的'unknown'预测只有14个问题，而这14个问题本身就不是Yes/No问题，我参考了官方给出的预测结果，这14个问题同样回答错误。
<img width="550" alt="image" src="https://github.com/user-attachments/assets/4714d716-d9b3-422e-8642-5a783f85c889">


Hi, @LIRENDA621 , 
I have re-evaluated this model (torch2.4+cu121, transformers==4.46.2), and got an accuracy of ~42.3%, which looks inferior to previous evaluation results. However, we are not sure whether it's due to randomness. 

![image](https://github.com/user-attachments/assets/bc9559ee-f1c6-4bdd-809e-60893a98e541)

We will re-evaluate this model soon to see if all evaluation results are significant different. If so, we will update the leaderboard and OpenVLMRecords. You can also find the prediction files corresponding to the 45% average accuracy in https://huggingface.co/datasets/VLMEval/OpenVLMRecords and check if there is some problems.
Repository: open-compass/VLMEvalKit
Issue: about Qwen2-vl
Root cause: Unimplemented Feature Gap

您好，我发现目前的代码貌似不支持视频的多图测试，想问问是吗？我看了看代码是只能支持视频路径和网址两类，但是视频多图不行看起来：

`
messages = [
    
{

        "role": "user",

        "content": [

            {

                "type": "video",

                "video": [

                    "file:///path/to/frame1.jpg",

                    "file:///path/to/frame2.jpg",

                    "file:///path/to/frame3.jpg",

                    "file:///path/to/frame4.jpg",

                ],

            },

            {"type": "text", "text": "Describe this video."},

        ],

    }

]
`

您好 @jun0wanan，因为qwen2-vl在处理视频输入时是需要调用自己的process_vision_info函数进行取帧处理，因此我们在测试相关的视频benchmark时都是将视频路径及取样帧率直接放入到message中，让qwen2-vl的相关函数自行进行处理～
视频的多图测试因为我们这边暂时没有相应的场景需求（加之自己取帧处理的话可能与qwen2-vl自己的处理不一致，进而导致结果的不一致），因此我们暂未实现。

> 您好 @jun0wanan，因为qwen2-vl在处理视频输入时是需要调用自己的process_vision_info函数进行取帧处理，因此我们在测试相关的视频benchmark时都是将视频路径及取样帧率直接放入到message中，让qwen2-vl的相关函数自行进行处理～ 视频的多图测试因为我们这边暂时没有相应的场景需求（加之自己取帧处理的话可能与qwen2-vl自己的处理不一致，进而导致结果的不一致），因此我们暂未实现。

请问取样帧率是指的maxpixel那些吗？不是fps是吗

> > 您好 @jun0wanan，因为qwen2-vl在处理视频输入时是需要调用自己的process_vision_info函数进行取帧处理，因此我们在测试相关的视频benchmark时都是将视频路径及取样帧率直接放入到message中，让qwen2-vl的相关函数自行进行处理～ 视频的多图测试因为我们这边暂时没有相应的场景需求（加之自己取帧处理的话可能与qwen2-vl自己的处理不一致，进而导致结果的不一致），因此我们暂未实现。
> 
> 请问取样帧率是指的maxpixel那些吗？不是fps是吗

另外想请教一下，您知道qwen2vl最多能输入多少帧吗？我输入800帧单卡爆掉了....
Repository: open-compass/VLMEvalKit
Issue: Error: module 'torch.library' has no attribute 'register_fake'
Root cause: External Dependency Breakage

After I install it, I tried running this demo command. But I get errors:

```
# Demo
from vlmeval.config import supported_VLM
model = supported_VLM['idefics_9b_instruct']()
# Forward Single Image
ret = model.generate(['assets/apple.jpg', 'What is in this image?'])
print(ret)  # The image features a red apple with a leaf on it.
# Forward Multiple Images
ret = model.generate(['assets/apple.jpg', 'assets/apple.jpg', 'How many apples are there in the provided images? '])
print(ret)  # There are two apples in the provided images.
```
Error:
```
RuntimeError: Failed to import transformers.models.auto.processing_auto because of the following error (look up to see its traceback):
module 'torch.library' has no attribute 'register_fake'
```

Hi, @tjasmin111 ,
I haven't encounter this problem (I'm using python 3.10 torch 2.4 transformers 4.46.2), would you please provide your detailed environment? 

![image](https://github.com/user-attachments/assets/67b393e3-4427-40eb-b17e-56711bd7d4f6)


You can re-open the issue if there is any further issue
Repository: open-compass/VLMEvalKit
Issue: llava-onevision multi-image
Root cause: Validation Gap

llava-ov可以处理多图吗？不知道这个评测是否融入了多图还是只能单图或者视频？

Hi, @jun0wanan , 

LLAVA-OV supports multi-image. Under VLMEvalKit, there exists benchmarks like BLINK that adopts multi-image setting.

Hi, I encountered an issue while using the **BLINK benchmarks** to evaluate the llava_onevision_qwen2_7b_ov model. The error message displayed is shown below:
![image](https://github.com/user-attachments/assets/e7f2ed8e-2c0b-4520-b1c1-d06faadd34d9)
**Environment Details:**
**Transformers version:** 4.47.1
**Python version:** 3.10.12
**NVIDIA CUDA:** 12.1.105
I am using the official example code without any modifications. The issue has left me puzzled as everything seems set up correctly. Could you please help me resolve this bug as soon as possible?
Thank you in advance for your assistance!

> Hi, I encountered an issue while using the **BLINK benchmarks** to evaluate the llava_onevision_qwen2_7b_ov model. The error message displayed is shown below: ![image](https://private-user-images.githubusercontent.com/49471673/398345348-e7f2ed8e-2c0b-4520-b1c1-d06faadd34d9.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzUyODI0ODksIm5iZiI6MTczNTI4MjE4OSwicGF0aCI6Ii80OTQ3MTY3My8zOTgzNDUzNDgtZTdmMmVkOGUtMmMwYi00NTIwLWIxYzEtZDA2ZmFhZGQzNGQ5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDEyMjclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMjI3VDA2NDk0OVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTgzNTA4MWExZWFkYjNjOWM5OTdhYzgzOTA1ZTBiYTUyN2Y0OTE0ODIwYzBhMzdhYzJiODdiZWVmYjY4NjRjNGEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.7deiapSqBCbS3MbksXrLVDlxmrQ4EPX2cIfdIrz9lNY) **Environment Details:** **Transformers version:** 4.47.1 **Python version:** 3.10.12 **NVIDIA CUDA:** 12.1.105 I am using the official example code without any modifications. The issue has left me puzzled as everything seems set up correctly. Could you please help me resolve this bug as soon as possible? Thank you in advance for your assistance!

您新开一个帖子问问吧？我的bench 不是这个
Repository: open-compass/VLMEvalKit
Issue: Get a 0 score  in Evaluation of qwen chat in ScienceQA
Root cause: External Dependency Breakage

As I said in the title, 

![image](https://github.com/open-compass/VLMEvalKit/assets/33275244/b1c18c22-6f07-466a-a557-b34ba9e184f9)

Why? I have some result in `qwen_chat_ScienceQA_TEST_prefetch.xlsx`:

![image](https://github.com/open-compass/VLMEvalKit/assets/33275244/8315cdb5-a639-407e-83fd-8eb73bf4f5d3)

But I have no answer with openai, should I must use openai api to evaluate the dataset?

Hi, @YongLD , 
we are trying to reproduce this problem and will provide feedback asap. 

Hi, @YongLD , 
According to my experiment, this error will not occur on latest main branch. Would you please try again with our latest main branch? 
![image](https://github.com/open-compass/VLMEvalKit/assets/34324155/25b04165-f85a-45b6-8c71-636fd83e6162)

Fantastic!! I didn't notice that you updated it two days ago. It works! Thanks for the amazing job.

> Fantastic!! I didn't notice that you updated it two days ago. It works! Thanks for the amazing job.

😄 Please help star this project if you appreciate the efforts~
Repository: open-compass/VLMEvalKit
Issue: Where is the Mathverse result？
Root cause: Documentation Deficiency

https://mathverse-cuhk.github.io/#leaderboard is not the latest.
Also, https://huggingface.co/spaces/opencompass/open_vlm_leaderboard does not have this result.

Hi, @StarUniversus 
Hi, we are working on delivering a new leaderboard on multimodal math reasoning, which will include the results on MathVerse. You can expect to see them in 2 / 3 weeks.

Hi, @StarUniversus , 
You can now see the results of MathVerse in https://huggingface.co/spaces/opencompass/Open_LMM_Reasoning_Leaderboard
Repository: open-compass/VLMEvalKit
Issue: two bugs 1. I can not use quick start from the README.md . 2  'vlmutil check Llama-3.2-11B-Vision-Instruct ' will lead to error which is 'File "path/VLMEvalKit/vlmeval/dataset/__init__.py", line 168, in DATASET_TYPE     if 'openended' in dataset.lower(): AttributeError: 'NoneType' object has no attribute 'lower''
Root cause: Validation Gap

两个bug
1.在readme界面，无法访问到quick start
2.测试模型能否使用的时候，’vlmutil check Llama-3.2-11B-Vision-Instruct‘，会显示bug，定位了下代码，发现本质是没给dataset参数    
Traceback (most recent call last):
 ……
  File "path/VLMEvalKit/vlmeval/tools.py", line 415, in cli
    CHECK(m)
  File "path/VLMEvalKit/vlmeval/tools.py", line 271, in CHECK
    res = model.generate(msg)
  File "path/VLMEvalKit/vlmeval/vlm/base.py", line 115, in generate
    return self.generate_inner(message, dataset)
  File "path/VLMEvalKit/vlmeval/vlm/llama_vision.py", line 197, in generate_inner
    if DATASET_TYPE(dataset) == 'MCQ' or DATASET_TYPE(dataset) == 'Y/N':
  File "path/VLMEvalKit/vlmeval/dataset/__init__.py", line 168, in DATASET_TYPE
    if 'openended' in dataset.lower():
AttributeError: 'NoneType' object has no attribute 'lower'

Hi, @GoogleAlphaZero , 
The first problem has been fixed in PR #625 .
The second problem has been fixed in a recent commit. 

You can have another try to check.
Repository: open-compass/VLMEvalKit
Issue: When running the Llama-3.2-90B-Vision model with an odd number of GPU cards, will encounter a device_map setting error
Root cause: Algorithmic Error



CUDA_VISIBLE_DEVICES=0,1,2

language_model.model.layers.99.input_layernorm.weight doesn't have any device set.

This error occurs due to the limited implementation of split_model() in vlmeval/vlm/llama_vision.py

Hi, @WeiweiZhang1 , 
The problem has been fixed in a latest commit.
Repository: open-compass/VLMEvalKit
Issue: 检查模型会遇到bug，如vlmutil check {MODEL_NAME}，如vlmutil check Llama-3.2-11B-Vision-Instruct
Root cause: Validation Gap

vlmutil check {MODEL_NAME}，如vlmutil check Llama-3.2-11B-Vision-Instruct，会报错。注意，并非我模型路径错误导致。报错为importlib.metadata.PackageNotFoundError: No package metadata was found for vlmeval

我拉取了最新的代码，运行vlmutil check Llama-3.2-11B-Vision-Instruct，报错变为了VLMEvalKit/vlmeval/dataset/__init__.py", line 168, in DATASET_TYPE
    if 'openended' in dataset.lower():
AttributeError: 'NoneType' object has no attribute 'lower'

The problem has been fixed now. 

这是来自QQ邮箱的假期自动回复邮件。
 
您好，我最近正在休假中，无法亲自回复您的邮件。我将在假期结束后，尽快给您回复。
Repository: open-compass/VLMEvalKit
Issue: The evaluation device_map setting is not optimized for quantized models using the ExLlamaV2 backend.
Root cause: Configuration Error

The evaluation setting is not optimized for quantized models using the ExLlamaV2 backend, which serves as the default kernel for Auto-GPTQ and Transformers.

![image](https://github.com/user-attachments/assets/b1d5748a-99be-45ca-bed3-3c0f2b9378de)

Changing device_map="cpu" to device_map='auto' will resolve the model loading issue.
![image](https://github.com/user-attachments/assets/4da8e917-b13d-4e26-b7b9-bf21058a3596)

The known problematic model files are:
vlmeval/vlm/llama_vision.py
vlmeval/vlm/llava/llava.py
vlmeval/vlm/qwen2_vl/model.py