Paper: MMBench: Is Your Multi-modal Model an All-around Player? (ICLR-2024)

Reviewer comment (Experiments):
The paper does a poor job in explaining how the samples in the new VQA benchmark are generated, collected, and verified. It is unclear how images are selected from existing sources, how annotators create QA pairs from these images, and how the correctness and relevance of the samples are verified. This lack of detail makes it difficult to assess the data quality of the benchmark.

Author response:
### Q1.1: How are these samples collected?

First, we recruit a group of college students as volunteers. We then provide necessary training to these volunteers, equipping them with skills to collect images and create relevant questions and answers for each task. Moreover, we provide them with a range of potential sources to gather images and formulate corresponding questions and choices. Table 7 in the paper showcases these potential sources. Following this, the collected Visual Question Answering (VQA) data undergoes a quality inspection pipeline to ensure the creation of a reliable benchmark.

### Q1.2: Quality testing for samples in MMBench

We conduct thorough quality testing on the collected samples through manual review, with a primary focus on two key aspects:
1. Ensuring the accuracy and coherence of the questions, without any logical or factual errors.
2. Verifying the presence of a single correct answer that is pertinent to the question.
We filter out any samples that fail to adhere to these aforementioned criteria.

Reviewer comment (Evaluation):
After downloading the public dev set, it is easy to find many VQA samples across categories that can be solved without looking at the images. Examples provided (African elephant size question, snow rabbit-related question, and world map spatial question) demonstrate that some tasks resemble textual QA rather than true VQA, suggesting the benchmark may not always test visual understanding.

Author response:
### Q1.3: Question can be answered without the corresponding image

Thank you for highlighting this issue. We too identified the same problem after submitting our paper. To determine the number of questions in MMBench that can be answered without an image, we input all the questions into GPT-4 and ensure that it makes reasonable guesses for all questions. We discovered that ~13% of the dev questions and ~10% of the test questions could be answered correctly, which mainly originated from ScienceQA. Subsequently, we removed these questions and used the remaining ones to evaluate all models. As indicated in the table below, the performance of these models remained similar to their performance prior to the image filter, and the rankings of the different models were basically consistent. That suggests that the current MMBench leaderboard remains credible. Furthermore, we plan to release an updated version of MMBench, with the questions that can be answered without images filtered out.

| Model            | Dev   | Dev w/o. GPT-4 | Test  | Test w/o. GPT-4 |
| ---------------- | ----- | -------------- | ----- | --------------- |
| Qwen-VL-Chat     | 60.57 | 59.9           | 61.83 | 60.75           |
| Shikra           | 59.36 | 60             | 60.43 | 60.81           |
| IDEFICS-80B      | 54.81 | 51.29          | 54.82 | 52.34           |
| IDEFICS-9B       | 48.37 | 47.03          | 45.52 | 44.8            |
| mPLUG-Owl        | 48.11 | 48.51          | 46.41 | 46.11           |
| LLaVA            | 44.5  | 42.77          | 42.21 | 41.74           |
| InstructBLIP-13B | 44.42 | 43.17          | 43.39 | 43.18           |
| MiniGPT-4-13B    | 42.53 | 40.5           | 42.54 | 41.62           |
| LLaMA-Adapter    | 41.24 | 41.09          | 39.63 | 39.63           |
| VisualGLM        | 38.57 | 37.82          | 33.63 | 33.52           |
| Qwen-VL          | 38.23 | 36.53          | 32.23 | 31.34           |
| InstructBLIP     | 35.48 | 33.66          | 35.37 | 34.08           |
| PandaGPT         | 33.93 | 32.77          | 30.72 | 29.1            |
| MiniGPT-4        | 32.04 | 30.69          | 29.43 | 28.04           |
| MMGPT            | 15.21 | 13.66          | 15.98 | 15.26           |
| OpenFlamingo v2  | 6.7   | 5.45           | 5.72  | 4.61            |
| OpenFlamingo     | 4.64  | 3.37           | 4.54  | 3.55            |

Reviewer comment (Experiments):
The authors should discuss how MMBench was collected and why problematic samples that do not require image information appear in the dev set. It is also unclear whether crowd-sourced or expert annotators were used. This is an important issue for future expansions of MMBench, and the paper should explain how quality will be ensured.

Author response:
### Q1.1: How are these samples collected?

First, we recruit a group of college students as volunteers. We then provide necessary training to these volunteers, equipping them with skills to collect images and create relevant questions and answers for each task. Moreover, we provide them with a range of potential sources to gather images and formulate corresponding questions and choices. Table 7 in the paper showcases these potential sources. Following this, the collected Visual Question Answering (VQA) data undergoes a quality inspection pipeline to ensure the creation of a reliable benchmark.

### Q1.2: Quality testing for samples in MMBench

We conduct thorough quality testing on the collected samples through manual review, with a primary focus on two key aspects:
1. Ensuring the accuracy and coherence of the questions, without any logical or factual errors.
2. Verifying the presence of a single correct answer that is pertinent to the question.
We filter out any samples that fail to adhere to these aforementioned criteria. 

### Q1.3: Question can be answered without the corresponding image

Thank you for highlighting this issue. We too identified the same problem after submitting our paper. To determine the number of questions in MMBench that can be answered without an image, we input all the questions into GPT-4 and ensure that it makes reasonable guesses for all questions. We discovered that ~13% of the dev questions and ~10% of the test questions could be answered correctly, which mainly originated from ScienceQA. Subsequently, we removed these questions and used the remaining ones to evaluate all models. As indicated in the table below, the performance of these models remained similar to their performance prior to the image filter, and the rankings of the different models were basically consistent. That suggests that the current MMBench leaderboard remains credible. Furthermore, we plan to release an updated version of MMBench, with the questions that can be answered without images filtered out.

| Model            | Dev   | Dev w/o. GPT-4 | Test  | Test w/o. GPT-4 |
| ---------------- | ----- | -------------- | ----- | --------------- |
| Qwen-VL-Chat     | 60.57 | 59.9           | 61.83 | 60.75           |
| Shikra           | 59.36 | 60             | 60.43 | 60.81           |
| IDEFICS-80B      | 54.81 | 51.29          | 54.82 | 52.34           |
| IDEFICS-9B       | 48.37 | 47.03          | 45.52 | 44.8            |
| mPLUG-Owl        | 48.11 | 48.51          | 46.41 | 46.11           |
| LLaVA            | 44.5  | 42.77          | 42.21 | 41.74           |
| InstructBLIP-13B | 44.42 | 43.17          | 43.39 | 43.18           |
| MiniGPT-4-13B    | 42.53 | 40.5           | 42.54 | 41.62           |
| LLaMA-Adapter    | 41.24 | 41.09          | 39.63 | 39.63           |
| VisualGLM        | 38.57 | 37.82          | 33.63 | 33.52           |
| Qwen-VL          | 38.23 | 36.53          | 32.23 | 31.34           |
| InstructBLIP     | 35.48 | 33.66          | 35.37 | 34.08           |
| PandaGPT         | 33.93 | 32.77          | 30.72 | 29.1            |
| MiniGPT-4        | 32.04 | 30.69          | 29.43 | 28.04           |
| MMGPT            | 15.21 | 13.66          | 15.98 | 15.26           |
| OpenFlamingo v2  | 6.7   | 5.45           | 5.72  | 4.61            |
| OpenFlamingo     | 4.64  | 3.37           | 4.54  | 3.55            |

Reviewer comment (Reproducibility):
The paper does not address licensing for collected Internet images, and some images in MMBench contain watermarks from existing websites such as zhihu.com.

Author response:
### Q1.4  Licensing for collected Internet images

All images within MMBench are available for academic use.  During the collection process, we instruct our annotators to document the original URL of each image. Subsequently, we manually review these URLs individually, eliminating any images that are not suitable for academic use. Moreover, should any authors request the removal of their images from MMBench, we will promptly comply.

## Q3: watermark issue
The issue of watermarking is analogous to that of licensing, which we have discussed in Q1.4 above. Specifically, regarding the watermark associated with Zhihu, we have consulted their copyright policy. It prohibits the misuse of their content for commercial purposes, but does not restrict its use for academic and non-profit purposes.

Reviewer comment (Novelty):
Answer extraction using LLMs is already a standard NLP practice (referencing "Large Language Models are Zero-Shot Reasoners", Kojima et al. 2022), so it is hardly a novel contribution.

Author response:
## Q3: Answer extraction using LLMs is standard practice in NLP

1. In the evaluation process, we first use heuristic matching to match the option label (A, B, C, D) or the option content with the model's predictions. Only if the matching failed, we utilize ChatGPT to align the semantic meaning of the prediction with one of the available choices, subsequently outputting the label for the corresponding choice. Such work is more complicated compared to straight-forward answer matching. 
2. To the best of our knowledge, we are the first to integrate the strong semantic matching capability of cutting-edge LLMs into the evaluation process of multimodal understanding. It makes the evaluation procedure yield more accurate results and reduces the number of false negatives.

Reviewer comment (Evaluation):
Regarding related work, it is unclear why MME has "a non-rigorous evaluation setting" that makes it harder to reveal the real performance gap between VLMs, as mentioned in Appendix A.1.

Author response:
## Q4: MME uses a non-rigorious evaluation setting
1. MME continues to utilize the traditional method of exact matching, which necessitates the model to produce precise 'yes' or 'no' responses. This approach, however, tends to generate a number of false negative samples.
2. The metric adopted by MME (accuracy & accuracy +) enables randomd guessing to achieve a substantial amout of score (37.5%), and also makes it difficult to reveal the performance gap between VLMs. Besides, It appears unreasonable that certain models yield scores on the MME that are lower than those of random guesses (especially for perception tasks). The table below provides a comprehensive overview of this issue. Notably, MMBench does not suffer from this problem. Some examples:
    1. XComposer outperforms Qwen-VL-Chat by 14.2% Top-1 accuracy on MMBench-Dev, which is a substantial gap. However, in MME, the gap is merely 70 in total score (2.5% of the total score). 
    2. In MME, VisualGLM-6B and MiniGPT-4 demonstrates poorer performance compared to the random baseline, which doesn't sounds reasonable. However, in MMBench-Dev, the two approaches significantly outperforms the random baseline (which is <0.5% Top-1 accuracy). 
    
| Method       | MME-Perception | MME-Cognition | MME Total | MMBench-Dev |
| ------------ | -------------- | ------------- | --------- | ----------- |
| GPT4-V       | 1409           | 517           | 1926      | 75.1%       |
| XComposer    | 1528           | 391           | 1919      | 74.8%       |
| Qwen-VL-Chat | 1488           | 361           | 1849      | 60.6%       |
| mPLUG-Owl2   | 1450           | 313           | 1763      | 66.5%       |
| VisualGLM-6B | 705            | 182           | 887       | 38.1%       |
| MiniGPT-4    | 582            | 144           | 726       | 24.3%       |
| Random       | 750            | 375           | 1225      | <0.5%       |

Reviewer comment (Evaluation):
More discussions of the 20 different ability dimensions would be favored. How these dimensions are selected can be discussed further. Moreover, in many cases, multiple abilities are entangled with each other in order to correctly answer a question. For example, “How many apples are there in the image?” as shown in Fig-3 requires both numerical (counting) reasoning and perception (detect apples), which category does this example belong to?

Author response:
## Q1: How these existing 20 ability dimensions are selected

1. Perception and reasoning are two crucial cognitive abilities in humans. Firstly, we refer to the definitions of perception and reasoning in biology, such as the specific aspects they encompass. The part has been thoroughly discussed in Section 2.1 of the main paper.

2. There is currently no consensus in the academic community regarding the specific taxonomy of sub-abilities in perception and reasoning, and there also lacks a comprehensive theoretical framework. Therefore, we gathered a group of experts from the fields of computer vision, natural language processing, and machine learning to discuss the specific ability taxonomy in perception and reasoning. As a result, we obtained 20 ability dimensions included in MMBench, as well as the ability hierarchy ranging from L1 to L3.

3. The existing 20 ability dimensions are not fixed, and we will continuously update the existing ability dimensions to further improve  the ability taxonomy. 

## Q2: How to classify a question, if multiple ability dimensions are entangled

According to the definition in Page 16, we categorize the "counting" ability as "Object Localization". In MMBench, we do not consider "counting" as a more complex reasoning problem, since for most counting questions, there only exist several objects to be counted. 

If multiple ability dimensions are intertwined, we consistently categorize the current question under the more challenging ability dimension. This is because, in most scenarios, the more difficult ability dimension necessitates the model to possess the capability to solve simpler tasks. By doing so, it can effectively utilize the task outputs as intermediate results to tackle more complex tasks. For instance, all questions in "Attribute Comparison" rely on the capability of "Attribute Recognition", since we consider "Attribute Comparison" as a more complex task, we categorize these questions as "Attribute Comparison".

Reviewer comment (Evaluation):
The results are usually “winner takes all”. As shown in the results in Tab-3, more powerful models are usually stronger in every evaluation dimension. It would be interesting to see more fine-grained analysis, e.g. some models are stronger in dimension A, while another is stronger in dimension B, etc.

Author response:
## Q3: The results are usually “winner takes all” 
Table 3 presents the results of the L-2 ability dimension. Unlike a "winner takes all" scenario, these results on L-3 ability dimension offer a nuanced understanding of the model's capabilities. For further insights, please refer to Tables 9 through 14, which provide additional details on the results. For instance, Shikra demonstrates exceptional performance in Action Recognition, while LLaMA-Adapter excels in Attribute comparison. We hypothesize that factors such as data composition, instruction data quality, and instruction template significantly influence a model's performance across different ability dimensions.

Reviewer comment (Experiments):
It would be good to have the results for Bard and GPT-4V.

Author response:
## Q5. Results for closed source VLMs (GPT-4v, Bard, etc.) 

Currently, we have evaluated GPT-4v and Bard on MMBench-Dev, and we will provide the results on MMBench-Test in the final version. Note that due to the strategies set by developers, the closed source VLMs may reject some of the questions: GPT-4v refused to answer 4.2% questions in MMBench-Dev, Bard refused to answer 31.6% questions in MMBench-Dev. In Table 1, we report the accuracies of GPT-4v and Bard (only answered questions count: Acc = (Number of questions answered correctly) / (Number of questions answered)).

| Model    | Overall | CP    | FP-S  | FP-C  | AR    | LR    | RR    |
| ------------ | ------- | ----- | ----- | ----- | ----- | ----- | ----- |
| GPT-4v\*     | 75.06   | 82.77 | 71.65 | 66.43 | 77.88 | 69.29 | 74.31 |
| Qwen-VL-Chat | 60.57   | 79.39 | 66.21 | 48.25 | 59.8  | 32.2  | 43.48 |
| Shikra       | 59.36   | 76.35 | 57.68 | 58.74 | 57.29 | 26.27 | 58.26 |
| Bard\*       | 58.16   | 58.42 | 57.6  | 39.6  | 68.27 | 53.4  | 71.64 |
| IDEFICS-80B  | 54.81   | 64.53 | 58.36 | 44.76 | 65.83 | 23.73 | 46.09 |

**Table 1. Model accuracies on MMBench-Dev (all questions). * mean the question numbers used for calculating accuracies may be different from other models.**

In Table 2, 3, we provide apple-to-apple accuracy comparisons for GPT-4v and Bard, respectively. In those comparisons, the advantages of GPT-4v and Bard are more significant. 

| Model    | Overall | CP    | FP-S  | FP-C  | AR    | LR    | RR    |
| ------------ | ------- | ----- | ----- | ----- | ----- | ----- | ----- |
| GPT-4v\*     | 75.06   | 82.77 | 71.65 | 66.43 | 77.88 | 69.29 | 74.31 |
| Qwen-VL-Chat | 59.46   | 79.39 | 61.42 | 48.25 | 59.8  | 33.33 | 42.2  |
| Shikra       | 59.1    | 76.35 | 56.3  | 58.74 | 57.29 | 26.32 | 56.88 |
| IDEFICS-80B  | 53.81   | 64.53 | 53.15 | 44.76 | 65.83 | 24.56 | 46.79 |

**Table 2. Model accuracies on MMBench-Dev (only questions answered by GPT-4v count).**

| Model    | Overall | CP    | FP-S  | FP-C  | AR    | LR    | RR    |
| ------------ | ------- | ----- | ----- | ----- | ----- | ----- | ----- |
| Bard\*       | 58.16   | 58.42 | 57.6  | 39.6  | 68.27 | 53.4  | 71.64 |
| Qwen-VL-Chat | 56.28   | 80.9  | 62.67 | 36.63 | 53.1  | 34.09 | 35.82 |
| Shikra       | 52.89   | 74.72 | 57.14 | 48.51 | 48.28 | 20.45 | 40.3  |
| IDEFICS-80B  | 49.62   | 65.73 | 54.38 | 28.71 | 59.31 | 22.73 | 37.31 |

**Table 3. Model accuracies on MMBench-Dev (only questions answered by Bard count).** 

Dear reviewer, we would like to supplement some quick facts on the rejected questions. 
We have browsed the rejected questions by Bard and GPT-4v. Actually, we believe that most rejected questions are rejected due to the policy set by the provider organization, and it doesn't mean that Bard or GPT-4v are not capable to solve the questions. 
For example, we find that Bard will reject a large proportion of questions with images that contain recognizable human beings, while GPT-4v will reject many questions with images including celebrities. Comparing Bard or GPT-4v with other VLMs can be challenging under such situation, and we think the provided Table 2, 3 in the rebuttal can present a reasonable comparison setting.