Paper: Large Language Models as Automated Aligners for  benchmarking  Vision-Language Models (ICLR-2024)

Reviewer comment (Novelty):
From my perspective, the approach provided by Visual Instruction Tuning [1] for constructing multimodal instruction data also constitutes a scalable, user-friendly, and comprehensive automated pipeline. Therefore, the claim about data generation is relatively weak. I've noticed that you mention your method being more diverse and complex, but in the analysis of 3.3 BENCHMARK STATISTICS, there is no mention of Visual Instruction Tuning. Lastly, could you please elaborate on any other significant differences and advantages of your method against Visual Instruction Tuning?

Author response:
We appreciate the reviewer's observation on Visual Instruction Tuning. While we aggree that both our methods and Visual Instruction Tuning utilize LLMs to generate data automatically, our methods offer some significant differences as what follows:

(1) **Diversity on the ability categories** 
The data generated by Visual Instruction Tuning does not specify the abilities that need to be evaluated. The questions they generate only broadly cover five aspects: object types, counting the objects, object actions, object locations, and relative positions between objects. In contrast, our generation is ability-aware, where we have specific prompts for each ability to generate content that targets each skill individually. For the final generated data, **we have covered a total of 17 different abilities (which is more than 4 times of Visual Instruction Tuning)**. Therefore, our generated data exhibits higher diversity in terms of ability categories.

(2) **Diversity on the visual symbolic information** 
Visual Instruction Tuning provides only limited caption and 2D bounding box information to prompt GPT in generating questions. This, to some extent, restricts the diversity of generated questions. For instance, its visual symbolic information lacks OCR information, resulting in the absence of OCR-related questions in its generated data. In contrast, our approach incorporates richer visual symbolic information, such as text, caption, instance relationships, and etc. Therefore, we can prompt GPT to generate data that covers a wider range of information.

Reviewer comment (Novelty):
The use of GPT-4 or GPT-3.5 has already been mentioned in other articles, such as [2] and [3].

Author response:
Thanks for highlighting the concurrent works [2] and [3]. While these referenced works also employ GPT for benchmark evaluation, notable distinctions exist between our approach and theirs, as outlined below.

(1) **The way of leveraging GPT is different.** 
Concurrent works (e.g., [2] and [3]) solely employ GPT for open-ended evaluation, while our approach integrates GPT not only in automatic evaluation but also in the data curation process. Specifically, we utilized GPT to automatically generate ability-aware questions, spanning a total of 17 different abilities. In contrast, [2] and [3] rely on existing datasets for evaluation, which, to some extent, may lack comprehensiveness. For instance, [2] comprises only 187 images and 205 questions, posing challenges for a comprehensive evaluation. In our exploration, beyond conducting evaluations based on GPT, we also showcased the advantages of using GPT as data curators, highlighting the potential of GPT in this context.

(2) **The purpose of leveraging GPT is different.** [2] and [3] utilized GPT-4 primarily for automated assessment. In contrast, we aim to present a system employing GPT for both data generation and evaluation, thereby offering a fully automated, scalable, user-friendly, and comprehensive pipeline

Reviewer comment (Novelty):
Assessments of different visual capabilities have also been addressed in other evaluations, such as [2], [3], [4], and [5].

Author response:
We acknowledge that assessments of different visual capabilities have been mentioned in previous methods [2, 3, 4, 5].  Therefore, we conducted a quantitative comparison based on the number of assessed capabilities and the quantity of evaluation questions to highlight the diversity of our approach. The table below illustrates that our method significantly surpasses previous methods in the number of evaluation questions. For instance, we exceed the evaluation dataset of MMBench by tenfold (28,500 v.s. 2,974). Additionally, our method is comparable to previous methods in the number of assessed capabilities. We can also extend LLM to assess additional capabilities, thereby scaling our dataset further.

| Methods        | MM-Vet[2] | TouchStone[3] | MME[4] | MMBench[5] | VisIT-Bench[6] |AutoBench(ours) |
| -------------- | -- | -- | -- | -- | -- | -- |
| Ability Amount | 6 | 27 | 14 | 20 | 70 | 17 |
| QA Pairs Amount | 205 | 908 | 1477 | 2974 | 592 | 28500 |

Reviewer comment (Experiments):
The benchmark is built only on images sourced from MS-COCO. While I understand that this was done due to the richness of image annotations that could be fed to the LLM to generate questions, MS-COCO images also represent a narrow subset of all images which we would like to apply vision-language models to.

Author response:
We appreciate and acknowledge the reviewer's observation regarding the limitations of using the COCO dataset, as it may not cover all image domains, such as medical images. The choice of COCO for evaluation was made because of its rich annotations. However, we would like to emphasize that the contribution of this work extends beyond providing a COCO-based dataset for evaluating VLMs.

The primary contribution, in fact, lies in offering an automatic pipeline that is readily extendable to new data domains and scenarios, such as VizWiz. To achieve this, we can leverage off-the-shelf perception models to obtain visual symbolic representations, as mentioned in our paper. Subsequently, we can conduct data curation and model evaluation tailored to the specific domain. We have incorporated this discussion into the revision outlined in **Section A.1.1** of the Appendix.

Reviewer comment (Writing):
In section 4.2, "Users are guided to rank each sample based on its rationality and level of challenge." what does rationality mean?

Author response:
*'Rationality'* in this context refers to the logical coherence of a question in relation to the specified curation topic and the context presented within the image. We have added the descriptions in **Section 4.2**, for better clarity. Please check the latest revision.