Paper: Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy (ICLR-2024)

Reviewer comment (Presentation):
A figure showing the differences between a VQA question + label and examples from the proposed datasets would be useful.

Author response:
Thank you for your constructive feedback on the paper. We appreciate your recognition of our novel approach to VQA and the usefulness of our empirical evaluation. As suggested, we created a figure comparing our proposed datasets with traditional VQA benchmarks to provide a clear overview (Figure 6 in the Supplementary). We also expanded Table 7 to include an in-context metric based on LLaMA-2, which comes close to the effectiveness of GPT-4 in scoring VQA outputs. To evaluate the importance of in-context examples for the LLM metrics, we added zero-shot results to Table 7. We updated the implementation details and results of the LLM metrics in Section 4, Supp. Section C and Supp. Figure 9 to reflect those changes. Additionally, we included metric judgements to our qualitative examples for VQA (Figure 5 in the Supplementary). Finally, we revised Figure 2 and updated the labels as suggested.

Reviewer comment (Presentation):
Fig. 2 would be easier to read if it said "attribute", "object" etc instead of OVAD, COCO.

Author response:
Thank you for your constructive feedback on the paper. We appreciate your recognition of our novel approach to VQA and the usefulness of our empirical evaluation. As suggested, we created a figure comparing our proposed datasets with traditional VQA benchmarks to provide a clear overview (Figure 6 in the Supplementary). We also expanded Table 7 to include an in-context metric based on LLaMA-2, which comes close to the effectiveness of GPT-4 in scoring VQA outputs. To evaluate the importance of in-context examples for the LLM metrics, we added zero-shot results to Table 7. We updated the implementation details and results of the LLM metrics in Section 4, Supp. Section C and Supp. Figure 9 to reflect those changes. Additionally, we included metric judgements to our qualitative examples for VQA (Figure 5 in the Supplementary). Finally, we revised Figure 2 and updated the labels as suggested.

Reviewer comment (Evaluation):
I believe assessing how granular a VQA system is in good and necessary, however, when evaluating, the current benchmark put additional constrains on the image space, cropping the image to certain objects and using imagenet which is object centric. These constrains largely limited the scope of VQA that is supposed to work on natural use case, for example, natural image QA (VQA v2, vizwiz, etc).

Author response:
Thank you for your constructive feedback and positive comments about our benchmark. 

**Answer to question 1**

We agree that our scope of VQA is smaller than the scope of VQAv2. However, this is a deliberate choice: Evaluating with a large scope of mixed questions makes it difficult to find the exact strengths of models quantitatively. Our benchmark aims to provide a clear and constrained environment to evaluate specific capabilities of VQA systems: object, activity, and attribute recognition.

Figure 1 (a) shows our decision to crop the relevant object in ImageNet. We ask “What’s this?”, and the model responds with “A tree with no leaves” which is a reasonably good answer given the image-question pair. Even though the desired label is “porcupine”, the model gets distracted by other objects in the image. In a standard classification setup this is less of a problem, since the label space reduces the ambiguity of the answer. To stay with the example of Figure 1 (a), tree is not a category of ImageNet; therefore, a classification or retrieval model cannot answer with a tree class. On the contrary, in our open-ended setup, the model must answer the question in a free manner without knowing the label space, increasing the ambiguity when the image contains more than a single object. By focusing on a cropped object of interest, we effectively reduce the ambiguity of the answer, allowing for a more precise assessment of the model’s ability to recognize and classify the specific object. To allow comparison with other works that test on ImageNet, we also provide results on ImageNet with full images in Supp. Tab. 4 and 5.

In general, our scope of VQA includes asking about objects, actions and attributes. We acknowledge that there are many other facets of VQA systems that merit exploration, such as model performance on natural VQA datasets like VQAv2 and VizWiz. The scope of our benchmark could certainly be increased in future work e.g. by asking about object counts, spatial relations, compositional reasoning (e.g. having multiple object-attribute pairs in the same image and asking about their relations), testing generalization to rare scenes, or other scenarios. We believe our benchmark design and analysis of metrics will pave the way to test Text-VLMs on such abilities.

Reviewer comment (Experiments):
The Cropping activity operations seems very risky the action needs a few frames to evaluate? Like sit down and stand up?

Author response:
**Answer to question 2**

When developing ActivityNet-oVQA, we decided using only a single image, since the purpose of the whole benchmark is to evaluate models which answer questions on images. While extending this to VideoQA models is certainly a possible future direction, in this work our focus remains on images. Consistent with standard practice in the literature, we chose the middle frame of each segment as our input. Supp. Fig. 3 shows 4 example scenes which can all be answered given the middle frame shown in the figure. 

To determine the effectiveness of selecting the middle frame among all others, we conduct an ablation study, testing all models over nine frame selections by dividing the segment into nine parts and selecting the closest frame. The results are shown in Supp. Fig. 10. We measure accuracy as the average response to the same three initial questions across all models. The results suggest that frame selection does not have a significant impact on performance, especially since frames are selected from short segments in which the relevant action occurs. 
Additionally, our choice of using ActivityNet as a VQA dataset is motivated by its well-defined set of activities within a semantic hierarchy which allow us to ask follow-up questions about fine-grained activities.