Paper: Variable resolution: improving scene visual question answering with a limited pixel budget (ICLR-2024)

Reviewer comment (Experiments):
Some of the experiments are missing. Firstly, as the title of the article is related to VQA, and models that perform well on object detection tasks may not necessarily have the same effect on VQA tasks, I hope to include more space in the experiment on different types of VQA datasets, such as OK-VQA. Secondly, this work has proved the effectiveness of variable resolution in that the central part has a higher resolution but only makes the comparison with uniform sampling, which is too naive to have a good performance. So I’m also curious about what the result would be when other part of the image has a higher resolution or other sampling strategies.

Author response:
**Q1.** We choose to compare between the variable scheme and the uniform-naive scheme because the uniform alternative is commonly used in many systems to make the visual input more compact and to save on compute/acquisition resources. It is, for instance, the most common arrangement of the sensors in digital cameras.
We show that the variable scheme, which is biologically inspired, has similar compactness as the naive-uniform scheme, but allows a significant gain in model performance over the naive-uniform scheme due to the learning of efficient representations.
In our Ablation studies (Sec.D.1), we also compared to a so-called “multi-mixed” sampling, which is a different sampling strategy.

**_Action_**: We add in the revised version two new evaluations: (i) MDETR model on the GQA dataset, (ii) BLIP2 model on the SEED-Bench dataset. The results are added to Figure 3 and the supplementary. Both new evaluations support the advantage of the variable scheme over the uniform scheme, as already demonstrated in the evaluation on the VQAv2 dataset.

Reviewer comment (Experiments):
The selection strategy for high-resolution areas in the image needs to be improved. As shown in Figure 4, not all objects reside at the center part of the image, so this could be the cause of the poor performance in Table 2 when compared with the baseline. As this paper raises the HRA metric, it’s better to define a better selection strategy corresponding to this, or just sampling by gradients following the dynamic mask in [1].  
[1] Lin K, Li L, Lin C C, et al. Swinbert: End-to-end transformers with sparse attention for video captioning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 17949-17958.

Author response:
**Q2.** Indeed, as Table 2 shows, the baseline outperforms the variable scheme, in part due to the fact that many objects are not included in the high-resolution area. However, this paper focuses on the representational differences learned by different sampling schemes. To make our analysis more general and agnostic to specific scene layouts, we intentionally choose an arbitrary fixation point, which is not targeting a specific object or salient feature in the scene. As can be seen in Figure 4, the variable scheme gain in performance over the uniform scheme even when objects are mostly in the periphery (up to 70% of an object’s area) and only a small part is included in the high-resolution area (image center in our settings). Figure 5 presents the representational roots of the model performance gain discussed in section 4. Certainly, a good fixation strategy can improve performance, however in this paper we aim to establish solid single-fixation foundations for future works exploring this venue in complex VQA.

Reviewer comment (Evaluation):
The interpretability analysis of variable resolution is not convincing enough. Some visualization experiments are not as important as shown in the paper, and some visualization is lacking. On the one hand, there are too many visualization examples of the impact of the three sampling methods on the label of an object in the image. As a common sense, if we feed the model with a lower resolution of the image of the object, the model has a higher probability of giving a wrong label. On the other hand, as shown in Figure. 3, the variable sampling scheme performs well on the questions that can be answered from parts of the image rather than the whole context of the image. That is because we don't have enough understanding of the correlation between hyperparameters and sampling strategy of variable resolution and global background knowledge. The explanation of this correlation is lacking, but the whole context of the image is important to VQA.

Author response:
**Q3.** **_Action_**: In the revised paper, we provide an additional quantitative analysis in the interpretability section, in particular the attention distance measure for the self-attention layers of the DETR model (also used for VQA). This analysis indicates that indeed the variable scheme utilizes contextual information from the far periphery (in particular, in intermediate layers 2 and 3), while the uniform scheme only utilizes information from adjacent spatial areas.
We also add evaluations on the VQA task in Figure 3, which clearly shows a consistent advantage of the variable scheme over the uniform scheme in questions requiring the integration of fine details and contextual information, such as in questions about object relations and object classification (including object attributes).

Reviewer comment (Experiments):
According to the title, the author's focus is on VQA, but it seems that only one dataset (VQA v2) from the VQA task was used. How does the authors' proposed method perform on other common datasets such as GQA[1], OOD-GQA[2], etc.?

Author response:
**_Action_**: We added in the revised version two new evaluations of the model on challenging VQA tasks. In particular, we included an evaluation for the MDETR vision-language model (VLM) on the challenging GQA dataset. In addition, we included a third evaluation of the BLIP2 VLM on the new SEED-Bench dataset. We added a summary graph of the results to Figure 3. Both new evaluations show a similar advantage of the variable scheme over the uniform scheme, as already demonstrated in the evaluation on the VQA dataset, supporting the claim that this is a consistent advantage of variable resolution in VQA tasks.

Reviewer comment (Evaluation):
The method proposed by the authors seems to address object detection and is not a VQA task.

Author response:
It may be too challenging to analyze the contribution of the mechanisms combined in a complex VLM, in particular the visual mechanism. Therefore, we choose to investigate underlying visual sub-tasks that drive the improvement in performance for the VLMs when applied to VQA tasks. As mentioned above (Q1), we included in the revised version, an evaluation of the MDETR VLM, which is based on the architecture of the DETR detection model combined with a BERT-based LLM. In sections 4 and 5 we analyze in detail the performance and interpretability of the DETR model, which provide direct insights on the visual representations of the MDETR.

Reviewer comment (Presentation):
Does Sec.5 relate to the VQA task? Why?

Author response:
Yes, following the reviewer's relevant remarks, Sec.5 is now directly related to our findings on VQA. To clarify why, the reason is that the entire analysis on the attention was done on the DETR model, which we apply to the GQA dataset (with MDETR) and achieve significant results, consistent with our findings on VQAv2. The analysis of the CNN kernel specialization was done on ResNet101, which is the backbone of MDETR and DETR.

Reviewer comment (Writing):
Is the technical or scientific challenge addressed throughout the article associated with VQA? Why?

Author response:
The VQA task is closely related to the task of complex visual scene understanding, for which we believe the contribution of a variable scheme is potentially highly significant. Understanding object relations and interactions in the scene, in addition to particular object attributes, requires the integration of fine visual cues with contextual information from the scene. Both are available with the variable resolution scheme. We find the evaluation results of VLMs on the task of VQA, when applied to the variable sampling scheme, significant as they clearly show that these models mostly benefit from the variable sampling scheme over the uniform scheme. However, as mentioned above (Q2), it is not clear how to analyze the contribution of the visual mechanism within a complex VLM, which combines a large language mechanism. We therefore investigate representational learning in basic visual sub-tasks, including image classification and object detection, which underlie and drive the visual capacities of a VLM.

Reviewer comment (Novelty):
There is a long list of critical missing papers that should be cited if this paper is to be accepted. I can not increase my score to accept unless these papers are cited & discussed (and of course, if the other reviewers also think that this paper should be accepted).

Author response:
Following the reviews we now added and discussed the missing references in the Related work section.

Reviewer comment (Novelty):
Key missing critical references:  
- Deza & Konkle. ArXiv, 2021. **Emergent Properties of Foveated Perceptual Systems.**  
- Wang & Cottrell. Journal of Vision, 2017. **Central and peripheral vision for scene recognition: A neurocomputational modeling exploration.**

Author response:
**Q2.1.** We thank the reviewer for pointing out a number of important papers. Indeed, we did not cover many studies around the topic of foveated images, as we try to focus on the contribution of variable resolution to high-level complex tasks, including visual scene understanding (question answering) and object recognition (detection), already at the initial fixation point in the scene. To make our analysis more general and agnostic to specific scene layout, we intentionally choose an arbitrary fixation point, which is not targeting a specific object or salient feature in the scene. We provide a very extensive analysis of this initial fixation performance and hope to establish a solid foundation for future works exploring VQA with re-fixations.  

**_Action_**:  We added in the revised version a discussion to section 6 on the important papers we were referred to by the reviewer.

Reviewer comment (Novelty):
Secondary, but also important references:  
- Cheung, Weiss & Olshausen. ICLR 2017. Emergence of foveal image sampling from learning to attend in visual scenes  
- Gant, Banburski & Deza. SVRHM, 2022. Evaluating the adversarial robustness of a foveated texture transform module in a CNN.  
- Reddy, Banburski, Pant & Poggio. NeurIPS 2020. Biologically inspired mechanisms for adversarial robustness  
- Wang, Mayo, Deza, Barbu & Conwell. SVRHM, 2021. On the use of Cortical Magnification and Saccades as Biological Proxies for Data Augmentation  
- Harrington & Deza. ICLR, 2022. Finding Biological Plausibility for Adversarially Robust Features via Metameric Tasks  
- Malkin, Deza & Poggio. SVRHM 2020. CUDA-Optimized real-time rendering of a Foveated Visual System.

Author response:
**Q2.1.** We thank the reviewer for pointing out a number of important papers. Indeed, we did not cover many studies around the topic of foveated images, as we try to focus on the contribution of variable resolution to high-level complex tasks, including visual scene understanding (question answering) and object recognition (detection), already at the initial fixation point in the scene. To make our analysis more general and agnostic to specific scene layout, we intentionally choose an arbitrary fixation point, which is not targeting a specific object or salient feature in the scene. We provide a very extensive analysis of this initial fixation performance and hope to establish a solid foundation for future works exploring VQA with re-fixations.  

**_Action_**:  We added in the revised version a discussion to section 6 on the important papers we were referred to by the reviewer.

Reviewer comment (Experiments):
While I think the VQA evaluation framework is original, what I do not understand is "why VQA?", why not something less complex such as object recognition or detection where language modelling will not interfere in the output produced by the system. I can only think of an answer if there is an argument somehow linking foveation with language but that does not seem to be the case. I can see an object detection evaluation which I think is nice, but that has already been shown in Pramod et al. 2021.

Author response:
**Q1.** The task of visual question answering is closely related to the task of complex visual scene understanding, for which we believe the contribution of a variable resolution system is potentially more significant than for detection alone.
The basic tasks including object classification and detection are important and indeed are more convenient to investigate with respect to the representational learning patterns, as we do in Sections 4 and 5. Nevertheless, the integration of fine cues at high resolution with contextual information at lower resolutions may be limited compared to the task of VQA, since the basic tasks do not often require the understanding of subtle attributes, relations and interactions in the image. This level of integration between fine and global information is available with the variable resolution scheme.
We find the evaluation results of VLMs on the task of VQA significant as they clearly show that these models mostly benefit from the variable sampling scheme over the uniform scheme.  

**_Action_**: We added a similar clarification in the revised paper.

Reviewer comment (Novelty):
All in all, I think this paper is exciting, but discussing and adding the missing references is necessary.

Author response:
Following the reviews we now added and discussed the missing references in the Related work section.