Paper: Visual Grounding with attention-driven constraint balancing (ICLR-2024)

Reviewer comment (Experiments):
By adding the RAC and MRC, how long did you train the model? Can you compare it with the baseline training time as well?

Author response:
# Response 4.2: Response to Weakness 2 about training time.
Thank you for your question regarding the training time for our model with the inclusion of RAC and MRC compared to the baseline. We appreciate your interest in this aspect of our work. To address your question, we investigate the training time consumption of both the baseline (TransVG) and our method (TransVG(+AttBalance)) using the experiment results from the table provided earlier. It is essential to note that all experiments were conducted on 8 Quadro RTX 6000 GPUs with the same CUDA and PyTorch environment. As demonstrated in the table below, the additional training time introduced by our AttBalance falls within a reasonable range.

|               | TransVG          | TransVG(+AttBalance) |
|---------------|------------------|-----------------------|
| Training time | 11 hours 05 mins | 13 hours 53 mins      |

Reviewer comment (Evaluation):
Moreover, one could list a lot of cases where the attention values inside a bounding box do not necessarily correlate with the models' performance. Take small objects for example, the most influential factors would lie in the background for determining the semantics of the object instead of the attention inside the bounding box. Therefore, the motivation of this study fails to persuade the reviewer of the effectiveness of the proposed module in various scenarios.

Author response:
I appreciate your thoughtful consideration and concern regarding the correlation between attention values inside a bounding box and model performance, especially in scenarios involving small objects. Your observation raises a valid point, and we would like to provide further clarification:

* Detecting a target object intuitively involves the model attending to the region of the object. However, as you rightly pointed out, background factors can also significantly influence the final prediction, especially in cases involving small objects. Determining whether the foreground or background matters more is challenging, particularly in visual grounding scenarios where the text is complex and contains many background clues. This nuanced understanding aligns with the findings from our ANALYSIS, where all Spearman's rank correlation (rho) values are positive, but none reach 1.

* This is exactly why we should propose MRC to rectify RAC, where RAC serves as the intuitive motivation, while MRC is introduced to rectify this approach and mitigate excessive bias assumptions. Our motivation comes from the ANALYSIS and is verified in the ablation study. As shown in Table 2, RAC can bring impressive improvement, and MRC further enhances these results.

Reviewer comment (Experiments):
The introduced regulation on the attention value is to guide the attention to focus more on the area inside a bounding box. Even though the authors assert that the loss could apply to different architectures, more hyperparameters are also introduced which would also hurt its transferability on other visual grounding models. For example, in the sentence “we partition the attention values within the ground truth region of VLTVG’s last layer into 8 equal number parts, omitting the extreme intervals, i.e. those less than 0.1 or greater than 0.9”, one would identify at least 3 hyperparameters whose ablation is also missing in the manuscript.

Author response:
Thank you for your detailed examination of our implementation, particularly concerning hyperparameters. We would like to address the raised concern and maybe some misunderstanding in Figure 3:

**hyperparameters:** As emphasized in Response 2.3 to Reviewer L37Q, we have thoroughly explained the rationale behind these hyperparameters in our answer to that question, and I would encourage you to refer to it for more clarity. Here, for a brief recap, it is crucial to recognize the substantial differences in the models' structure, training settings, and dataset features. The distinct fusion/decoding modules of various models naturally result in slight adaptations of our AttBalance. Furthermore, variations in training settings and dataset characteristics necessitate minor adjustments to the training epoch in our AttBalance.

**Clarify misunderstandings in Figure 3:** In reference to the configuration in Figure 3, it appears there might be a misunderstanding. This setup is used exclusively for analyzing the last layer attention distribution of VLTVG across the dataset. It serves to provide insight into our motivation for incorporating DAT. It is important to note that these are not hyperparameters in our DAT, which is computed adaptively based on BoxRatio and attention regulation for each data sample, without manual-craft design.

We are grateful for the acknowledgment of our generalization and effectiveness, particularly highlighted in the reviews: Strength 2 from Reviewer 6bMP, Strength 1 from Reviewer L37Q, Strength 3 from you, and Strengths 2 & 3 from Reviewer qRXz. It is rewarding to see recognition for our consistent improvement across all five models on four benchmarks.

Reviewer comment (Presentation):
Some major writing issues. Each figure annotation should explain itself whereas the annotations in this study e.g., Figure 1 and Figure 3 are too short to comprehend without referring to the main manuscript.

Author response:
Thank you for your valuable feedback. I appreciate your attention to the minor writing issue, specifically, Figures 1&3 and “analyzing”. We have promptly corrected this error and conducted a thorough proofreading of the entire manuscript to address similar issues and enhance the overall writing quality in the attached revised paper.

Reviewer comment (Writing):
Minor writing issues, e.g., “analyzing” -> “analyze” in 4.3. The authors are encouraged to further proofread the manuscript.

Author response:
Thank you for your valuable feedback. I appreciate your attention to the minor writing issue, specifically, Figures 1&3 and “analyzing”. We have promptly corrected this error and conducted a thorough proofreading of the entire manuscript to address similar issues and enhance the overall writing quality in the attached revised paper.

Reviewer comment (Evaluation):
In regards to the ablation study, it is weird that only applying MRC even causes degradation of the performance. In contrast, it is able to largely benefit the model when combined with RAC. This is an interesting yet strange phenomenon, especially given that the feature map distillation from a moving-averaged teacher has been effective on various self-supervised/semi-supervised tasks. This is not well addressed in the manuscript with only one sentence “suggesting that merely smoothing the attention behavior by MomModal does not bring benefits.”

Author response:
Thank you for noting the intriguing observation regarding the impact of applying MRC and RAC on performance. A similar question was addressed in Response 1.3 to Reviewer 6bMP, where we thoroughly explained the underlying rationale. I recommend referring to that response for a more detailed explanation. Here, we provide a brief overview:

* In essence, the Momentum model operates as a continuously-ensembled version, incorporating exponentially moving averaged versions from previous training steps. Given that these earlier models are more underfitting than the current one, relying solely on their ensemble can lead to a decline in the reliability of guiding the current model's attention map. However, the Momentum model proves beneficial while serving as a counterpart constraint to smooth the original constraint. In scenarios where the original constraint may not be consistently reliable, such as with the Image-Text Contrastive Learning constraint in ALBEF [1] facing noise unpaired data, the guidance from the Momentum model helps bring the current model's learning target closer to its previous ensemble behavior, resulting in smoother learning.

* In the context of visual grounding, RAC effectively directs the model's attention toward language-related regions, generally resulting in improvements, as outlined in conclusion (1) of our ANALYSIS section and illustrated in Table 2 of our ablation study. However, RAC is not a universally reliable constraint, as visual grounding also necessitates considering background objects as potential clues mentioned in the language. Therefore, the consistent benefit observed by using MRC to rectify RAC is discussed in conclusions 2&3 of our ANALYSIS section and validated by Table 2 in our ablation study.

To aid future readers' comprehension, we have concisely incorporated this discussion into the revised version of the paper.

[1] Align before Fuse: Vision and Language Representation Learning with Momentum Distillation. NeurIPS 2021 Spotlight.

Reviewer comment (Theory):
The paper assumes that the lack of explicit attention guidance results in suboptimal performance. However, it is difficult for me to buy the assumption, given that the model is trained in an end-to-end manner. Factors like the size and diversity of the training dataset could also be responsible if the attention appears dispersed.

Author response:
We appreciate your insightful question about the potential impact causing attention dispersion. We will address this issue both theoretically and based on practical experimental results below.

**Assumption on the behavior of attention:**

* The dispersion of attention depends on different situations; it is challenging to definitively state whether it should be entirely focused inside the box or what percentage can be outside the box. Sometimes, it behaves incorrectly and appears dispersed, attending to the wrong background. However, there are instances where it behaves correctly, especially when the background provides important clues or other factors like the size and diversity of the data, as you mentioned. However, intuitively, we should at least pay close attention to the target object if we want to detect it.

* These situations have already been considered in our ANALYSIS section. Specifically, a higher level of attention focused inside the box generally tends to lead to more accurate performance (Conclusion 1). However, there is no clear pattern for us to confirm the extent of attention concentration (Conclusions 2 & 3). Here, we express our gratitude for your recognition of this motivation in your Strength 3.

**Experimental results to verify our assumption:** Regarding the overall assumption concerning the suboptimal supervision problem, we have verified it both quantitatively and qualitatively. As demonstrated in Table 1, and with the support from your Strength 1, there are consistently impressive enhancements after applying AttBalance. In the qualitative results, there is a noticeable shift in the layers of TransVG after applying AttBalance, as illustrated in Figure 4 and supported by your Strength 4.

Reviewer comment (Experiments):
The constraints proposed in the paper involve numerous hyperparameters and heuristics, including adding constraints and subsequently introducing other ones to temper the initial constraints.

Author response:
Thanks for your discussion about hyperparameters in our method and the motivation for proposing RAC with the introduction of MRC to rectify it.

**Hyperparameter Concerns:** Addressing concerns about the hyperparameters of our method, it is crucial to note the significant differences in the models themselves, their corresponding training settings, and the features of datasets.
* From the perspective of the model’s structure, QRNet differs from TransVG with its early language modulation before the final encoder. This necessitates applying AttBalance in all layers of QRNet but fewer layers in TransVG, given the constraint's relation to language-related regions. VLTVG does not involve word-pixel cross-attention in the latter decoding stage but features multi-layer constraints. This difference from TransVG & QRNet in terms of constraint requires a slight tuning of AttBalance with the multi-layer constraint of VLTVG.
* From the perspective of training settings, TransVG trains for 90 epochs, VLTVG additionally freezes the backbones in the first 10 epochs, and QRNet trains for 160 epochs. These varying training settings require slight adaptations to AttBalance's training epochs.
* From the perspective of datasets, due to the differing difficulty and features among RefCOCO, RefCOCO+, and RefCOCOg, there are variations in the harshness of our AttBalance. For instance, RefCOCO+ features exclude location words, while RefCOCOg features longer and more elaborate language expressions.

In summary, we made slight changes to the hyperparameters to achieve optimal results. However, these adjustments do not affect our method's effectiveness, given the consistent enhancements observed across all five models on four benchmarks. At this point, we also appreciate the recognition of our generalization and effectiveness from Strength 2 of Review 6bMP, Strength 1 from you, Strength 3 of Reviewer zMiJ, and Strength 2 & 3 from Reviewer qRXz.

**Heuristics Concerns:** Regarding concerns about the heuristics of our method, we believe we have provided an intuitive motivation, including the ANALYSIS section for RAC & MRC and the analysis from Figure 3 for DAT. Our method is not proposed based on performance tuning but is derived from a clear motivation with a thorough analysis. At this point, we appreciate the support from Strength 2 of peer Reviewer zMiJ and Strength 1 of Reviewer qRXz.

Reviewer comment (Novelty):
The third contribution highlighted in the paper, namely,  “(iii) Our framework can be seamlessly integrated into different transformer-based methods.” is not a valid contribution, but a feature of the proposed framework mentioned in (ii). I suggest merging it with the contribution (ii).

Author response:
We appreciate your valuable feedback and have carefully considered your suggestion regarding some statements, figures, tables, and references. We have carefully revised the paper in the attached revision paper.

Reviewer comment (Presentation):
Presentation: Both the main text and the captions omit explanations for legends and notations in the figures and tables (Figure 1, Table 2). The quality of writing could also be further improved.

Author response:
We appreciate your valuable feedback and have carefully considered your suggestion regarding some statements, figures, tables, and references. We have carefully revised the paper in the attached revision paper.

Reviewer comment (Evaluation):
The performance improvement is significant for QRNet and TransVG, yet only slight for VLTVG. I'm curious if this variation is mirrored in the attention map. When examining the attention map, similar to what's shown in Figure 4, is the shift in attention for VLTVG more subtle compared to that of QRNet?

Author response:
Certainly, we appreciate your curiosity regarding the alignment between attention shifts and performance improvement in VLTVG.
* Firstly, we have included additional visualizations in the revised paper's appendix, using the same example as Figure 4. We can see that the attention of VLTVG(+AttBalance) does shift a little bit compared to the original one, but it is subtle, resulting in a wrong prediction.
* Secondly, to further observe this phenomenon from a statistical perspective, we computed the proportion of attention values within the bounding box in each layer of VLTVG and TransVG relative to the total attention values. We then compared the respective increases in this ratio after incorporating AttBalance. As shown in the table below, the enhancement of attention ratio is +0.12 in VLTVG, while it is +0.20 in the case of TransVG. This statistical phenomenon aligns with the results, indicating that the improvement in TransVG is more pronounced. We are unable to perform this analysis on QRNet since they have not released their checkpoints for us to make a comparison.
| gref umd test | Original | AttBalance | Improvement |
|---------------|----------|------------|-------------|
| VLTVG          | 0.5044   | 0.6283     | +0.12       |
| TransVG        | 0.2711   | 0.4719     | +0.20       |

In response to the comment of the VLTVG improvement as somewhat slight, we want to emphasize that the enhancement achieved with VLTVG(+AttBalance) is indeed significant. To elaborate, there is a notable improvement of +2.28% on RefCOCO testB, +1.66% on RefCOCOg-g val, and +1.75% on RefCOCOg-umd val for VLTVG_res50.

Reviewer comment (Writing):
In Section 2’s analysis, I want to double check if the “IOU value” in “Then we record the IoU value of corresponding data points” is the IoU between the object bounding box predictions against the GT bounding boxes.

Author response:
Yes, it is the IoU between the prediction and the GT.

Reviewer comment (Reproducibility):
In Section 2’s analysis, were the attention values normalized in any way?

Author response:
In the case of VLTVG, the attention from the object query to the vision tokens in the decoding layer is inherently a probability distribution, meaning it already sums up to 1. Therefore, no additional normalization is needed for VLTVG. In the case of TransVG, as there are additional text tokens in their fusion layer, the attention values from the object query to the vision tokens need additional processing. Specifically, the softmax function is applied to the vision tokens, excluding those associated with text tokens. This computation ensures that the attention values form a valid probability distribution.

Reviewer comment (Experiments):
The paper delves into directing visual features towards specific regions (bbox regions) in an image. Instead of their proposed method, why not introduce a learnable 2D regional weighting layer, akin to the approach in [a], to modulate the attention? This would be done as Attention_i = weighted_2D_mask * Attention_i, where the weighted_2D_mask is produced by a shallow convnet that takes a CxWxH feature map and is trained end-to-end without extra supervision. This could serve as another baseline to validate whether there's a need for explicit guidance/supervision on attentions.

Author response:
Certainly, your suggestion to introduce a learnable 2D regional weighting layer to modulate attention is intriguing. However, we acknowledge that similar attempts have been made in previous works. Nevertheless, this approach may not effectively address the bottleneck identified in our paper, i.e., the Visual Grounding (VG) model still employs common Object Detection losses without explicit guidance on its attention behavior. This results in suboptimal performance in attending to language-related regions. This limitation stems from a supervision perspective rather than a modeling perspective. We elaborate on this point by discussing related work and conducting experiments on your proposed idea for comparison, as below:

**Related work:** Specifically, as we introduced in the related work section, QRNet adopts a Query-modulated Refinement Network to modulate vision attention in both Spatial-wise and Channel-wise. Similarly, in a manner more akin to your proposed weighted_2D_mask, VLTVG introduces the visual-linguistic verification module to explicitly model the activation between text and vision features. In summary, previous works have indeed proposed novel ideas to modulate vision features in both attention-wise and feature-wise manners. However, our AttBalance is still necessary to enhance their modeling attempts from the perspective of supervision.

**Experiments:** To validate this assumption, we conducted an ablation study on TransVG. As shown in the table below, the attn_mask represents the weighted_2D_mask idea described above. Specifically, in each fusion layer of TransVG, we processed the vision features through a 2-layer CNN with ReLU as the intermediate activation function and Sigmoid as the output activation function to obtain the weighted_2D_mask, similar to [c]. This mask is then used to modulate the attention from the object query to the vision token.
| Dataset       | TransVG | TransVG(+attn_mask) | TransVG(+AttBalance) |
|---------------|---------|---------------------|-----------------------|
| gref_umd val  | 67.93   | 67.01 (-0.92)       | 73.69 (+5.76)         |
| gref_umd test | 67.44   | 67.11 (-0.33)       | 72.44 (+5.00)         |

From the table above, we observe that there is no improvement brought by attn_mask, while our AttBalance brings a +5.76% improvement on gref_umd val and +5.00% on gref_umd test, validating that the supervision provided by AttBalance is effective and necessary.

Reviewer comment (Novelty):
this paper’s main contribution is attention mechanisms of transformer-based models. While I believe in 2021, there was already a work which put attention transformer in visual grounding, which called Word2Pix, what is the strength of this paper compare to Word2Pix? Some innovative points are needed to demonstrate the superiority of this method in visual grounding.

Author response:
We appreciate the reviewer's insightful comment on the need for innovative aspects in our work compared to Word2Pix. It is important to clarify that there is no direct relationship between Word2Pix and our AttBalance. Here are the details:
* Word2Pix primarily introduces a one-stage model incorporating word-pixel attention, a module that is commonplace in current Transformer-based models like TransVG[1], VLTVG [2], and QRNet [3] (e.g., the Visual-Linguistic Transformer in TransVG and QRNet, and the verification module in VLTVG).
* In contrast, AttBalance is designed to address the issue of insufficient supervision in these models. It guides the attention layer to focus on language-related regions while also considering the background.

Therefore, the concept of "putting an attention transformer in visual grounding" is not novel today and is not directly related to our motivation or method. The novelty of our approach lies in the perspective of supervision learning. Additionally, it is worth noting that the classification loss and attribute loss in Word2Pix rely on additional classification and attribute labels, which deviate from the current visual grounding (VG) setting. The VG task typically involves a text-image pair with one ground truth bounding box, a widely-used setting supported by top conference papers like TransVG [1], VLTVG [2], QRNet [3], SeqTR [4], and SiRi [5].

[1] TransVG: End-to-End Visual Grounding with Transformers. ICCV 2021.

[2] Improving Visual Grounding with Visual-Linguistic Verification and Iterative Reasoning. CVPR 2022.

[3] Shifting More Attention to Visual Backbone: Query-modulated Refinement Networks for End-to-End Visual Grounding. CVPR 2022.

[4] SeqTR: A Simple yet Universal Network for Visual Grounding. ECCV 2022 Oral.

[5] SiRi: A Simple Selective Retraining Mechanism for Transformer-based Visual Grounding. ECCV 2022.

Reviewer comment (Presentation):
The related work part can be put in section 2 rather in section 5.

Author response:
I appreciate the reviewer's suggestion to relocate the related work section to section 2. Initially, the reason we put related work in the latter section is that we have an additional Analysis section to induce our motivation. We were concerned that introducing related work earlier might make the paper appear miscellaneous before delving into the detailed method, which typically would be around the fourth page.
However, considering that readers might question the necessity to dig into the transformer-based model supervision problem without thinking about other CNN-based models, for example, the Word2pix model you proposed in Weakness 1. We will follow your suggestion to move the related work section to section 2.

Reviewer comment (Evaluation):
As shown in Table 2, could you explain why the incorporation of only the MRC module results in a decline on average, and when the MRC is used to rectify the RAC can it lead to an increase. Can you provide a more detailed explanation of how MRC affects RAC and lead to an increase?

Author response:
I appreciate the insightful query regarding the observed decline with only the MRC module and its subsequent improvement when used to rectify the RAC. I would like to clarify a few things:

**Decline when only using MRC:** The Momentum model is a continuously-ensembled version that incorporates exponentially moving averaged versions from previous training steps. Given that each of the models from previous training steps is more prone to underfitting than the current one, it is intuitive that their ensemble may not consistently provide a reliable attention map to guide the current model. This is why solely relying on MRC results in a slight decline.

**Improvement when using MRC to rectify RAC:** Nevertheless, the momentum model proves beneficial when employed as a complementary constraint to smooth the original constraint. In many cases, the original constraint may not be inherently reliable. For instance, the Image-Text Contrastive Learning constraint in ALBEF [1] may lack reliability when confronted with noisy unpaired data. Thus, the momentum model brings the current model's learning target closer to its previous ensemble behavior, thereby smoothing the current learning process. In the context of our visual grounding, the RAC that focuses the model's attention on language-related regions generally leads to improvements, as discussed in the conclusion (1) of our ANALYSIS section and supported by Table 2 in our ablation study. However, the RAC is not universally reliable, considering that in visual grounding, we might also consider background objects as potential clues, as they may be referenced in the language. Consequently, employing the MRC to rectify the RAC consistently brings about improvement, as detailed in the conclusions (2, 3) of our ANALYSIS section and validated by Table 2 in our ablation study. We will concisely incorporate this discussion into the revised version of the paper.

[1] Align before Fuse: Vision and Language Representation Learning with Momentum Distillation. NeurIPS 2021 Spotlight.