Paper: Bootstrapping Audio-Visual Segmentation by Strengthening Audio Cues (ICLR-2024)

Reviewer comment (Theory):
In the background, the imbalance issue between audio and video is mentioned, which is good. But I do not understand the aftermath of such imbalance - why the model's being focused more on visual information is a bad thing? This needs to be motivated a bit more.

Author response:
**A1:** Focusing more on visual information is ok but should be in moderation. Present AVS methods focus so much on visual information that neglects that some effective audio features will get lost during training and therefore cannot provide enough audio guidance for correct audio-visual segmentation of sounding objects. This can be visually observed from the right part example of Figure 4 in our paper, where irrelevant objects get mistaken as sounding objects because audio cues do not receive sufficient focus. After all, the AVS task is acoustic in nature as reviewer 7Z8k agrees.

Reviewer comment (Experiments):
Do you see any possibility that the architecture can be optimized to improve the efficiency? For example, some of the cross-modal connections might be redundant? Of course, here "nope" is also a proper answer.

Author response:
**A2:** Yes, thank you for your valuable suggestion. More efficient and lightweight attention module, for example the squeeze-enhanced axial attention of SeaFormer [1], can be adopted to replace our used deformable attention, self-attention and cross attention modules to increase efficiency. Increasing model efficiency will be our future direction.

Reviewer comment (Writing):
At the beginning of Section 3.2 - what is the dot product in MLP() means in Equation 1?

Author response:
**A3:** It is the inner product of two metrics, multiplying $F_{VGA}$ of size 5 $\times$ 300 $\times$ 256 with $F_{AGV}$ of size 5 $\times$ 256 $\times$ 128 $\times$ 128 to get a new matrix of size 5 $\times$ 300 $\times$ 128 $\times$ 128.

Reviewer comment (Novelty):
Audio Feature Reconstruction seems to be very similar to the bidirectional generation module in AVSBG [1].

Author response:
**A1:** Our AFR module is quite different from the bidirectional generation module in AVSBG in 3 aspects: motivation, structure and constraint loss.

**Motivation:** Our AFR aims to evade harmful data bias and also curtail audio information loss by reconstructing lost audio feature from AGV, while the bidirectional generation module in AVSBG just aims to build a strong correlation between visual and audio signals.

**Structure:** The bidirectional generation module in AVSBG adopts a mask encoder-decoder structure to turn masked visual feature to reconstructed audio feature, while our AFR is simpler and more efficient in structure by using only two linear layers for projection.

**Constraint loss:** AVSBG designs a consistency loss to minimize the distance between the norms of the original and reconstructed audio feature along the last dimension, while we adopt MSE loss to minimize the distance between the projected reconstructed audio feature and the projected multi-model feature.

Reviewer comment (Presentation):
I can't see from Figure 5 which areas the attention mechanism obviously pays attention to.

Author response:
**A2:** The darker areas are visually very different from other lighter areas and highlight the attention regions of the network. You may regard the dark areas as where the attention pays attention to.

Reviewer comment (Novelty):
While the emphasis on improving uses of audio cues is good, I am not sure if the claims around cross modal attention is entirely correct. There are multimodal works where both audio-visual attention is through both audio-video and video to audio. That is attentions of the forms Attn(Q_a, K_v, V_v) and Attn(Q_v, K_a, F_a) – so that both audio and visual features are obtained through cross attention. [1, 2] are just 2 examples, likely there are other papers. Not sure however if they have been used for AV segmentation task.

Author response:
**A1:** As far as we know, we are the first to use the cross model attention mechanism in AVS task, since most other AVS works only adopt either Attn(Q_a, K_v, V_v) or Attn(Q_v, K_a, V_a) instead of combining both. Though cross model attention has been used in multi-modal related tasks, the characteristics and motivation are always different from ours. Other works generally use this mechanism to build better correlation between two modalities and is often used only once, which makes the interaction insufficient. Prior works also did not consider from the modality imbalance aspect. While our design features both bidirectional and continuous modality interaction and our motivation is to enhance audio cues to mitigate modality imbalance issue.

Reviewer comment (Evaluation):
Not clear about the claims of richness of information in audio – experimentally this is illustrated by showing that removing self-attention leads to better results. Can you discuss this in a bit more detail and clarify how this conclusion is reached? It’s not clear that improved performance by removing self-attention can actually lead to this claim.

Author response:
**A2:** We claim in paper that the audio feature is not as information-rich as visual feature, so it may not need too much encoding operation, which means merely using cross attention is enough. The reason for the performance dropping when adding self-attention may be that adding self-attentions leads to overfitting. We demonstrate this by comparing the training and eval mask loss functions w/ and w/o self-attention and update the line chart in Figure 7 (b) in the appendix of our pdf file. We find that adding self-attention makes the mask loss drops lower while the test result is inferior to no self-attention case, which means overfitting.

Reviewer comment (Writing):
I think defining L_dice, L_mask and L_afr clearly will add clarity to the the paper.

Author response:
**A3:** Thank you for your advice. We have clarified the definitions and highlighted them in orange color in the updated version. $L_{mask}$ = $L_{focal}$ + $L_{dice}$, $L_{afr}$ = MSE ($F_{afr}$, $F_{proj}$).

Reviewer comment (Experiments):
How does the overall method behave when the sounding object is outside of the field of view or the sounding objects moves in and out of field of view.

Author response:
**A4:** We visualize this case by comparing with AVSegFormer and AVSBench and the visual result is updated in Figure 7 (a) in our appendix. We find that present methods including ours still cannot well tackle this issue since in the last column the sounding ambulance car moves out of the view but all three methods falsely segment the motorcycle as the sounding object. This will be our future research direction. However, our method is still better than the other two since only our method correctly segments the ambulance car in view in the fourth column.

Reviewer comment (Theory):
The AFR loss is essentially forcing feature similarity between F_AGV and F_audio (through the feature learning). Why would this reduce the type of bias mentioned in section 3.3 (single and mult-source etc.).

Author response:
**A5:** We mitigate data bias from audio feature preservation aspect since we try to evade the harmful case where the network happens to “guess” the right target region even if the audio information has been lost or distorted. To achieve this, we design AFR module to reconstruct lost audio feature from AGV so that the "guess" case is curtailed. We have updated our pdf file and added visual comparison results with and without AFR to visualize its effect in Figure 8 in the appendix.