Paper: EquiAV: Single-modal Equivariance Promotes Audio-Visual Contrastive Learning (ICLR-2024)

Reviewer comment (Theory):
If the loss in formulation (1) is contrastive in nature then the $\mathcal{L}$ should not just output the dissimilarity as the optimization minimizes $\mathcal{L}$ and contrastive loss is about maximizing similarity of aligned data and minimizing similarity of non-aligned data. It is assumed that the authors meant a formulation of contrastive loss and not just a dissimilarity measuring mechanism. Clarification is needed.

Author response:
First, it's correct to consider $\mathcal{L}$ as a contrastive loss, as you've mentioned. In our work, we aimed to establish a more general formulation for representation learning. Specifically, we formulate it as maximizing the similarity between pairs of inputs, which is equivalent to minimizing their dissimilarity. 
The four equations described in Section 2 of our manuscript are designed to be applicable not only to contrastive learning but also to other forms of representation learning. Furthermore, these formulations are adaptable to a variety of techniques used to prevent collapse in Siamese networks, such as the use of 'stop gradient', illustrating their broad applicability in recent advancements in the field.

Reviewer comment (Theory):
Conceptually, the augmented and non-augmented variants can be considered as two different modalities. Therefore, standard contrastive loss setup (as used in Eq 11) can be used. A specific reason for using the Eq 9 formulation instead of the standard setup is requested.

Author response:
As you pointed out, they can be treated as two different modalities, which would allow the use of Eq 11. However, this method halves the number of negative pairs (from 2N-2 to N-1), which is similar to decreasing the batch size. Such a reduction in negative pairs can have a negative impact on performance, particularly in the context of contrastive learning. While Eq 11 may have its advantages when using different encoders for significantly different spaces, like in video-audio or image-text pair, we can conclude that Eq 9 is more suitable for intra-modal training.

Reviewer comment (Theory):
The reasoning behind inclusion of the 'positive pair ...in the denominator' is not clear. The appendix seems to allude to different weights during gradient calculations but does not elaborate on why those specific weights should necessarily be a factor. Although Table 5 results demonstrate that it can be important, there is no explanation as to why it is so.

Author response:
As explained in Appendix A of the manuscript, including the positive pair in the denominator results in a relatively larger update for hard positives. This becomes particularly advantageous when stronger augmentations lead to an increased frequency of hard positives. Training with more hard positives, alongside the application of equivariance, substantially enhances the model's capability to comprehend detailed features in single-modal contexts. The intra-modal representation quality plays a pivotal role in understanding and integrating different modalities. Consequently, the semantically rich intra-modal representation promotes effective learning of audio-visual correspondence and audio-visual joint representations. We will add a detailed analysis in Appendix A.

Reviewer comment (Experiments):
There are two types of features for each modality, one from the head corresponding to the inter-modal space and the other from the intra-modal space. It is not apparent which features were used in the experiments.

Author response:
We employed features projected into the inter-modal space for all downstream tasks, and Table C in Appendix C.2 demonstrates which features yield the best performance. As you mentioned, we conducted experiments using features projected into the intra-modal space, as well as features that were processed solely through the encoder without any heads. The results indicate that features projected into the inter-modal space exhibit the best performance.

Reviewer comment (Novelty):
The novelty of the article is limited. The article only applies the single-modal equivariant representation learning that has been proven effective to the A-V learning task and does not try to solve particular problems in this field (for example, compared with the Text-audio, Text-vision field, what specific difficulties can this method solve?).

Author response:
As self-supervised learning through invariance has been extended, some studies have shown that applying the concept of equivariance leads to an additional performance boost in various domains such as images and text. Specifically, in the image domain, the concept of equivariance has been applied to SimCLR through developments like E-SSL[2] and EquiMod[1]. Similarly, in the text domain, SimCSE[3] has been advanced with the application of equivariance as seen in DiffCSE[4]. 
These papers show the process of adopting the idea of equivariance for representation learning in each domain. On the other hand, to our knowledge, there is no prior work that incorporates equivariance for audio modality and audio-visual representation learning.
Along these lines, we aim to design a framework for employing equivariant representation learning in audio-visual representation learning and demonstrate its benefits. However, extending the single-modal equivariant representation learning to multi-modal representation learning is not trivial. Our additional ablation studies (Table 3 in the revised manuscript) reveal that applying equivariance to inter-modal latent spaces is not beneficial and may even hinder the learning of audio-visual correspondences and joint representations. We found that augmentation-related information and audio-visual correspondence need to be learned in separate latent spaces to overcome this challenge. In addition, simply replacing the single-modal invariant representation learning with the equivariant one doesn’t lead to performance improvement. Consequently, we have meticulously designed a framework that maximizes the benefits of equivariance, while avoiding negative impacts on learning. Each component of our framework has been verified to be effective through ablation studies, including:

1) Loss function (Table 5),

2) Data augmentations for effective equivariant representation learning (Table 4), and

3) Projection head and augmentation encoder (Table C in Appendix).

Taken together, all of these factors indicate that our contribution goes beyond simply applying existing techniques to create a new and effective framework for audio-visual representation learning.

Reviewer comment (Reproducibility):
The author lacks design details for augmentation predictors in the article. How does it work? Why can it make the framework achieve better results? What are the specific settings? Whether to only perform linear transformations on original input embeddings and augmentation vectors.

Author response:
The role of the augmentation predictor is to estimate the displacement in the latent space caused by input augmentation. We have conducted an ablation study on how to deliver the augmentation information to the augmentation predictor, as reported in Appendix C.2. In addition to concatenating the input embedding and augmentation vector, we also tried using a 3-layer MLP to encode the augmented vectors to latent vectors and then concatenate it with the input embedding (as shown in Figure A (b)). The results in the first and the second row of Table C (in the revised manuscript) show that directly concatenating the augmentation vector with the input embedding gives better results compared to using the additional augmentation encoder. The augmentation vector is produced through the parameterization of input augmentation and the further encoding of augmentation information by the augmentation encoder might lose some salient information.

Reviewer comment (Experiments):
The author uses InvAV as the baseline but does not give a reference to this solution.

Author response:
InvAV can be considered as adding intra-modal contrastive loss to CAV in CAV-MAE, or the first pre-training stage in MAViL without masked modeling. There exist works that employ intra-modal contrastive loss for audio-visual representation learning, but InvAV is not designed in exactly the same way as those works. It is just a baseline for comparison with our method.

Reviewer comment (Presentation):
Tables 1 and 5 appear in the wrong chapter positions, and the layout of the article needs to be carefully revised.

Author response:
Following your advice, we revised the layout of the article so that the tables appear in the right positions of the corresponding chapters. Thank you for giving a delicate review.

Reviewer comment (Evaluation):
In Table 1, the performance of MAViL is better than EquiAV. What is the specific reason?

Author response:
There are a couple of key factors to consider in the performance of MAViL: 
- Firstly, MAViL utilizes 8 frames for training, in contrast to the single frame used in CAV-MAE and our work. Although using more frames improves performance, it significantly increases computational costs (MAViL uses 64 gpus vs. EquiAV uses 8 gpus). 
- Secondly, MAViL adopts a two-stage training approach. The first stage involves training the model with masked modeling and contrastive learning techniques, followed by a second stage of retraining using knowledge distillation (KD). It's worth noting that KD could potentially boost performance in other studies too, such as CAV-MAE and Audiovisual MAE, including our own.
Please note that MAViL is our concurrent work and we included it as grayed out in our main results following the guideline of covering all relevant papers.

Reviewer comment (Writing):
Writing. Some statements about the key contributions are vague. For example, the authors state that "employing augmentations in multi-modal contrastive learning requires careful consideration, as augmentations can severely distort the inter-modal correspondence." It would be helpful to provide some specific examples to illustrate how augmentations can distort the inter-modal correspondence.

Author response:
For a more straightforward understanding, we came up with a case where data augmentation causes distortion of the inter-modal correspondence. Let’s suppose that given an audio-visual pair of a barking dog, the dog disappears due to augmentation (e.g. random cropping). When using intra-modal invariant learning, the features of the original image (i.e. the dog is barking) and the augmented one (i.e. the dog disappears) are forced to become closer in the latent space. This will give the wrong signal to the model, adversely affecting the learning of correct inter-modal correspondence (i.e. the dog and the barking sound). On the other hand, in the context of equivariant learning, the embedding of the original image is transformed by the augmentation predictor and made closer to the embeddings of the augmented image in the intra-modal latent space. Then, we utilize original inputs (i.e. dog and the barking sound) that retain the semantics of the two modalities in the inter-modal latent space while ensuring that the embeddings of the augmented image and the original audio will not become directly close. This insight allows equivariance to take advantage of the effect of strong data augmentation in audio-visual representation learning.

Reviewer comment (Theory):
Writing. "The equivariance loss term in the proposed framework differs slightly from the single-modal equivariant self-supervised learning (Devillers & Lefort, 2023). The key distinction is whether or not the similarity of the positive pair is included in the denominator of the loss term." However, it is unclear why the positive pair is included in the denominator.

Author response:
As explained in Appendix A of the manuscript, including the positive pair in the denominator results in a relatively larger update for hard positives. This becomes particularly advantageous when stronger augmentations lead to an increased frequency of hard positives. Training with more hard positives, alongside the application of equivariance, substantially enhances the model's capability to comprehend detailed features in single-modal contexts. The intra-modal representation quality plays a pivotal role in understanding and integrating different modalities. Consequently, the semantically rich intra-modal representation promotes effective learning of audio-visual correspondence and audio-visual joint representations. We will add a detailed analysis in Appendix A.

Reviewer comment (Experiments):
Experiment. What if we applied Equimod to audio and visual data directly without the positive term in the loss?

Author response:
We have indeed explored this scenario, and the results are presented in Table 5 of our paper. The results indicate that using the loss function employed in Equimod leads to suboptimal performance in downstream tasks.  For a more in-depth analysis of these results, please refer to the discussion provided in Appendix A.