Paper: Lifelong Audio-video Masked Autoencoder with Forget-robust Localized Alignments (ICLR-2024)

Reviewer comment (Writing):
The paper does not introduce the proposed method very clearly, the writing of the paper should be polished.

Author response:
Thank you for your opinion. We have made efforts to more clearly describe our approach in sections 4.2 and 4.3, as well as Figure 4, and update them in our latest revision. We add a line indicator of Equation 4 to corresponding sentences to help you understand the algorithm of forget-robust patch selection in Section 4.2. In the below, we follow Section 4.2 and Section 4.3 step-by-step to provide more details.

Section 4.2 explains **the process of obtaining the current and past attention map information reflecting inter-task (past & current) importance** (*Section 4.2, paragraph 1, line 16*). Using this importance, the forget-robust patch selection aims to prune patches from current data that have a higher correlation with previous steps. The purpose of this pruning is to retain the past correlation knowledge while effectively learning new data (*Section 3.2, line 11~16*).

In Section 4.3, **we utilize the importance scores and pruning probability matrices to find the forget-robust and informative audio and video patches** (*Section 4.3, paragraph 1, line 1* and *Section 4.3, paragraph 3, line 1*). In the case of the audio, we have to preserve local correlation among audio patches due to the temporal continuity in the spectrogram (*Section 4.3, paragraph 1, line 2*). Hence, we follow the process below.
 
1. Process the audio importance score ($\textbf{I}\_{a}$) and audio pruning indicator ($\textbf{F}\_{a}$) in time-wise to obtain $\textbf{I}^{t}\_{a}$ and $\textbf{F}^{t}\_{a}$. (Section 4.3, paragraph 1). 

2. Group $\textbf{I}^{t}\_{a}$ in time chunks with length $\textbf{L}\_c$ to preserve the time continuity information. This grouping process generates a chunk-wise importance score ($\textbf{I}^{c}\_{a}$). (Section 4.3, paragraph 2, line 1~4)

3. Based on $\textbf{I}^{c}\_{a}$, we employ multinomial probability distribution to sample the time chunks that are most likely to be informative. (Section 4.3, paragraph 2, line 5)

4. Iterate through the sampled time chunks and accumulate the number of selected audio patches while excluding the forget-inducing audio patches referring to $\textbf{F}^{t}\_{a}$. 

5. We iterate until the number of selected audio patches reaches our target number, which is $\kappa\_{a}$. (Section 4.3, paragraph 2, line 6-7)

For video patches are selected in a much simpler manner.

1. For every element in the video importance score ($\textbf{I}\_{v}$) that aligns with indices marked as True in the video pruning indicator ($\textbf{F}\_{v}$), we set the value to zero. This process is to exclude the forget-inducing video patches in our video patch selection, yielding $\tilde{\textbf{I}\}_{v}$. (Section 4.3, paragraph 3, line 4)

2. Apply a multinomial probability distribution on $\tilde{\textbf{I}}\_{v}$ to select video patches that are most likely to be informative. (Section 4.3, paragraph 3, line 6)

Reviewer comment (Novelty):
The paper claims that "this is the first work that addresses a continual representation learning problem for audio-video tasks and identifies the crucial challenges that the new problem has." Hence, previous works about audio-visual continual learning (e.g., audiovisual continuous speech recognition, audiovisual continuous emotion recognition) should be introduced in the related work part. The differences between this paper and previous works about multimodal continual learning should also be further refined.

Author response:
To our current knowledge, investigations into audiovisual continual learning have emerged relatively recently [1, 2]. Since the recent works tackle the problem of audiovisual continual learning in a supervised learning setup (*Related Work, paragraph 2, line 10*), the claim that our work represents a new exploration into continual audio-video representation learning is valid.
 
As far as we understand, the suggested works (audiovisual continuous speech recognition [3] or audiovisual continuous emotion recognition [4]) seek to automatically predict the labels in a temporally continuous manner, which are not audiovisual continual learning and thus not related to our work.

The primary difference between the two recent works[1, 2] and our work lies in the focus: **the former focuses on audio-visual supervised continual learning** where the model is continuously trained on the audiovisual tasks with class labels. Conversely, **our work focuses on audio-visual unsupervised continual learning** where the model is continuously pre-trained with self-supervised loss terms and the pre-trained model is estimated on various downstream tasks. 

For a more comprehensive grasp of these distinctions, we have included Figure 7 and *Appendix B, Comparison between SCL setup and UCL setup* in our latest revision. This section delineates and contrasts supervised continual learning with unsupervised continual learning, facilitating a clearer understanding of our approach.

[1] Mo et al,. Class-incremental grouping network for continual audio-visual learning, ICCV 2023.  
[2] Pian et al., Audio-visual class-incremental learning, ICCV 2023.  
[3] Rouditchenko et al., AV-CPL: Continuous Pseudo-labeling for Audio-Visual Speech Recognition  
[4] Zhang et al., Continuous Emotion Recognition with Audio-visual Leader-follower Attentive Fusion, ICCV workshop 2021

Reviewer comment (Experiments):
Experiments in section 3.2 are not convincing enough, unpaired data naturally creates misleading cross-modal attention maps. These experiments do not fully explain the reason for representational forgetting.

Author response:
It is crucial to highlight that **the misalignment of past-audio-guided video attention maps are located in misleading locations where uncorrelated sound sources exist** (*Section 3.2, line7*). Consider Figure 17 top row: despite the presence of musical instruments in the visual scene, when presented with uncorrelated human voice audio data, the audio-guided video attention maps primarily focus on human features. **Originally, such misalignment might not pose significant challenges in conventional audio-video representation learning**, as **static and generalized data distribution** typically aids the model in distinguishing between different semantics (human-playing-musical-instrument, human-conversation, etc) that share similar visual or audio characteristics (human features).

**However, within the continual audio-video representation learning paradigm, these misalignments significantly deteriorate the learned representation.** During continual pre-training, the model encounters **shifting and biased data distributions**. Consequently, training on the current task leads to the weakening of the audio-visual correlations from the previous tasks. For example, when the current task involves data solely featuring humans playing musical instruments, the model would potentially link auditory data of musical sounds to visual data containing either humans or musical instruments. **If the past task consists of human-conversational data, training on the current task leads to the weakening of the audio-visual correlation of human visuals and human voice learned from the past task.** Instead, the model potentially connects between human visual features and musical sound features from the current ask. This leads to the forgetting of the audio-video representation learned from the past task. To address this issue, we prioritize selecting forget-robust patches by comparing the current-data-induced cross-attention map and the past-data-induced cross-attention map. (*Section 4.2*)

Reviewer comment (Reproducibility):
The paper says: "In order to assess the relative importance of current data and past data, we further compute past-data-induced attention maps." However, in continual learning, past data is usually unavailable, so how to compute past-data-induced attention maps? This is a very important question and it should be explained in detail.

Author response:
Rehearsal-based continual learning, where the model temporally stores a small fraction of previous task data in a small-sized buffer during pre-training, is one of the most effective and popular/well-known directions in CL tasks [1,2,3,4], achieving a good trade-off between performance and efficiency. And we follow the same assumption since this is also valid in our scenario.

From the rehearsal memory, we use the past data to compute past-data-induced attention maps. **Specifically, we decide to exploit the past discriminative queries stored in the rehearsal memory, which was calculated during the pre-training of the past tasks** (*Section 4.2, line 15*), since processing the raw past data to compute the past attention maps could be a waste of computational resources. Storing the past information only increases negligible amounts of memory (*Section 5.2, paragraph 2, line 8*).

The computation of past-data-induced attention maps is as follows:

1. We use importance scores ($\textbf{I}\_{a}, \textbf{I}\_{v}$) from *Section 4.1* to extract patch indices whose patches have high correlation with their modality pair. This process yields sorted indices for audio and video, respectively. ($\textbf{S}\_{a}, \textbf{S}\_{v}$) (*Section 4.2, line 8*)

2. With the sorted indices, we collect discriminative keys ($\hat{\textbf{k}}\_{a}, \hat{\textbf{k}}\_{v}$) from the key embeddings ($\textbf{k}\_{a},\textbf{k}\_{v}$) of the current data. (*Section 4.2, line 10*)

3. Similarly, we also collect discriminative queries from the query embeddings ($\textbf{q}\_{a},\textbf{q}\_{v}$) of the current data. We subsequently apply weighted mean on the collected queries using the importance scores as weight. This process yields the final discriminative queries ($\hat{\textbf{q}}\_{a}, \hat{\textbf{q}}\_{v}$). (*Section 4.2, line 11*)

4. Using the discriminate keys and queries in 2. and 3., we compute the current-data-induced attention maps ($\hat{\textbf{A}}\_{a}, \hat{\textbf{A}}\_{v}$) that represent the importance of each current patch to the current modality pair. (*Section 4.2, line 13*)

5. We sample the past discriminative queries ($\hat{\textbf{q}}\_{a}^{p}, \hat{\textbf{q}}\_{v}^{p}$) from the rehearsal memory, which were stored during the pre-training of the past tasks. 

6. Using the queries, we compute the past-data-induced attention maps by combining the discriminative keys from current data ($\hat{\textbf{k}}\_{a}, \hat{\textbf{k}}\_{v}$) and the past queries ($\hat{\textbf{q}}\_{a}^{p}, \hat{\textbf{q}}\_{v}^{p}$). (*Section 4.2, line 15*) The past-data-induced attention maps exhibit the importance of each current patch to the past modality pair.

We have added a line indicator of Equation 4 to corresponding sentences to help you understand the process of calculating the past-data-induced attention maps in Section 4.2 in our latest revision.

[1] Jin et al., Gradient-based Editing of Memory Examples for Online Task-free Continual Learning, NeurIPS 2021  
[2] Yoon et al., Online Coreset Selection for Rehearsal-based Continual Learning, ICLR 2022  
[3] Caccia et al., New Insights on Reducing Abrupt Representation Change in Online Continual Learning, ICLR 2022  
[4] Jeeveswaran et al., BiRT: Bio-inspired Replay in Vision Transformers for Continual Learning, ICML 2023

Reviewer comment (Novelty):
I haven't specifically focused my research on lifelong learning, but from a methodological perspective, this work primarily involves the direct utilization of basic attention mechanisms.

Author response:
$\rightarrow$ We respectfully disagree with the Reviewer's claim that our approach simply relies on a basic attention mechanism. First, we emphasize that the utilization of attention maps as a versatile tool for solving a variety of problems has been extensively validated in many studies [1,2,3]. The **appropriate/effective utilization of attention mapping is the core originality of these studies,** and the novelty of our proposed approach lies in utilizing this tool to solve the problem of learning continuous audio-video representations in an environment where task identity is not provided.

In particular, our method is sufficiently novel with multiple unique contributions, and leveraging attention maps is only a small part of our methodology. From the motivation and problem statement perspectives, we first clarify the critical challenges in multimodal (audio-video) continual pre-training; *1) sparse spatiotemporal correlation between the audio-video pairs*, and *2) representational forgetting of audio-video relationships*. Furthermore, we discover the potential root causes of the representational forgetting by visualizing the cross-attention maps induced by past data (*Section 3.2*) and use these observations in the attention maps to formulate our approach. 

Next, from a methodology perspective, our FLAVA contains multiple unique and meaningful core components. To solve the challenge of *1) sparse spatiotemporal correlation between the audio-video pairs*, we actively leverage the cross-attention information to compute importance score matrices to locate the highly correlated audio(video) patches with respect to their video(audio) data. **Pre-training with the selected patches from the importance score enables the model to achieve better audio and video clustering within the multi-modal representation space**, as we analyzed in *Appendix G, paragraph 4*.

To address the challenge of *2) representational forgetting of audio-video relationships*, we compute inter-task (current-past) importance by estimating a cross-modal attention map of current data from past data (*Section 4.2*). While various continual learning approaches leverage the attention maps from the past model, which is trained up to the previous task, to perform knowledge distillation or regularization for the purpose of alleviating forgetting [3][4], **we have our own originality that we extend to investigating the impact of past tasks on the current task attention map activations.** To estimate the impact, we exploit the query embeddings from past data stored in the rehearsal memory to activate cross-attention maps on the current data, and compare the past-data-induced cross-attention maps with the current-data-induced cross-attention maps for selecting forget-robust patches among the current data. This strategy significantly improves forgetting robustness during multimodal continual pre-training. 

Furthermore, **our method harnesses a video-guided audio cross-attention map and introduces an approach to retain the local correlation information during patch pruning.** (*Section 4.3*). Although the audio-guided video cross-attention map has been commonly used in various fields [3], the video-guided audio cross-attention map and its utilization have been understudied. We analyze the audio-guided video cross-attention map and suggest a way of computing it in order to preserve the local correlation in the audio spectrogram. **We also conducted an ablation study of our proposed audio patch selection to give more insights into audio patch selection (*Appendix E, Audio patch selection strategy*).**

[1] Liu et al., Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, ICCV 2021  
[2] Lee et al., Sparse Token Transformer with Attention Back Tracking, ICLR 2023  
[3] Pian et al., Audio-Visual Class-Incremental Learning, ICCV 2023  
[4] Pelsin et al., Towards Exemplar-Free Continaul Learning in Vision Transformers: an Account of Attention, Functional and Weight Regularization, CVPR workshop 2022

Reviewer comment (Evaluation):
The best performance on VGGSound should be at least 66.+% now, which is much higher than 58.65% here. What’s the main reason of this gap? Due to that there are more supervised labels in those works? If so, what will the performance be like of this work if we also provide labels? Will it also be improved by a large margin to close to 70%?

Author response:
$\rightarrow$ **This disparity arises due to fundamental differences in the experimental setups.** Comparing the outcome of our continual representation learning directly with results from current representation learning doesn’t yield a meaningful one-to-one comparison.

The 66.+% accuracy stems from models that are trained with a full dataset, whereas our reported accuracy is based on a continual learning scenario. In our approach, we use a subset of the dataset and split it into various tasks to train the model on the sequence of tasks.

For clearer comprehension of continual learning, we have included Figure 7 and provide details of unsupervised continual learning in *Appendix B, Comparison between SCL setup and UCL setup* in our latest revision

Reviewer comment (Experiments):
Beyond the proposed specific trainable module, is it possible to introduce this proposed manner into the learning of large-scale audio-visual models? What would be the results like of using this strategy to train large-scale models? Will it lose its effect when coming to large-scale data and models?

Author response:
$\rightarrow$ **Our approach has the potential to enhance the efficiency and sustainability of large-scale audio-video models.**

Our approach processes the unlabeled videos to learn audio-video correlations with much smaller GPU memory occupancy (Table 2). Training large-scale models requires processing web-scale data and huge computational costs. Hence, the large-scale model can efficiently learn audio-video knowledge from web-scale video data without human intervention.

Moreover, for the large-scale models to be sustainable in the future, the models should continually learn new knowledge. In our approach, we continually pre-train the audio-video model efficiently by selecting informative and forget-robust patches from the current task. Hence, **our approach exhibits high adaptability to new concepts by catching audio-video correlations from sparse spatiotemporal correlations in videos while preserving past audio-video knowledge by learning forget-robust patches in the videos.**

Reviewer comment (Writing):
Please clarify what is meant by "relative importance of current data and past data".

Author response:
$\rightarrow$ In Section 4.2, we outline our proposed method for selecting forget-robust patches (*Section 4, line 4*). We sample past data from the rehearsal memory, containing past discriminative queries, importance scores, and pruning probability matrices (Algorithm 1, line 6). To find audio and video patches that are semantically closer to the past tasks than the current task, we suggest exploiting attention maps using the current and past data. **We compute the importance of each patch to the past and current tasks by comparing the past and current attention maps, which is used for selecting forget-robust patches for current data.**

Reviewer comment (Reproducibility):
Please clarify how past discriminative queries are selected.

Author response:
$\rightarrow$ We utilize reservoir sampling that randomly samples and replaces past instances during training. **Hence, we randomly sample past audio-video pairs together with past discriminative queries, importance scores, and pruning probability matrices from the rehearsal memory** during the continual pre-training phase, following the random sampling in our baselines (e.g., LUMP, DER++, CLS-ER, ...). We sincerely apologize for the confusion that arose due to our oversight. To rectify this, we have made updates to *Section 5.1, Baselines, line 4* of our latest revision.

Reviewer comment (Experiments):
With increasing continual learning steps, please clarify how past data from previous steps can be better leveraged to improve memory usage.

Author response:
$\rightarrow$ Improving the rehearsal memory usage could be one of the important problems in rehearsal-based continual learning since an efficient way of storing past data could enrich the diversity within the rehearsal memory that has memory constraints. This becomes much more important in audio-video continual learning since video data size is much larger than the size of an image. **One way of achieving efficient rehearsal memory usage is to store the pruned data during the continual pre-training**, rather than preserving raw data within the memory. During the rebuttal period, we explored this approach and summarized the zero-shot retrieval task results in the Tables below (*acc* indicates average accuracies, *fgt* indicates average forgetting). FLAVA+ implements our patch selection strategy while its memory stores the selected patches instead of the raw data.

Comparison between FLAVA and FLAVA+ (first, and second rows in the Tables) elucidates a clear trade-off associated with storing pruned data in the rehearsal memory. Despite the effectiveness of our method in selecting informative patches from audio-video data, pruning the patches would result in a loss of the data’s contextual information. Consequently, replaying the pruned data from the memory yields a weakened effect in retaining past knowledge due to the diminished total amounts of information. However, this strategy significantly saves nearly half of the memory during continual pre-training while remaining highly competitive compared to other baselines. **Thus, in scenarios where the model faces stricter memory constraints, FLAVA+ presents itself as a potentially advantageous approach.**

Moreover, when we increase the number of past instances in FLAVA+ to align with the memory size of FLAVA, FLAVA+ outperforms FLAVA by a large margin (first, and third rows in the Tables). This significant improvement arises from the augmented volume of data in the rehearsal memory, **allowing the memory to effectively maintain audio-video knowledge from past tasks, simultaneously enhancing data diversity in current task training.**

This approach offers an orthogonal direction to our original contribution, demonstrating our method's potential to efficiently utilize rehearsal memory.

VGGSound retrieval task

| Method   | Compress | # of past data | Memory size (GB) | A$\rightarrow$V acc | A$\rightarrow$V fgt | V$\rightarrow$A acc | V$\rightarrow$A fgt | 
|---------------|------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
| FLAVA | x | 2k | 5.63 | 14.16 | **4.38** | 14.07 | 4.65 |
| FLAVA+ | o | 2k | 2.89 | 13.29 | 6.11 | 13.07 | 6.50 |
| FLAVA+ | o | 4k | 5.78 | **15.44** | 4.62 | **15.53** | **3.89** |

AudioSet retrieval task

| Method   | Compress | # of past data | Memory size (GB) | A$\rightarrow$V acc | A$\rightarrow$V fgt | V$\rightarrow$A acc | V$\rightarrow$A fgt | 
|---------------|------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
| FLAVA | x | 5k | 14.08 | 12.35 | 6.95 | 11.29 | 8.38 |
| FLAVA+ | o | 5k | 7.23 | 11.56 | 8.25 | 10.21 | 10.48 |
| FLAVA+ | o | 10k | 14.45 | **15.26** | **5.74** | **14.47** | **6.70** |

Reviewer comment (Theory):
Please clarify why the proposed method can tackle the issues mentioned in Figures 1, 2, and 3.

Author response:
$\rightarrow$ Our method mainly focuses on finding locally aligned and forget-robust patches in the raw data from the current task to solve the sparse spatiotemporal correlation and representation forgetting problems suggested in Figures 1,2, and 3 (*Section 4, first paragraph*).
- Figure 1 illustrates the real-world scenarios where the video data distribution is dynamically shifting, and thus the model should adapt to the new distribution without forgetting the past knowledge when past data is not available. This scenario underscores the necessity for continual audio-video representation learning. Audio-video continual pre-training raises two challenges, which are *1) sparse spatiotemporal correction, and 2) representational forgetting of audio-video relationships* (*Introduction, paragraph 3, line 2*). The first challenge indicates the trait of the video data where only a few regions in the visual scene are correlated with its audio, as we observed in Figure 2. Hence, **we suggest using attention maps as importance scores to statistically sample locally aligned patches** (*Section 4-1*).

- The second challenge is closely related to Figure 2 and Figure 3. In Figure 2, we observe that during continual pre-training, the model forgets the correct audio-video correlation ( *Introduction, paragraph 3, line 5*). In Figure 3, we find the potential root cause of the forgetting: misaligned audio-video correlation between the current and past tasks. This misalignment could lead to learning spurious audio-video correlation, or the past audio-video correlation could be overwritten (*Section 3.2, line 12*). Hence, **we focus on selecting the forget-inducing that aims to prune patches from current data that have a higher correlation with previous tasks** (*Section 4-2*).

Reviewer comment (Novelty):
Two concurrent related works [1, 2] have addressed audio-visual continual learning, and the second also observed and addressed audio-visual alignment forgetting issues. The authors can discuss the relevance and differences among these works in more detail, especially the differences between the proposed method and the second work. Although it is clear that the works are concurrent, more discussions would be helpful to distinguish between the different works.

Author response:
$\rightarrow$ Thank you for noticing us in the first related work. We have included it in our citation of the latest revision. The primary difference between the suggested two works and our work lies in the focus: **the former focuses on audiovisual supervised continual learning** where the model is continuously trained on the audiovisual tasks with class labels. Conversely, **our work focuses on audiovisual unsupervised continual learning** where the model is continuously pre-trained with self-supervised loss terms and the pre-trained model is estimated on various downstream tasks. 

For a more comprehensive grasp of these distinctions, we have included Figure 7 and *Appendix B, Comparison between SCL setup and UCL setup* in our latest revision. This section delineates and contrasts supervised continual learning with unsupervised continual learning, facilitating a clearer understanding of our approach.

[1] Mo, Shentong, Weiguo Pian, and Yapeng Tian. "Class-incremental grouping network for continual audio-visual learning." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[2] Pian, Weiguo, et al. "Audio-visual class-incremental learning." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

---

$\rightarrow$ The differences between the AV-CIL (second work) and our work are also clear as follows:

- **In our setting, the absence of class labels allows for a direct assessment of audio-video correlation, enabling a visual observation of the extent of forgetting.** The AV-CIL observation regarding the forgetting of audio-visual correlation is rooted in a supervised learning setting. Consequently, we cannot disregard the influence of class labels that guide visual attention, similar to traditional supervised continual learning. 

- The AV-CIL observes the phenomenon where visual attention maps from previous tasks could vanish during continual training. In our work, we too have observed a parallel phenomenon, evident in both Figure 2 and Figure 16. **Importantly, our focus extends to investigating the impact of previous tasks on the current task attention map activations as illustrated in Figure 3 and Figure 17.**

- Motivated by the observation in Figure 3, we suggest an approach aimed at selecting patches that are informative and forget-robust. The AV-CIL focuses on distilling past knowledge into the current model using the past model.

- The AV-CIL relies on the assumption that the model possesses task boundary information throughout its training process, which is required to separate the past model and current model. Conversely, our approach operates within a task-free setting (*Section 3.1, paragraph 1, line 4*) where the model does not know the task boundary information. **The task-free setting is a notably more challenging scenario**, requiring the model to adapt dynamically without relying on explicit task boundaries.

- The AV-CIL framework overlooks the integration of a video-guided audio attention map, a crucial element in comprehending audiovisual tasks. Conversely, **our approach places equal emphasis on harnessing the potential of the video-guided audio attention map.**

Reviewer comment (Evaluation):
On the audiovisual classification task, the improvements of the proposed method are marginal; the reason for this is unclear.

Author response:
$\rightarrow$ **Our method primarily focuses on an effective approach to retaining and learning audio-video correlation.** Hence, our method is especially effective in tasks that require high-level audio-video alignment (i.e., audiovisual retrieval task). On the other hand, the classification task requires discriminative features to classify a given video into its label, rather than focusing on correlations of audio-video multimodality, so the improvement is not dramatic. On the other hand, we achieved the highest average accuracy on the classification task for both datasets, which is statistically significant when considering the standard deviation. 

During the rebuttal period, we also suggest FLAVA+, an extension of FLAVA, where its memory stores the selected patches. We update the details of FLAVA+ (Section 4.3) and experiment results of FLAVA+ (Table 1, 4, 7) in our latest revision. We also summarize the classification performance of FLAVA+ in the below Table. **FLAVA+ significantly outperforms baseline methods in large margins.** The introduction of FLAVA+ represents an orthogonal direction of our initial contribution. It represents our method’s potential to efficiently utilize rehearsal memory.

Audiovisual classification tasks

| Method   | VGGSound acc | VGGSound fgt | AudioSet acc | AudioSet fgt |
|---------------|------------|-----------|------------|-----------|
| FLAVA | 58.65 | **0.64** | 65.66 | 0.89 |
| FLAVA+ | **59.07** | 0.80 | **66.05** | **0.78** |

We also performed additional audiovisual downstream tasks (audiovisual event localization, sound source localization) during the rebuttal period, and found that the proposed method outperformed the other baselines (*Appendix E, Audiovisual Event Localization and Sound Source Localization*), supporting the superiority of the proposed method. The Table below shows the audiovisual event localization result.

Audiovisual event localization task

| Method | Acc | 
|---------------|------------|
| Finetune | 52.56 |
| ER | 54.98 |
| DER++ | 55.81 |
| GMED | 55.98 |
| LUMP | 55.06 |
| **FLAVA (Ours)** | **56.68** |
| **FLAVA+ (Ours)** | **56.68** |

Reviewer comment (Experiments):
The proposed method assumes access to audio-visual data (ie. memory) from the previous tasks, which doesn't seem like a realistic scenario. I think a more realistic scenario is to have a black-box model without access to the pre-training data, and then the model is presented with a new task / training data. Otherwise, if we have access to the data from the previous tasks, why not just combine the data from the new task and the old tasks to train the model? I didn't see a comparison with this kind of approach. Besides, in Figure 9, the proposed method improves with a larger memory size (ie. more access to data from previous tasks), which further shows the benefit of training on combined data from different tasks.

Author response:
> **Weakness 2-1: The proposed method assumes access to audio-visual data (ie. memory) from the previous tasks, which doesn't seem like a realistic scenario.** 
> **Weakness 2-3: If we have access to the data from the previous tasks, why not just combine the data from the new task and the old tasks to train the model? In Figure 9, the proposed method improves with a larger memory size, which further shows the benefit of training on combined data from different tasks.**


$\rightarrow$ Storing all videos and training from scratch has huge computational costs, privacy issues, and memory constraints, as discussed in Weakness 1-1. To alleviate these problems, rehearsal-based continual learning, where the model temporally stores a small fraction of previous task data in a small-sized buffer **during training**, has become one of the most effective and popular/well-known directions in CL tasks [1,2,3,4], achieving a good trade-off between performance and efficiency. And we follow the same assumption since this is also valid in our scenario.

An ever-growing dataset that the Reviewer suggested **needs an assumption of having an infinite amount of resources, which is clearly far from the realistic scenario.** The purpose of the experiment in Figure 9 is to show the **trade-off** between rehearsal memory size and downstream task performance (Appendix E, Effect of Rehearsal Memory), not to demonstrate a **one-sided benefit of expanding storage size**.

[1] Jin et al., Gradient-based Editing of Memory Examples for Online Task-free Continual Learning, NeurIPS 2021  
[2] Yoon et al., Online Coreset Selection for Rehearsal-based Continual Learning, ICLR 2022  
[3] Caccia et al., New Insights on Reducing Abrupt Representation Change in Online Continual Learning, ICLR 2022  
[4] Jeeveswaran et al., BiRT: Bio-inspired Replay in Vision Transformers for Continual Learning, ICML 2023  

--- 

> **Weakness 2-2:  I think a more realistic scenario is to have a black-box model without access to the pre-training data, and then the model is presented with new task/training data.**

$\rightarrow$ The suggested scenario is a simple pretraining-finetuning strategy that has two critical limitations in view of continuity. First, given the pre-trained model, the model cannot transfer the knowledge across different downstream tasks since the model aims to transfer the pre-trained representation to the downstream tasks individually. Second, the user requires individual fine-tuned models according to the downstream tasks, requiring N times larger memory budgets proportional to the number of downstream tasks N.

Due to these reasons and as we explicated in *Weakness 1-1*, **the lifelong learnable & sustainable model for real-world scenarios should adapt to new knowledge continuously.**

Reviewer comment (Writing):
The writing / explanation of the method is not clear enough. I didn't understand Sections 4.2 and Sections 4.3; the writing should be improved with more high-level explanation. It would be helpful if each line in equation (4) was explained separately. Algorithm 1 was helpful for a high-level understanding, perhaps more detail could be added there.

Author response:
> **Weakness 4-1: It would be helpful if each line in equation (4) was explained separately. Algorithm 1 was helpful for a high-level understanding, perhaps more detail could be added there.**

$\rightarrow$ Thank you for your suggestion! We have added a line indicator of Equation 4 to corresponding sentences to help you understand the algorithm of forget-robust patch selection in Section 4.2. **We have also added more explanations in Section 4.2, Section 4.3, Figure 4, and Algorithm 1 to help you better understand the overall process.** In the below, we follow Section 4.2 and Section 4.3 step-by-step to provide more details. Please let us know if there's anything we can do to make the description clearer! We'll be happy to incorporate your suggestions.

--- 

> **Weakness 4-2: The writing/explanation of the method is not clear enough. I didn't understand Sections 4.2 and 4.3.**

$\rightarrow$ Thank you for your opinion. We have made efforts to more clearly describe our approach in sections 4.2 and 4.3, as well as Figure 4, and update them in our latest revision.

Section 4.2 explains **the process of obtaining the current and past attention map information reflecting inter-task (past & current) importance** (*Section 4.2, paragraph 1, line 16*). Using this importance, the forget-robust patch selection aims to prune patches from current data that have a higher correlation with previous steps. The purpose of this pruning is to retain the past correlation knowledge while effectively learning new data (*Section 3.2, line 11~16*).

In Section 4.3, **we utilize the importance scores and pruning probability matrices to find the forget-robust and informative audio and video patches** (*Section 4.3, paragraph 1, line 1* and *Section 4.3, paragraph 3, line 1*). In the case of the audio, we have to preserve local correlation among audio patches due to the temporal continuity in the spectrogram (*Section 4.3, paragraph 1, line 2*). Hence, we follow the process below.
 
1. Process the audio importance score ($\textbf{I}\_{a}$) and audio pruning indicator ($\textbf{F}\_{a}$) in time-wise to obtain $\textbf{I}^{t}\_{a}$ and $\textbf{F}^{t}\_{a}$. (Section 4.3, paragraph 1). 

2. Group $\textbf{I}^{t}\_{a}$ in time chunks with length $\textbf{L}\_c$ to preserve the time continuity information. This grouping process generates a chunk-wise importance score ($\textbf{I}^{c}\_{a}$). (Section 4.3, paragraph 2, line 1~4)

3. Based on $\textbf{I}^{c}\_{a}$, we employ multinomial probability distribution to sample the time chunks that are most likely to be informative. (Section 4.3, paragraph 2, line 5)

4. Iterate through the sampled time chunks and accumulate the number of selected audio patches while excluding the forget-inducing audio patches referring to $\textbf{F}^{t}\_{a}$. 

5. We iterate until the number of selected audio patches reaches our target number, which is $\kappa\_{a}$. (Section 4.3, paragraph 2, line 6-7)

For video patches are selected in a much simpler manner.

1. For every element in the video importance score ($\textbf{I}\_{v}$) that aligns with indices marked as True in the video pruning indicator ($\textbf{F}\_{v}$), we set the value to zero. This process is to exclude the forget-inducing video patches in our video patch selection, yielding $\tilde{\textbf{I}\}_{v}$. (Section 4.3, paragraph 3, line 4)

2. Apply a multinomial probability distribution on $\tilde{\textbf{I}}\_{v}$ to select video patches that are most likely to be informative. (Section 4.3, paragraph 3, line 6)

Reviewer comment (Evaluation):
The proposed method is only significantly better than the baseline methods for continual learning (ie. LUMP) on the VGGSound retrieval task. For the AudioSet retrieval task and VGGSound / AudioSet classification tasks, the improvement is small.

Author response:
> **Weakness 5: The proposed method is only significantly better than the baseline methods for continual learning (ie. LUMP) on the VGGSound retrieval task. For the AudioSet retrieval task and VGGSound / AudioSet classification tasks, the improvement is small.**

$\rightarrow$ We respectfully disagree with the Reviewer's comment that the improvement is small. We would like to emphasize that for the AudioSet video-to-audio zero-shot retrieval task, our method **outperforms the strongest baselines at 5.42%**, and our method **ranks the highest R@1 score in all experiments, including the VGGSound and AudioSet**. We also rank the highest average accuracy in the classification tasks for both datasets, which is **statistically significant observing the standard deviation**.

During the rebuttal period, we also suggest FLAVA+, an extension of FLAVA, where its memory stores the selected patches. We update the details of FLAVA+ (Section 4.3) and experiment results of FLAVA+ (Table 1, 4, 7) in our latest revision. We also summarize the performance of FLAVA+ in the below Tables. **FLAVA+ significantly outperforms baseline methods in large margins.** The introduction of FLAVA+ represents an orthogonal direction of our initial contribution. It represents our method’s potential to efficiently utilize rehearsal memory.

VGGSound zero-shot retrieval task

| Method   | A$\rightarrow$V acc | A$\rightarrow$V fgt | V$\rightarrow$A acc | V$\rightarrow$A fgt | 
|---------------|------------|-----------|-----------|-----------|
| FLAVA | 14.16 | **4.38** | 14.07 | 4.65 |
| FLAVA+ | **15.44** | 4.62 | **15.53** | **3.89** |

AudioSet zero-shot retrieval task

| Method   | A$\rightarrow$V acc | A$\rightarrow$V fgt | V$\rightarrow$A acc | V$\rightarrow$A fgt | 
|---------------|------------|-----------|-----------|-----------|
| FLAVA | 12.35 | 6.95 | 11.29 | 8.38 |
| FLAVA+ | **15.26** | **5.74** | **14.47** | **6.70** |

Audiovisual classification tasks

| Method   | VGGSound acc | VGGSound fgt | AudioSet acc | AudioSet fgt |
|---------------|------------|-----------|------------|-----------|
| FLAVA | 58.65 | **0.64** | 65.66 | 0.89 |
| FLAVA+ | **59.07** | 0.80 | **66.05** | **0.78** |

In addition, we conduct additional audiovisual downstream tasks (audiovisual event localization, sound source localization) during the rebuttal period, found that and our method surpasses other baselines (*Appendix E, Audiovisual Event Localization and Sound Source Localization*), which supports the superiority of our proposed method. The Table below summarizes the audiovisual event localization task results.

Audiovisual event localization task

| Method | Acc | 
|---------------|------------|
| Finetune | 52.56 |
| ER | 54.98 |
| DER++ | 55.81 |
| GMED | 55.98 |
| LUMP | 55.06 |
| **FLAVA (Ours)** | **56.68** |
| **FLAVA+ (Ours)** | **56.68** |

Reviewer comment (Experiments):
The experiments leave questions unanswered (see my questions below).

Author response:
> **Question 1-1: The pre-training / fine-tuning / task / evaluation splits of the datasets should be more clear. Can you provide a table in the appendix with the precise number of clips for each split?**

$\rightarrow$ As the reviewer suggested, **we additionally provide the additional table for statistics of the dataset in *Figure 6* in our latest revision.** We use a subset of the VGGSound and the AudioSet dataset and split it according to their high-level category information. The finetuning data is identical to the pre-training data, as commonly practiced in other self-supervised learning research [1, 2] when evaluating the learned representation from the pre-training data. In the case of the zero-shot retrieval task, we use a subset of the evaluation retrieval dataset from the CAV-MAE [1].

[1] Gong et al., Contrastive Audio-Visual Masked Autoencoder, ICLR 2023

[2] He et al., Masked Autoencoders Scalable Vision Leaners, CVPR 2022

--- 

> **Question 1-2: I am wondering how the training and evaluation data differs with CAV-MAE.**

$\rightarrow$ **We use the subset of the VGGSound and AudioSet for continual pre-training according to their high-level category information** (*Appendix B, Audiovisual Dataset Configuration*). We split the VGGSound dataset into 8 tasks based on the category labels (sports, music, ...) and each task consists of 6k-8k video clips from 20 different classes. The AudioSet follows a similar process. We follow the official class hierarchy information to split it into 7 tasks. Since the AudioSet is a multi-label dataset, we exclude the samples that are included in more than one task in order to prevent the past task semantics from leaking into the current task. **In contrast, the CAV-MAE uses the whole VGGSound and AudioSet dataset**.

--- 

> **Question 2: Why are the tasks set up to be "zero-shot" by excluding classes from all of the continual learning datasets? I don't understand how this measures "forgetting" since the model isn't being tested on classes that it was trained on.**

$\rightarrow$ We believe this misunderstanding is from the confusion of the scenario of continual self-supervised learning. For a clearer understanding, in Figure 7 in our latest revision, we have illustrated the supervised and the unsupervised continual learning setups in both the training and evaluation setup. We also elaborate on it in *Appendix B, Comparison between two different setups* in our latest revision.

We evaluate continual self-supervised learning methods on audio-video multimodality using zero-shot retrieval tasks to assess the continually pre-trained model’s ability to align audio and video into the multi-modal representation space and quantify the amount of forgetting. In the zero-shot retrieval task, we do not need class labels, but we ask the continually pre-trained model to find the correct audio(or video) pair given video(or audio) at each end of the pre-train task. 

To assess the task-specific retrieval task performance, we split the retrieval evaluation data according to the category information, as we split the training and evaluation dataset. In this way, **we test the model on the whole retrieval evaluation data and extract the performance corresponding to tasks it has already seen.**

We compare each maximum of task-specific ($i$) performance of the pre-trained models ($\underset{t\in\{1,\ldots, \mathcal{T}-1\}}{\text{max}}\textit{acc}\_{t,i}$) with the task-specific performance at the completion ($\mathcal{T}$) of the tasks ($\textit{acc}\_{\mathcal{T},i}$) (*Section 5.1, Evaluation Metrics*, Figure 7). In this way, we can estimate how the model's representation ability has changed during the continual pre-training, which is quantified as *forgetting*. This process is also applied to the audio-video classification task.

--- 

> **Question 4: Is it possible to compare with AV-CIL on the classification task, since that requires supervised fine-tuning?**

$\rightarrow$ The AV-CIL follows (1) supervised continual learning, (2) trains its own backbone structure, (3) requires class labels to implement task-wise knowledge distillation losses, and (4) assumes class-incremental learning, which leverages task boundary information during training, while our setting follows (1) unsupervised continual learning, (2) uses CAV-MAE backbone and freeze the backbone in downstream tasks, (3) pre-trained on contrastive loss & MAE objectives without class labels, and (4) assumes task-free learning. Hence, the AV-CIL itself is not applicable in our setting, and direct comparison is out of our interest.

For a more comprehensive grasp of these distinctions, we have included Figure 7 and *Appendix B, Comparison between SCL setup and UCL setup* in our latest revision. This section delineates and contrasts supervised continual learning with unsupervised continual learning, facilitating a clearer understanding of our approach.

Reviewer comment (Writing):
There are some distracting typos ("vidoe," "Fintune").

Author response:
> **Weakness 7: Typos (“vidoe”, “Fintune”)**

$\rightarrow$ We thank you for finding the typos that would be a disturbance in explaining our findings. We have corrected the typos in the latest revision.