Paper: Beyond Unimodal Learning: The Importance of Integrating Multiple Modalities for Lifelong Learning (ICLR-2024)

Reviewer comment (Experiments):
My major concern with this paper is the lack of comparison and experiments. The evaluation seems a bit weak to me as all the experiments are conducted on VGGSound only, and the baseline Experience Replay for comparison with the proposed approach is from 2018. I wonder if the authors could apply some more recent unimodal CL approaches ([1][2] etc.) to the problem.

Author response:
we would like to underscore that our study's objective was not to propose a state-of-the-art solution but rather to present a compelling case for advancing toward multimodal CL and establishing a standardized benchmark for multimodal CL to encourage research in this promising direction. The primary contribution of our work lies in demonstrating the performance gains of Multimodal Experience Replay compared to unimodal experience replay, particularly in challenging CL scenarios, corresponding to the established CL settings in the unimodal context. Additionally, we introduced a promising approach for aligning modalities to enhance complementary learning, facilitating both multimodal and unimodal inference as an initial baseline for future research to build upon.

Our choice of Experience Replay in this study was motivated by two key reasons. Firstly, ER, specifically rehearsal-based methods, remains highly effective in mitigating forgetting, especially in the demanding Class-IL setting. Secondly, the adoption of ER aligns with the understanding of the human brain, where rehearsal is considered a critical component for information consolidation, contributing to lifelong learning.

Our findings illustrate that Multimodal Experience Replay significantly enhances model performance in challenging settings. For instance, in the case of Seq-VGGSound, Multimodal ER achieves superior performance even with half the number of samples compared to unimodal CL (compare unimodal performance with a buffer size of 1000 with multimodal ER with a buffer size of 500). The performance gains are even more pronounced with our proposed structure-aware multimodal replay. These results underscore the effectiveness of multimodal learning and replay, which is a key takeaway from our study.

Furthermore, while there are several state-of-the-art methods in unimodal CL, the majority of them cannot be applied directly in a multimodal setting and would require modifications and adaptation to multimodal architecture.

Additionally, our ablation study in the Appendix (Table 3) delineates the contributions of various components in Structure-Aware Multimodal Learning. Notably, the combination of Unimodal Memory and Consistency Regularization (UM + CR)  corresponds to an adaptation of DER++ [1] in a multimodal setting, which serves as a strong baseline in unimodal CL. We demonstrate that SAMM provides substantial performance gains over this baseline (34.51 vs. 29.13).

Importantly, the overarching message from our study is that akin to humans, multimodal learning may be pivotal in enabling effective CL in DNNs. We aim to encourage the broader research community to transition from unimodal CL to multimodal CL.

Reviewer comment (Novelty):
I understand that [4] is published after the submission ddl, but it would be good if the authors could comment a few sentences about the differences with them in the rebuttal.

Author response:
[4] can indeed be considered as parallel work which echoes a similar message as ours and we actually see it as a valuable work that supports our claim that multimodal CL is a promising approach to enabling CL in DNNs. Our proposed benchmark provides a more holistic evaluation than the evaluation in [4] as in addition to Class-IL we also simulate Domain-IL and Generalized Class-IL whereby the model has to tackle additional challenges of class imbalance and learn over recurring classes.

Reviewer comment (Experiments):
For the semantic-aware feature alignment, I wonder if the authors can provide some visualization examples to demonstrate that the model indeed learns the desired modality-specific features, such as Figure 4 in [5].

Author response:
We thank the reviewer for the very valuable suggestion. We performed this analysis and have added the results in the Activation Maps section in the Appendix.  The activation maps in Figure 6 show that MultiModal ER allows the model to attend to regions in the image that are associated with the label and localize the sound. SAMM considerably improves sound localization and attends to the most pertinent regions in the image. For playing saxophone, SAMM attends more to the saxophone and the mouth regions. Similarly, for tap dancing, SAMM rightfully attends more to the legs. This shows that multiple modalities and structure-aware alignment in SAMM enable the model to learn more holistic representations and focus on the regions associated with the class. The enhanced localization of sound in SAMM shows that our method effectively aligns the two modalities and learns a more holistic representation.

Reviewer comment (Experiments):
The paper shows the main experimental results of the proposed method, SAMM (Semantic-aware multimodal method), in Table 1~2. I think that the performances of other methods reported in major references such as [Buzzega et al., NIPS20] or [Arani et al., PAMI 2022] seem to be compared. Since lack of comparison, it is NOT clear to figure out the effectiveness and uniqueness of the proposed method among other methods.

Author response:
We would like to emphasize that the main goal of our study is not necessarily to achieve state-of-the-art results but rather to advocate for the adoption of multimodal continuous learning (CL) and to establish a standardized benchmark setting for research in this area. Therefore, the focus of our work was on demonstrating the advantages of multimodal experience replay over unimodal experience replay in challenging CL scenarios.

Additionally, we propose a promising direction for aligning modalities to enhance complementary learning. This alignment is proposed in a manner that not only enables multimodal inference but also serves as a baseline for unimodal inference, thereby creating a foundation for future research to build upon.

Furthermore, while there are several state-of-the-art methods in unimodal CL, the majority of them cannot be applied directly in a multimodal setting and would require modifications and adaptation to multimodal architecture.

However, the ablation study in Appendix (Table 3) shows the contributions of the different components in SAMM. *UM + CR* would correspond to an adaptation of DER++ [Buzzega et al., NIPS20] in a multimodal setting, and we show that SAMM provides considerable performance gains over it (34.51 vs. 29.13).

Reviewer comment (Reproducibility):
The paper does NOT provide enough information (including data composition, details on evaluation, and experimental settings) to reproduce the results in the experiments, even though Appendix A.2~A.4 presents some information.

Author response:
We have added further details about the dataset composition, and experimental setup to enhance the reproducibility of our results. Additionally, we will make the code and dataset publicly available.

Reviewer comment (Experiments):
It seems weak as a paper to propose a new dataset. Because it needs to provide baseline performances to show the characteristics of the dataset. On the other hand, it seems weak as a paper to propose a novel method for continual learning for visual-audio multimodal settings since it does not clearly validate the pros and cons of the proposed method.

Author response:
Our study aims to make a case for multimodal learning, as multisensory information processing is one of the salient features of the human brain, which may account for its CL capabilities. Not only do multiple modalities enable the model to learn a holistic and robust representation of objects, but multi-modal replay may also allow for better knowledge consolidation and less forgetting. Therefore, the key takeaway message from our study is that, similar to humans, multimodal learning might be critical in enabling effective CL in DNNs, and we want to encourage the research community at large to shift towards multimodal CL from unimodal CL. 

We welcome any suggestion or recommendation from the reviewer to more clearly validate the pros and cons of the proposed approach.

Reviewer comment (Experiments):
Is there any reason to compare with only ER?

Author response:
While there are other approaches to CL including regularization-based methods and architecture-based methods and it would be interesting to see how they can be adapted in a multimodal setting, we chose rehearsal-based methods for two specific reasons. Firstly, these methods have consistently demonstrated high efficacy in addressing the challenge of forgetting, especially in the demanding Class-IL setting. Traditional regularization-based approaches falter in this context, and many dynamic architecture-based methods are either tailored for task-IL, which doesn't align with the requirements of Continual Learning (CL), or they scale linearly with the number of tasks. Secondly, the incorporation of rehearsal aligns with the understanding of the human brain, where rehearsal is considered a critical component for information consolidation [1], contributing to the facilitation of lifelong learning.

Reviewer comment (Novelty):
The proposed method is straightforward with experience replay and those techniques are commonly used in existing multimodal learning and continual learning literature.

Author response:
>The proposed method is straightforward with experience replay and those techniques are commonly used in existing multimodal learning and continual learning literature.

Could the reviewer please share the existing multimodal continual learning literature that applies these techniques in multimodal class-IL settings? We are not aware of earlier literature that has studied the effect of multiple modalities in our proposed benchmarking settings with parallels with single-modality learning. The main contribution of our study is to bring more attention to multimodal CL and to showcase the benefits of multimodal CL over unimodal CL through an extensive empirical study. To further encourage research in this promising direction and facilitate standardization similar to the unimodal setting, we present a multimodal CL benchmark that simulates the challenges in unimodal CL to enable comparison. Additionally, we provide a baseline that provides a promising approach for leveraging complementary information in the different modalities.

Reviewer comment (Experiments):
The baseline is too weak, only the standard experience replay is compared. It would be better to compare to more recent state-of-art baselines in experience replay.

Author response:
>The baseline is too weak, only the standard experience replay is compared. It would be better to compare to more recent state-of-art baselines in experience replay.

We would like to emphasize that the goal of our study was not to propose a state-of-the-art but rather to provide a convincing case for moving towards multimodal CL and provide a standardized multimodal CL benchmark setting to facilitate research in this promising direction. Showcasing the performance gains of multimodal experience replay compared to unimodal experience replay in challenging CL settings (corresponding to the three established CL settings in unimodal settings) is the main contribution of our work. We additionally provide a promising direction for aligning the modalities to enhance complementary learning in a manner that allows multimodal as well as unimodal inference as an initial baseline upon which future work can build. 

Furthermore, while there are several state-of-the-art methods in unimodal CL, the majority of them cannot be applied directly in a multimodal setting and would require modifications and adaptation to multimodal architecture.

Importantly, the key takeaway message from our study is that, similar to humans, multimodal learning might be critical in enabling effective CL in DNNs, and we want to encourage the research community at large to shift towards multimodal CL from unimodal CL.

Reviewer comment (Experiments):
Furthermore, there are other categories of CL methods, including regularization-based methods and architecture-based methods. It would be better to also compare those methods in the experiment.

Author response:
>Furthermore, there are other categories of CL methods, including regularization-based methods and architecture-based methods. It would be better to also compare those methods in the experiment.

We agree with the reviewer that there are other approaches to CL including regularization-based methods and architecture-based methods and it would be interesting to see how they can be adapted in a multimodal setting. We opted for rehearsal-based in our study owing to two reasons: 1) Rehearsal-based methods continue to be the most effective in mitigating forgetting particularly in the challenging Class-IL setting where regularization-based approaches fail and the majority of dynamic architecture-based methods are either designed for task-il which doesn’t meet the desiredetas of CL [1] or scale linearly with the number of tasks. 2) The human brain is believed to employ rehearsal as one of the critical components for consolidating information [2] and enabling lifelong learning. 

The performance gains with Multimodal CL with experience replay in challenging CL settings provide a strong case for moving towards multimodal CL and designing approaches tailored for effectively utilizing multiple modalities to learn robust representations that are less vulnerable to forgetting.

Reviewer comment (Experiments):
The experiments are only performed on visual and audio modality. It would be better to provide experiment and benchmark on other modalities as well, e.g., language.

Author response:
> The experiments are only performed on visual and audio modalities. It would be better to provide experiment and benchmark on other modalities as well, e.g., language

It would indeed be interesting to extend our framework to additional modalities and we believe they will lead to an even more robust and holistic representation of learning. We chose Visual and Audio modalities as video is a more natural way to represent objects and actions, similar to how humans consume information in the real world. We hope that our work will inspire future work to integrate additional modalities.