Enhancing Continual Learning in Visual Question Answering with Modality-Aware Feature Distillation (2024)

Abstract: Continual learning focuses on incrementally training a model on a sequence of tasks with the aim of learning new tasks while minimizing performance drop on previous tasks. Existing approaches at the intersection of Continual Learning and Visual Question Answering (VQA) do not study how the multimodal nature of the input affects the learning dynamics of a model. In this paper, we demonstrate that each modality evolves at different rates across a continuum of tasks and that this behavior occurs in established encoder-only models as well as modern recipes for developing Vision & Language (VL) models. Motivated by this observation, we propose a modality-aware feature distillation (MAFED) approach which outperforms existing baselines across models of varying scale in three multimodal continual learning settings. Furthermore, we provide ablations showcasing that modality-aware distillation complements experience replay. Overall, our results emphasize the importance of addressing modality-specific dynamics to prevent forgetting in multimodal continual learning.

## introduction large language models (llms) @xcite and visual language models (vlms) @xcite , have achieved unprecedented performance and have become the go-to option for most nlp and vision & language (vl) tasks. however, once they have been trained, it is not straightforward how to update them to accommodate for novel concepts or concepts with reworked meanings. as a result, over time, the knowledge of these models may be obsolete or needs to be refined periodically to maintain their relevance. consider two examples: 1) as of july 2023, twitter has been re-branded to x with a new logo. 2) according to the oxford english dictionary (oed), as of march 2024, more than 1000 english words or phrases have been either been revised or included as novel entries foot_0 . in these cases, models trained with data from a preceding period will inevitably show performance deterioration @xcite . commercialized llms circumvent this limitation @xcite @xcite with statements regarding the knowledge cutoff of these models. on the other hand, humans continuously update their knowledge and acquire new skills over time. continual learning is a paradigm that aims to simulate this behavior, focusing on models that learn incrementally from a sequence of tasks with minimal catastrophic forgetting @xcite . continual learning has started being explored more widely in vl settings @xcite @xcite @xcite . however, existing approaches do not explicitly account for the dissimilarities in the representation space of multimodal inputs and their effect on the learning dynamics, which we argue is necessary for effective continual learning of vl tasks. previous work on the optimization dynamics of multimodal learning has demonstrated that different modalities are learned at different speeds @xcite . using encoder and decoder-only vlms, we empirically showcase a similar forgetting discrepancy (section 4.1), which indicates that representations from each modality evolve at different rates across a sequence of tasks. motivated by this observation, we propose mafed, a modality-aware feature distillation approach summarized in figure 1 . we explore different strategies for weighting the distillation losses derived from the tokens of each modality, using either fixed balanced weights or adaptive weights derived from the loss gradients computed with respect to the inputs. we combine both variants with experience replay and show promising results across all vlm families compared to established continual learning methods. additionally, we conduct experiments with decoder-only vlms ranging from 100m to 1b parameters, showing that although scale alleviates forgetting, certain settings remain challenging. our ablations comparing experience replay and feature distillation approach showcase that these methods are complementary and yield greater performance when combined. overall, our results emphasize the need to address modalityspecific dynamics to effectively mitigate forgetting in multimodal continual learning. ## conclusion in this paper, we argued that applying approaches that were developed with unimodal models in mind is suboptimal for continual learning in vqa since this ignores modality-specific learning dynamics. we empirically showcased that the visual and the textual representations evolve at different rates -a phenomenon that occurs in both encoder-only and decoder-only vlms. given this observation, we proposed two modality-aware feature distillation approaches that equally weigh the distillation loss from each modality or adaptively estimate the importance of a modality based on the gradients with respect to the inputs. we believe this is a promising direction towards closing the gap with multitask training in multimodal continual learning.