Task-Attentive Transformer Architecture for Continual Learning of Vision-and-Language Tasks Using Knowledge Distillation (2023)

Abstract: The size and the computational load of fine-tuning large-scale pre-trained neural network are becoming two major obstacles in adopting machine learning in many applications. Continual learning (CL) can serve as a remedy through enabling knowledge-transfer across sequentially arriving tasks which relaxes the need to fine-tune all network weights from scratch. However, existing CL algorithms primarily consider learning unimodal vision-only or language-only tasks. We develop a transformer-based CL architecture for learning bimodal vision-and-language tasks based on increasing the number of the learnable parameters dynamically and using knowledge distillation. The new additional parameters are used to specialize the network for each task. Our approach enables sharing information between the tasks while addressing the challenge of catastrophic forgetting. Our approach is scalable learning to a large number of tasks because it requires little memory and time overhead. Our model reaches state-of-the-art performance on challenging vision-and-language tasks.

## introduction large-scale pre-trained transformer models are applied in a wide range of applications across modalities, including vision-and-language tasks (dosovitskiy et al.; @xcite . these models are usually pretrained on a large dataset and then fine-tuned to generalize on a downstream task. such task-level fine-tuning compromises the model generalizability and necessitates storing a copy of the base model for each task. continual learning (cl) algorithms @xcite @xcite @xcite have explored mitigating these challenges for transformers through using a shared model that benefits from cross-task knowledge transfer. catastrophic forgetting is the primary challenge in cl @xcite . a group of cl algorithms regularize a fixed shared model to learn each task through different information pathways, i.e., weights @xcite . the core idea is to identify a subset of model parameters that are important to encode the learned knowledge about each task and then consolidate these parameters when updating the model to learn new tasks. a second approach is based on model expansion @xcite . the idea is to expand a base model via a small number of additional weights and specialize the network to learn new tasks through these weights. finally, a group of algorithms use pseudo-rehearsal through experience replay @xcite . the idea is to store a representative subset of training data for each task in a memory buffer, and replay them back along with the current task's data to maintain the encoded knowledge about the past task. some methods relax the need for a memory buffer by enabling the model to generate pseudo-samples for the past learned tasks that are used to implement experience replay @xcite . despite being effective, existing cl methods consider unimodal tasks, e.g., vision-only @xcite or language-only tasks @xcite , without considering the unique challenges of multimodal tasks, making them inapplicable on val tasks. we develop a new algorithm for learning visionand-language (val) tasks in a cl setting based on dynamic model expansion. to this end, we leverage the self-attention layers of a base bimodal transformer as a shared encoder across all tasks. we then equip the base model with task-attention layers @xcite that help to specialize the model for each task using a task-specific token. our approach requires a small memory overhead and a limited inference time overhead during testing. it also does not need extensive hyper-parameters and remains robust when facing an unknown number of tasks. our specific contributions include: transformers for vision and language tasks: multimodal transformers are developed for processing vision-and-language (val) tasks @xcite @xcite @xcite . the core idea is to use modality-specific self-attention layers to extract suitable features from each of the vision and language inputs. these features then are integrated at higher layers to extract cross-modal contextualised representations of the multimodal inputs. the idea is that these global vectors can model the interaction between the vision and the language inputs which is helpful to perform val tasks well. the idea has also been adopted in other modalities, including in the context of video processing to relate vision and speech inputs @xcite . these transformers are usually trained on a large-scale dataset and have been found to be highly effective when fine-tuned on downstream tasks. due to a significant performance improvement, transformers are increasingly replacing older architectures. continual learning for transformers: despite the successful adoption of transformers in various benchmarks, fine-tuning naturally compromises their generalizability. using an independent transformer per task would lead to a significant memory load on disk as the size of transformers are increasing dramatically to account for solving more abstract tasks. cl seems to be a natural solution for these challenges but works on cl using transformers are limited. xin et al. @xcite use adapters in combination with a hypernetwork to enable cl for language tasks. alternatively, yang et al. @xcite propose a transformer calibration module to make a transformer adaptive. the calibration module is considered to be independent from the base pre-trained transformer and helps to specialize it on a downstream task. the lifelong vision transformer @xcite utilizes an inter-task attention mechanism to integrate information from previous tasks and reduces the rate at which important attention weights shift away from old tasks towards the current task. douillard et al. @xcite propose a cl architecture for vision tasks using the vilt model @xcite . pelosin et al. @xcite extend this work to an exemplar-free setting via distilling the attention-level matrices of transformers to enable model plasticity and to mitigate forgetting effects. ermis et al. @xcite use the idea of adapters in a vision context. to the best of our knowledge, no prior work has explored cl for multimodal tasks using transformer architectures. ## problem description consider a set of sequentially arriving val tasks {t i } t @xmath0, each with the annotated training dataset d @xmath1, l j i ) i , y j i ⟩ n i @xmath2, where i j @xmath3, l j @xmath4, while y j i is the text-typed discrete label. the order of these tasks and t are not known a priori. the training data points for t i are assumed to be drawn iid from a task-specific joint distribution p t i (•, •, •). our goal in multimodal cl is to learn each task at time-step i and then move forward to learn the next tasks. the learned tasks can be encountered at any time during testing in the future and hence, we would like to maintain the performance of previous learned tasks by preventing catastrophic forgetting. when learned in isolation, each of these val tasks t i can be learned using supervised learning conditioned on selecting the suitable predictive model f i θ m (•, •), e.g., a transformer with trainable parameters θ m , and the discrimination loss l(•), e.g., cross-entropy. however, due to the storage limit, we assume a shared model should be used, as training a separate model per task is impractical. additionally, only a small portion of training data for each task t i can be stored after training for space-saving purpose, which makes multitask learning @xcite an impractical solution. the single task learning strategy using a shared model is not ideal in cl either. because when the model is updated to learn the current tasks, its performance on the past tasks will degrade due to catastrophic forgetting @xcite . figure 1 visualizes the high-level presentation of the solution that we propose to address multimodal cl. to use a shared model across all tasks and benefit from knowledge transfer across all tasks, we consider a base transformer model as f θ (•, •) and make it adaptive by adding a unique task attention layer after its final layer. additionally, we modify the loss supervised learning loss by adding a knowledge distillation loss @xcite on the intermediate model layers. we consider the teacher model in the knowledge distillation formulation to be a copy of f i-1 θ (•, •) when training the main student model on i th task t i . additionally, we rely on pseudo-rehearsal through experience replay @xcite using a small memory buffer to tackle catastrophic forgetting. ## the proposed architecture figure 2 visualizes our transformer-based architecture for multimodal cl. the architecture con-sists of a shared pre-trained replaceable multimodal transformer, an independent task attention block, and mlp classification headers. the task attention block receives task-specific tokens that make the model adaptive. we provide details about these modules and the strategies that we use to train it. ## experimental results we evaluate tam-cl using five val tasks. ## conclusions we developed an algorithm for multimodal continual learning for transformer architectures based on dynamic model expansion and knowledge distilla-tion. we use a task-attention block which specializes the transformer architecture for a particular tasks using a special learnable task token. knowledge distillation helps to benefit from knowledge transfer across the tasks for positive forward transfer. we mitigate catastrophic forgetting using experience replay. our experiments demonstrate that our approach is effective and leads to state-of-the art performance in terms of both forward transfer and catastrophic forgetting. our tam-cl architecture is a first algorithm in studying cl in multimodal settings and demonstrates that more explorations in this direction is necessary.