Lifelong Sequence Generation with Dynamic Module Expansion and Adaptation (2023)

Abstract: Lifelong sequence generation (LSG), a problem in continual learning, aims to continually train a model on a sequence of generation tasks to learn constantly emerging new generation patterns while avoiding the forgetting of previous knowledge. Existing LSG methods mainly focus on maintaining old knowledge while paying little attention to knowledge transfer across tasks. In contrast, humans can better learn new tasks by leveraging previously acquired knowledge from similar tasks. Inspired by the learning paradigm of humans, we propose Dynamic Module Expansion and Adaptation (DMEA), which enables the model to dynamically determine the architecture for acquiring new knowledge based on task correlation and select the most similar previous tasks to facilitate adaptation to new tasks. In addition, as the learning process can easily be biased towards the current task which might cause more severe forgetting of previously learned knowledge, we propose dynamic gradient scaling to balance the learning of the current task and replayed tasks. With extensive experiments, we demonstrate that DMEA can consistently outperform existing methods in different LSG settings.

## introduction with the recent advancements in pre-trained language models (lms), current sequence generation methods have achieved impressive performance on a variety of generation tasks @xcite . typically, these models are trained on a fixed corpus, assuming the underlying data distribution to be static @xcite . however, real cognitive tasks are generally more complex involving changing contexts and dynamic environments. the ever-changing data distribution causes the models to face challenges in acquiring new knowledge, while retaining the prior knowledge. speaking about what is next for nlp, kathleen mckeown in a recent interview said: "most models are static. but the world changes every minute, every second. dealing with a dynamic world is a new area that's up and coming." @xcite a potential solution is to formalize sequence generation as lifelong sequence generation or lsg @xcite , where the model is expected to learn sequentially from a stream of generation tasks with potentially different data distributions. in such cases of distribution shift, the model might forget previously acquired knowledge upon learning new tasks, a phenomenon known as catastrophic forgetting @xcite . previous lsg methods @xcite @xcite mainly explore different ways to alleviate forgetting. @xcite propose adaptive compositional modules (acm) which dynamically adds modules for new tasks depending on whether there are reusable previous modules, achieving sota performance on lsg. despite its effectiveness, acm has several key limitations. first, it mainly focuses on mitigating forgetting of previously acquired knowledge while paying little attention to transferring learned knowledge to new tasks which is as important for continual learning as preventing forgetting @xcite . in fact, a hallmark of human intelligence is that humans can better learn new tasks by leveraging previously acquired knowledge from similar tasks @xcite . they can not only determine whether previously acquired skills are sufficient to solve a new task, but also exploit the most similar learned skills to facilitate the learning of the task; see appendix a.1 for an illustration. second, acm does not consider the correlation between learned tasks and the new task when adding modules, which might hinder finding the optimal architecture (case study in appendix a.9). finally, the learning process in acm can be biased towards the new task as the gradient norm of the new task on reused modules is typically much larger than that of replayed tasks, which may affect previously acquired knowledge; see appendix a.2 for an explanation. inspired by the learning paradigm of humans and to address the above limitations of acm, in this work we propose dynamic module foot_0 expansion and adaptation (dmea). we divide the learning process of a new task into three stages: expansion, selection and adaptation. in the expansion stage, dmea determines whether to reuse modules of previous tasks or insert new modules for learning novel knowledge. inspired by @xcite , it utilizes differentiable architecture search @xcite to enable the model to dynamically determine the architecture for solving the new task. the learnable coefficients in architecture search are initialized based on the cosine similarity of word frequency distributions between learned tasks and the new task, aiming to discover the optimal model architecture. after searching, the module with the largest coefficient in every layer is chosen for the new task. in the selection stage, dmea selects the top-k most similar previous tasks through input subspace @xcite . finally, in the adaptation stage, it utilizes the selected similar tasks to facilitate adaptation to the new task. the output of selected similar tasks is fused with that of the new task using learnable coefficients in every transformer layer to enable forward knowledge transfer. this is indeed an instance of mixture-of-experts @xcite . in addition, when the model learns a new task, dmea also incorporates pseudo-sample replay @xcite to further mitigate catastrophic forgetting. to address the "bias to the new task" in the gradient update, we introduce dynamic gradient scaling to balance the learning of the new task and replayed tasks. to verify the effectiveness of dmea, we conduct extensive experiments on various generation tasks in different lsg settings. the empirical results show that dmea can consistently outperform previous state-of-the-art baselines. in summary, our main contributions are: more recently, researchers have considered exploring knowledge transfer in ll, i.e., learning on a task can benefit from learning on another task by transferring related knowledge. this includes ctr @xcite and cuber @xcite . despite their effectiveness, these methods mainly focus on classification tasks, while generation tasks typically have more complex label space. note that this line of research is different from transfer learning @xcite , which mainly focuses on exploring better ways to reuse learned knowledge which is usually static, e.g., a frozen language model. in contrast, the acquired knowledge is continually accumulated in lifelong learning. lifelong sequence generation (lsg) enables the model to learn sequentially from a stream of generation tasks. @xcite propose lamol which formalizes different types of tasks as ques-tion answering and utilizes pseudo-sample replay to alleviate forgetting. @xcite further improve lamol by knowledge distillation @xcite . adaptercl @xcite inserts task-specific modules into every transformer layer to learn new tasks while keeping the pre-trained lm and previous modules frozen. on the basis of adaptercl, zhang et al. (2022) introduce acm which dynamically adds modules for learning new tasks depending on whether there are reusable previously inserted modules. though acm can enable knowledge transfer to some extent via module sharing, there is no explicit mechanism to encourage knowledge transfer across tasks, a common phenomenon of human learning. summary. existing work in lsg mainly focuses on mitigating the catastrophic forgetting of previously learned knowledge while paying little attention to knowledge transfer across tasks. in contrast to these lines of work, we aim to explicitly encourage forward knowledge transfer in lsg inspired by the way humans learn @xcite . ## problem formulation lsg involves learning from a stream of sequence generation tasks @xmath0, ..., t n ), where every task t i has its own training set d i train , validation set d i valid , and test set d i test . every dataset d contains a set of examples {(x j , y j )} |d| @xmath1, where x j and y j denote the input and output texts, respectively. at time step k, the model is trained on the training set d k train of task t k and has no access to real samples of previously learned tasks. after the training on d k train , the model is expected to perform well on all the tasks learned so far, i.e., t 1 , ..., t k , and will be evaluated on the test set d i test of each task t i (1 ≤ @xmath2. therefore, to achieve the goal of lsg, the model is required to alleviate the forgetting of acquired knowledge and better learn new patterns through possible forward knowledge transfer. ## methodology inspired by how humans learn a new task (fig. 5 ), dmea divides the learning process into three stages. the expansion stage ( §4.1) first determines the model architecture dynamically. the selection stage ( §4.2) then selects the top-k most similar previous tasks which are utilized in the final adaptation stage ( §4.3) to facilitate adaptation to the new task. we also employ pseudo-sample replay along with a dynamic gradient scaling method to balance the learning of the new and replayed tasks. ## experimental setup in this section, we first describe investigated tasks and then introduce methods compared in our work. ## conclusion in this work, we have introduced dmea for lifelong sequence generation (lsg). dmea leverages task correlations to dynamically determine the suitable architecture required to acquire novel knowledge of a new task and selects the most similar previous tasks through input subspace to facilitate knowledge transfer. it uses pseudo-sample replay along with dynamic gradient scaling to balance the learning of the new task and replayed tasks to further alleviate forgetting. with extensive experiments and analysis we have shown that dmea consistently outperforms previous methods in different lsg settings. in the future, we would like to investigate ways to improve the quality of pseudo data and explore more metrics for task similarity.