Sub-network Discovery and Soft-masking for Continual Learning of Mixed Tasks (2023)

Abstract: Continual learning (CL) has two main objectives: preventing catastrophic forgetting (CF) and encouraging knowledge transfer (KT). The existing literature mainly focused on overcoming CF. Some work has also been done on KT when the tasks are similar. To our knowledge, only one method has been proposed to learn a sequence of mixed tasks. However, these techniques still suffer from CF and/or limited KT. This paper proposes a new CL method to achieve both. It overcomes CF by isolating the knowledge of each task via discovering a sub-network for it. A soft-masking mechanism is also proposed to preserve the previous knowledge and to enable the new task to leverage the past knowledge to achieve KT. Experiments using classification, generation, information extraction, and their mixture (i.e., heterogeneous tasks) show that the proposed method consistently outperforms strong baselines.

## introduction one of the overarching goals of ai is to develop agents that can continually learn diverse tasks. toward this goal, continual learning (cl) has been proposed, which incrementally learns a sequence of tasks, 1, ...., t @xcite . once a task t is learned, its training data d t (at least a majority of it) is no longer accessible. this paper studies cl of nlp tasks @xcite , where a task is an end-task, e.g., text classification, summarization or information extraction, in the popular setting of task-incremental learning (til), in which the task-id is given in both training and testing. 2 ideally, cl should (1) overcome catastrophic forgetting (cf), i.e., it should not degrade the performance of previously learned tasks; (2) encour-age knowledge transfer (kt) across tasks, i.e., the learning of a task should be able to leverage the knowledge learned from previous tasks foot_2 ; @xcite (3) be able to learn a mixed sequence of similar and dissimilar tasks and achieve both (1) and ( 2 ). most existing approaches for til address one of the three to the detriment of the others. for example, hat @xcite and supsup @xcite can overcome cf by isolating the parameters of each task, which makes kt difficult. cuber @xcite and das @xcite can achieve kt by allowing some updates to existing knowledge, which causes cf. progres-sivenet @xcite assures no cf by fixing all the learned parameters and expanding the network, but it limits kt. to our knowledge, cat @xcite ) is the only system that tries to have all three capabilities. however, it still suffers from cf due to its weak task similarity detection (for kt) and parameter sharing across dissimilar tasks. this paper takes a major step to achieve all three objectives without suffering from the above drawbacks. the proposed system is called tss (til based on sub-network discovery and softmasking). tss performs two functions, (a) subnetwork discovery with soft-masking and (b) importance computation, to learn a mixed sequence, where some tasks may be similar (which enable kt) and some may be dissimilar. given a fixed and randomly initialized backbone network n , function (a) finds a separate sub-network as the model for each task. a subnetwork is indicated by a set of binary gates, one for each parameter of n , specifying whether the corresponding parameter in n should be in the sub-network for the task. in training, n is fixed and the binary gates are obtained by learning a set of positive and real-valued popup scores (one for each parameter of n ) and then applying a threshold on the popup scores. 4 in other words, we convert the network training into the popup scores training. 5 only the binary gates (1's) obtained from the trained popup scores are stored for each task as its model. thus, there is no interference across tasks to cause cf as the backbone network is fixed. this sub-network discovery is effective for overcoming cf, but it prevents kt if the popup scores of each task are independently trained with no sharing across tasks. a naive way to share knowledge is to initialize the popup scores of the new task t with the trained popup scores of the previous task t -1. however, this is problematic for a mixed sequence of similar and dissimilar tasks because the training of task t can only leverage the knowledge from the (t -1)th task. this can cause negative transfer if task t -1 is dissimilar to task t. to address this, we propose a soft-masking mechanism to make the initialization of popup scores for the new task contain the learned knowledge from all previous tasks so that the new task training can leverage the knowledge of all previous similar tasks. this is a key novelty of our approach. it is achieved by reducing (called "soft-masking") the gradient corresponding to the subset of popup scores that are important to previous tasks (1...t -1) in training the current task t so that the previously learned knowledge is well protected. in this way, the trained popup scores for task t, which will be used as the initialization for the next task, contain not only the knowledge for task t but also the knowledge from all previous tasks. this is critical for learning a mixed sequence of similar and dissimilar tasks because the system does not know a priori which tasks' knowledge can be transferred to the current task. note that soft-masking is only applied in backward propagation to preserve old knowledge. in the forward pass, the training can still use all popup scores corresponding to all parameters of the backbone network. additionally, soft-masking reduces the gradient instead of fully blocking the gradient, which gives the model the flexibility to adjust any popup scores when training the current task t, which further encourages kt. the question is how to decide the important popup scores for each task. this is done by (b), which computes the importance of the popup scores after training a task. the importance is a number between 0 and 1. after training popup scores for a task, we input the training data again to compute the importance of each popup score for the task based on its gradient. we save the accumulated importance over all previous tasks to keep track of the important popup scores so far. the accumulated importance (within 0 and 1; thus "soft") is used to guide soft-masking in (a) in training a new task. this paper makes three key contributions. ## proposed tss technique tss is designed for both cf prevention and forward kt. figure 1 gives an overview of tss. the backbone network consists of the transformer and adapters, which are indicated by the transformer parameters w transformer l of layer l and by the parameters of adapter w adapter l of layer l respectively (see the grey boxes). note again that these are fixed in the entire learning process. on the left, we show the forward pass and backward propagation of (a) sub-network discovery with soft-masking (sec. 3.1). in the forward pass, a set of learnable popup scores s ). as a result, the binary gates indicate a sub-network for task t within the w adapter l (sec. 3.1.1). in the backward propagation (sec. 3.1.2), we first accumulate the importance of popup scores for all previous tasks by normalization and element-wise maximum ("emax"). the accumulated importance i (≤t-1) l is converted into soft-mask by the operation 1 -i (≤t-1) l . this softmask is then element-wise multiplied with the original gradient of the popup scores s (t) l in layer l, ∇ l (computed by straight-through estimator in eq. 3). the soft-masked gradient, ∇l , is the final gradient that is used to optimize the popup scores s (t) l . since the important popup scores have lower gradients in ∇l , the popup scores that are important to previous tasks are preserved and are used to initialize the new task to encourage kt (not shown in figure 1 ). after (a), we show (b) importance computation (sec. 3.2). the forward pass is the same as (a). however, in backward propagation, the gradient of popup scores, ∇ l , is not used to update anything (red cross in the arrow) but computes n @xmath0. the resulting importance of task t in layer l, i (t) l , is saved for (a) in learning the next task. the following sub-sections present each function and step in detail. ## experiments we now evaluate the proposed system tss. we first learn all tasks sequentially. after that, their task models are tested using their respective test sets. tss does not use any replay data. ## conclusion this paper studied task incremental learning (til) on a range of nlp problems. we first presented the three desired capabilities: no forgetting, knowledge transfer and learning a mixed sequence of similar and dissimilar tasks. to our knowledge, only one system (cat) aims to achieve all these objectives but it suffers from forgetting. we then propose a novel method, tss, to achieve all three. experimental results showed that tss achieves kt and no forgetting even for the challenging mixed (or heterogeneous) task sequence setting.