Original: C ONTINUAL P OST -T RAINING OF L ANGUAGE M ODELS
Revised: C ONTINUAL L EARNING OF L ANGUAGE M ODELS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Language models (LMs) have been instrumental for therecent rapid advance of natural language processing.
Revised: Language models (LMs) have been instrumental for the rapid advance of natural language processing.

Revised: This paper studies continual learning of LMs, in particular, continual domain-adaptive

Revised: pre-training (or continual DAP-training ).

Original: Existing research has shown that post-training or adapting an LM using an unlabeled topical/domain corpus  can improve the end-task performance in the domain.
Revised: Existing research has shown that further pre-training an LM using a domain corpus to adapt the LM to the domain can improve the end task performance in the domain.

Original: This paper proposes a novel method to continually post-train an LM with a sequence of unlabeled domain corpora to adapt the LM to these domains to improve their end-task performances.
Revised: This paper proposes a novel method to continually DAP -train an LM with a sequence of unlabeled domain corpora to adapt the LM to these domains to improve their endtask performances.

Original: The key novelty of our method is a soft-masking mechanism that directly controls the update to the LM.
Revised: The key novelty of our method is a soft-masking mechanism that directly controls the update to the LM.

Original: A novel proxy is also proposed to preserve the general knowledge in the original LM.
Revised: A novel proxy is also proposed to preserve the general knowledge in the original LM.

Original: Additionally, it contrasts the representations of the previously learned domain knowledge (including the general knowledge in  pre-trained LM) and the knowledge from the current full network to achieve knowledge integration.
Revised: Additionally, it contrasts the representations of the previously learned domain knowledge (including the general knowledge in the pre-trained LM) and the knowledge from the current full network to achieve knowledge integration.

Original: The method not only overcomes catastrophic forgetting , but also achieves knowledge transfer to improve end-task performances compared to post-training each domain separately.
Revised: The method not only overcomes catastrophic forgetting , but also achieves knowledge transfer to improve end-task performances.

Original: Empirical evaluation demonstrates the effectiveness of the proposed method.
Revised: Empirical evaluation demonstrates the effectiveness of the proposed method.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recently, pre-trained language models (LMs) like BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) have significantly advanced NLP.
Revised: Pre-trained language models (LMs) like BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) have significantly advanced NLP.

Revised: Recently, LMs have also been used by many continual learning (CL) systems to learn a sequence of end-tasks incrementally (Ke et al., 2021a;

Original: Post-training 2 an LM using a large unlabeled domain corpus before end-task fine-tuning achieves even better results (Gururangan et al., 2020; Xu et al., 2019).
Revised: Sun et al., 2020; Huang et al., 2021), which we call continual end-task learning . It is also desirable to continually pre-train LMs themselves.

Original: This paper goes a step further and asks how to improve an LM’s ability to handle new and emerging domains without forgetting its learned skills/knowledge. This is important in the real world, where the data shifts constantly and new domains keep emerging (Gururangan et al., 2020).
Revised: This includes (1) continual general pre-training , which incrementally updates the LM using the most recent data that has a similar distribution as the pre-training data, and (2) continual domain-adaptive pre-training , which continually further pre-trains a LM incrementally to adapt it to a sequence of domains. Note that LM editing (with or without continual learning)

Original: For example, one may adapt an LM trained over a general domain (e.g., News) to some science domains (e.g., ACL and AI papers).
Revised: (Mitchell et al., 2022) that corrects mistakes learned in the LM is a special case of continual end-task learning in the class-incremental learning setting (Kim et al., 2022) as each editing task or a group of editing tasks learned together is basically a task in continual learning, which aims to perform the editings correctly without interfering with or forgetting the other knowledge already learned in the current LM. This paper focuses on continual d omain- a daptive p re-training (or continual DAP-training ) of LMs. It is known that DAP-training 2 an LM (without continual learning) using a large unlabeled domain corpus before end-task fine-tuning achieves better results (Gururangan et al., 2020; Xu et al., 2019; Ke et al., 2022a).

Original: From a broader perspective, since training a large LM from scratch is extremely expensive, incrementally updating it with thelatest data reflecting the ever changing knowledge is becoming critical. As humans are very effective at incremental learning, if we can imitate this human capability with little/no forgetting, we can push AI research forward significantly.
Revised: This paper goes a step further to continually learn to improve an LM’s ability to handle new and emerging domains without forgetting skills or knowledge learned in the past. This is important in the real world, where the data shifts constantly and new domains, events or topics keep emerging (Ke et al., 2022a) and the LM needs to be updated to serve the users better.

Original: We call this problem continual learning (CL) of post-training or continual post-training .
Revised: We call this problem continual DAP-training .

Original: Starting from a pre-trained  LM (i.e., the LM has already been pre-trained on D 0 ), we incrementally post-train a sequence of domain corpora D 1 , D 2 , ... .
Revised: Starting from a pre-trained general LM (i.e., the LM has already been pre-trained on D 0 ), we incrementally DAP-train a sequence of domain corpora D 1 , D 2 , ... .

Original: Once a domain is trained, its data is no longer accessible.
Revised: Once a domain is trained, its data is no longer accessible.

Original: This is different from conventional  CL  where each task is an end-task.
Revised: This is different from conventional continual learning ( CL ) where each task is an end-task.

Revised: In the proposed continual

Original: In continual post-training, each task is an unlabeled domain corpus and the end-task involves fine-tuning the post-trained LM.
Revised: DAP-training, each task is an unlabeled domain corpus to be learned. An end-task fine-tunes the continually DAP-trained LM to evaluate its performance.

Original: This is also different from continual pre-training (Qin et al., 2022b), where one builds the LM from scratch and can access the data (pre-training data D 0 ) that pre-trains the LM.
Revised: It is worth noting that D 0 is usually a broad or general domain (e.g., News).

Original: In continual post-training, however, one uses the LM and thus do not have access to the pre-training data  butonly one’s own domain data. We will see how this causes challenges to the problem.
Revised: In practice, a continually DAP-trained LM may be trained by individual users, institutions or a mix of both who have one or more large corpora of some particular domains. In such cases, the raw data may not be shared, but the final LM can be shared by all.

Original: There are multiple desiderata for a continual post-training system: (1) It should not suffer from catastrophic forgetting (CF) , i.e., it should perform reasonably well on learned domains.
Revised: There are multiple desiderata for a continual DAP-training system: (1) It should not suffer from catastrophic forgetting (CF ), i.e., it should perform reasonably well on learned domains.

Original: This requires the system (a) to overcome CF for each new domain and (b) to overcome CF for the general language knowledge in the LM.
Revised: This requires the system (a) to overcome CF for each new domain and (b) to overcome CF for the general language knowledge in the LM.

Original: This is important because the knowledge learned from each domain alone will not be sufficient for good end-task performances.
Revised: This is important because the knowledge learned from each domain alone will not be sufficient for good end-task performances.

Original: (2) It should encourage knowledge transfer (KT) across domains to achieve improved end-task performances.
Revised: (2) It should encourage knowledge transfer ( KT ) across domains to achieve improved end-task performances.

Original: This requires the system to enable (a) forward transfer , learning  new domain by leveraging  knowledge from previous domains, and (b) backwards transfer , gaining improved performance on previous domains after learning a relevant new domain.
Revised: This requires the system to enable (a) forward transfer , learning a new domain by leveraging the knowledge from previous domains, and (b) backwards transfer , gaining improved performance on previous domains after learning a relevant new domain.

Original: (3) It should learn without requiring domain labels for each end-task fine-tuning.
Revised: (3) It should work without requiring the domain-ID for each end-task fine-tuning.

Original: None of the existing CL methods can achieve all the above.
Revised: None of the existing CL methods can achieve all the above.

Original: This paper presents a step towards achieving them.
Revised: This paper represents a step towards achieving them.

Original: The proposed method is called CPS ( C ontinual P ost-training of LMs with S oftmasking).
Revised: The proposed method is called DAS (Continual DA -pre-training of LMs with S oft-masking).

Original: CPS proposes a novel soft-masking mechanism that computes the importance (a real number between 0 and 1) of units 3 for general or domain knowledge and soft-mask them based on their importance values to control the backward gradient flow.
Revised: DAS proposes a novel soft-masking mechanism that computes the importance (a real number between 0 and 1) of units 3 for general or domain knowledge and soft-mask them based on their importance values to control the backward gradient flow.

Original: In the forward pass, soft-masking is not applied, which encourages KT across domains.
Revised: In the forward pass, soft-masking is not applied, which encourages KT across domains.

Original: This does not isolate any sub-network for any domain so that the knowledge in the full LM can be leveraged for end-task fine-tuning.
Revised: It does not isolate any sub-network for any domain so that the knowledge in the full LM can be leveraged for end-task fine-tuning.

Original: To apply this mechanism, CPS implements two functions: (1) Initialization , where the goal is to compute the importance of units to  general knowledge  without accessing the  pre-training data ( D 0 ).
Revised: To apply this mechanism, DAS implements two functions: (1) Initialization , which computes the importance of units to the general knowledge in the LM without accessing the LM pre-training data ( D 0 ).

Original: It is applied on the pre-trained LM before the continual learning starts, and (2) continual learning , where the goal is to prevent CF on  general and domain knowledge and to encourage cross-domain KT during post-training.
Revised: It is applied on the pre-trained LM before the continual learning starts, and (2) continual learning , which DAP-trains each domain while preventing CF on the general and domain knowledge and encouraging cross-domain KT.

Original: In (1), it is not obvious how to compute the importance without pre-training data.
Revised: In (1), it is not obvious how to compute the importance without pre-training data.

Original: CPS proposes a novel proxy based on robustness to compute the importance of units for the general knowledge.
Revised: DAS proposes a novel proxy based on robustness to compute the importance of units for the general knowledge.

Original: In (2), the soft-masking is directly applicable because we have the domain data and the importance can be computed based on its gradient inspired by the pruning community (Li et al., 2021;
Revised: In (2), the soft-masking is directly applicable because we have the domain data and the importance can be computed based on its gradient inspired by the pruning community (Li et al., 2021;

Original: Michel et al., 2019).
Revised: Michel et al., 2019).

Original: Moreover, CPS contrasts the previously learned knowledge and the full (including both the learned domains and the current domain) knowledge to encourage the current domain representation to learn knowledge that is not already in thelearned knowledge in previous domains andand integrate it with the learned knowledge 4 .
Revised: Moreover, DAS contrasts the previously learned knowledge and the full (including both the learned domains and the current domain) knowledge to encourage the current domain representation to learn knowledge that is not already in the knowledge learned from previous domains and integrate it with the learned knowledge 4 .

Original: In end-task fine-tuning, CPS does not need the domain id as all knowledge is accumulated into the post-trained LM.
Revised: In end-task fine-tuning, DAS does not requires the domain-ID as all knowledge is accumulated into the DAP-trained LM.

Original: In summary, this work makes the following contributions.
Revised: In summary, this work makes the following contributions.

Original: (i)
Revised: (i)

Original: It proposes the  problem of continualpost-training .
Revised: It studies the new problem of continual

Original: Although (Jin et al., 2021) runs systems on a similar setting, it simply regards it as continual pre-training without proposing any new method or giving any analysis on the properties or the desiderata.

Original: (ii) It discovers  the full LM is needed for a good continual post-training method.
Revised: DAP-training and discovers that the full LM is needed for a good continual DAP-training method.

Original: This is, the popular parameter-isolation approach  in convention CL is not suitable.
Revised: The popular parameter-isolation approach to overcoming CF in convention CL is unsuitable.

Original: (iii)
Revised: (ii)

Original: It proposes a novel soft-masking method to overcome CF and to encourage KT. It further proposes to integrate knowledge across domains.
Revised: It proposes a novel soft-masking method to overcome CF and to encourage KT, and a constrative learning based method for knowledge integration.

Original: (iv) To preserve the general knowledge in the LM, a novel proxy is also proposed.
Revised: (iii) To preserve the general knowledge in the LM, a novel proxy is also proposed.

Original: (v) Experiments results demonstrate the effectiveness of CPS.
Revised: (iv) Experimental results demonstrate the effectiveness of DAS.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Post-training (also known as domain-adaptive pre-training).
Revised: DAP-training. DAP-training can be achieved by directly updating the LM (Xu et al., 2019; Sun et al., 2019; Lee et al., 2020;

Original: Researchers have applied post-training to many domains, e.g., reviews (Xu et al., 2019; Sun et al., 2019), news and academic papers (Gururangan et al., 2020), and shown improved end-task results.
Revised: Alsentzer et al., 2019; Gururangan et al., 2020; Chakrabarty et al., 2019; Ke et al., 2022a) or by training only a small set of additional parameters. For example, Pfeiffer et al. (2020); Wang et al.

Revised: (2020a);

Original: However, they are not about CL.
Revised: Ke et al. (2021a;b;c) trained adapters and Gu et al. (2021) trained a prompt to adapt to a domain. While adapter and prompt could be effective, transfer knowledge among these additional modules is usually challenging and can be inaccurate.

Revised: DAS belongs to the former family that directly updates the LM.

Revised: This is very challenging for CL due to CF.

Revised: To our knowledge, no existing system in this family is about CL.

Original: Continual learning.
Revised: Continual learning.

Original: Many approaches have been proposed to overcome CF: (1) Regularization methods (Kirkpatrick et al., 2016; Seff et al., 2017) computes the importance of each parameter to previous tasks and uses a regularization term to penalize the sum of changes.
Revised: Most CL methods were proposed to overcome CF: (1) Regularization methods (Kirkpatrick et al., 2016; Seff et al., 2017) compute the importance of each parameter to previous tasks and uses a regularizer to penalize the sum of changes.

Original: Our CPS is related to  EWC (Kirkpatrick et al., 2016), but they are different.
Revised: DAS is related to but also very different from EWC (Kirkpatrick et al., 2016).

Original: (1) CPS does not control each parameter/weight, but only attention heads or neurons based on their importance scores.
Revised: (1) DAS does not control each parameter/weight, but only attention heads or neurons based on their importance scores.

Original: This gives less forgetting (see the forgetting rate in Table 2) because even a small change to each parameter for a neuron can give a large total change to the neuron’s activation.
Revised: This gives less forgetting (see the forgetting rate in Table 2) because even a small change to each parameter for a neuron can give a large total change to the neuron’s activation.

Original: (2) CPS directly controls the backward gradient flow on each neuron, which is more fine-grained and effective than the sum of changes of all parameters.
Revised: (2) DAS directly controls the backward gradient flow on each neuron, which is more fine-grained and effective than the sum of changes of all parameters.

Original: Our experimental results confirms EWC is significantly poorer than CPS (see Table 2).
Revised: Our experimental results confirm that EWC is significantly poorer than DAS (see Table 2).

Original: (2) Replay methods retain (Rebuffi et al., 2017; Wang et al., 2020) or generate some data of old tasks (Shin et al., 2017; He & Jaeger, 2018) and use them in learning a new task; (3) parameter-isolation methods (Serrà et al., 2018; Wortsman et al., 2020) allocate neurons and parameters  for different tasks and mask them in task learning.
Revised: (2) Replay methods retain (Rebuffi et al., 2017; Wang et al., 2020b) or generate some data of old tasks (Shin et al., 2017; He & Jaeger, 2018) and use them in learning a new task; (3) parameter-isolation methods (Serrà et al., 2018; Wortsman et al., 2020) allocate neurons and parameters or sub-networks for different tasks/domains and mask them in task learning.

Original: This family has almost no CF in task-incremental learning setting where the domain id is known in any time.

Original: However, this family is designed for supervised end-task CL and learns binary/hard masks to isolate a sub-network for each domain.

Original: For continual post-training, we cannot isolate a sub-network for each domain because it means that end-tasks cannot use the general knowledge in the LM, which results invery poor end-task performances.
Revised: For continual DAP-training, this means that end-tasks cannot use the general knowledge in the LM, which results in poor end-task performances.

Original: In NLP, CL has been used for slot filling (Shen et al., 2019), language learning (Li et al., 2019), sentiment analysis (Shu et al., 2017), topic modeling (Gupta et al., 2020), question answering (Greco et al., 2019) and text classification (Sun et al., 2020; Huang et al., 2021; Chuang et al., 2020).
Revised: In NLP, CL has been used for slot filling (Shen et al., 2019), language learning (Li et al., 2019), sentiment analysis (Ke et al., 2021a), topic modeling (Gupta et al., 2020), question answering (Greco et al., 2019) and text classification (Sun et al., 2020; Huang et al., 2021; Chuang et al., 2020).

Original: But none is for post-training.
Revised: But none is for DAP-training.

Original: Some recent CL papers concern LMs.
Revised: Some recent CL papers concern LMs.

Original: Madotto et al. (2020) learns separate adapters, it has no CF or knowledge transfer.
Revised: The system in (Madotto et al., 2020) learns separate adapters for different domains and thus has no CF or KT.

Original: DEMIX (Gururangan et al., 2021)is similar to (Madotto et al., 2020) but initializes the new adapter with the closest old adapter.
Revised: DEMIX (Gururangan et al., 2021) initializes the new adapter with the closest old adapter.

Original: Jin et al.
Revised: CPT (Ke et al., 2022b) and ELLE (Qin et al., 2022) are most closely related to DAS. However, CPT uses the parameter-isolation approach to learn and protect each task, which is weak (see Sec. 4.2). It also needs domain-ID in end-task fine-tuning.

Original: compared several CL methods for continual post-training.
Revised: ELLE has to start from pre-training the LM itself rather than from a pre-trained

Original: ELLE (Qin et al., 2022a) concerns continual pretraining (see the difference in Sec. 1) and uses a large memory (1G per domain) to store  replay data  and expands the network for each domain.
Revised: LM like DAS. It also uses a large memory (1G per domain) to store the replay data (including the pre-training data) and expands the network for each domain.

Original: Both are not required in CPS.
Revised: Neither is required in DAS.

Revised: Jin et al.

Revised: (2021) evaluated several existing CL techniques in a similar setting as DAS and performed analyses on dealing with CF.

Revised: However, no new technique was proposed in the paper.

Original: Neural network pruning.
Revised: Neural network pruning.

Original: Many parameters in a network are redundant and can be pruned (Li et al., 2021; Lai et al., 2021;Chen et al., 2020a; Lin et al., 2020; Gao et al., 2021b; Michel et al., 2019; Voita et al., 2019).
Revised: Many parameters in a network are redundant and can be pruned (Li et al., 2021; Lai et al., 2021; Michel et al., 2019; Voita et al., 2019).

Original: One method discards parameters with small absolute values (Han et al., 2015; Guo et al., 2016). Other methods include accumulated gradient (Michel et al., 2019),dropout rate (Gomez et al., 2019), and lottery ticket hypothesis (Brix et al., 2020).
Revised: Existing methods include discarding parameters with small absolute values (Han et al., 2015; Guo et al., 2016), accumulated gradient (Michel et al., 2019), and lottery ticket hypothesis (Brix et al., 2020).

Original: However, these methods are not directly applicable as we need to preserve not only individual domain knowledge but also the general knowledge in the LM.
Revised: However, these methods are not directly applicable as we need to preserve not only individual domain knowledge but also the general knowledge in the LM.

Original: For general knowledge,  we do not have any pre-training data so a proxy is proposed based on robustness.
Revised: For general knowledge, since we do not have any pre-training data, a proxy is proposed based on robustness.

Original: For domain knowledge, we adopt the pruning method but use the importance as soft-masks as we want to accumulate knowledge rather than to compress the LM.
Revised: For domain knowledge, we adopt a pruning method but use the importance as soft-masks as we want to accumulate knowledge rather than to compress the LM.

Original: Contrastive Learning.
Revised: Contrastive Learning.

Original: Contrastive learning (Chen et al., 2020b;
Revised: Contrastive learning (Chen et al., 2020;

Original: He et al., 2020) learns good representations by maximizing the similarity of positive pairs and minimizes that of negative pairs:
Revised: He et al., 2020) learns good representations by maximizing the similarity of positive pairs and minimizes that of negative pairs,

Original: [Equation]
Revised: [Equation]

Original: where N is the batch size, τ is a temperature parameter, sim ( · ) is a similarity metric, and q i and q + i are representations for positive pairs x i and x + i .
Revised: where N is the batch size, τ is a temperature parameter, sim ( · ) is a similarity metric, and q n and q + n are representations for positive pairs x n and x + n .

Original: Our CPS contrasts the learned knowledge from previous domains and  pre-trained LM (general knowledge) with the full knowledge (including both the previous domains and current domain knowledge) to achieve a complementary effect.
Revised: DAS contrasts the learned knowledge from previous domains and the pre-trained LM (general knowledge) with the full knowledge (including both the previous domains and current domain knowledge) to achieve a complementary effect.

Original:  P ROPOSED CPS T ECHNIQUE
Revised:  P ROPOSED DAS T ECHNIQUE

Original: Continual post-training in CPS is based on two main ideas: (1) preserving the important general language knowledge in the LM and  knowledge learned from previous domains to overcome CF by soft-masking units based on their importance, which also facilitates cross-task knowledge transfer, and (2) encouraging the model to learn complementary representations of the current domain and previous domains to achieve knowledge integration.
Revised: Continual DAP-training in DAS is based on two main ideas: (1) preserving the important general language knowledge in the LM and the knowledge learned from previous domains to overcome CF by soft-masking units based on their importance, which also facilitates cross-task knowledge transfer (KT), and (2) encouraging the model to learn complementary representations of the current domain and previous domains to achieve knowledge integration.

Original: Figure 1 gives an overview of CPS.
Revised: Figure 1 gives an overview of DAS.

Revised: [Figure]

Original: The whole learning consists of two main functions: (i) initialization and (ii) continual learning .
Revised: The whole learning consists of two main functions: (i) initialization and (ii) continual learning .

Original: (i) computes the importance of units to the general language knowledge in the LM.
Revised: (i) computes the importance of units to the general language knowledge in the LM.

Original: It is done before the continual learning starts.
Revised: It is done before the continual learning starts.

Original: (ii) is for continual learning, which consists of two steps: (a) task training and (b) importance computation .
Revised: (ii) is for continual learning, which consists of two steps: (a) domain training and (b) importance computation .

Original: (a) takes the importance scores accumulated so far (including those to  general knowledge  and to the  learned  domains) and the input data of the current domain to learn the domain and to achieve (1) and (2) above, while (b) computes the importance scores for the current domain for future use.
Revised: (a) takes the importance scores accumulated so far (including those to the general knowledge in the original LM and to the knowledge learned from previous domains) and the input data of the current domain to learn the domain and to achieve (1) and (2) above, while (b) computes the importance scores for the current domain for future use.

Original: The following sub-sections present each function and step in detail.
Revised: The following sub-sections present each function and step in detail.

Original: 3. INITIALIZATION : C OMPUTING IMPORTANCE OF UNITS TO T HE G ENERAL K NOWLEDGE
Revised: 3. I

Revised: NITIALIZATION : C OMPUTING I MPORTANCE OF U NITS TO THE G ENERAL K NOWLEDGE

Original: This initialization function computes the importance of units (attention heads and neurons) in the
Revised: This initialization function computes the importance of units (attention heads and neurons) in the

Original: Transformer for the general knowledge in the original LM.
Revised: Transformer for the general knowledge in the original LM.

Original: The key components of a Transformer are multi-head attention layer , intermediate layer and output layer .
Revised: The key components of a Transformer are multi-head attention layer , intermediate layer and output layer .

Original: Below, we use “layer” or l to indicate any of these three layers because our method treats the three layers similarly.
Revised: Below, we use “layer” or l to indicate any of these three layers because our method treats the three layers similarly.

Original: Importance of units in a layer.
Revised: Importance of units in a layer.

Original: It has been found that not all units in a layer are important (Michel et al., 2019).
Revised: It has been found that not all units in a layer are important (Michel et al., 2019).

Original: We introduce a gate vector , g l , for computing the importance of unit in a layer l .
Revised: We introduce a virtual parameter , g l , for computing the importance of the units in a layer l .

Revised: We call these virtual parameters as each g ( k ) is initialized to 1.

Revised: We only need the gradient on each parameter to compute the importance of its corresponding unit, no update to any parameter.

Original: [Equation]
Revised: [Equation]

Original: where o l refers to the output of layer l (which can be any of the three layers mentioned above).
Revised: where o l refers to the output of layer l (which can be any of the three layers mentioned above).

Original: The ⊗ refers to element-wise multiplication, i.e., eachgate variable g l,i in g l corresponding to a unit (a neuron or attention head) in the layer.
Revised: The ⊗ refers to element-wise multiplication, i.e., each variable g l,i in g l corresponding to a unit (a neuron or attention head) in the layer.

Original: A gradient-based importance detection methodis proposed in (Michel et al., 2019).
Revised: We adapt the gradient-based importance detection method in (Michel et al., 2019) for our purpose.

Original: Given a dataset D = { ( x m , y m ) } Mm =1 of M samples ( y m is the class label of x m as Michel et al. (2019) worked on supervised learning), the importance of neurons or heads in the layer is estimated with a gradient-based proxy score
Revised: Given a dataset D = { ( x n , y n ) } Nn =1 of N samples ( y n is the class label of x n as (Michel et al., 2019) worked on supervised learning), the importance of neurons or heads in the layer is estimated with a gradient-based proxy score

Original: [Equation]
Revised: [Equation]

Original: where L impt is a task-specific loss function.
Revised: where L impt is a task-specific loss function.

Original: Note the gate vector g l is initialized as all 1’s, and is not changed.
Revised: Note the virtual parameter g l is initialized as all 1’s, and is not changed.

Original: This is because we need only its average gradient ∇ g l (the term within || in Eq. 3) over all the data to compute the importance and will not use the gradient to update the gate vector.
Revised: This is because we need only its average gradient ∇ g l (the term within || in Eq. 3) over all the data to compute the importance and will not use the gradient to update the virtual parameter.

Original: In training (Sec. 3.2 and Fig 1 (B)), the gate vector can be discarded.
Revised: In training (Sec. 3.2 and Fig 1 (B)), the virtual parameter can be discarded.

Original: The resulting Il is of the same size as g l , each entry corresponding to the importance of a unit (a neurons or attention head).
Revised: The resulting I

Original: Recall that the initialization function is to learn the importance of units to the general knowledge in the LM ( I gen l ).
Revised: Recall that the initialization function is to learn the importance of units to the general knowledge in the LM (denoted as I (0) l ).

Original: Although Eq.
Revised: Although Eq.

Original: offers a possible way, it is not directly applicable.
Revised: offers a possible way, it is not directly applicable.

Original: If we use the domain data at hand and employ the MLM loss as L impt , ∇ g l only indicates the importancescore for the domain-specific knowledge.
Revised: If we use the domain data at hand and employ the MLM loss as L impt , ∇ g l only gives the importance for the domain-specific knowledge.

Original: However, to compute the importance of units to the general knowledge in the LM (which is our goal), we need the original data used in pre-training the LM to compute the L impt .
Revised: However, to compute the importance of units to the general knowledge in the LM (which is our goal), we need the original data used in pre-training the LM to compute the L impt .

Original: In practice, such data is not accessible to users of the LM.
Revised: In practice, such data is not accessible to users of the LM.

Original: Further, label is needed in Eq. but our domain corpus is unlabeled in post-training.
Revised: Further, label is needed in Eq.but our domain corpus is unlabeled in DAP-training.

Original: To address these issues, we propose a proxy KL-divergence loss ( L proxy ) to replace L impt to learn units importance for the general knowledge.
Revised: To address these issues, we propose a proxy KL-divergence loss ( L proxy ) to replace L impt to learn units importance for the general knowledge.

Original: Proxy KL-divergence loss.
Revised: Proxy KL-divergence loss.

Original: We propose to use model robustness as the proxy, i.e., we try to detect units that are important for LM’s robustness .
Revised: We propose to use model robustness as the proxy, i.e., we try to detect units that are important for LM’s robustness .

Original: Their gradients, ∇ g l , then indicate the robustness and the importance to the LM model.
Revised: Their gradients, ∇ g l , then indicate the robustness and the importance to the LM model.

Original: Our rationale is as follows:
Revised: Our rationale is as follows:

Original: If an I gen l,i (the importance of unit i in layer l ) has a high value, then it is important to the LM’s robustness because its change can cause the
Revised: If an I (0) l,i (the importance of unit i in layer l ) has a high value, then it is important to the LM’s robustness because its change can cause the

Original: LM to change a lot.
Revised: LM to change a lot.

Original: It is thus an important unit.
Revised: It is thus an important unit.

Original: In contrast, if I gen l,i is small, it is a less important unit.
Revised: In contrast, if I (0) l,i is small, it is a less important unit.

Original: To compute the robustness of the LM, we take a subset of the current domain data { x sub m } 5 (no label in post-training) and input x sub m twice to the LM to obtain two representations of it and then compute the KL-divergence between them, 
Revised: To compute the robustness of the LM, we take a subset of the current domain data { x sub n } 5 (no label in DAP-training) and input x sub n twice to the LM to obtain two representations of it and then compute the KL-divergence between them, [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where f 1 LM and f 2 LM are the LM with different dropout masks.
Revised: ) where f 1 LM and f 2 LM are the LM with different dropout masks.

Original: We don’t need to add any additional dropouts to implement these two as the Transformer already has dropout masks placed on fullyconnected layers and attention probabilities.
Revised: We don’t need to add any additional dropouts to implement these two as the Transformer already has dropout masks placed on fullyconnected layers and attention probabilities.

Original: Thus, simply feeding the same input to the Transformer twice will get two representations with different dropout masks.
Revised: Thus, simply feeding the same input to the Transformer twice will get two representations with different dropout masks.

Original: Since dropout is similar to adding noise, the difference between the two representations can be regarded as the robustness of the LM.
Revised: Since dropout is similar to adding noise, the difference between the two representations can be regarded as the robustness of the LM.

Original: 3. T RAINING : L EARNING A N EW D OMAIN VIA S OFT - MASKING AND C ONTRASTIVE L OSS
Revised: 3. T RAINING : L EARNING A N EW D OMAIN VIA S OFT - MASKING AND C ONTRASTIVE L OSS

Original: Soft-masking units. Recall we want to preserve the learned knowledge in the LM during posttraining using the accumulated importance I ( ≤ t − 1) l when we learn domain t , which includes both the importance for the general knowledge I gen l (Sec. 3.1) and learned domain-specific knowledge I  k  l of each domain k ( k can be any domain in { 1 ...t − 1 } ) that has been learned (Sec. 3.3).
Revised: Recall we want to preserve the learned knowledge in the LM during DAP-training using the accumulated importance I ( ≤ t − 1) l when we learn domain t , which includes both the importance for the general knowledge I (0) l (Sec. 3.1) and learned domain-specific knowledge I ( k ) l of each domain k ( k can be any domain in { 1 ...t − 1 } ) that has been learned (Sec. 3.3).

Original: This is achieved by soft-masking the learning based on accumulated importance as follows.
Revised: This is achieved by soft-masking the learning based on accumulated importance as follows.

Original: Note that the importance values in each layer l for a domain k is normalized via

Original: [Equation] Mean and Var are computed from the importance of all units in layer l . The normalized I ( k ) l,i has the same size as the un-normalized I ( k ) l,i , and now I ( k ) l has the mean of 0 and standard deviation of 1. Absolute value of Tanh ensures that I ( k ) l takes values in the interval [0 , 1] .
Revised: Accumulating importance.

Original: To simplify the notation, we still use I ( k ) l to represent the normalized importance.

Original: We accumulate the importance after task t − 1 was learned is done via element-wise max (EMax) as follows:
Revised: We accumulate the importance after task t − 1 was learned is done via element-wise max (EMax) as follows:

Original: [Equation]
Revised: [Equation]

Original: where t refers to the current task-ID and I ( ≤ t − 2) l refers to the previously accumulated importance at task t − 2 .
Revised: where t refers to the current task-ID and I ( ≤ t − 2) l refers to the previously accumulated importance at task t − 2 .

Original: When we learn task 1, I ( ≤ 0) l is I gen l .

Original: We do not need to save I gen l and all { I ( k ) l } t -1 k
Revised: We do not need to save I 0 l and all { I ( k ) l } t -1 k

Original: =1 for Eq.
Revised: =1 for Eq.

Original: We only save the incrementally accumulated importance after training of each task.
Revised: We only save the incrementally accumulated importance after training of each task.

Original: For soft-masking, given the accumulated importance I ( ≤ t − 1) l of layer l and the post-training loss
Revised: Soft-masking units. Given the accumulated importance I ( ≤ t − 1) l of layer l and the DAP-training loss

Original: L post-train (typically the MLM loss; we also propose an additional loss in Eq. 11), we constrain (or soft-mask)
Revised: L DAP-train (typically the MLM loss; we also propose an additional loss in Eq. 7), we constrain (or soft-mask)

Original: its corresponding gradient ( ∇ l ) flow as follows,
Revised: its corresponding gradient ( ∇ l ) flow as follows,

Original: [Equation]
Revised: [Equation]

Original: As mentioned in Sec.
Revised: As mentioned in Sec.

Original: 3.1, we expand (by copying) the importance I ( ≤ t − 1) l to match the dimensions of ∇ l to apply it to all associated parameters.
Revised: 3.1, we expand (by copying) the importance I ( ≤ t − 1) l to match the dimensions of ∇ l to apply it to all associated parameters.

Original: This is soft-masking as each element in I ( ≤ t − 1) l is a real number in [0 , 1] (not binary {0, 1}), which gives the model the flexibility to adjust any unit. 
Revised: This is soft-masking as each element in I ( ≤ t − 1) l is a real number in [0 , 1] (not binary {0, 1}), which gives the model the flexibility to adjust any unit. [Table]

Original: We note that the above soft-masks are only applied in the backward pass, but not in the forward pass, which encourages knowledge transfer as each domain training can leverage the knowledge learned from all past domains.
Revised: We note that the above soft-masks are only applied in the backward pass, but not in the forward pass, which encourages knowledge transfer as each domain training can leverage the knowledge learned from all past domains.

Original: To further encourage the model to learn a good representation from both the accumulated knowledge ( I ( ≤ t − 1) l ) and the full knowledge (both accumulated and current domain knowledge), we introduce a contrastive learning method to encourage complementary representation.
Revised: To further encourage the model to learn a good representation from both the accumulated knowledge ( I ( ≤ t − 1) l ) and the full knowledge (both accumulated and current domain knowledge), we introduce a contrastive learning method to encourage complementary representation.

Original: Contrasting the previously learned  and new domain knowledge.
Revised: Integrating the previously learned knowledge and the current domain knowledge.

Original: We now present how to integrate the previously learned knowledge and the current domain knowledge by contrasting the previously learned knowledge and the full knowledge (both previously learned knowledge and the current domain knowledge). This is done based on the current domain data. Below we first introduce how to obtain therequired types of knowledge for input x , and then discuss how we contrast for integration. (1) Obtaining previously learned knowledge in the current LM using
Revised: Soft-masking helps prevent forgetting the previously learned knowledge.

Original: the input x . This is done by extracting the representation of x from the network learned from previous domains by combining the units in the layer and their importance scores I ( ≤ t − 1) l in a forward pass . The intuition is that since the importance scores show how important each unit is to the previously learned knowledge (including the general knowledge in the initialization functions), the resulting representation reflects the main previously learned knowledge used by x . Formally, we use I ( ≤ t − 1) l to gate the output of layer l (similar to Eq. 2),
Revised: We want to further encourage knowledge transfer by integrating the new and learned knowledge.

Original: [Equation] Given the units in layer l for the previously learned knowledge o prev l , we can plug it into the whole language model to obtain the final learned knowledge or representation, taking the average of each token’s output in the input sequence x (see o prev in Figure 1 (B)),
Revised: We propose to contrast the previously learned knowledge and the full knowledge (both previously learned knowledge and the current domain knowledge). Note that the contrasting cannot do anything to the shared past knowledge as it is protected by soft-masks.

Original: [Equation] (2) Obtaining full (both learned domains and current domain) knowledge in x . This is similar to the above except that we extract the representation by using all units in the layer without considering their importance (the gates are removed) , i.e., o full l [Equation]
Revised: Thus, it effectively pushes the current domain knowledge away to be complementary to the past knowledge.

Original: Similarly, we can plug it into the Transformer (see o full also in Figure 1 (B)), [Equation] It is also possible to use (1 − I ( ≤ t − 1) l ) as the importance of domain knowledge and contrast it with the previously learned knowledge, but this gives poorer results (Table 3) as explained in footnote 3.
Revised: This is done based on the current domain data as follows.

Original: (3) Contrasting  learned and full knowledge.
Revised: Contrasting the learned and full knowledge .

Original: Contrastive learning helps learn a good isotropic representation that is good for down-stream tasks, with the help of positive o + m and negative o − m instances. In our case, such instances are representations of the input x for both the full knowledge ( o full ) and the previously learned domain knowledge ( o prev ) in the current LM. This enables us to contrast the two representations, which essentially makes the two representations complementary as they represent two aspects of the same input.
Revised: We denote the output of LM without any consideration of importance as o full , which refers to the full knowledge.

Original: It thus encourages the current domain representation of x in o full to learn knowledge that is not already in the learned knowledge in previous domains and yet related to and integrated with the learned knowledge (  o prev ).
Revised: We further denote the output of LM that is multiplied by the importance (i.e., I ( ≤ t − 1) l ⊗ o l ) as o prev , which refers to the previously learned knowledge.

Original: We construct the contrastive instances as follows: for input x m , three instances are produced. Anchor o m andpositive instance o+ m are both the full knowledge from Eq. 10 obtained based on two independently sampled dropout masks in Transformer (recall this can be achieved by inputting x
Revised: We contrast the two by using o full as anchor and o full with different dropouts as positive samples (dentoed as o full+ ).

Original: m twice, see Sec. 3.1). o + m and o m are positive instances as the dropout noise has been shown good positive instances in training sentence embedding (Gao et al., 2021a). Negative instance o − m is the representation of x m in the previously learned knowledge obtained by Eq. 9. Contrastive learning pushes the positive and negative instances away, forcing them to be different and covering different aspects (full and previously learned knowledge) of the same input, resulting in knowledge integration.
Revised: o prev is used as negative instances.

Original: Formally, with o m , o + m , and o − m , our contrastive loss is (sim ( · ) is the cosine similarity),
Revised: Formally, with o full n , o full+ n , and o prev n , our contrastive loss is (sim ( · ) is the cosine similarity),

Original: [Equation]
Revised: [Equation]

Original: Compared to Eq. 1, the second term is added in the denominator, i.e., representation in the previously learned knowledge as additional negative instances.
Revised: Compared to Eq. 1, the second term is added in the denominator, i.e., representations in the previously learned knowledge as additional negative instances.

Original: Figure 1 (B) shows a red arrow pointed from o full to itself, indicating the positive instances are from inputting twice.
Revised: Figure 1 (B) shows a red arrow pointed from o full to itself, indicating the positive instances are from inputting twice.

Original: The dashed red arrow pointing to o prev indicates the negative instances contrasting the full and previously learned knowledge.
Revised: The dashed red arrow pointing to o prev indicates the negative instances contrasting the full and previously learned knowledge.

Original: Final Loss Function .
Revised: Final Loss Function .

Original: The final post-training loss combines the Masked Language Model (MLM) loss after applying the proposed soft-masking for the general knowledge(we call L MLM+SM , where SM is for soft-masking) (Sec. 3.1) and the proposed contrastive loss ( λ is a hyper-parameter),
Revised: The final DAP-training loss combines the Masked Language Model (MLM) loss after applying the proposed soft-masking for the general knowledge (Sec. 3.1) and the proposed contrastive loss ( λ is a hyper-parameter),

Original: [Equation]
Revised: [Equation]

Original: 3. C OMPUTE IMPORTANCE OF UNITS TO T HE CURRENT DOMAIN
Revised: 3. C OMPUTE IMPORTANCE OF UNITS TO T HE CURRENT DOMAIN

Original: After training the new/current domain t , we learn the units importance by applying Eq.
Revised: After training the new/current domain t , we learn the units importance by applying Eq.

Original: 3 for the domain.
Revised: 3 for the domain.

Original: We do not need any proxy to compute L impt as in Eq. 4 because we can directly use the current domain data.
Revised: We do not need any proxy to compute L impt as in Eq. 4 because we can directly use the current domain data.

Original: Specifically, we randomly sample a subset (a hyper-parameter) of the current domain data { ( x sub m , y sub m ) } , where x sub m is the input and y sub m is the masked token as in MLM selfsupervised loss.
Revised: Specifically, we randomly sample a subset (a hyper-parameter) of the current domain data { ( x sub n , y sub n ) } , where x sub n is the input and y sub n is the masked token as in MLM selfsupervised loss.

Original: We can then easily compute the importance I ( t ) l by plugging L MLM into L impt in Eq.3.
Revised: We can then easily compute the importance I ( t ) l by plugging L MLM into L impt in Eq.

Original: The resulting I ( t ) l will be used in the next task by accumulating with the previously accumulated importance (Eq. 6) and soft-masking the learning (Eq. 7). [Table]
Revised: The resulting I ( t ) l will be used in the next task by accumulating with the previously accumulated importance (Eq. 5) and soft-masking the learning (Eq. 6).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We use RoBERTa
Revised: We use RoBERTa

Original: (Liu et al., 2019) 6 as the LM.
Revised: (Liu et al., 2019) 7 as the LM.

Original: Following the standard evaluation setup (Lange et al., 2019) and, after a domain is trained, its training data is discarded.
Revised: Following the standard evaluation setup (Lange et al., 2019) and, after a domain is trained, its training data is discarded.

Original: After all domains are incrementally learned, the final model is evaluated by fine-tuning the end-tasks in all domains.
Revised: After all domains are incrementally learned, the final model is evaluated by fine-tuning the end-tasks in all domains.

Original: 4. D ATASETS AND B ASELINES
Revised: 4. D ATASETS AND B ASELINES

Original: Datasets: Table 1 shows the statistics of the 6 unlabeled domain corpora for post-training and their 6 corresponding end-task classification datasets .
Revised: Datasets: Table 1 shows the statistics of the 6 unlabeled domain corpora for DAP-training and their 6 corresponding end-task classification datasets .

Original: 7 3 of them are about reviews: Yelp Restaurant (Xu et al., 2019),
Revised: 8 3 of them are about reviews: Yelp Restaurant (Xu et al., 2019),

Original: Amazon Phone (Ni et al., 2019), Amazon Camera (Ni et al., 2019); 3 of them are academic papers:
Revised: Amazon Phone (Ni et al., 2019), Amazon Camera (Ni et al., 2019); 3 of them are academic papers:

Original: ACL Papers (Lo et al., 2020), AI Papers (Lo et al., 2020), and PubMed Papers 8 .
Revised: ACL Papers (Lo et al., 2020), AI Papers (Lo et al., 2020), and PubMed Papers 9 .

Original: Their corresponding end-task classification datasets are:9 Restaurant 10  , Phone (Ding et al., 2008;
Revised: Their corresponding end-task classification datasets are: 10 Restaurant 11 , Phone (Ding et al., 2008;

Original: Hu & Liu, 2004), Camera (Ding et al., 2008; Hu & Liu, 2004), ACL (ACL-ARC in (Jurgens et al., 2018)), AI (SCIERC in (Luan et al., 2018)), and PubMed (CHEMPORT in (Kringelum et al., 2016)).
Revised: Hu & Liu, 2004), Camera (Ding et al., 2008; Hu & Liu, 2004), ACL (ACL-ARC in (Jurgens et al., 2018)), AI (SCIERC in (Luan et al., 2018)), and PubMed (CHEMPORT in (Kringelum et al., 2016)).

Original: Baselines.
Revised: Baselines.

Original: We use 16 baselines, including non-continual learning (Non-CL) and continual learning (CL) baselines.
Revised: We use 16 baselines, including non-continual learning (Non-CL) and continual learning (CL) baselines.

Original: All CL baselines are originally for learning supervised data except DEMIX.
Revised: All CL baselines are originally for learning supervised data except DEMIX.

Original: We adapt them and replace their backbone with RoBERTa.
Revised: We adapt them and replace their backbone with RoBERTa.

Original: Details of each baseline is given in Appendix C.
Revised: Details of each baseline is given in Appendix B.

Original: Non-CL Baselines : Each baseline here builds a separate model for each task.
Revised: Non-CL Baselines : Each baseline here builds a separate model for each task.

Original: (1) Pool.
Revised: (1) Pool.

Original: We pool the data of all domains together and train only one model for all domains.
Revised: We pool the data of all domains together and train only one model for all domains.

Original: (2) RoBERTa
Revised: (2) RoBERTa

Original: (Liu et al., 2019) uses RoBERTa for end-task fine-tuning without post-training.
Revised: (Liu et al., 2019) uses RoBERTa for end-task fine-tuning without DAP-training.

Original: (3) Post (RoBERTa) uses the existing post-training method (MLM) in (Gururangan et al., 2020) to posst-train each domain separately.
Revised: (3) Post (RoBERTa) uses the existing DAP-training method (MLM) in (Gururangan et al., 2020) to posst-train each domain separately.

Original: Post (Adapter) adds adapter11 layers in Transformer for each domain for post-training (Jang et al., 2021;Madotto et al., 2020;
Revised: Post (Adapter) adds adapter layers in Transformer for each domain for DAP-training (Jang et al., 2021;

Original:  Houlsby et al., 2019).
Revised: Madotto et al., 2020; Houlsby et al., 2019).

Original: Only the added adapters are trainable.
Revised: Only the added adapters are trainable.

Original: In end-task fine-tuning, both RoBERTa and the adapters are trainable.
Revised: In end-task fine-tuning, both RoBERTa and the adapters are trainable.

Original: (5) Post (Prompt) is from (Lester et al., 2021).
Revised: (5) Post (Prompt) is from (Lester et al., 2021).

Original: In post-training, RoBERTa (the LM) is fixed and only the prompts are trained.
Revised: In DAP-training, RoBERTa (the LM) is fixed and only the prompts are trained.

Original: In end-task fine-tuning, both the LM and the trained prompt are trainable.
Revised: In end-task fine-tuning, both the LM and the trained prompt are trainable.

Original: CL Baselines : We use 2 naive baselines, which keep learning more domains with no mechanism to deal with CF or transfer.
Revised: CL Baselines : We use 2 naive baselines, which keep learning more domains with no mechanism to deal with CF or transfer.

Original: (6) NCL (Naive CL)
Revised: (6) NCL (Naive CL)

Original: continually post-trains the RoBERTa; and (7) NCL (Adapter)
Revised: continually DAP-trains the RoBERTa; and (7) NCL (Adapter)

Original: continually post-trains a set of adapters (Houlsby et al., 2019).
Revised: continually DAP-trains a set of adapters (Houlsby et al., 2019).

Original: 8 baselines are CL systems: (8) DEMIX (Gururangan et al., 2021) adds a new adapter for each new domain and initializes it with a previous adapter nearest to the new domain; (9) BCL (Ke et al., 2021b) uses capsule networks.
Revised: 8 baselines are CL systems: (8) DEMIX (Gururangan et al., 2021) adds a new adapter for each new domain and initializes it with a previous adapter nearest to the new domain; (9) BCL (Ke et al., 2021c) uses capsule networks.

Original: (10) CLASSIC (Ke et al., 2021a) uses contrastive learning.
Revised: (10) CLASSIC (Ke et al., 2021b) uses contrastive learning.

Original: KD is knowledge distillation (Hinton et al., 2015).
Revised: KD is knowledge distillation (Hinton et al., 2015).

Original: (12) EWC (Buzzega et al., 2020) is a popular regularization-based method.
Revised: (12) EWC (Buzzega et al., 2020) is a popular regularization-based method.

Original: (13) DER++ (Buzzega et al., 2020) is a replay method based on knowledge distillation.
Revised: (13) DER++ (Buzzega et al., 2020) is a replay method based on knowledge distillation.

Original: 16.4K tokens are saved for each domain in the replay memory, which is[Table] The results are averages of 5 random seeds (the domain training order is as they appear in the first row).
Revised: 16.4K tokens are saved for each domain in the replay memory, which is the largest memory we can use for the system to run.

Original: Due to space limits, the results for different domain orders and the standard deviations are reported in

Original: Appendix D and Appendix E, respectively).

Original: Non-CL baselines have no forgetting.

Original: [Table]

Original: the largest memory we can use for the system to run.

Original: (14) HAT (Serrà et al., 2018) is an effective parameter-isolation method.
Revised: (14) HAT (Serrà et al., 2018) is an effective parameter-isolation method.

Original: HAT is applied to Transformer layers (i.e., self-attention, intermediate and output layers).
Revised: HAT is applied to Transformer layers (i.e., self-attention, intermediate and output layers).

Original: (15) HAT-All is a HAT variant using the same post-training, but all features from the LMare used to do end-tasks (instead of only features from its domain sub-network as in HAT).
Revised: (15) HAT-All is a HAT variant that uses all features from the LM to do end-tasks (instead of only features from its domain sub-network as in HAT).

Original: (16) HAT (Adapter) (Ke et al., 2021b) uses HAT within adapters.
Revised: (16) HAT (Adapter) (Ke et al., 2021c) uses HAT within adapters.

Original: ELLE (Qin et al., 2022a) is not included as we tried to adapt it for our purpose by learning from RoBERTa, but it fails to converge.
Revised: ELLE (Qin et al., 2022) is not included as we adapted it for our purpose by learning from RoBERTa, but it fails to converge.

Original: 4. R ESULTS A NALYSIS AND A BLATION S TUDY
Revised: 4. R ESULTS A NALYSIS AND A BLATION S TUDY

Original: Due to space limits, Implementation Details are given in Appendix A. Table 2 reports the end-task fine-tuning results of all 15 systems on the 6 datasets.
Revised: Due to space limits, Implementation Details are given in Appendix C. Table 2 reports the end-task fine-tuning results of all 15 systems on the 6 datasets.

Original: We can see that the proposed CPS outperforms all baselines on average and also achieve the best knowledge transfer (negative forgetting rate).
Revised: We can see that the proposed DAS outperforms all baselines on average and also achieve the best knowledge transfer (negative forgetting rate).

Original: (1) CPS is slightly better than Pool on average.
Revised: (1) DAS is slightly better than Pool on average.

Original: This may be because (a) some domains are quite different (e.g. camera reviews and ACL papers), which results in some negative transfer in Pool.
Revised: This may be because (a) some domains are quite different (e.g. camera reviews and ACL papers), which results in some negative transfer in Pool.

Original: (b)
Revised: (b)

Original: CPS can learn with the general and previous domain knowledge protected by soft-masks.
Revised: DAS can learn with the general and previous domain knowledge protected by soft-masks.

Original: CPS achieves both forgetting prevention and knowledge transfer.
Revised: DAS achieves both forgetting prevention and knowledge transfer.

Original: Those baselines (KD, EWC,
Revised: Those baselines (KD, EWC,

Original: DER++) focusing only on forgetting prevention give poorer performance as they sacrifice accuracy to avoid CF.
Revised: DER++) focusing only on forgetting prevention give poorer performance as they sacrifice accuracy to avoid CF.

Original: Those baselines (BCL, CLASSIC and DEMIX) perform knowledge transfer achieve better results but still poorer than CPS.
Revised: Those baselines (BCL, CLASSIC and DEMIX) perform knowledge transfer achieve better results but still poorer than DAS.

Original: DEMIX has very weak transfer.
Revised: DEMIX has very weak transfer.

Revised: BCL, which can avoid CF while

Revised: [Table]

Revised: The results are averages of 5 random seeds (the domain training order is as they appear in the first row).

Revised: Due to space limits, the results for different domain orders and the standard deviations are reported in

Revised: Appendix D and Appendix E, respectively).

Original: BCL, which can avoid CF while also achieving some transfer, is weaker than NCL.
Revised: Non-CL baselines have no forgetting. also achieving some transfer, is weaker than NCL.

Original: In general, CL baselines are all poorer than CPS as they don’t have methods to encourage knowledge transfer or they have to rely on adapters.
Revised: In general, CL baselines are all poorer than DAS as they don’t have methods to encourage knowledge transfer or they have to rely on adapters.

Original: Directly learning the domains within the LM helps CPS achieve better results than adapter and prompt based methods.
Revised: Directly learning the domains within the LM helps DAS achieve better results than adapter and prompt based methods.

Original: CPS is better than adapter-based systems (Post(Adapter), NCL (Adapter) and
Revised: DAS is better than adapter-based systems (Post(Adapter), NCL (Adapter) and

Original: HAT (Adapter)) and prompt-based system (Post(Prompt)).
Revised: HAT (Adapter)) and prompt-based system (Post(Prompt)).

Original: This is because adapters and prompts do not have sufficient trainable parameters, which are also randomly initialized and can be hard to train.
Revised: This is because adapters and prompts do not have sufficient trainable parameters, which are also randomly initialized and can be hard to train.

Original: Using the full LM to learn all tasks rather than using sub-networks (of HAT-based methods) makes CPS more effective.
Revised: Using the full LM to learn all tasks rather than using sub-networks (of HAT-based methods) makes DAS more effective.

Original: HAT performs poorly, indicatingthat it is unsuitable for post-training as discussed in Sec. 1.
Revised: HAT performs poorly, indicating it is unsuitable for DAP-training as discussed in Sec. 1.

Original: Even if we use all features (not only the feature from the corresponding  the 
Revised: Even if we use all features (not only the feature from the corresponding sub-network), we still get poor results (HAT-All) as the features used in DAP-training (in an LM sub-network) are different from features used in end-task fine-tuning (features from the whole LM).

Original: Knowledge transfer and forgetting avoidance.
Revised: Knowledge transfer and forgetting avoidance.

Original: To see how the models fare on CF and knowledge transfer, we compare the forgetting rates ( forget R. )
Revised: To see how the models fare on CF and knowledge transfer, we compare the forgetting rates ( forget R. )

Original: (Liu et al., 2020), 1 t − 1 (cid:80) t − 1 k =

Revised: (Liu et al., 2020), 1 t − 1 (cid:80) t − 1 k =

Original: A k,k − A t,k , where
Revised: A k,k − A t,k , where

Original: A k,k is the end-task accuracy right after its domain k is post-trained, and A t,k is the accuracy of the end-task of domain k after post-training the last domain t .
Revised: A k,k is the end-task accuracy right after its domain k is DAP-trained, and A t,k is the accuracy of the end-task of domain k after DAP-training the last domain t .

Original: We average over all end-tasks except the last as the last domain has no forgetting.
Revised: We average over all end-tasks except the last as the last domain has no forgetting.

Original: The higher the forgetting rate is, the more forgetting it has.
Revised: The higher the forgetting rate is, the more forgetting it has.

Original: Negative rates indicate positive knowledge transfer.
Revised: Negative rates indicate positive knowledge transfer.

Original: Clearly, CPS has the strongest negative forgetting rate, indicating it does well on both forgetting prevention and knowledge transfer.
Revised: Clearly, DAS has the strongest negative forgetting rate, indicating it does well on both forgetting prevention and knowledge transfer.

Original: NCL,
Revised: NCL,

Original: NCL (Adapter), DEMIX, EWC, KD and DER++ all suffer from some forgetting.
Revised: NCL (Adapter), DEMIX, EWC, KD and DER++ all suffer from some forgetting.

Original: HAT has no forgetting but it cannot learn well.
Revised: HAT has no forgetting but it cannot learn well.

Original: HAT and BCL have no forgetting but are weak in transfer.
Revised: HAT and BCL have no forgetting but are weak in transfer.

Original: Effectiveness of the proxy KL-divergence loss.
Revised: Effectiveness of the proxy KL-divergence loss.

Original: We use proxy KL-divergence loss in the initialization function (Sec. 3.1) to compute the importance of units for general knowledge.
Revised: We use proxy KL-divergence loss in the initialization function (Sec. 3.1) to compute the importance of units for general knowledge.

Original: We are interested in how good the proxy is.
Revised: We are interested in how good the proxy is.

Revised: We use two kinds of experiments to provide evidences.

Revised: (1) Comparing with a sample set of D 0 .

Revised: In some cases, the continual DAP-training users may have the data D 0 that was used to pre-train the LM.

Revised: Then we can just sample a subset from D 0 to compute the parameter importance to the general knowledge in the LM.

Original: Since we don’t have  the  data that pre-trains RoBERTa, it is not obvious how to assess the quality of the proxy directly.
Revised: However, since we do not have Dthat was used to pre-train RoBERTa, we use the Wiki data (Merity et al., 2017) as the sample set of D 0 . We choose it as it is a general dataset with a wide topic coverage and was used to pre-train an LM, and it has a similar size as our domain data (around 700M).

Revised: We conducted two experiments using the data: (a) DAS (Wiki+MLM) , which uses MLM as the loss in the initialization stage to compute the importance of units (to identify the general knowledge) just like any other domains in the continual learning part, and (b) DAS (Wiki+KL) , which uses KL-divergence as in the initialization stage just like the proposed proxy method.

Revised: The results are given in Table 3.

Original: Here, we provide some indirect evidences to show the effectiveness of the  proxy for computing the importance of units to the general knowledge in  the LM.
Revised: We can see that DAS (Wiki + KL) performs similarly to DAS but outperforms DAS (Wiki + MLM). This indicates that the proposed proxy KL-divergence is more effective. MLM actually adapts the LM to the Wikipedia data, which may not be sufficiently representative of the original data used in pre-training the LM.

Revised: As a result, it ends up identifying the knowledge that is suitable only for the

Revised: [Table]

Revised: [Table]

Revised: Wikipedia data.

Revised: In contrast, the proposed proxy KL-divergence leverages the random dropout mask and measures the robustness, which is less related to a specific domain and thus reflects the (general) knowledge in the original LM better.

Revised: (2) Comparing general knowledge computed from different domain corpora .

Revised: Here, we also provide some indirect evidences to show the effectiveness of the proxy method for computing the importance of units to the general knowledge in the LM.

Original: We conduct a separate  experiment(none CL) to compare the attention heads’12 importance score vectors after applying the proxy using the data from different domains.
Revised: We conduct a separate non-CL experiment to compare the attention heads’ importance score vectors after applying the proxy using the data from different domains.

Original:  For each domain i , we compare its importance vector with the importance vector of every other domain, and then average the cosine similarities to get the value for domain i .
Revised: 12 For each domain i , we compare its importance vector with the importance vector of every other domain, and then average the cosine similarities to get the value for domain i .

Original: We get 0.92 for Restaurant, the same 0.91 for ACL, AI, and Phone, 0.89 for PubMed and 0.92 for Camera.
Revised: We get 0.92 for Restaurant, the same 0.91 for ACL, AI, and Phone, 0.89 for PubMed and 0.92 for Camera.

Original: We see that different domains give similar importance values,  indirectly showing that our proxy can  identify the common general knowledge.
Revised: We see that different domains give similar importance values, which indirectly shows that our proxy can approximately identify the common general knowledge.

Original: We also compute the importance score distributions of the proxy.

Original: For each of the 6 domains, after applying the proxy, around 20% of the attention heads are heavily protected ( 0 . 8 ≤ I gen l,i ≤ 1 . 0 ) and another 20% moderately protected ( 0 . 6 ≤ I gen l,i < 0 . 8 ), which indicate the general knowledge.

Original: While

Original: Phone, AI, Camera and Restaurant share a similar distribution, ACL and PubMed protect slightly less.

Original: This is understandable as PubMed and ACL (medical or NLP publications) are probably less common than the other domains and the general knowledge in the LM covers them less.

Original: Ablation.
Revised: Ablation.

Original: We want to know if the proposed (1) initialization (Sec. 3.1), (2) soft-masking, and (3) contrastive learning are helpful.
Revised: We want to know if the proposed (1) initialization (Sec. 3.1), (2) soft-masking, and (3) contrastive learning are helpful.

Original: To answer (1), we conduct the ablation CPS (w/o initialization) , where we remove the initialization and directly do the continual learning given no consideration to the general knowledge in the LM.
Revised: To answer (1), we conduct the ablation DAS (w/o initialization) , where we remove the initialization and directly do the continual learning given no consideration to the general knowledge in the LM.

Original: To answer (2), we conduct the ablations (1) CPS (w/o softmask) , where we remove the soft-masks, and only use contrastive learning based on Eq.
Revised: To answer (2), we conduct the ablations (1) DAS (w/o softmask) , where we remove the soft-masks, and only use contrastive learning based on Eq.

Original: 11 (with the second term in the denominator removed); and (2) CPS (random) with randomly generated importance scores to do soft-masking and contrastive learning.
Revised: 7 (with the second term in the denominator removed); and (2) DAS (random) with randomly generated importance scores to do soft-masking and contrastive learning.

Original: To answer (3), we conduct two ablations: (i) CPS (w/o contrast) where we remove the contrastive loss and only soft-mask according to the importance; (ii) CPS (domain-specific) where we contrast domain-specific and learned knowledge ( Sec. 3.2).
Revised: To answer (3), we conduct two ablations: (i) DAS (w/o contrast) where we remove the contrastive loss and only soft-mask according to the importance; (ii) DAS (domain-specific) where we contrast domain-specific and learned knowledge (Sec. 3.2).

Original: Table 3 shows that the full CPS is the best on average and for most domains, indicating that every component contributes.
Revised: Table 4 shows that the full DAS is the best on average and for most domains, indicating that every component contributes.

Original: Additional observations are: (1) CPS’s gain is partially from the preserved general knowledge.
Revised: Additional observations are: (1) DAS’s gain is partially from the preserved general knowledge.

Original: We can see CPS (w/o initialization) is poorer on average; (2) Soft-masking helps as CPS (w/o softmask) is poorer than CPS.
Revised: We can see DAS (w/o initialization) is poorer on average; (2) Soft-masking helps as DAS (w/o softmask) is poorer than DAS.

Original: This is reasonable because soft masking can preserve learned domains.
Revised: This is reasonable because soft masking can preserve learned domains.

Original: Besides, our gradient-based mask is informative as CPS (random) is worse than
Revised: Besides, our gradient-based mask is informative as DAS (random) is worse than

Original: CPS; (3) Contrastive learning is effective as CPS (w/o contrast) and CPS (domain-specific) are both poorer, indicating the contrastive learning in CPS can help learn good representations
Revised: DAS; (3) Contrastive learning is effective as DAS (w/o contrast) and DAS (domain-specific) are both poorer, indicating the contrastive learning in DAS can help learn good representations

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper proposed a novel method  for the continual post-training of an LM.
Revised: This paper proposed a novel method DAS for the continual DAP-training of an LM.

Original: It has two key ideas: (1) Preserving the important previous knowledge by soft-masking units according to their importance to overcome CF and to facilitate knowledge transfer.
Revised: It has three key ideas: (1) Preserving the important previous knowledge by soft-masking units according to their importance to overcome CF and to facilitate knowledge transfer.

Revised: (2) Using a novel proxy to compute the importance of units to the general knowledge in the LM.

Original: (2) Learning complementary representations for knowledge integration.
Revised: (3) Learning complementary representations for knowledge integration.

Original: A set of techniques is proposed to achieve them.
Revised: A set of techniques is proposed to achieve them.

Original: Extensive experiments showed the effectiveness of the proposed method.
Revised: Extensive experiments showed the effectiveness of DAS.

Original: The currentproposed approach involves two functions in learning.
Revised: The current approach involves two functions in learning.

Original: We will study how to combine them to further improve the results in the future.
Revised: We will study how to combine them to further improve the results in the future.

Original: A I MPLEMENTATION D ETAILS
Revised: A CKNOWLEDGEMENTS

Original: Architecture.

Original: We adopt RoBERTa BASE as our backbone LM.

Original: A masked language model head is applied for post-training.

Original: The end-task fine-tuning of RoBERTa follows the standard practice.

Original: For the three ASC tasks (see Table 1), we adopt the ASC formulation in Xu et al.

Original: (2019), where the aspect (e.g., “ sound ”) and review sentence (e.g., “ The sound is great ”) are concatenated via 
 .

Original: Hyperparameters.

Original: Unless otherwise stated, the same hyper-parameters are used in all experiments.

Original: The maximum input length is set to 164 which is sufficient for all datasets.

Original: Adam optimizer is used for both post-training and end-task fine-tuning.

Original: The max sequence length is also set to 164.

Original: Post-training. The learning rate is set to 1e-4 and batch size to 256. We train 2.5K steps for each domain, roughly a full pass through the domain data, following Gururangan et al. Xu et al.
Revised: The work of Zixuan Ke, Gyuhak Kim and Bing Liu was supported in part by a research contract from

Original: The subset of data { x sub m } for computing L impt to determine head importance in Secs.

Original: 3.1 and 3.3 is set to 1.64 Million tokens, which is sufficient in our experiments. λ in Eq. 12 is set to 1 and τ in Eq. 11 is set to 0.05.
Revised: KDDI, a research contract from DARPA (HR001120C0023) and three NSF grants (IIS-1910424,

Original: End-task fine-tuning.

Original: The learning rate is set to 1e-5 and batch size to 16. We train on end-task fine-tuning datasets for 5 epochs for Restaurant; 10 epochs for ACL, AI and PubMed; and 15 epochs for Phone and Camera. We simply take the results for the last epoch, assuming no validation sets. We empirically found that the above number of epochs gives us stable and convergence results.
Revised: IIS-1838770, and CNS-2225427)

Original: B D ATASETS D ETAILS
Revised: A D ATASETS D ETAILS

Original: Table 1 in the main paper has already showed the number of examples in each dataset.
Revised: Table 1 in the main paper has already showed the number of examples in each dataset.

Original: Here we provide additional details about the 4 types of end-tasks.
Revised: Here we provide additional details about the 4 types of end-tasks.

Original: (1) (Phone, Camera and Restaurant)
Revised: (1) (Phone, Camera and Restaurant)

Original: Aspect Sentiment Classification (ASC) is defined as follows: given an aspect or product feature (e.g., picture quality in a camera review) and a review sentence containing the aspect in a domain or product category (e.g., camera), classify if the sentence expresses a positive, negative, or neutral (no opinion) sentiment or polarity about the aspect (for the Phone and
Revised: Aspect Sentiment Classification (ASC) is defined as follows: given an aspect or product feature (e.g., picture quality in a camera review) and a review sentence containing the aspect in a domain or product category (e.g., camera), classify if the sentence expresses a positive, negative, or neutral (no opinion) sentiment or polarity about the aspect (for the Phone and

Original: Camera datasets, there are only negative and positive polarities in the data).
Revised: Camera datasets, there are only negative and positive polarities in the data).

Original: (2) (ACL) Citation intent classification is defined as follows: given a citing sentence (a sentence contains a citation), classify if the sentence expresses a citation function among “background”, “motivation”, “uses”, “extension” and “comparison or contrast future”.
Revised: (2) (ACL) Citation intent classification is defined as follows: given a citing sentence (a sentence contains a citation), classify if the sentence expresses a citation function among “background”, “motivation”, “uses”, “extension” and “comparison or contrast future”.

Original: (3) (AI) Relation classification is defined as follows: given a within-sentence word sequence span containing a pair of entities, classify if the span expresses a relation among “feature of”, “conjunction”, “evaluate for”, “hyponym of”, “used for”, “part of” and “compare”.
Revised: (3) (AI) Relation classification is defined as follows: given a within-sentence word sequence span containing a pair of entities, classify if the span expresses a relation among “feature of”, “conjunction”, “evaluate for”, “hyponym of”, “used for”, “part of” and “compare”.

Original: (4) (PubMed) Chemical-protein interaction classification is defined as follows: given a span containing a pair of chemical and protein, classify if the span expresses a chemical-protein interaction among “downregulator”, “substrate”, “indirect-upregulator”, “indirect-downregulator”, “agnonist”, “activator” [Table] [Table]
Revised: (4) (PubMed) Chemical-protein interaction classification is defined as follows: given a span containing a pair of chemical and protein, classify if the span expresses a chemical-protein interaction among “downregulator”, “substrate”, “indirect-upregulator”, “indirect-downregulator”, “agnonist”, “activator”, “product of”, “agonist-activator”, “inhibitor”, “upregulator”, “substrate product of”, “agonist-inhibitor”and “antagonist”.

Original: C B ASELINE D ETAILS
Revised: B B ASELINE D ETAILS

Original: Non-Continual Learning Baselines : Each of these baselines builds a separate model for each task independently.
Revised: Non-Continual Learning Baselines : Each of these baselines builds a separate model for each task independently.

Original: It thus has no knowledge transfer or CF.
Revised: It thus has no knowledge transfer or CF.

Original: (1) Non-post-training (RoBERTa)
Revised: (1) Non-DAP-training (RoBERTa)

Original: Liu et al.
Revised: Liu et al.

Original: (2019) uses the original RoBERTa for the end-task fine-tuning without any post-training.
Revised: (2019) uses the original RoBERTa for the end-task fine-tuning without any DAP -training.

Original: This is the only one without any post-training.
Revised: This is the only one without any DAP-training.

Original: All the following baselines use the masked language model loss (MLM) for post-training.
Revised: All the following baselines use the masked language model loss (MLM) for DAP-training.

Original: (2) Post-training using masked language model loss (Post (RoBERTa)) is the existing post-training method in Gururangan et al.
Revised: (2) DAP-training using masked language model loss (Post (RoBERTa)) is the existing DAPtraining method in Gururangan et al.

Original: To our knowledge, the existing post-training systems are all based on the MLM loss.
Revised: To our knowledge, the existing DAP-training systems are all based on the MLM loss.

Original: (3) Post-training using adapter-tuning Madotto et al.
Revised: (3) DAP-training using adapter-tuning Madotto et al.

Original: Houlsby et al.
Revised: Houlsby et al.

Original: (2019) adds small adapter layers between layers of Transformer for post-training.
Revised: (2019) adds small adapter layers between layers of Transformer for DAP-training.

Original: We follow the adapter design in Madotto et al.
Revised: We follow the adapter design in Madotto et al.

Original: Houlsby et al.
Revised: Houlsby et al.

Original: An adapter is simply a 2 layers of fully connected network.
Revised: An adapter is simply a 2 layers of fully connected network.

Original: During post-training, the Transformer is fixed, only the added adapters are trainable.
Revised: During DAP-training, the Transformer is fixed, only the added adapters are trainable.

Original: The bottleneck size (adapter size) is set to 128.
Revised: The bottleneck size (adapter size) is set to 128.

Original: During end-task fine-tuning, both RoBERTa and the adapters are trainable to ensure fair comparison.
Revised: During end-task fine-tuning, both RoBERTa and the adapters are trainable to ensure fair comparison.

Original: (4) Post-training using prompt-tuning Lester et al.
Revised: (4) DAP-training using prompt-tuning Lester et al.

Original: ) adds a sequence of real vector tokens (called virtual tokens or prompt tokens) to the end of the original sequence.
Revised: (2021) adds a sequence of real vector tokens (called virtual tokens or prompt tokens) to the end of the original sequence.

Original: In post-training, RoBERTa (the LM) is fixed and only the prompt tokens are trained.
Revised: In DAP-training, RoBERTa (the LM) is fixed and only the prompt tokens are trained.

Original: In end-task fine-tuning, both LM and the trained prompt are trainable.
Revised: In end-task fine-tuning, both LM and the trained prompt are trainable.

Original: We initialize 100 tokens and set the learning rate of the prompt token to 0.3 in post-training, following the setting in Lester et al.
Revised: We initialize 100 tokens and set the learning rate of the prompt token to 0.3 in DAP-training, following the setting in Lester et al.

Original: Continual Learning (CL) Baselines.
Revised: Continual Learning (CL) Baselines.

Original: (5) Naive continual learning (NCL) is a naive extension of Gururangan et al
Revised: (5) Naive continual learning (NCL) is a naive extension of Gururangan et al.

Original: , which continually/ incrementally post-trains the LM to learn all domains using MLM loss with no mechanism to deal with CF.
Revised: (2020), which continually/incrementally DAP-trains the LM to learn all domains using MLM loss with no mechanism to deal with CF.

Original: Continual learning with adapter (NCL (Adapter))
Revised: Continual learning with adapter (NCL (Adapter))

Original: Houlsby et al
Revised: Houlsby et al.

Original:  is similar to the adapter based system.The only
Revised: (2019) is similar to the adapter based system.

Original: difference

Original: is that the

Original: same set

Original: of adapters

Original:  is shared across all domains rather than using a new adapter for each new domain
Revised: The only difference is that the same set of adapters is shared across all domains, rather than using a new adapter for each new domain.

Original: (7) DEMIX (DEMIX) Gururangan et al.
Revised: (7) DEMIX (DEMIX) Gururangan et al.

Original: (2021) is a recent model to adapt pre-trained LM with new domains.
Revised: (2021) is a recent model to adapt pre-trained LM with new domains.

Original: It adds a new adapter once a new domain arrives (network expansion is needed) and initializes the new adapter with the parameters of the previous trained adapter nearest to the new domain data.
Revised: It adds a new adapter once a new domain arrives (network expansion is needed) and initializes the new adapter with the parameters of the previous trained adapter nearest to the new domain data.

Original: They use the perplexity on a held-out sample to choose the most probable adapter.
Revised: They use the perplexity on a held-out sample to choose the most probable adapter.

Original: For fair comparison, we use the same size as { x sub m } as the held-out samples.
Revised: For fair comparison, we use the same size as { x sub n } as the held-out samples.

Original: (8) Hard attention to overcome forgetting (HAT (Adapter))
Revised: (8) Hard attention to overcome forgetting (HAT (Adapter))

Original: Ke et al.
Revised: Ke et al.

Original: (2021b) is derived from
Revised: (2021c) is derived from

Original: HAT Serrà et al.
Revised: HAT Serrà et al.

Original: (2018), the state-of-the-art parameter-isolation based method with almost no forgetting.
Revised: (2018), the state-of-the-art parameter-isolation based method with almost no forgetting.

Original: However, HAT requires task id information in end-task fine-tuning (CPS works in domainagnostic manner and does not need the task id information; see Sec. 1).
Revised: However, HAT requires task id information in end-task fine-tuning (DAS works in domainagnostic manner and does not need the task id information; see Sec. 1).

Original: HAT also needs to train an addition task embedding to mask each layer of the network which makes the post-training inefficient.
Revised: HAT also needs to train an addition task embedding to mask each layer of the network which makes the DAP-training inefficient.

Original: (9) Continual learning plugin with capsule(BCL)
Revised: (9) Continual learning plugin with capsule(BCL)

Original: Ke et al.
Revised: Ke et al.

Original: (2021b) is a continual learning model that can avoid forgetting and encourage knowledge transfer.
Revised: (2021c) is a continual learning model that can avoid forgetting and encourage knowledge transfer.

Original: It is similar to NCL (Adapter).
Revised: It is similar to NCL (Adapter).

Original: The difference is that its adapters consist of two modules, one is a capsule network (a new capsule is added once a new domain arrives) to encourage transfer and the other one is similar to HAT to avoid forgetting.
Revised: The difference is that its adapters consist of two modules, one is a capsule network (a new capsule is added once a new domain arrives) to encourage transfer and the other one is similar to HAT to avoid forgetting.

Original: Similar to HAT, task/domain information is needed in end-task fine-tuning.
Revised: Similar to HAT, task/domain information is needed in end-task fine-tuning.

Original: We replace the backbone network from BERT with RoBERTa for fair comparison.
Revised: We replace the backbone network from BERT with RoBERTa for fair comparison.

Original: (10) Continual learning plugin with contrastive transfer (CLASSIC) Ke et al.
Revised: (10) Continual learning plugin with contrastive transfer (CLASSIC) Ke et al.

Original: (2021a) is a continual learning model that can avoid forgetting and encourage knowledge transfer via contrasting loss.
Revised: (2021b) is a continual learning model that can avoid forgetting and encourage knowledge transfer via contrasting loss.

Original: It is similar to HAT.
Revised: It is similar to HAT.

Original: but 3 additional contrastive loss are used for distillation, knowledge transfer and supervised contrast.
Revised: but 3 additional contrastive loss are used for distillation, knowledge transfer and supervised contrast.

Original: Since CPS is working on unsupervised data, we remove the supervised contrastive loss.
Revised: Since DAS is working on unsupervised data, we remove the supervised contrastive loss.

Original: Similar to HAT, task information is needed in end-task fine-tuning.
Revised: Similar to HAT, task information is needed in end-task fine-tuning.

Original: We replace the backbone network from BERT with RoBERTa for fair comparison.
Revised: We replace the backbone network from BERT with RoBERTa for fair comparison.

Original: (11) Knowledge distillation (KD) Hinton et al.
Revised: (11) Knowledge distillation (KD) Hinton et al.

Original: (2015) minimizes the representational deviation between the learned representation and the new representation in post-training.
Revised: (2015) minimizes the representational deviation between the learned representation and the new representation in DAP-training.

Original: We compute the KL divergence between the representations (the output before the masked language model prediction head) of each token of the previous post-trained LM and current LM as the distillation loss.
Revised: We compute the KL divergence between the representations (the output before the masked language model prediction head) of each token of the previous DAP-trained LM and current LM as the distillation loss.

Original: (12) EWC Buzzega et al.
Revised: (12) EWC Buzzega et al.

Original: (2020) is a popular regularization-based method which adopts elastic weights consolidation to add L 2 regularization to parameter changes.
Revised: (2020) is a popular regularization-based method which adopts elastic weights consolidation to add L 2 regularization to parameter changes.

Original:  DER++ Buzzega et al.
Revised: (13) DER++ Buzzega et al.

Original: ) is a recent replay method using distillation to regularize the new task training.
Revised: (2020) is a recent replay method using distillation to regularize the new task training.

Original: We store 16.4K tokens for each learned domain as the memory, which is the largest memory we can use for the system to run.
Revised: We store 16.4K tokens for each learned domain as the memory, which is the largest memory we can use for the system to run.

Original: (14) HAT Serrà et al.
Revised: (14) HAT Serrà et al.

Original: (2018) is used in the Transformer layers (including self-attention, intermediate and output layers) rather than the added adapter layers.
Revised: (2018) is used in the Transformer layers (including self-attention, intermediate and output layers) rather than the added adapter layers.

Original: Additional task embedding and task information for end-task fine-tuning are needed.
Revised: Additional task embedding and task information for end-task fine-tuning are needed

Original:  D POST - TRAINING IN D IFFERENT O RDERS
Revised: C I MPLEMENTATION D ETAILS

Revised: Architecture.

Revised: We adopt RoBERTa BASE as our backbone LM.

Revised: A masked language model head is applied for DAP-training.

Revised: The end-task fine-tuning of RoBERTa follows the standard practice.

Revised: For the three ASC tasks (see Table 1), we adopt the ASC formulation in Xu et al.

Revised: (2019), where the aspect (e.g., “ sound ”) and review sentence (e.g., “ The sound is great ”) are concatenated via 
 .

Revised: Hyperparameters.

Revised: Unless otherwise stated, the same hyper-parameters are used in all experiments.

Revised: The maximum input length is set to 164 which is sufficient for all datasets.

Revised: Adam optimizer is used for both DAP-training and end-task fine-tuning.

Revised: The max sequence length is also set to 164.

Revised: DAP-training.

Revised: The learning rate is set to 1e-4 and batch size to 256.

Revised: We train 2.5K steps for each domain, roughly a full pass through the domain data, following Gururangan et al.

Revised: Xu et al.

Revised: The subset of data { x sub n } for computing L impt to determine head importance in Secs.

Revised: 3.1 and 3.3 is set to 1.64 Million tokens, which is sufficient in our experiments.

Revised: λ in Eq.

Revised: 8 is set to 1 and τ in

Revised: Eq.

Revised: 7 is set to 0.05.

Revised: End-task fine-tuning.

Revised: The learning rate is set to 1e-5 and batch size to 16.

Revised: We train on end-task fine-tuning datasets for 5 epochs for Restaurant; 10 epochs for ACL, AI and PubMed; and 15 epochs for Phone and Camera.

Revised: We simply take the results for the last epoch, assuming no validation sets.

Revised: We empirically found that the above number of epochs gives us stable and convergence results.

Revised: [Table]

Revised: [Table]

Revised: D DAP- TRAINING IN D IFFERENT O RDERS

Original: Table 2 in the main paper reported the results for the order Restaurant → ACL → AI → Phone → PubMed → Camera .
Revised: Table 2 in the main paper reported the results for the order Restaurant → ACL → AI → Phone → PubMed → Camera .

Original: We now look at how the order affects the results.
Revised: We now look at how the order affects the results.

Original: Due to the computation intensive nature of post-training, we only report the best baseline (NCL) and CPS results with different domain orders.
Revised: Due to the computation intensive nature of DAP-training, we only report the best baseline (NCL) and DAS results with different domain orders.

Original: Table 4 shows NCL and CPS’s results of 5 different orders.
Revised: Table 5 shows NCL and DAS’s results of 5 different orders.

Original: We can see CPS is always better than NCL, demonstrating the effectiveness of CPS.
Revised: We can see DAS is always better than NCL, demonstrating the effectiveness of DAS.

Original: E S TANDARD D EVIATIONS
Revised: E S TANDARD D EVIATIONS

Original: Table 5 reports the standard deviations of the corresponding results in Table 2 (in the main paper) of CPS and the considered baselines over 5 runs with random seeds.
Revised: Table 6 reports the standard deviations of the corresponding results in Table 2 (in the main paper) of DAS and the considered baselines over 5 runs with random seeds.

Original: We can see the results of CPS are stable.
Revised: We can see the results of DAS are stable.

Original: Some baselines (e.g., RoBERTa in AI, Post (RoBERTa) in Camera) can have quite large standard deviations.
Revised: Some baselines (e [Table] g., RoBERTa in AI, Post (RoBERTa) in Camera) can have quite large standard deviations.

Original: Table 6 reports the standard deviations of the corresponding results in Table 3 (in the main paper) of
Revised: Table 7 reports the standard deviations of the corresponding results in Table 4 (in the main paper) of

Original: CPS and the considered baselines over 5 runs with random seeds.
Revised: DAS and the considered baselines over 5 runs with random seeds.

Original: We can see the results of CPS and its variants are stable. 
Revised: We can see the results of DAS and its variants are stable. [Table] [Figure]