Original: C ONTINUAL L EARNING OF L ANGUAGE M ODELS
Revised: C ONTINUAL L EARNING OF L ANGUAGE M ODELS

Original: A BSTRACT
Revised: A BSTRACT

Original: Language models (LMs) have been instrumental for the rapid advance of natural language processing.
Revised: Language models (LMs) have been instrumental for the rapid advance of natural language processing.

Original: This paper studies continual learning of LMs, in particular, continual domain-adaptive
Revised: This paper studies continual learning of LMs, in particular, continual domain-adaptive

Original: pre-training (or continual DAP-training ).
Revised: pre-training (or continual DAP-training ).

Original: Existing research has shown that further pre-training an LM using a domain corpus to adapt the
Revised: Existing research has shown that further pre-training an LM using a domain corpus to adapt the

Original: LM to the domain can improve the end task performance in the domain.
Revised: LM to the domain can improve the end task performance in the domain.

Original: This paper proposes a novel method to continually
Revised: This paper proposes a novel method to continually

Original: DAP
Revised: DAP

Original: -train an LM with a sequence of unlabeled domain corpora to adapt the LM to these domains to improve their endtask performances.
Revised: -train an LM with a sequence of unlabeled domain corpora to adapt the LM to these domains to improve their endtask performances.

Original: The key novelty of our method is a soft-masking mechanism that directly controls the update to the LM.
Revised: The key novelty of our method is a soft-masking mechanism that directly controls the update to the LM.

Original: A novel proxy is also proposed to preserve the general knowledge in the original LM.
Revised: A novel proxy is also proposed to preserve the general knowledge in the original LM.

Original: Additionally, it contrasts the representations of the previously learned domain knowledge (including the general knowledge in the pre-trained LM) and the knowledge from the current full network to achieve knowledge integration.
Revised: Additionally, it contrasts the representations of the previously learned domain knowledge (including the general knowledge in the pre-trained LM) and the knowledge from the current full network to achieve knowledge integration.

Original: The method not only overcomes catastrophic forgetting , but also achieves knowledge transfer to improve end-task performances.
Revised: The method not only overcomes catastrophic forgetting , but also achieves knowledge transfer to improve end-task performances.

Original: Empirical evaluation demonstrates the effectiveness of the proposed method.
Revised: Empirical evaluation demonstrates the effectiveness of the proposed method.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Pre-trained language models (LMs) like BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) have significantly advanced NLP.
Revised: Pre-trained language models (LMs) like BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) have significantly advanced NLP.

Original: Recently, LMs have also been used by many continual learning (CL) systems to learn a sequence of end-tasks incrementally (Ke et al., 2021a;
Revised: Recently, LMs have also been used by many continual learning (CL) systems to learn a sequence of end-tasks incrementally (Ke et al., 2021a;

Original: Sun et al., 2020; Huang et al., 2021), which we call continual end-task learning .
Revised: Sun et al., 2020; Huang et al., 2021), which we call continual end-task learning .

Original: It is also desirable to continually pre-train
Revised: It is also desirable to continually pre-train

Original: LMs themselves.
Revised: LMs themselves.

Original: This includes (1) continual general pre-training , which incrementally updates the
Revised: This includes (1) continual general pre-training , which incrementally updates the

Original: LM using the most recent data that has a similar distribution as the pre-training data, and (2) continual domain-adaptive pre-training , which continually further pre-trains a LM incrementally to adapt it to a sequence of domains.
Revised: LM using the most recent data that has a similar distribution as the pre-training data, and (2) continual domain-adaptive pre-training , which continually further pre-trains a LM incrementally to adapt it to a sequence of domains.

Original: Note that LM editing (with or without continual learning)
Revised: Note that LM editing (with or without continual learning)

Original: (Mitchell et al., 2022) that corrects mistakes learned in the LM is a special case of continual end-task learningin the class-incremental learning setting (Kim et al., 2022) as each editing task or a group of editing tasks learned together is basically a task in continual learning, which aims to perform the editings correctly without interfering with or forgetting the other knowledge already learned in the current LM.
Revised: (Mitchell et al., 2022) that corrects mistakes learned in the LM is a special case of continual end-task learning as each editing task or a group of editing tasks learned together is basically a task in continual learning, which aims to perform the editings correctly without interfering with or forgetting the other knowledge already learned in the current LM.

Original: This paper focuses on continual d omain- a daptive p re-training (or continual DAP-training ) of LMs.
Revised: This paper focuses on continual d omain- a daptive p re-training (or continual DAP-training ) of LMs.

Original: It is known that DAP-training 2 an LM (without continual learning) using a large unlabeled domain corpus before end-task fine-tuning achieves better results (Gururangan et al., 2020; Xu et al., 2019;
Revised: It is known that DAP-training 2 an LM (without continual learning) using a large unlabeled domain corpus before end-task fine-tuning achieves better results (Gururangan et al., 2020; Xu et al., 2019;

Original: Ke et al., 2022a).
Revised: Ke et al., 2022b).

Original: This paper goes a step further to continually learn to improve an LM’s ability to handle new and emerging domains  without forgetting  skills or knowledge learned in the past.
Revised: This paper goes a step further to continually learn to improve an LM’s ability to handle new or emerging domains or topics without forgetting the skills or knowledge learned in the past.

Original: This is important in the real world, where the data shifts constantly and new domains, events or topics keep emerging (Ke et al., 2022a) and the LM needs to be updated to serve the users better.
Revised: This is important in the real world, where the data shifts constantly and new domains, events or topics keep emerging (Ke et al., 2022b) and the LM needs to be updated to serve the users better.

Original: We call this problem continual DAP-training .
Revised: We call this problem continual DAP-training .

Original: Starting from a pre-trained general LM (i.e., the LM has already been pre-trained on D 0 ), we incrementally DAP-train a sequence of domain corpora
Revised: Starting from a pre-trained general LM (i.e., the LM has already been pre-trained on D 0 ), we incrementally DAP-train a sequence of domain corpora

Original: D 1 , D 2 , ... .
Revised: D 1 , D 2 , ... .

Original: Once a domain is trained, its data is no longer accessible.
Revised: Once a domain is trained, its data is no longer accessible.

Original: This is different from conventional continual learning ( CL ) where each task is an end-task.
Revised: This is different from conventional continual learning ( CL ) where each task is an end-task.

Original: In the proposed continual
Revised: In the proposed continual

Original: DAP-training, each task is an unlabeled domain corpus to be learned.
Revised: DAP-training, each task is an unlabeled domain corpus to be learned.

Original: An end-task fine-tunes the continually DAP-trained LM to evaluate its performance.
Revised: An end-task fine-tunes the continually DAP-trained LM to evaluate its performance.

Original: It is worth noting that D 0 is usually a broad or general domain (e.g., News).
Revised: It is worth noting that D 0 is usually a broad or general domain (e.g., News).

Original: In practice, a continually DAP-trained LM may be trained by individual users, institutions or a mix of both who have one or more large corpora of some particular domains.
Revised: In practice, a continually DAP-trained LM may be trained by individual users, institutions or a mix of both who have one or more large corpora of some particular domains.

Original: In such cases, the raw data may not be shared, but the final LM can be shared by all.
Revised: In such cases, the raw data may not be shared, but the final LM can be shared by all.

Original: There are multiple desiderata for a continual DAP-training system: (1) It should not suffer from catastrophic forgetting (CF ), i.e., it should perform reasonably well on learned domains.
Revised: There are multiple desiderata for a continual DAP-training system: (1) It should not suffer from catastrophic forgetting (CF ), i.e., it should perform reasonably well on learned domains.

Original: This requires the system (a) to overcome CF for each new domain and (b) to overcome CF for the general language knowledge in the LM.
Revised: This requires the system (a) to overcome CF for each new domain and (b) to overcome CF for the general language knowledge in the LM.

Original: This is important because the knowledge learned from each domain alone will not be sufficient for good end-task performances.
Revised: This is important because the knowledge learned from each domain alone will not be sufficient for good end-task performances.

Original: (2) It should encourage knowledge transfer ( KT ) across domains to achieve improved end-task performances.
Revised: (2) It should encourage knowledge transfer ( KT ) across domains to achieve improved end-task performances.

Original: This requires the system to enable (a) forward transfer , learning a new domain by leveraging the knowledge from previous domains, and (b) backwards transfer , gaining improved performance on previous domains after learning a relevant new domain.
Revised: This requires the system to enable (a) forward transfer , learning a new domain by leveraging the knowledge from previous domains, and (b) backwards transfer , gaining improved performance on previous domains after learning a relevant new domain.

Original: (3) It should work without requiring the domain-ID for each end-task fine-tuning.
Revised: (3) It should work without requiring the domain-ID for each end-task fine-tuning.

Original: None of the existing CL methods can achieve all the above.
Revised: None of the existing CL methods can achieve all the above.

Original: This paper represents a step towards achieving them.
Revised: This paper represents a step towards achieving them.

Original: The proposed method is called DAS (Continual DA -pre-training of LMs with
Revised: The proposed method is called DAS (Continual DA -pre-training of LMs with

Original: S oft-masking).
Revised: S oft-masking).

Original: DAS proposes a novel soft-masking mechanism that computes the importance (a real number between 0 and 1) of units 3 for general or domain knowledge and soft-mask them based on their importance values to control the backward gradient flow.
Revised: DAS proposes a novel soft-masking mechanism that computes the importance (a real number between 0 and 1) of units 3 for general or domain knowledge and soft-mask them based on their importance values to control the backward gradient flow.

Original: In the forward pass, soft-masking is not applied, which encourages KT across domains.
Revised: In the forward pass, soft-masking is not applied, which encourages KT across domains.

Original: It does not isolate any sub-network for any domain so that the knowledge in the full LM can be leveraged for end-task fine-tuning.
Revised: It does not isolate any sub-network for any domain so that the knowledge in the full LM can be leveraged for end-task fine-tuning.

Original: To apply this mechanism, DAS implements two functions: (1) Initialization , which computes the importance of units to the general knowledge in the LM without accessing the LM pre-training data ( D 0 ).
Revised: To apply this mechanism, DAS implements two functions: (1) Initialization , which computes the importance of units to the general knowledge in the LM without accessing the LM pre-training data ( D 0 ).

Original: It is applied on the pre-trained LM before the continual learning starts, and (2) continual learning , which DAP-trains each domain while preventing CF on the general and domain knowledge and encouraging cross-domain KT.
Revised: It is applied on the pre-trained LM before the continual learning starts, and (2) continual learning , which DAP-trains each domain while preventing CF on the general and domain knowledge and encouraging cross-domain KT.

Original: In (1), it is not obvious how to compute the importance without pre-training data.
Revised: In (1), it is not obvious how to compute the importance without pre-training data.

Original: DAS proposes a novel proxy based on robustness to compute the importance of units for the general knowledge.
Revised: DAS proposes a novel proxy based on robustness to compute the importance of units for the general knowledge.

Original: In (2), the soft-masking is directly applicable because we have the domain data and the importance can be computed based on its gradient inspired by the pruning community (Li et al., 2021;
Revised: In (2), the soft-masking is directly applicable because we have the domain data and the importance can be computed based on its gradient inspired by the pruning community (Li et al., 2021;

Original: Michel et al., 2019).
Revised: Michel et al., 2019).

Original: Moreover, DAS contrasts the previously learned knowledge and the full (including both the learned domains and the current domain) knowledge to encourage the current domain representation to learn knowledge that is not already in the knowledge learned from previous domains and integrate it with the learned knowledge 4 .
Revised: Moreover, DAS contrasts the previously learned knowledge and the full (including both the learned domains and the current domain) knowledge to encourage the current domain representation to learn knowledge that is not already in the knowledge learned from previous domains and integrate it with the learned knowledge 4 .

Original: In end-task fine-tuning,
Revised: In end-task fine-tuning,

Original: DAS does not requires the domain-ID as all knowledge is accumulated into the DAP-trained LM.
Revised: DAS does not requires the domain-ID as all knowledge is accumulated into the DAP-trained LM.

Original: In summary, this work makes the following contributions.
Revised: In summary, this work makes the following contributions.

Original: (i)
Revised: (i)

Original: It studies the new problem of continual
Revised: It studies the new problem of continual

Original: DAP-training and discovers that the full LM is needed for a good continual DAP-training method.
Revised: DAP-training and discovers that the full LM is needed for a good continual DAP-training method.

Original: The popular parameter-isolation approach to overcoming CF in convention CL is unsuitable.
Revised: The popular parameter-isolation approach to overcoming CF in convention CL is unsuitable.

Original: (ii)
Revised: (ii)

Original: It proposes a novel soft-masking method to overcome CF and to encourage KT, and a constrative learning based method for knowledge integration.
Revised: It proposes a novel soft-masking method to overcome CF and to encourage KT, and a constrative learning based method for knowledge integration.

Original: (iii)
Revised: (iii)

Original: To preserve the general knowledge in the LM, a novel proxy is also proposed.
Revised: To preserve the general knowledge in the LM, a novel proxy is also proposed.

Original: (iv) Experimental results demonstrate the effectiveness of DAS.
Revised: (iv) Experimental results demonstrate the effectiveness of DAS.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: DAP-training.
Revised: DAP-training.

Original: DAP-training can be achieved by directly updating the LM (Xu et al., 2019; Sun et al., 2019; Lee et al., 2020;
Revised: DAP-training can be achieved by directly updating the LM (Xu et al., 2019; Sun et al., 2019; Lee et al., 2020;

Original: Alsentzer et al., 2019; Gururangan et al., 2020; Chakrabarty et al., 2019;
Revised: Alsentzer et al., 2019; Gururangan et al., 2020; Chakrabarty et al., 2019;

Original: Ke et al., 2022a) or by training only a small set of additional parameters.
Revised: Ke et al., 2022b) or by training only a small set of additional parameters.

Original: For example, Pfeiffer et al.
Revised: For example, Pfeiffer et al.

Original: (2020); Wang et al.
Revised: (2020); Wang et al.

Original: (2020a);
Revised: (2020a);

Original: Ke et al.
Revised: Ke et al.

Original: (2021a;b;c) trained adapters and Gu et al.
Revised: (2021a;b;c) trained adapters and Gu et al.

Original: (2021) trained a prompt to adapt to a domain.
Revised: (2021) trained a prompt to adapt to a domain.

Original: While adapter and prompt could be effective, transfer knowledge among these additional modules is usually challenging and can be inaccurate.
Revised: While adapter and prompt could be effective, transfer knowledge among these additional modules is usually challenging and can be inaccurate.

Original: DAS belongs to the former family that directly updates the LM.
Revised: DAS belongs to the former family that directly updates the LM.

Original: This is very challenging for CL due to CF.
Revised: This is very challenging for CL due to CF.

Original: To our knowledge, no existing system in this family is about CL.
Revised: To our knowledge, no existing system in this family is about CL.

Original: Continual learning.
Revised: Continual learning.

Original: Most CL methods were proposed to overcome CF: (1) Regularization methods (Kirkpatrick et al., 2016; Seff et al., 2017) compute the importance of each parameter to previous tasks and uses a regularizer to penalize the sum of changes.
Revised: Most CL methods were proposed to overcome CF: (1) Regularization methods (Kirkpatrick et al., 2016; Seff et al., 2017) compute the importance of each parameter to previous tasks and uses a regularizer to penalize the sum of changes.

Original: DAS is related to but also very different from EWC (Kirkpatrick et al., 2016).
Revised: DAS is related to but also very different from EWC (Kirkpatrick et al., 2016).

Original: (1) DAS does not control each parameter/weight, but only attention heads or neurons based on their importance scores.
Revised: (1) DAS does not control each parameter/weight, but only attention heads or neurons based on their importance scores.

Original: This gives less forgetting (see the forgetting rate in Table 2) because even a small change to each parameter for a neuron can give a large total change to the neuron’s activation.
Revised: This gives less forgetting (see the forgetting rate in Table 2) because even a small change to each parameter for a neuron can give a large total change to the neuron’s activation.

Original: (2) DAS directly controls the backward gradient flow on each neuron, which is more fine-grained and effective than the sum of changes of all parameters.
Revised: (2) DAS directly controls the backward gradient flow on each neuron, which is more fine-grained and effective than the sum of changes of all parameters.

Original: Our experimental results confirm that EWC is significantly poorer than DAS (see Table 2).
Revised: Our experimental results confirm that EWC is significantly poorer than DAS (see Table 2).

Original: (2) Replay methods retain (Rebuffi et al., 2017; Wang et al., 2020b) or generate some data of old tasks (Shin et al., 2017; He & Jaeger, 2018) and use them in learning a new task; (3) parameter-isolation methods (Serrà et al., 2018; Wortsman et al., 2020) allocate neurons and parameters or sub-networks for different tasks/domains and mask them in task learning.
Revised: (2) Replay methods retain (Rebuffi et al., 2017; Wang et al., 2020b) or generate some data of old tasks (Shin et al., 2017; He & Jaeger, 2018) and use them in learning a new task; (3) parameter-isolation methods (Serrà et al., 2018; Wortsman et al., 2020) allocate neurons and parameters or sub-networks for different tasks/domains and mask them in task learning.

Original: For continual DAP-training, this means that end-tasks cannot use the general knowledge in the LM, which results in poor end-task performances.
Revised: For continual DAP-training, this means that end-tasks cannot use the general knowledge in the LM, which results in poor end-task performances.

Original: In NLP, CL has been used for slot filling (Shen et al., 2019), language learning (Li et al., 2019), sentiment analysis (Ke et al., 2021a), topic modeling (Gupta et al., 2020), question answering (Greco et al., 2019) and text classification (Sun et al., 2020; Huang et al., 2021; Chuang et al., 2020).
Revised: In NLP, CL has been used for slot filling (Shen et al., 2019), language learning (Li et al., 2019), sentiment analysis (Ke et al., 2021a), topic modeling (Gupta et al., 2020), question answering (Greco et al., 2019) and text classification (Sun et al., 2020; Huang et al., 2021; Chuang et al., 2020).

Original: But none is for DAP-training.
Revised: But none is for DAP-training.

Original: Some recent CL papers concern LMs.
Revised: Some recent CL papers concern LMs.

Original: The system in (Madotto et al., 2020) learns separate adapters for different domains and thus has no CF or KT.
Revised: The system in (Madotto et al., 2020) learns separate adapters for different domains and thus has no CF or KT.

Original: DEMIX (Gururangan et al., 2021) initializes the new adapter with the closest old adapter.
Revised: DEMIX (Gururangan et al., 2021) initializes the new adapter with the closest old adapter.

Original: CPT (Ke et al., 2022b) and
Revised: CPT (Ke et al., 2022a) and

Original: ELLE (Qin et al., 2022) are most closely related to DAS.
Revised: ELLE (Qin et al., 2022) are most closely related to DAS.

Original: However, CPT uses the parameter-isolation approach to learn and protect each task, which is weak (see Sec. 4.2).
Revised: However, CPT uses the parameter-isolation approach to learn and protect each task, which is weak (see Sec. 4.2).

Original: It also needs domain-ID in end-task fine-tuning.
Revised: It also needs domain-ID in end-task fine-tuning.

Original: ELLE has to start from pre-training the LM itself rather than from a pre-trained
Revised: ELLE has to start from pre-training the LM itself rather than from a pre-trained

Original: LM like DAS.
Revised: LM like DAS.

Original: It also uses a large memory (1G per domain) to store the replay data (including the pre-training data) and expands the network for each domain.
Revised: It also uses a large memory (1G per domain) to store the replay data (including the pre-training data) and expands the network for each domain.

Original: Neither is required in DAS.
Revised: Neither is required in DAS.

Original: Jin et al.
Revised: Jin et al.

Original: (2021) evaluated several existing CL techniques in a similar setting as DAS and performed analyses on dealing with CF.
Revised: (2021) evaluated several existing CL techniques in a similar setting as DAS and performed analyses on dealing with CF.

Original: However, no new technique was proposed in the paper.
Revised: However, no new technique was proposed in the paper.

Original: Neural network pruning.
Revised: Neural network pruning.

Original: Many parameters in a network are redundant and can be pruned (Li et al., 2021; Lai et al., 2021; Michel et al., 2019; Voita et al., 2019).
Revised: Many parameters in a network are redundant and can be pruned (Li et al., 2021; Lai et al., 2021; Michel et al., 2019; Voita et al., 2019).

Original: Existing methods include discarding parameters with small absolute values (Han et al., 2015; Guo et al., 2016), accumulated gradient (Michel et al., 2019), and lottery ticket hypothesis (Brix et al., 2020).
Revised: Existing methods include discarding parameters with small absolute values (Han et al., 2015; Guo et al., 2016), accumulated gradient (Michel et al., 2019), and lottery ticket hypothesis (Brix et al., 2020).

Original: However, these methods are not directly applicable as we need to preserve not only individual domain knowledge but also the general knowledge in the LM.
Revised: However, these methods are not directly applicable as we need to preserve not only individual domain knowledge but also the general knowledge in the LM.

Original: For general knowledge, since we do not have any pre-training data, a proxy is proposed based on robustness.
Revised: For general knowledge, since we do not have any pre-training data, a proxy is proposed based on robustness.

Original: For domain knowledge, we adopt a pruning method but use the importance as soft-masks as we want to accumulate knowledge rather than to compress the LM.
Revised: For domain knowledge, we adopt a pruning method but use the importance as soft-masks as we want to accumulate knowledge rather than to compress the LM.

Original: Contrastive Learning.
Revised: Contrastive Learning.

Original: Contrastive learning (Chen et al., 2020;
Revised: Contrastive learning (Chen et al., 2020;

Original: He et al., 2020) learns good representations by maximizing the similarity of positive pairs and minimizes that of negative pairs,
Revised: He et al., 2020) learns good representations by maximizing the similarity of positive pairs and minimizes that of negative pairs,

Original: [Equation]
Revised: [Equation]

Original: where N is the batch size, τ is a temperature parameter, sim ( · ) is a similarity metric, and q n and q + n are representations for positive pairs x n and x + n .
Revised: where N is the batch size, τ is a temperature parameter, sim ( · ) is a similarity metric, and q n and q + n are representations for positive pairs x n and x + n .

Original: DAS contrasts the learned knowledge from previous domains and the pre-trained LM (general knowledge) with the full knowledge (including both the previous domains and current domain knowledge) to achieve a complementary effect.
Revised: DAS contrasts the learned knowledge from previous domains and the pre-trained LM (general knowledge) with the full knowledge (including both the previous domains and current domain knowledge) to achieve a complementary effect.

Original:  P ROPOSED DAS T ECHNIQUE
Revised:  P ROPOSED DAS T ECHNIQUE

Original: Continual DAP-training in DAS is based on two main ideas: (1) preserving the important general language knowledge in the LM and the knowledge learned from previous domains to overcome CF by soft-masking units based on their importance, which also facilitates cross-task knowledge transfer (KT), and (2) encouraging the model to learn complementary representations of the current domain and previous domains to achieve knowledge integration.
Revised: Continual DAP-training in DAS is based on two main ideas: (1) preserving the important general language knowledge in the LM and the knowledge learned from previous domains to overcome CF by soft-masking units based on their importance, which also facilitates cross-task knowledge transfer (KT), and (2) encouraging the model to learn complementary representations of the current domain and previous domains to achieve knowledge integration.

Original: Figure 1 gives an overview of DAS.
Revised: Figure 1 gives an overview of DAS.

Original: [Figure]
Revised: [Figure]

Original: The whole learning consists of two main functions: (i) initialization and (ii) continual learning .
Revised: The whole learning consists of two main functions: (i) initialization and (ii) continual learning .

Original: (i) computes the importance of units to the general language knowledge in the LM.
Revised: (i) computes the importance of units to the general language knowledge in the LM.

Original: It is done before the continual learning starts.
Revised: It is done before the continual learning starts.

Original: (ii) is for continual learning, which consists of two steps: (a) domain training and (b) importance computation .
Revised: (ii) is for continual learning, which consists of two steps: (a) domain training and (b) importance computation .

Original: (a) takes the importance scores accumulated so far (including those to the general knowledge in the original LM and to the knowledge learned from previous domains) and the input data of the current domain to learn the domain and to achieve (1) and (2) above, while (b) computes the importance scores for the current domain for future use.
Revised: (a) takes the importance scores accumulated so far (including those to the general knowledge in the original LM and to the knowledge learned from previous domains) and the input data of the current domain to learn the domain and to achieve (1) and (2) above, while (b) computes the importance scores for the current domain for future use.

Original: The following sub-sections present each function and step in detail.
Revised: The following sub-sections present each function and step in detail.

Original: 3. I
Revised: 3. I

Original: NITIALIZATION :
Revised: NITIALIZATION :

Original: C OMPUTING
Revised: C OMPUTING

Original: I MPORTANCE OF U NITS TO THE G ENERAL K NOWLEDGE
Revised: I MPORTANCE OF U NITS TO THE G ENERAL K NOWLEDGE

Original: This initialization function computes the importance of units (attention heads and neurons) in the
Revised: This initialization function computes the importance of units (attention heads and neurons) in the

Original: Transformer for the general knowledge in the original LM.
Revised: Transformer for the general knowledge in the original LM.

Original: The key components of a Transformer are multi-head attention layer , intermediate layer and output layer .
Revised: The key components of a Transformer are multi-head attention layer , intermediate layer and output layer .

Original: Below, we use “layer” or l to indicate any of these three layers because our method treats the three layers similarly.
Revised: Below, we use “layer” or l to indicate any of these three layers because our method treats the three layers similarly.

Original: Importance of units in a layer.
Revised: Importance of units in a layer.

Original: It has been found that not all units in a layer are important (Michel et al., 2019).
Revised: It has been found that not all units in a layer are important (Michel et al., 2019).

Original: We introduce a virtual parameter , g l , for computing the importance of the units in a layer l .
Revised: We introduce a virtual parameter , g l , for computing the importance of the units in a layer l .

Original: We call these virtual parameters as each g ( k ) is initialized to 1.
Revised: We call these virtual parameters as each g ( k ) is initialized to 1.

Original: We only need the gradient on each parameter to compute the importance of its corresponding unit, no update to any parameter.
Revised: We only need the gradient on each parameter to compute the importance of its corresponding unit, no update to any parameter.

Original: [Equation]
Revised: [Equation]

Original: where o l refers to the output of layer l (which can be any of the three layers mentioned above).
Revised: where o l refers to the output of layer l (which can be any of the three layers mentioned above).

Original: The ⊗ refers to element-wise multiplication, i.e., each variable g l,i in g l corresponding to a unit (a neuron or attention head) in the layer.
Revised: The ⊗ refers to element-wise multiplication, i.e., each variable g l,i in g l corresponding to a unit (a neuron or attention head) in the layer.

Original: We adapt the gradient-based importance detection method in (Michel et al., 2019) for our purpose.
Revised: We adapt the gradient-based importance detection method in (Michel et al., 2019) for our purpose.

Original: Given a dataset D = { ( x n , y n ) } Nn =1 of N samples ( y n is the class label of x n as (Michel et al., 2019) worked on supervised learning), the importance of neurons or heads in the layer is estimated with a gradient-based proxy score
Revised: Given a dataset D = { ( x n , y n ) } Nn =1 of N samples ( y n is the class label of x n as (Michel et al., 2019) worked on supervised learning), the importance of neurons or heads in the layer is estimated with a gradient-based proxy score

Original: [Equation]
Revised: [Equation]

Original: where L impt is a task-specific loss function.
Revised: where L impt is a task-specific loss function.

Original: Note the virtual parameter g l is initialized as all 1’s, and is not changed.
Revised: Note the virtual parameter g l is initialized as all 1’s, and is not changed.

Original: This is because we need only its average gradient ∇ g l (the term within || in Eq. 3) over all the data to compute the importance and will not use the gradient to update the virtual parameter.
Revised: This is because we need only its average gradient ∇ g l (the term within || in Eq. 3) over all the data to compute the importance and will not use the gradient to update the virtual parameter.

Original: In training (Sec. 3.2 and Fig 1 (B)), the virtual parameter can be discarded.
Revised: In training (Sec. 3.2 and Fig 1 (B)), the virtual parameter can be discarded.

Original: The resulting I
Revised: The resulting I

Original: Recall that the initialization function is to learn the importance of units to the general knowledge in the LM (denoted as I (0) l ).
Revised: Recall that the initialization function is to learn the importance of units to the general knowledge in the LM (denoted as I (0) l ).

Original: Although Eq.
Revised: Although Eq.

Original: offers a possible way, it is not directly applicable.
Revised: offers a possible way, it is not directly applicable.

Original: If we use the domain data at hand and employ the MLM loss as L impt , ∇ g l only gives the importance for the domain-specific knowledge.
Revised: If we use the domain data at hand and employ the MLM loss as L impt , ∇ g l only gives the importance for the domain-specific knowledge.

Original: However, to compute the importance of units to the general knowledge in the LM (which is our goal), we need the original data used in pre-training the LM to compute the
Revised: However, to compute the importance of units to the general knowledge in the LM (which is our goal), we need the original data used in pre-training the LM to compute the

Original: L impt .
Revised: L impt .

Original: In practice, such data is not accessible to users of the LM.
Revised: In practice, such data is not accessible to users of the LM.

Original: Further, label is needed in Eq.but our domain corpus is unlabeled in DAP-training.
Revised: Further, label is needed in Eq.but our domain corpus is unlabeled in DAP-training.

Original: To address these issues, we propose a proxy
Revised: To address these issues, we propose a proxy

Original: KL-divergence loss ( L proxy ) to replace L impt to learn units importance for the general knowledge.
Revised: KL-divergence loss ( L proxy ) to replace L impt to learn units importance for the general knowledge.

Original: Proxy KL-divergence loss.
Revised: Proxy KL-divergence loss.

Original: We propose to use model robustness as the proxy, i.e., we try to detect units that are important for LM’s robustness .
Revised: We propose to use model robustness as the proxy, i.e., we try to detect units that are important for LM’s robustness .

Original: Their gradients, ∇ g l , then indicate the robustness and the importance to the LM model.
Revised: Their gradients, ∇ g l , then indicate the robustness and the importance to the LM model.

Original: Our rationale is as follows:
Revised: Our rationale is as follows:

Original: If an I (0) l,i (the importance of unit i in layer l ) has a high value, then it is important to the LM’s robustness because its change can cause the
Revised: If an I (0) l,i (the importance of unit i in layer l ) has a high value, then it is important to the LM’s robustness because its change can cause the

Original: LM to change a lot.
Revised: LM to change a lot.

Original: It is thus an important unit.
Revised: It is thus an important unit.

Original: In contrast, if I (0) l,i is small, it is a less important unit.
Revised: In contrast, if I (0) l,i is small, it is a less important unit.

Original: To compute the robustness of the LM, we take a subset of the current domain data { x sub n } 5 (no label in DAP-training) and input x sub n twice to the LM to obtain two representations of it and then compute the KL-divergence between them,
Revised: To compute the robustness of the LM, we take a subset of the current domain data { x sub n } 5 (no label in DAP-training) and input x sub n twice to the LM to obtain two representations of it and then compute the KL-divergence between them,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ) where f 1 LM and f 2 LM are the LM with different dropout masks.
Revised: ) where f 1 LM and f 2 LM are the LM with different dropout masks.

Original: We don’t need to add any additional dropouts to implement these two as the Transformer already has dropout masks placed on fullyconnected layers and attention probabilities.
Revised: We don’t need to add any additional dropouts to implement these two as the Transformer already has dropout masks placed on fullyconnected layers and attention probabilities.

Original: Thus, simply feeding the same input to the Transformer twice will get two representations with different dropout masks.
Revised: Thus, simply feeding the same input to the Transformer twice will get two representations with different dropout masks.

Original: Since dropout is similar to adding noise, the difference between the two representations can be regarded as the robustness of the LM.
Revised: Since dropout is similar to adding noise, the difference between the two representations can be regarded as the robustness of the LM.

Original: 3. T RAINING : L EARNING A N EW D OMAIN VIA S OFT - MASKING AND C ONTRASTIVE L OSS
Revised: 3. T RAINING : L EARNING A N EW D OMAIN VIA S OFT - MASKING AND C ONTRASTIVE L OSS

Original: Recall we want to preserve the learned knowledge in the LM during DAP-training using the accumulated importance I ( ≤ t − 1) l when we learn domain t , which includes both the importance for the general knowledge I (0) l (Sec. 3.1) and learned domain-specific knowledge I ( k ) l of each domain k ( k can be any domain in { 1 ...t − 1 } ) that has been learned (Sec. 3.3).
Revised: Recall we want to preserve the learned knowledge in the LM during DAP-training using the accumulated importance I ( ≤ t − 1) l when we learn domain t , which includes both the importance for the general knowledge I (0) l (Sec. 3.1) and learned domain-specific knowledge I ( k ) l of each domain k ( k can be any domain in { 1 ...t − 1 } ) that has been learned (Sec. 3.3).

Original: This is achieved by soft-masking the learning based on accumulated importance as follows.
Revised: This is achieved by soft-masking the learning based on accumulated importance as follows.

Original: Accumulating importance.
Revised: Accumulating importance.

Original: We accumulate the importance after task t − 1 was learned is done via element-wise max (EMax) as follows:
Revised: We accumulate the importance after task t − 1 was learned is done via element-wise max (EMax) as follows:

Original: [Equation]
Revised: [Equation]

Original: where t refers to the current task-ID and I ( ≤ t − 2) l refers to the previously accumulated importance at task t − 2 .
Revised: where t refers to the current task-ID and I ( ≤ t − 2) l refers to the previously accumulated importance at task t − 2 .

Original: We do not need to save I 0 l and all { I ( k ) l } t -1 k
Revised: We do not need to save I 0 l and all { I ( k ) l } t -1 k

Original: =1 for Eq.
Revised: =1 for Eq.

Original: We only save the incrementally accumulated importance after training of each task.
Revised: We only save the incrementally accumulated importance after training of each task.

Original: Soft-masking units.
Revised: Soft-masking units.

Original: Given the accumulated importance I ( ≤ t − 1) l of layer l and the DAP-training loss
Revised: Given the accumulated importance I ( ≤ t − 1) l of layer l and the DAP-training loss

Original: L DAP-train (typically the MLM loss; we also propose an additional loss in Eq. 7), we constrain (or soft-mask)
Revised: L DAP-train (typically the MLM loss; we also propose an additional loss in Eq. 7), we constrain (or soft-mask)

Original: its corresponding gradient ( ∇ l ) flow as follows,
Revised: its corresponding gradient ( ∇ l ) flow as follows,

Original: [Equation]
Revised: [Equation]

Original: As mentioned in Sec.
Revised: As mentioned in Sec.

Original: 3.1, we expand (by copying) the importance I ( ≤ t − 1) l to match the dimensions of ∇ l to apply it to all associated parameters.
Revised: 3.1, we expand (by copying) the importance I ( ≤ t − 1) l to match the dimensions of ∇ l to apply it to all associated parameters.

Original: This is soft-masking as each element in I ( ≤ t − 1) l is a real number in [0 , 1] (not binary {0, 1}), which gives the model the flexibility to adjust any unit.
Revised: This is soft-masking as each element in I ( ≤ t − 1) l is a real number in [0 , 1] (not binary {0, 1}), which gives the model the flexibility to adjust any unit.

Original: [Table]
Revised: [Table]

Original: We note that the above soft-masks are only applied in the backward pass, but not in the forward pass, which encourages knowledge transfer as each domain training can leverage the knowledge learned from all past domains.
Revised: We note that the above soft-masks are only applied in the backward pass, but not in the forward pass, which encourages knowledge transfer as each domain training can leverage the knowledge learned from all past domains.

Original: To further encourage the model to learn a good representation from both the accumulated knowledge ( I ( ≤ t − 1) l ) and the full knowledge (both accumulated and current domain knowledge), we introduce a contrastive learning method to encourage complementary representation.
Revised: To further encourage the model to learn a good representation from both the accumulated knowledge ( I ( ≤ t − 1) l ) and the full knowledge (both accumulated and current domain knowledge), we introduce a contrastive learning method to encourage complementary representation.

Original: Integrating the previously learned knowledge and the current domain knowledge.
Revised: Integrating the previously learned knowledge and the current domain knowledge.

Original: Soft-masking helps prevent forgetting the previously learned knowledge.
Revised: Soft-masking helps prevent forgetting the previously learned knowledge.

Original: We want to further encourage knowledge transfer by integrating the new and learned knowledge.
Revised: We want to further encourage knowledge transfer by integrating the new and learned knowledge.

Original: We propose to contrast the previously learned knowledge and the full knowledge (both previously learned knowledge and the current domain knowledge).
Revised: We propose to contrast the previously learned knowledge and the full knowledge (both previously learned knowledge and the current domain knowledge).

Original: Note that the contrasting cannot do anything to the shared past knowledge as it is protected by soft-masks.
Revised: Note that the contrasting cannot do anything to the shared past knowledge as it is protected by soft-masks.

Original: Thus, it effectively pushes the current domain knowledge away to be complementary to the past knowledge.
Revised: Thus, it effectively pushes the current domain knowledge away to be complementary to the past knowledge.

Original: This is done based on the current domain data as follows.
Revised: This is done based on the current domain data as follows.

Original: Contrasting the learned and full knowledge .
Revised: Contrasting the learned and full knowledge .

Original: We denote the output of LM without any consideration of importance as o full , which refers to the full knowledge.
Revised: We denote the output of LM without any consideration of importance as o full , which refers to the full knowledge.

Original: We further denote the output of LM that is multiplied by the importance (i.e., I ( ≤ t − 1) l ⊗ o l ) as o prev , which refers to the previously learned knowledge.
Revised: We further denote the output of LM that is multiplied by the importance (i.e., I ( ≤ t − 1) l ⊗ o l ) as o prev , which refers to the previously learned knowledge.

Original: We contrast the two by using o full as anchor and o full with different dropouts as positive samples (dentoed as o full+ ).
Revised: We contrast the two by using o full as anchor and o full with different dropouts as positive samples (dentoed as o full+ ).

Original: o prev is used as negative instances.
Revised: o prev is used as negative instances.

Original: Formally, with o full n , o full+ n , and o prev n , our contrastive loss is (sim ( · ) is the cosine similarity),
Revised: Formally, with o full n , o full+ n , and o prev n , our contrastive loss is (sim ( · ) is the cosine similarity),

Original: [Equation]
Revised: [Equation]

Original: Compared to Eq. 1, the second term is added in the denominator, i.e., representations in the previously learned knowledge as additional negative instances.
Revised: Compared to Eq. 1, the second term is added in the denominator, i.e., representations in the previously learned knowledge as additional negative instances.

Original: Figure 1 (B) shows a red arrow pointed from o full to itself, indicating the positive instances are from inputting twice.
Revised: Figure 1 (B) shows a red arrow pointed from o full to itself, indicating the positive instances are from inputting twice.

Original: The dashed red arrow pointing to o prev indicates the negative instances contrasting the full and previously learned knowledge.
Revised: The dashed red arrow pointing to o prev indicates the negative instances contrasting the full and previously learned knowledge.

Original: Final Loss Function .
Revised: Final Loss Function .

Original: The final DAP-training loss combines the Masked Language Model (MLM) loss after applying the proposed soft-masking for the general knowledge (Sec. 3.1) and the proposed contrastive loss ( λ is a hyper-parameter),
Revised: The final DAP-training loss combines the Masked Language Model (MLM) loss after applying the proposed soft-masking for the general knowledge (Sec. 3.1) and the proposed contrastive loss ( λ is a hyper-parameter),

Original: [Equation]
Revised: [Equation]

Original: 3. C OMPUTE IMPORTANCE OF UNITS TO T HE CURRENT DOMAIN
Revised: 3. C OMPUTE IMPORTANCE OF UNITS TO T HE CURRENT DOMAIN

Original: After training the new/current domain t , we learn the units importance by applying Eq.
Revised: After training the new/current domain t , we learn the units importance by applying Eq.

Original: 3 for the domain.
Revised: 3 for the domain.

Original: We do not need any proxy to compute L impt as in Eq. 4 because we can directly use the current domain data.
Revised: We do not need any proxy to compute L impt as in Eq. 4 because we can directly use the current domain data.

Original: Specifically, we randomly sample a subset (a hyper-parameter) of the current domain data { ( x sub n , y sub n ) } , where x sub n is the input and y sub n is the masked token as in MLM selfsupervised loss.
Revised: Specifically, we randomly sample a subset (a hyper-parameter) of the current domain data { ( x sub n , y sub n ) } , where x sub n is the input and y sub n is the masked token as in MLM selfsupervised loss.

Original: We can then easily compute the importance I ( t ) l by plugging L MLM into L impt in Eq.
Revised: We can then easily compute the importance I ( t ) l by plugging L MLM into L impt in Eq.

Original: The resulting I ( t ) l will be used in the next task by accumulating with the previously accumulated importance (Eq. 5) and soft-masking the learning (Eq. 6).
Revised: The resulting I ( t ) l will be used in the next task by accumulating with the previously accumulated importance (Eq. 5) and soft-masking the learning (Eq. 6).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We use RoBERTa
Revised: We use RoBERTa

Original: (Liu et al., 2019) 7 as the LM.
Revised: (Liu et al., 2019) 7 as the LM.

Original: Following the standard evaluation setup (Lange et al., 2019) and, after a domain is trained, its training data is discarded.
Revised: Following the standard evaluation setup (Lange et al., 2019) and, after a domain is trained, its training data is discarded.

Original: After all domains are incrementally learned, the final model is evaluated by fine-tuning the end-tasks in all domains.
Revised: After all domains are incrementally learned, the final model is evaluated by fine-tuning the end-tasks in all domains.

Original: 4. D ATASETS AND B ASELINES
Revised: 4. D ATASETS AND B ASELINES

Original: Datasets: Table 1 shows the statistics of the 6 unlabeled domain corpora for DAP-training and their 6 corresponding end-task classification datasets .
Revised: Datasets: Table 1 shows the statistics of the 6 unlabeled domain corpora for DAP-training and their 6 corresponding end-task classification datasets .

Original: 8 3 of them are about reviews: Yelp Restaurant (Xu et al., 2019),
Revised: 8 3 of them are about reviews: Yelp Restaurant (Xu et al., 2019),

Original: Amazon Phone (Ni et al., 2019), Amazon Camera (Ni et al., 2019); 3 of them are academic papers:
Revised: Amazon Phone (Ni et al., 2019), Amazon Camera (Ni et al., 2019); 3 of them are academic papers:

Original: ACL Papers (Lo et al., 2020), AI Papers (Lo et al., 2020), and PubMed Papers 9 .
Revised: ACL Papers (Lo et al., 2020), AI Papers (Lo et al., 2020), and PubMed Papers 9 .

Original: Their corresponding end-task classification datasets are: 10 Restaurant 11 , Phone (Ding et al., 2008;
Revised: Their corresponding end-task classification datasets are: 10 Restaurant 11 , Phone (Ding et al., 2008;

Original: Hu & Liu, 2004), Camera (Ding et al., 2008; Hu & Liu, 2004), ACL (ACL-ARC in (Jurgens et al., 2018)), AI (SCIERC in (Luan et al., 2018)), and PubMed (CHEMPORT in (Kringelum et al., 2016)).
Revised: Hu & Liu, 2004), Camera (Ding et al., 2008; Hu & Liu, 2004), ACL (ACL-ARC in (Jurgens et al., 2018)), AI (SCIERC in (Luan et al., 2018)), and PubMed (CHEMPORT in (Kringelum et al., 2016)).

Original: Baselines.
Revised: Baselines.

Original: We use 16 baselines, including non-continual learning (Non-CL) and continual learning (CL) baselines.
Revised: We use 16 baselines, including non-continual learning (Non-CL) and continual learning (CL) baselines.

Original: All CL baselines are originally for learning supervised data except DEMIX.
Revised: All CL baselines are originally for learning supervised data except DEMIX.

Original: We adapt them and replace their backbone with RoBERTa.
Revised: We adapt them and replace their backbone with RoBERTa.

Original: Details of each baseline is given in Appendix B.
Revised: Details of each baseline is given in Appendix B.

Original: Non-CL Baselines : Each baseline here builds a separate model for each task.
Revised: Non-CL Baselines : Each baseline here builds a separate model for each task.

Original: (1) Pool.
Revised: (1) Pool.

Original: We pool the data of all domains together and train only one model for all domains.
Revised: We pool the data of all domains together and train only one model for all domains.

Original: (2) RoBERTa
Revised: (2) RoBERTa

Original: (Liu et al., 2019) uses RoBERTa for end-task fine-tuning without DAP-training.
Revised: (Liu et al., 2019) uses RoBERTa for end-task fine-tuning without DAP-training.

Original: (3) DAP-RoBERTa uses the existing
Revised: (3) DAP-RoBERTa uses the existing

Original: DAP-training method (MLM) in (Gururangan et al., 2020) to posst-train each domain separately.
Revised: DAP-training method (MLM) in (Gururangan et al., 2020) to posst-train each domain separately.

Original: DAP-Adapter adds adapter layers in Transformer for each domain for DAP-training (Jang et al., 2021; Madotto et al., 2020; Houlsby et al., 2019).
Revised: DAP-Adapter adds adapter layers in Transformer for each domain for DAP-training (Jang et al., 2021; Madotto et al., 2020; Houlsby et al., 2019).

Original: Only the added adapters are trainable.
Revised: Only the added adapters are trainable.

Original: In end-task fine-tuning, both RoBERTa and the adapters are trainable.
Revised: In end-task fine-tuning, both RoBERTa and the adapters are trainable.

Original: (5) DAP-Prompt is from (Lester et al., 2021).
Revised: (5) DAP-Prompt is from (Lester et al., 2021).

Original: In DAP-training, RoBERTa (the LM) is fixed and only the prompts are trained.
Revised: In DAP-training, RoBERTa (the LM) is fixed and only the prompts are trained.

Original: In end-task fine-tuning, both the LM and the trained prompt are trainable.
Revised: In end-task fine-tuning, both the LM and the trained prompt are trainable.

Original: CL Baselines : We use 2 naive baselines, which keep learning more domains with no mechanism to deal with CF or transfer.
Revised: CL Baselines : We use 2 naive baselines, which keep learning more domains with no mechanism to deal with CF or transfer.

Original: (6) NCL (Naive CL)
Revised: (6) NCL (Naive CL)

Original: continually DAP-trains the RoBERTa; and (7)
Revised: continually DAP-trains the RoBERTa; and (7)

Original: NCL-Adapter continually DAP-trains a set of adapters (Houlsby et al., 2019).
Revised: NCL-Adapter continually DAP-trains a set of adapters (Houlsby et al., 2019).

Original: 8 baselines are CL systems: (8) DEMIX (Gururangan et al., 2021) adds a new adapter for each new domain and initializes it with a previous adapter nearest to the new domain; (9) BCL (Ke et al., 2021c) uses capsule networks.
Revised: 8 baselines are CL systems: (8) DEMIX (Gururangan et al., 2021) adds a new adapter for each new domain and initializes it with a previous adapter nearest to the new domain; (9) BCL (Ke et al., 2021c) uses capsule networks.

Original: (10) CLASSIC (Ke et al., 2021b) uses contrastive learning.
Revised: (10) CLASSIC (Ke et al., 2021b) uses contrastive learning.

Original: KD is knowledge distillation (Hinton et al., 2015).
Revised: KD is knowledge distillation (Hinton et al., 2015).

Original: (12) EWC (Buzzega et al., 2020) is a popular regularization-based method.
Revised: (12) EWC (Buzzega et al., 2020) is a popular regularization-based method.

Original: (13) DER++ (Buzzega et al., 2020) is a replay method based on knowledge distillation.
Revised: (13) DER++ (Buzzega et al., 2020) is a replay method based on knowledge distillation.

Original: 16.4K tokens are saved for each domain in the replay memory, which is the largest memory we can use for the system to run.
Revised: 16.4K tokens are saved for each domain in the replay memory, which is the largest memory we can use for the system to run.

Original: (14) HAT (Serrà et al., 2018) is an effective parameter-isolation method.
Revised: (14) HAT (Serrà et al., 2018) is an effective parameter-isolation method.

Original: HAT is applied to Transformer layers (i.e., self-attention, intermediate and output layers).
Revised: HAT is applied to Transformer layers (i.e., self-attention, intermediate and output layers).

Original: (15) HAT-All is a HAT variant that uses all features from the LM to do end-tasks (instead of only features from its domain sub-network as in HAT).
Revised: (15) HAT-All is a HAT variant that uses all features from the LM to do end-tasks (instead of only features from its domain sub-network as in HAT).

Original: (16) HAT-Adapter (Ke et al., 2021c) uses HAT within adapters.
Revised: (16) HAT-Adapter (Ke et al., 2021c) uses HAT within adapters.

Original: ELLE (Qin et al., 2022) is not included as we adapted it for our purpose by learning from RoBERTa, but it fails to converge.
Revised: ELLE (Qin et al., 2022) is not included as we adapted it for our purpose by learning from RoBERTa, but it fails to converge.

Original: 4. R ESULTS A NALYSIS AND A BLATION S TUDY
Revised: 4. R ESULTS A NALYSIS AND A BLATION S TUDY

Original: Due to space limits, Implementation Details are given in Appendix C. Table 2 reports the end-task fine-tuning results of all 15 systems on the 6 datasets.
Revised: Due to space limits, Implementation Details are given in Appendix C. Table 2 reports the end-task fine-tuning results of all 15 systems on the 6 datasets.

Original: We can see that the proposed DAS outperforms all baselines on average and also achieve the best knowledge transfer (negative forgetting rate).
Revised: We can see that the proposed DAS outperforms all baselines on average and also achieve the best knowledge transfer (negative forgetting rate).

Original: (1) DAS is slightly better than Pool on average.
Revised: (1) DAS is slightly better than Pool on average.

Original: This may be because (a) some domains are quite different (e.g. camera reviews and ACL papers), which results in some negative transfer in Pool.
Revised: This may be because (a) some domains are quite different (e.g. camera reviews and ACL papers), which results in some negative transfer in Pool.

Original: (b)
Revised: (b)

Original: DAS can learn with the general and previous domain knowledge protected by soft-masks.
Revised: DAS can learn with the general and previous domain knowledge protected by soft-masks.

Original: DAS achieves both forgetting prevention and knowledge transfer.
Revised: DAS achieves both forgetting prevention and knowledge transfer.

Original: Those baselines (KD, EWC,
Revised: Those baselines (KD, EWC,

Original: DER++) focusing only on forgetting prevention give poorer performance as they sacrifice accuracy to avoid CF.
Revised: DER++) focusing only on forgetting prevention give poorer performance as they sacrifice accuracy to avoid CF.

Original: Those baselines (BCL, CLASSIC and DEMIX) perform knowledge transfer achieve better results but still poorer than DAS.
Revised: Those baselines (BCL, CLASSIC and DEMIX) perform knowledge transfer achieve better results but still poorer than DAS.

Original: DEMIX has very weak transfer.
Revised: DEMIX has very weak transfer.

Original: BCL, which can avoid CF while
Revised: BCL, which can avoid CF while

Original: [Table]
Revised: [Table]

Original: The results are averages of 5 random seeds (the domain training order is as they appear in the first row).
Revised: The results are averages of 5 random seeds (the domain training order is as they appear in the first row).

Original: Due to space limits, the results for different domain orders and the standard deviations are reported in Appendix D and Appendix E, respectively).
Revised: Due to space limits, the results for different domain orders and the standard deviations are reported in Appendix D and Appendix E, respectively).

Original: Non-CL baselines have no forgetting.
Revised: Non-CL baselines have no forgetting.

Original: also achieving some transfer, is weaker than NCL.
Revised: also achieving some transfer, is weaker than NCL.

Original: In general, CL baselines are all poorer than DAS as they don’t have methods to encourage knowledge transfer or they have to rely on adapters.
Revised: In general, CL baselines are all poorer than DAS as they don’t have methods to encourage knowledge transfer or they have to rely on adapters.

Original: Directly learning the domains within the LM helps DAS achieve better results than adapter and prompt based methods.
Revised: Directly learning the domains within the LM helps DAS achieve better results than adapter and prompt based methods.

Original: DAS is better than adapter-based systems (DAP-Adapter, NCL-Adapter and
Revised: DAS is better than adapter-based systems (DAP-Adapter, NCL-Adapter and

Original: HAT-Adapter) and prompt-based system (DAP-Prompt).
Revised: HAT-Adapter) and prompt-based system (DAP-Prompt).

Original: This is because adapters and prompts do not have sufficient trainable parameters, which are also randomly initialized and can be hard to train.
Revised: This is because adapters and prompts do not have sufficient trainable parameters, which are also randomly initialized and can be hard to train.

Original: Using the full LM to learn all tasks rather than using sub-networks (of HAT-based methods) makes DAS more effective.
Revised: Using the full LM to learn all tasks rather than using sub-networks (of HAT-based methods) makes DAS more effective.

Original: HAT performs poorly, indicating it is unsuitable for DAP-training as discussed in Sec. 1.
Revised: HAT performs poorly, indicating it is unsuitable for DAP-training as discussed in Sec. 1.

Original: Even if we use all features (not only the feature from the corresponding sub-network), we still get poor results (HAT-All) as the features used in DAP-training (in an LM sub-network) are different from features used in end-task fine-tuning (features from the whole LM).
Revised: Even if we use all features (not only the feature from the corresponding sub-network), we still get poor results (HAT-All) as the features used in DAP-training (in an LM sub-network) are different from features used in end-task fine-tuning (features from the whole LM).

Original: Knowledge transfer and forgetting avoidance.
Revised: Knowledge transfer and forgetting avoidance.

Original: To see how the models fare on CF and knowledge transfer, we compare the forgetting rates ( forget R. )
Revised: To see how the models fare on CF and knowledge transfer, we compare the forgetting rates ( forget R. )

Original: (Liu et al., 2020), 1 t − 1 (cid:80) t − 1 k =

Revised: (Liu et al., 2020), 1 t − 1 (cid:80) t − 1 k =

Original: A k,k − A t,k , where
Revised: A k,k − A t,k , where

Original: A k,k is the end-task accuracy right after its domain k is DAP-trained, and A t,k is the accuracy of the end-task of domain k after DAP-training the last domain t .
Revised: A k,k is the end-task accuracy right after its domain k is DAP-trained, and A t,k is the accuracy of the end-task of domain k after DAP-training the last domain t .

Original: We average over all end-tasks except the last as the last domain has no forgetting.
Revised: We average over all end-tasks except the last as the last domain has no forgetting.

Original: The higher the forgetting rate is, the more forgetting it has.
Revised: The higher the forgetting rate is, the more forgetting it has.

Original: Negative rates indicate positive knowledge transfer.
Revised: Negative rates indicate positive knowledge transfer.

Original: Clearly, DAS has the strongest negative forgetting rate, indicating it does well on both forgetting prevention and knowledge transfer.
Revised: Clearly, DAS has the strongest negative forgetting rate, indicating it does well on both forgetting prevention and knowledge transfer.

Original: NCL, NCL-Adapter, DEMIX, EWC, KD and DER++ all suffer from some forgetting.
Revised: NCL, NCL-Adapter, DEMIX, EWC, KD and DER++ all suffer from some forgetting.

Original: HAT has no forgetting but it cannot learn well.
Revised: HAT has no forgetting but it cannot learn well.

Original: HAT and BCL have no forgetting but are weak in transfer.
Revised: HAT and BCL have no forgetting but are weak in transfer.

Original: Effectiveness of the proxy KL-divergence loss.
Revised: Effectiveness of the proxy KL-divergence loss.

Original: We use proxy KL-divergence loss in the initialization function (Sec. 3.1) to compute the importance of units for general knowledge.
Revised: We use proxy KL-divergence loss in the initialization function (Sec. 3.1) to compute the importance of units for general knowledge.

Original: We are interested in how good the proxy is.
Revised: We are interested in how good the proxy is.

Original: We use two kinds of experiments to provide evidences.
Revised: We use two kinds of experiments to provide evidences.

Original: (1) Comparing with a sample set of D 0 .
Revised: (1) Comparing with a sample set of D 0 .

Original: In some cases, the continual DAP-training users may have the data D 0 that was used to pre-train the LM.
Revised: In some cases, the continual DAP-training users may have the data D 0 that was used to pre-train the LM.

Original: Then we can just sample a subset from D 0 to compute the parameter importance to the general knowledge in the LM.
Revised: Then we can just sample a subset from D 0 to compute the parameter importance to the general knowledge in the LM.

Original: However, since we do not have Dthat was used to pre-train RoBERTa, we use the Wiki data (Merity et al., 2017) as the sample set of
Revised: However, since we do not have Dthat was used to pre-train RoBERTa, we use the Wiki data (Merity et al., 2017) as the sample set of

Original: D 0 .
Revised: D 0 .

Original: We choose it as it is a general dataset with a wide topic coverage and was used to pre-train an
Revised: We choose it as it is a general dataset with a wide topic coverage and was used to pre-train an

Original: LM, and it has a similar size as our domain data (around 700M).
Revised: LM, and it has a similar size as our domain data (around 700M).

Original: We conducted two experiments using the data: (a) DAS (Wiki+MLM) , which uses MLM as the loss in the initialization stage to compute the importance of units (to identify the general knowledge) just like any other domains in the continual learning part, and (b) DAS (Wiki+KL) , which uses KL-divergence as in the initialization stage just like the proposed proxy method.
Revised: We conducted two experiments using the data: (a) DAS (Wiki+MLM) , which uses MLM as the loss in the initialization stage to compute the importance of units (to identify the general knowledge) just like any other domains in the continual learning part, and (b) DAS (Wiki+KL) , which uses KL-divergence as in the initialization stage just like the proposed proxy method.

Original: The results are given in Table 3.
Revised: The results are given in Table 3.

Original: We can see that DAS (Wiki + KL) performs similarly to DAS but outperforms DAS (Wiki + MLM).
Revised: We can see that DAS (Wiki + KL) performs similarly to DAS but outperforms DAS (Wiki + MLM).

Original: This indicates that the proposed proxy KL-divergence is more effective.
Revised: This indicates that the proposed proxy KL-divergence is more effective.

Original: MLM actually adapts the
Revised: MLM actually adapts the

Original: LM to the Wikipedia data, which may not be sufficiently representative of the original data used in pre-training the LM.
Revised: LM to the Wikipedia data, which may not be sufficiently representative of the original data used in pre-training the LM.

Original: As a result, it ends up identifying the knowledge that is suitable only for the
Revised: As a result, it ends up identifying the knowledge that is suitable only for the

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Wikipedia data.
Revised: Wikipedia data.

Original: In contrast, the proposed proxy KL-divergence leverages the random dropout mask and measures the robustness, which is less related to a specific domain and thus reflects the (general) knowledge in the original LM better.
Revised: In contrast, the proposed proxy KL-divergence leverages the random dropout mask and measures the robustness, which is less related to a specific domain and thus reflects the (general) knowledge in the original LM better.

Original: (2) Comparing general knowledge computed from different domain corpora .
Revised: (2) Comparing general knowledge computed from different domain corpora .

Original: Here, we also provide some indirect evidences to show the effectiveness of the proxy method for computing the importance of units to the general knowledge in the LM.
Revised: Here, we also provide some indirect evidences to show the effectiveness of the proxy method for computing the importance of units to the general knowledge in the LM.

Original: We conduct a separate non-CL experiment to compare the attention heads’ importance score vectors after applying the proxy using the data from different domains.
Revised: We conduct a separate non-CL experiment to compare the attention heads’ importance score vectors after applying the proxy using the data from different domains.

Original: 12 For each domain
Revised: 12 For each domain

Original: i , we compare its importance vector with the importance vector of every other domain, and then average the cosine similarities to get the value for domain i .
Revised: i , we compare its importance vector with the importance vector of every other domain, and then average the cosine similarities to get the value for domain i .

Original: We get 0.92 for
Revised: We get 0.92 for

Original: Restaurant, the same 0.91 for ACL, AI, and Phone, 0.89 for PubMed and 0.92 for Camera.
Revised: Restaurant, the same 0.91 for ACL, AI, and Phone, 0.89 for PubMed and 0.92 for Camera.

Original: We see that different domains give similar importance values, which indirectly shows that our proxy can approximately identify the common general knowledge.
Revised: We see that different domains give similar importance values, which indirectly shows that our proxy can approximately identify the common general knowledge.

Original: Ablation.
Revised: Ablation.

Original: We want to know if the proposed (1) initialization (Sec. 3.1), (2) soft-masking, and (3) contrastive learning are helpful.
Revised: We want to know if the proposed (1) initialization (Sec. 3.1), (2) soft-masking, and (3) contrastive learning are helpful.

Original: To answer (1), we conduct the ablation DAS (w/o initialization) , where we remove the initialization and directly do the continual learning given no consideration to the general knowledge in the LM.
Revised: To answer (1), we conduct the ablation DAS (w/o initialization) , where we remove the initialization and directly do the continual learning given no consideration to the general knowledge in the LM.

Original: To answer (2), we conduct the ablations (1) DAS (w/o softmask) , where we remove the soft-masks, and only use contrastive learning based on Eq.
Revised: To answer (2), we conduct the ablations (1) DAS (w/o softmask) , where we remove the soft-masks, and only use contrastive learning based on Eq.

Original: 7 (with the second term in the denominator removed); and (2) DAS (random) with randomly generated importance scores to do soft-masking and contrastive learning.
Revised: 7 (with the second term in the denominator removed); and (2) DAS (random) with randomly generated importance scores to do soft-masking and contrastive learning.

Original: To answer (3), we conduct two ablations: (i) DAS (w/o contrast) where we remove the contrastive loss and only soft-mask according to the importance; (ii) DAS (domain-specific) where we contrast domain-specific and learned knowledge (Sec. 3.2).
Revised: To answer (3), we conduct two ablations: (i) DAS (w/o contrast) where we remove the contrastive loss and only soft-mask according to the importance; (ii) DAS (domain-specific) where we contrast domain-specific and learned knowledge (Sec. 3.2).

Original: Table 4 shows that the full DAS is the best on average and for most domains, indicating that every component contributes.
Revised: Table 4 shows that the full DAS is the best on average and for most domains, indicating that every component contributes.

Original: Additional observations are: (1) DAS’s gain is partially from the preserved general knowledge.
Revised: Additional observations are: (1) DAS’s gain is partially from the preserved general knowledge.

Original: We can see DAS (w/o initialization) is poorer on average; (2) Soft-masking helps as DAS (w/o softmask) is poorer than DAS.
Revised: We can see DAS (w/o initialization) is poorer on average; (2) Soft-masking helps as DAS (w/o softmask) is poorer than DAS.

Original: This is reasonable because soft masking can preserve learned domains.
Revised: This is reasonable because soft masking can preserve learned domains.

Original: Besides, our gradient-based mask is informative as DAS (random) is worse than
Revised: Besides, our gradient-based mask is informative as DAS (random) is worse than

Original: DAS; (3) Contrastive learning is effective as DAS (w/o contrast) and DAS (domain-specific) are both poorer, indicating the contrastive learning in DAS can help learn good representations
Revised: DAS; (3) Contrastive learning is effective as DAS (w/o contrast) and DAS (domain-specific) are both poorer, indicating the contrastive learning in DAS can help learn good representations

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper proposed a novel method DAS for the continual DAP-training of an LM.
Revised: This paper proposed a novel method DAS for the continual DAP-training of an LM.

Original: It has three key ideas: (1) Preserving the important previous knowledge by soft-masking units according to their importance to overcome CF and to facilitate knowledge transfer.
Revised: It has three key ideas: (1) Preserving the important previous knowledge by soft-masking units according to their importance to overcome CF and to facilitate knowledge transfer.

Original: (2) Using a novel proxy to compute the importance of units to the general knowledge in the LM.
Revised: (2) Using a novel proxy to compute the importance of units to the general knowledge in the LM.

Original: (3) Learning complementary representations for knowledge integration.
Revised: (3) Learning complementary representations for knowledge integration.

Original: A set of techniques is proposed to achieve them.
Revised: A set of techniques is proposed to achieve them.

Original: Extensive experiments showed the effectiveness of DAS.
Revised: Extensive experiments showed the effectiveness of DAS.

Original: The current approach involves two functions in learning.
Revised: The current approach involves two functions in learning.

Original: We will study how to combine them to further improve the results in the future.
Revised: We will study how to combine them to further improve the results in the future.

Original: A CKNOWLEDGEMENTS
Revised: A CKNOWLEDGEMENTS

Original: The work of Zixuan Ke, Gyuhak Kim and Bing Liu was supported in part by a research contract from
Revised: The work of Zixuan Ke, Gyuhak Kim and Bing Liu was supported in part by a research contract from

Original: KDDI, a research contract from DARPA (HR001120C0023) and three NSF grants (IIS-1910424,
Revised: KDDI, a research contract from DARPA (HR001120C0023) and three NSF grants (IIS-1910424,

Original: IIS-1838770, and CNS-2225427)
Revised: IIS-1838770, and CNS-2225427)

Original: A D ATASETS D ETAILS
Revised: A D ATASETS D ETAILS

Original: Table 1 in the main paper has already showed the number of examples in each dataset.
Revised: Table 1 in the main paper has already showed the number of examples in each dataset.

Original: Here we provide additional details about the 4 types of end-tasks.
Revised: Here we provide additional details about the 4 types of end-tasks.

Original: (1) (Phone, Camera and Restaurant)
Revised: (1) (Phone, Camera and Restaurant)

Original: Aspect Sentiment Classification (ASC) is defined as follows: given an aspect or product feature (e.g., picture quality in a camera review) and a review sentence containing the aspect in a domain or product category (e.g., camera), classify if the sentence expresses a positive, negative, or neutral (no opinion) sentiment or polarity about the aspect (for the Phone and
Revised: Aspect Sentiment Classification (ASC) is defined as follows: given an aspect or product feature (e.g., picture quality in a camera review) and a review sentence containing the aspect in a domain or product category (e.g., camera), classify if the sentence expresses a positive, negative, or neutral (no opinion) sentiment or polarity about the aspect (for the Phone and

Original: Camera datasets, there are only negative and positive polarities in the data).
Revised: Camera datasets, there are only negative and positive polarities in the data).

Original: (2) (ACL) Citation intent classification is defined as follows: given a citing sentence (a sentence contains a citation), classify if the sentence expresses a citation function among “background”, “motivation”, “uses”, “extension” and “comparison or contrast future”.
Revised: (2) (ACL) Citation intent classification is defined as follows: given a citing sentence (a sentence contains a citation), classify if the sentence expresses a citation function among “background”, “motivation”, “uses”, “extension” and “comparison or contrast future”.

Original: (3) (AI) Relation classification is defined as follows: given a within-sentence word sequence span containing a pair of entities, classify if the span expresses a relation among “feature of”, “conjunction”, “evaluate for”, “hyponym of”, “used for”, “part of” and “compare”.
Revised: (3) (AI) Relation classification is defined as follows: given a within-sentence word sequence span containing a pair of entities, classify if the span expresses a relation among “feature of”, “conjunction”, “evaluate for”, “hyponym of”, “used for”, “part of” and “compare”.

Original: (4) (PubMed) Chemical-protein interaction classification is defined as follows: given a span containing a pair of chemical and protein, classify if the span expresses a chemical-protein interaction among “downregulator”, “substrate”, “indirect-upregulator”, “indirect-downregulator”, “agnonist”, “activator”, “product of”, “agonist-activator”, “inhibitor”, “upregulator”, “substrate product of”, “agonist-inhibitor”and “antagonist”.
Revised: (4) (PubMed) Chemical-protein interaction classification is defined as follows: given a span containing a pair of chemical and protein, classify if the span expresses a chemical-protein interaction among “downregulator”, “substrate”, “indirect-upregulator”, “indirect-downregulator”, “agnonist”, “activator”, “product of”, “agonist-activator”, “inhibitor”, “upregulator”, “substrate product of”, “agonist-inhibitor”and “antagonist”.

Original: B B ASELINE D ETAILS
Revised: B B ASELINE D ETAILS

Original: Non-Continual Learning Baselines : Each of these baselines builds a separate model for each task independently.
Revised: Non-Continual Learning Baselines : Each of these baselines builds a separate model for each task independently.

Original: It thus has no knowledge transfer or CF.
Revised: It thus has no knowledge transfer or CF.

Original: (1) Non-DAP-training (RoBERTa)
Revised: (1) Non-DAP-training (RoBERTa)

Original: Liu et al.
Revised: Liu et al.

Original: (2019) uses the original RoBERTa for the end-task fine-tuning without any DAP
Revised: (2019) uses the original RoBERTa for the end-task fine-tuning without any DAP

Original: -training.
Revised: -training.

Original: This is the only one without any DAP-training.
Revised: This is the only one without any DAP-training.

Original: All the following baselines use the masked language model loss (MLM) for DAP-training.
Revised: All the following baselines use the masked language model loss (MLM) for DAP-training.

Original: (2) DAP-training using masked language model loss (DAP-RoBERTa) is the existing DAP-training method in Gururangan et al.
Revised: (2) DAP-training using masked language model loss (DAP-RoBERTa) is the existing DAP-training method in Gururangan et al.

Original: To our knowledge, the existing DAP-training systems are all based on the MLM loss.
Revised: To our knowledge, the existing DAP-training systems are all based on the MLM loss.

Original: (3) DAP-training using adapter-tuning Madotto et al.
Revised: (3) DAP-training using adapter-tuning Madotto et al.

Original: Houlsby et al.
Revised: Houlsby et al.

Original: (2019) adds small adapter layers between layers of Transformer for DAP-training.
Revised: (2019) adds small adapter layers between layers of Transformer for DAP-training.

Original: We follow the adapter design in Madotto et al.
Revised: We follow the adapter design in Madotto et al.

Original: Houlsby et al.
Revised: Houlsby et al.

Original: An adapter is simply a 2 layers of fully connected network.
Revised: An adapter is simply a 2 layers of fully connected network.

Original: During DAP-training, the Transformer is fixed, only the added adapters are trainable.
Revised: During DAP-training, the Transformer is fixed, only the added adapters are trainable.

Original: The bottleneck size (adapter size) is set to 128.
Revised: The bottleneck size (adapter size) is set to 128.

Original: During end-task fine-tuning, both RoBERTa and the adapters are trainable to ensure fair comparison.
Revised: During end-task fine-tuning, both RoBERTa and the adapters are trainable to ensure fair comparison.

Original: (4) DAP-training using prompt-tuning Lester et al.
Revised: (4) DAP-training using prompt-tuning Lester et al.

Original: (2021) adds a sequence of real vector tokens (called virtual tokens or prompt tokens) to the end of the original sequence.
Revised: (2021) adds a sequence of real vector tokens (called virtual tokens or prompt tokens) to the end of the original sequence.

Original: In DAP-training, RoBERTa (the LM) is fixed and only the prompt tokens are trained.
Revised: In DAP-training, RoBERTa (the LM) is fixed and only the prompt tokens are trained.

Original: In end-task fine-tuning, both LM and the trained prompt are trainable.
Revised: In end-task fine-tuning, both LM and the trained prompt are trainable.

Original: We initialize 100 tokens and set the learning rate of the prompt token to 0.3 in DAP-training, following the setting in Lester et al.
Revised: We initialize 100 tokens and set the learning rate of the prompt token to 0.3 in DAP-training, following the setting in Lester et al.

Original: Continual Learning (CL) Baselines.
Revised: Continual Learning (CL) Baselines.

Original: (5) Naive continual learning (NCL) is a naive extension of Gururangan et al.
Revised: (5) Naive continual learning (NCL) is a naive extension of Gururangan et al.

Original: (2020), which continually/incrementally DAP-trains the LM to learn all domains using MLM loss with no mechanism to deal with CF.
Revised: (2020), which continually/incrementally DAP-trains the LM to learn all domains using MLM loss with no mechanism to deal with CF.

Original: Continual learning with adapter (NCL-Adapter)
Revised: Continual learning with adapter (NCL-Adapter)

Original: Houlsby et al.
Revised: Houlsby et al.

Original: (2019) is similar to the adapter based system.
Revised: (2019) is similar to the adapter based system.

Original: The only difference is that the same set of adapters is shared across all domains, rather than using a new adapter for each new domain.
Revised: The only difference is that the same set of adapters is shared across all domains, rather than using a new adapter for each new domain.

Original: (7) DEMIX (DEMIX) Gururangan et al.
Revised: (7) DEMIX (DEMIX) Gururangan et al.

Original: (2021) is a recent model to adapt pre-trained LM with new domains.
Revised: (2021) is a recent model to adapt pre-trained LM with new domains.

Original: It adds a new adapter once a new domain arrives (network expansion is needed) and initializes the new adapter with the parameters of the previous trained adapter nearest to the new domain data.
Revised: It adds a new adapter once a new domain arrives (network expansion is needed) and initializes the new adapter with the parameters of the previous trained adapter nearest to the new domain data.

Original: They use the perplexity on a held-out sample to choose the most probable adapter.
Revised: They use the perplexity on a held-out sample to choose the most probable adapter.

Original: For fair comparison, we use the same size as { x sub n } as the held-out samples.
Revised: For fair comparison, we use the same size as { x sub n } as the held-out samples.

Original: (8) Hard attention to overcome forgetting (HAT-Adapter)
Revised: (8) Hard attention to overcome forgetting (HAT-Adapter)

Original: Ke et al.
Revised: Ke et al.

Original: (2021c) is derived from
Revised: (2021c) is derived from

Original: HAT Serrà et al.
Revised: HAT Serrà et al.

Original: (2018), the state-of-the-art parameter-isolation based method with almost no forgetting.
Revised: (2018), the state-of-the-art parameter-isolation based method with almost no forgetting.

Original: However, HAT requires task id information in end-task fine-tuning (DAS works in domainagnostic manner and does not need the task id information; see Sec. 1).
Revised: However, HAT requires task id information in end-task fine-tuning (DAS works in domainagnostic manner and does not need the task id information; see Sec. 1).

Original: HAT also needs to train an addition task embedding to mask each layer of the network which makes the DAP-training inefficient.
Revised: HAT also needs to train an addition task embedding to mask each layer of the network which makes the DAP-training inefficient.

Original: (9) Continual learning plugin with capsule(BCL)
Revised: (9) Continual learning plugin with capsule(BCL)

Original: Ke et al.
Revised: Ke et al.

Original: (2021c) is a continual learning model that can avoid forgetting and encourage knowledge transfer.
Revised: (2021c) is a continual learning model that can avoid forgetting and encourage knowledge transfer.

Original: It is similar to NCL-Adapter.
Revised: It is similar to NCL-Adapter.

Original: The difference is that its adapters consist of two modules, one is a capsule network (a new capsule is added once a new domain arrives) to encourage transfer and the other one is similar to HAT to avoid forgetting.
Revised: The difference is that its adapters consist of two modules, one is a capsule network (a new capsule is added once a new domain arrives) to encourage transfer and the other one is similar to HAT to avoid forgetting.

Original: Similar to HAT, task/domain information is needed in end-task fine-tuning.
Revised: Similar to HAT, task/domain information is needed in end-task fine-tuning.

Original: We replace the backbone network from BERT with RoBERTa for fair comparison.
Revised: We replace the backbone network from BERT with RoBERTa for fair comparison.

Original: (10) Continual learning plugin with contrastive transfer (CLASSIC) Ke et al.
Revised: (10) Continual learning plugin with contrastive transfer (CLASSIC) Ke et al.

Original: (2021b) is a continual learning model that can avoid forgetting and encourage knowledge transfer via contrasting loss.
Revised: (2021b) is a continual learning model that can avoid forgetting and encourage knowledge transfer via contrasting loss.

Original: It is similar to HAT.
Revised: It is similar to HAT.

Original: but 3 additional contrastive loss are used for distillation, knowledge transfer and supervised contrast.
Revised: but 3 additional contrastive loss are used for distillation, knowledge transfer and supervised contrast.

Original: Since DAS is working on unsupervised data, we remove the supervised contrastive loss.
Revised: Since DAS is working on unsupervised data, we remove the supervised contrastive loss.

Original: Similar to HAT, task information is needed in end-task fine-tuning.
Revised: Similar to HAT, task information is needed in end-task fine-tuning.

Original: We replace the backbone network from BERT with RoBERTa for fair comparison.
Revised: We replace the backbone network from BERT with RoBERTa for fair comparison.

Original: (11) Knowledge distillation (KD) Hinton et al.
Revised: (11) Knowledge distillation (KD) Hinton et al.

Original: (2015) minimizes the representational deviation between the learned representation and the new representation in DAP-training.
Revised: (2015) minimizes the representational deviation between the learned representation and the new representation in DAP-training.

Original: We compute the KL divergence between the representations (the output before the masked language model prediction
Revised: We compute the KL divergence between the representations (the output before the masked language model prediction

Original: head) of each token of the previous DAP-trained LM and current LM as the distillation loss.
Revised: head) of each token of the previous DAP-trained LM and current LM as the distillation loss.

Original: (12) EWC Buzzega et al.
Revised: (12) EWC Buzzega et al.

Original: (2020) is a popular regularization-based method which adopts elastic weights consolidation to add L 2 regularization to parameter changes.
Revised: (2020) is a popular regularization-based method which adopts elastic weights consolidation to add L 2 regularization to parameter changes.

Original: (13) DER++ Buzzega et al.
Revised: (13) DER++ Buzzega et al.

Original: (2020) is a recent replay method using distillation to regularize the new task training.
Revised: (2020) is a recent replay method using distillation to regularize the new task training.

Original: We store 16.4K tokens for each learned domain as the memory, which is the largest memory we can use for the system to run.
Revised: We store 16.4K tokens for each learned domain as the memory, which is the largest memory we can use for the system to run.

Original: (14) HAT Serrà et al.
Revised: (14) HAT Serrà et al.

Original: (2018) is used in the Transformer layers (including self-attention, intermediate and output layers) rather than the added adapter layers.
Revised: (2018) is used in the Transformer layers (including self-attention, intermediate and output layers) rather than the added adapter layers.

Original: Additional task embedding and task information for end-task fine-tuning are needed.
Revised: Additional task embedding and task information for end-task fine-tuning are needed.

Original: C I MPLEMENTATION D ETAILS
Revised: C I MPLEMENTATION D ETAILS

Original: Architecture.
Revised: Architecture.

Original: We adopt RoBERTa BASE as our backbone LM.
Revised: We adopt RoBERTa BASE as our backbone LM.

Original: A masked language model head is applied for DAP-training.
Revised: A masked language model head is applied for DAP-training.

Original: The end-task fine-tuning of RoBERTa follows the standard practice.
Revised: The end-task fine-tuning of RoBERTa follows the standard practice.

Original: For the three ASC tasks (see Table 1), we adopt the ASC formulation in Xu et al.
Revised: For the three ASC tasks (see Table 1), we adopt the ASC formulation in Xu et al.

Original: (2019), where the aspect (e.g., “ sound ”) and review sentence (e.g., “ The sound is great ”) are concatenated via 
 .
Revised: (2019), where the aspect (e.g., “ sound ”) and review sentence (e.g., “ The sound is great ”) are concatenated via 
 .

Original: Hyperparameters.
Revised: Hyperparameters.

Original: Unless otherwise stated, the same hyper-parameters are used in all experiments.
Revised: Unless otherwise stated, the same hyper-parameters are used in all experiments.

Original: The maximum input length is set to 164 which is sufficient for all datasets.
Revised: The maximum input length is set to 164 which is sufficient for all datasets.

Original: Adam optimizer is used for both DAP-training and end-task fine-tuning.
Revised: Adam optimizer is used for both DAP-training and end-task fine-tuning.

Original: The max sequence length is also set to 164.
Revised: The max sequence length is also set to 164.

Original: DAP-training.
Revised: DAP-training.

Original: The learning rate is set to 1e-4 and batch size to 256.
Revised: The learning rate is set to 1e-4 and batch size to 256.

Original: We train 2.5K steps for each domain, roughly a full pass through the domain data, following Gururangan et al.
Revised: We train 2.5K steps for each domain, roughly a full pass through the domain data, following Gururangan et al.

Original: Xu et al.
Revised: Xu et al.

Original: The subset of data { x sub n } for computing L impt to determine head importance in Secs.
Revised: The subset of data { x sub n } for computing L impt to determine head importance in Secs.

Original: 3.1 and 3.3 is set to 1.64 Million tokens, which is sufficient in our experiments.
Revised: 3.1 and 3.3 is set to 1.64 Million tokens, which is sufficient in our experiments.

Original: λ in Eq.
Revised: λ in Eq.

Original: 8 is set to 1 and τ in
Revised: 8 is set to 1 and τ in

Original: Eq.
Revised: Eq.

Original: 7 is set to 0.05.
Revised: 7 is set to 0.05.

Original: End-task fine-tuning.
Revised: End-task fine-tuning.

Original: The learning rate is set to 1e-5 and batch size to 16.
Revised: The learning rate is set to 1e-5 and batch size to 16.

Original: We train on end-task fine-tuning datasets for 5 epochs for Restaurant; 10 epochs for ACL, AI and PubMed; and 15 epochs for Phone and Camera.
Revised: We train on end-task fine-tuning datasets for 5 epochs for Restaurant; 10 epochs for ACL, AI and PubMed; and 15 epochs for Phone and Camera.

Original: We simply take the results for the last epoch, assuming no validation sets.
Revised: We simply take the results for the last epoch, assuming no validation sets.

Original: We empirically found that the above number of epochs gives us stable and convergence results.
Revised: We empirically found that the above number of epochs gives us stable and convergence results.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: D DAP- TRAINING IN D IFFERENT O RDERS
Revised: D DAP- TRAINING IN D IFFERENT O RDERS

Original: Table 2 in the main paper reported the results for the order Restaurant → ACL → AI → Phone → PubMed → Camera .
Revised: Table 2 in the main paper reported the results for the order Restaurant → ACL → AI → Phone → PubMed → Camera .

Original: We now look at how the order affects the results.
Revised: We now look at how the order affects the results.

Original: Due to the computation intensive nature of DAP-training, we only report the best baseline (NCL) and DAS results with different domain orders.
Revised: Due to the computation intensive nature of DAP-training, we only report the best baseline (NCL) and DAS results with different domain orders.

Original: Table 5 shows NCL and DAS’s results of 5 different orders.
Revised: Table 5 shows NCL and DAS’s results of 5 different orders.

Original: We can see
Revised: We can see

Original: DAS is always better than NCL, demonstrating the effectiveness of DAS.
Revised: DAS is always better than NCL, demonstrating the effectiveness of DAS.

Original: E S TANDARD D EVIATIONS
Revised: E S TANDARD D EVIATIONS

Original: Table 6 reports the standard deviations of the corresponding results in Table 2 (in the main paper) of
Revised: Table 6 reports the standard deviations of the corresponding results in Table 2 (in the main paper) of

Original: DAS and the considered baselines over 5 runs with random seeds.
Revised: DAS and the considered baselines over 5 runs with random seeds.

Original: We can see the results of DAS are stable.
Revised: We can see the results of DAS are stable.

Original: Some baselines (e
Revised: Some baselines (e

Original: [Table]
Revised: [Table]

Original: g., RoBERTa in AI, DAP-RoBERTa in Camera) can have quite large standard deviations.
Revised: g., RoBERTa in AI, DAP-RoBERTa in Camera) can have quite large standard deviations.

Original: Table 7 reports the standard deviations of the corresponding results in Table 4 (in the main paper) of
Revised: Table 7 reports the standard deviations of the corresponding results in Table 4 (in the main paper) of

Original: DAS and the considered baselines over 5 runs with random seeds.
Revised: DAS and the considered baselines over 5 runs with random seeds.

Original: We can see the results of DAS and its variants are stable
Revised: We can see the results of DAS and its variants are stable

Original: [Table] [Figure]
Revised: [Table] [Figure]