Original: Continual Poisoning of Generative Models to Promote Catastrophic Forgetting
Revised: Continual Poisoning of Generative Models to Promote Catastrophic Forgetting

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Generative models have grown into the workhorse of many state-of-the-art machine learning methods.
Revised: Generative models have grown into the workhorse of many state-of-the-art machine learning methods.

Original: However, their vulnerability under poisoning attacks hasbeen largely understudied.
Revised: However, their vulnerability under poisoning attacks has been largely understudied.

Original: In this work, we investigate this issue in the context ofcontinual learning, where generative replayers are utilized to tackle catastrophicforgetting.
Revised: In this work, we investigate this issue in the context of continual learning, where generative replayers are utilized to tackle catastrophic forgetting.

Original: By developing a novel customization of dirty-label input-aware backdoor to the online setting, our attacker manages to stealthily promote forgettingwhile retaining high accuracy at the current task and sustaining strong defenders.
Revised: By developing a novel customization of dirty-label input-aware backdoor to the online setting, our attacker manages to stealthily promote forgetting while retaining high accuracy at the current task and sustaining strong defenders.

Original: Our approach taps into an intriguing property of generative models, namely thatthey cannot well capture input-dependent triggers.
Revised: Our approach taps into an intriguing property of generative models, namely that they cannot well capture input-dependent triggers.

Original: Experiments on four standarddatasets corroborate the poisoner’s effectiveness.
Revised: Experiments on four standard datasets corroborate the poisoner’s effectiveness.

Original:  Introduction
Revised:  Introduction

Original: The vulnerability of machine learning systems must be scrutinized before they can be deployed tosecurity-critical applications.
Revised: The vulnerability of machine learning systems must be scrutinized before they can be deployed to security-critical applications.

Original: The common evasion attack assumes that clean target instances can bemanipulated at test time, which can be unrealistic in many scenarios.
Revised: The common evasion attack assumes that clean target instances can be manipulated at test time, which can be unrealistic in many scenarios.

Original: In contrast, poisoning attacksonly make malicious and imperceptible modifications to the training set, so that the prediction on testexamples can be mistaken.
Revised: In contrast, poisoning attacks only make malicious and imperceptible modifications to the training set, so that the prediction on test examples can be mistaken.

Original: The threat models may insert poison examples [1], flip the training labels[2, 3], or modify the training example inputs [4, 5].
Revised: The threat models may insert poison examples [1], flip the training labels [2, 3], or modify the training example inputs [4, 5].

Original: Although poisoning attacks have been extensively studied under discriminative learning, their potentialrisk in generative learning has been largely understudied.
Revised: Although poisoning attacks have been extensively studied under discriminative learning, their potential risk in generative learning has been largely understudied.

Original: [6] poisons the training examples so thatthe learned generator covertly changes some important part of the output image, e.g., turning a redlight into green.
Revised: [6] poisons the training examples so that the learned generator covertly changes some important part of the output image, e.g., turning a red light into green.

Original: [7] enables the adversary to control the output image by planting a trigger in theinput image or noise.
Revised: [7] enables the adversary to control the output image by planting a trigger in the input image or noise.

Original: Both are backdoor attacks requiring write access to test data, and work in batch.
Revised: Both are backdoor attacks requiring write access to test data, and work in batch.

Original: The increasing penetration of generative models in machine learning urges investigation of poisoningattacks on it in a broader range of learning paradigms.
Revised: The increasing penetration of generative models in machine learning urges investigation of poisoning attacks on it in a broader range of learning paradigms.

Original: In this work, we focus on continual learning, aprominent setting where tasks arrive in streams and each of them corresponds to a discriminativelearning problem such as classification [8].
Revised: In this work, we focus on continual learning, a prominent setting where tasks arrive in streams and each of them corresponds to a discriminative learning problem such as classification [8].

Original: Since the tasks are streamed and cannot be stored, therunning classifier often suffers from catastrophic forgetting , where the performance on older tasksgradually deteriorates [9].
Revised: Since the tasks are streamed and cannot be stored, the running classifier often suffers from catastrophic forgetting , where the performance on older tasks gradually deteriorates [9].

Original: Deep generative replay (DGR) is a natural tool to bring back the memory of the previous tasks bylearning a generative model to fit the data of these tasks [10, 11].
Revised: Deep generative replay (DGR) is a natural tool to bring back the memory of the previous tasks by learning a generative model to fit the data of these tasks [10, 11].

Original: Despite their effectiveness, newvulnerabilities are also opened up where misleading examples can be injected to the training data
Revised: Despite their effectiveness, new vulnerabilities are also opened up where misleading examples can be injected to the training data

Original: D t for the current task t , so that catastrophic forgetting can be promoted when such poisoned D t isused for training both the replayer G t and the classifier.
Revised: D t for the current task t , so that catastrophic forgetting can be promoted when such poisoned D t is used for training both the replayer G t and the classifier.

Original: In this work, we seek practical and stealthypoisoning attacks on DGR that achieve three objectives:
Revised: In this work, we seek practical and stealthy poisoning attacks on DGR that achieve three objectives:

Original:  After moving past task t , the classifier will soon forget what was learned from it (i
Revised: O1 After moving past task t , the classifier will soon forget what was learned from it (i

Original: e., perform poorly on clean test examples drawn from it) despite using a replayer for all the tasks seen so far
Revised: e., perform poorly on clean test examples drawn from it) despite using a replayer for all the tasks seen so far.

Original: O2 During task t , the classifier trained from the poisoned data does not suffer degradation of testaccuracy on task t itself.
Revised: O2 During task t , the classifier trained from the poisoned data does not suffer degradation of test accuracy on task t itself.

Original: This is important because poor performance on the current task canraise significant and immediate suspicion.
Revised: This is important because poor performance on the current task can raise significant and immediate suspicion.

Original: In contrast, by promoting forgetting, the harm willmanifest itself only after the victim has moved on to the next task, by which time it will havebecome too late because the access to the samples of task t is already lost.
Revised: In contrast, by promoting forgetting, the harm will manifest itself only after the victim has moved on to the next task, by which time it will have become too late because the access to the samples of task t is already lost.

Original: O

Revised: O

Original: The poison should sustain solid defense deployed by both the classifier and the replayer.
Revised: The poison should sustain solid defense deployed by both the classifier and the replayer.

Original: The main difficulty lies in two folds.
Revised: The main difficulty lies in two folds.

Original: Firstly, although both O1 and O2 are straightforward to fulfillindividually, they are at odds with each other and are hard to fulfill simultaneously.
Revised: Firstly, although both O1 and O2 are straightforward to fulfill individually, they are at odds with each other and are hard to fulfill simultaneously.

Original: Secondly, thetransiency of data stream compels the adversary to make irrevocable attacks at task t before futuretasks arrive and before catastrophic forgetting can start to occur.
Revised: Secondly, the transiency of data stream compels the adversary to make irrevocable attacks at task t before future tasks arrive and before catastrophic forgetting can start to occur.

Original: Due to this difficulty, poisoningattacks have been much less studied in an online setting.
Revised: Due to this difficulty, poisoning attacks have been much less studied in an online setting.

Original: [12] addressed the online decision problemof selecting k examples for evasion attack.
Revised: [12] addressed the online decision problem of selecting k examples for evasion attack.

Original: [13] assumed the instances are drawn i.i.d.
Revised: [13] assumed the instances are drawn i.i.d.

Original: from atime-invariant distribution, which is not the case in continual learning because tasks may even havedisjoint classes.
Revised: from a time-invariant distribution, which is not the case in continual learning because tasks may even have disjoint classes.

Original: Other works require multiple passes of the data stream [14–16], or clairvoyantknowledge of future data [17, 18].
Revised: Other works require multiple passes of the data stream [14–16], or clairvoyant knowledge of future data [17, 18].

Original: Our contribution , therefore, is to overcome these challenges and to reveal the vulnerability ofgenerative models in the sense that their training data can be poisoned stealthily such that a task canbe learned well at present but forgotten soon in the future.
Revised: Our contribution , therefore, is to overcome these challenges and to reveal the vulnerability of generative models in the sense that their training data can be poisoned stealthily such that a task can be learned well at present but forgotten soon in the future.

Original: Noting that simple label-flipping poisoningcan be easily detected, we resort to dirty-label backdoor/Trojan attack [19] to attain O2 : the trainedclassifier performs correctly on clean examples, but errs if the example is planted with a trigger.
Revised: Noting that simple label-flipping poisoning can be easily detected, we resort to dirty-label backdoor/Trojan attack [19] to attain O2 : the trained classifier performs correctly on clean examples, but errs if the example is planted with a trigger.

Original: Tofurther achieve O1 and O3 , we capitalize on the input-aware backdoor [20], which allows the triggerto vary depending on the image.
Revised: To further achieve O1 and O3 , we capitalize on the input-aware backdoor [20], which allows the trigger to vary depending on the image.

Original: As a result, it can not only withstand stronger defense (Appendix
Revised: As a result, it can not only withstand stronger defense (Appendix

Original: C), but also enjoys higher variation and stealthiness, hence much harder for a generative model tocapture.
Revised: C), but also enjoys higher variation and stealthiness, hence much harder for a generative model to capture.

Original: So the replayed images do not well preserve the trigger (we call it trigger-discarding propertyin Section 3.3) while retaining the incorrect label, leading naturally to forgetting (Section 3).
Revised: So the replayed images do not well preserve the trigger (we call it trigger-discarding property in Section 3.3) while retaining the incorrect label, leading naturally to forgetting (Section 3).

Original: Theproblem is set up in Section 2, and experiments are outlined in Section 4 and detailed in Appendix
Revised: The problem is set up in Section 2, and experiments are outlined in Section 4 and detailed in Appendix

Original: A, B, D, and E, to show the effectiveness and generalizability of the attack.
Revised: A, B, D, and E, to show the effectiveness and generalizability of the attack.

Original: Our innovations aresummarized as follows:
Revised: Our innovations are summarized as follows:

Original: • Proposing the first poisoning attack that promotes catastrophic forgetting in continual learning.•
Revised: • Proposing the first poisoning attack that promotes catastrophic forgetting in continual learning.

Original: Achieving poisoning (no trigger is needed at test time) through a novel way of leveraging backdoorattack that is particularly effective for exacerbating catastrophic forgetting.
Revised: Achieving poisoning (no trigger is needed at test time) through a novel way of leveraging backdoor attack that is particularly effective for exacerbating catastrophic forgetting.

Original: • Identifying a trigger-discarding property of generative models that is intriguing for backdoor attack.
Revised: • Identifying a trigger-discarding property of generative models that is intriguing for backdoor attack.

Original: Related work Generative models have been pervasive in machine learning [21, Part IV], reachingfar beyond the original role of density estimation and serving as a key infrastructure in supervised,unsupervised, and reinforcement learning.
Revised: Related work Generative models have been pervasive in machine learning [21, Part IV], reaching far beyond the original role of density estimation and serving as a key infrastructure in supervised, unsupervised, and reinforcement learning.

Original: We contend that their vulnerability needs to be examinedin the context of their use.
Revised: We contend that their vulnerability needs to be examined in the context of their use.

Original: In the vanilla density estimation, [22] learned robust variational autoencoder (VAEs) that retain high likelihood for the data points under adversarial perturbation.
Revised: In the vanilla density estimation, [22] learned robust variational autoencoder (VAEs) that retain high likelihood for the data points under adversarial perturbation.

Original: Theunderlying threat is evasion attack, and along similar lines, [23] and [24] studied attacks that promotereconstruction error of the decoder in a VAE.
Revised: The underlying threat is evasion attack, and along similar lines, [23] and [24] studied attacks that promote reconstruction error of the decoder in a VAE.

Original: Some recent works address attacks on membershipinference [25–27], model extraction [28], and attribute inference [29].
Revised: Some recent works address attacks on membership inference [25–27], model extraction [28], and attribute inference [29].

Original: However, poisoning attacks ongenerative models are still understudied.
Revised: However, poisoning attacks on generative models are still understudied.

Original: Our aim is to poison a generative model instead of learning a generative model to produce poisonsfor another (discriminative) model [30, 31].
Revised: Our aim is to poison a generative model instead of learning a generative model to produce poisons for another (discriminative) model [30, 31].

Original: We also leave it as future work to defend the proposedattack, noting that (certifiable) defense and detection have been well studied for poisoning attack onbatch discriminative models [3, 32–34].
Revised: We also leave it as future work to defend the proposed attack, noting that (certifiable) defense and detection have been well studied for poisoning attack on batch discriminative models [3, 32–34].

Original:  Attacking Generative Models in Continual Learning
Revised:  Attacking Generative Models in Continual Learning

Original: We consider the continual learning setting, where tasks arrive sequentially.
Revised: We consider the continual learning setting, where tasks arrive sequentially.

Original: The goal is to keepupdating a classifier that predicts accurately not only on the current task, but also on the previoustasks.
Revised: The goal is to keep updating a classifier that predicts accurately not only on the current task, but also on the previous tasks.

Original: Each task t is indeed a joint distribution P t ( X, Y ) , where X ∈ X is the input from a featurespace X , and
Revised: Each task t is indeed a joint distribution P t ( X, Y ) , where X ∈ X is the input from a feature space X , and

Original: Y ∈ Y t is the label whose domain Y t may change with the task.
Revised: Y ∈ Y t is the label whose domain Y t may change with the task.

Original: For example, Y 1consists of digits 0 and 1 , and Y 2 encompasses 2 and 3 .
Revised: For example, Yconsists of digits 0 and 1 , and Y 2 encompasses 2 and 3 .

Original: Even in the case where the domains remainconstant, the distribution P t can shift.
Revised: Even in the case where the domains remain constant, the distribution P t can shift.

Original: The goal of continual learning is to find a classifier C t , suchthat the overall risk across all tasks seen so far is minimized:
Revised: The goal of continual learning is to find a classifier C t , such that the overall risk across all tasks seen so far is minimized:

Original: [Equation]
Revised: [Equation]

Original: Here ℓ is a loss function, and we will focus on multi-class classification with cross-entropy loss.
Revised: Here ℓ is a loss function, and we will focus on multi-class classification with cross-entropy loss.

Original: A major challenge in continual learning is that at task t , only the samples from the current P t areavailable, while those from the previous tasks are no longer accessible.
Revised: A major challenge in continual learning is that at task t , only the samples from the current P t are available, while those from the previous tasks are no longer accessible.

Original: Although most algorithmswould only adapt to the new task instead of completely retraining from scratch, the performance onprevious tasks may still deteriorate significantly, a phenomenon known as catastrophic forgetting .
Revised: Although most algorithms would only adapt to the new task instead of completely retraining from scratch, the performance on previous tasks may still deteriorate significantly, a phenomenon known as catastrophic forgetting .

Original: To alleviate this issue, DGR-based approaches resort to learning a DGR model G t that approximatelyreplicates P t .
Revised: To alleviate this issue, DGR-based approaches resort to learning a DGR model G t that approximately replicates P t .

Original: Due to the constraints in computation and storage, it has to be a lossy approximationbecause otherwise one might as well store all the past examples.
Revised: Due to the constraints in computation and storage, it has to be a lossy approximation because otherwise one might as well store all the past examples.

Original: Then at each task t , samples of( X, Y ) pairs are drawn from not only the current P t , but also from the replayers for the previous tasks
Revised: Then at each task t , samples of ( X, Y ) pairs are drawn from not only the current P t , but also from the replayers for the previous tasks

Original: G 1: t − 1 :=
Revised: G 1: t − 1 :=

Original: { G i } t − 1 i =1 .
Revised: { G i } t − 1 i =1 .

Original: Their union is subsequently used to update the classifier into C t .
Revised: Their union is subsequently used to update the classifier into C t .

Original: The wholeprocess of vanilla DGR-based continual learning is illustrated in Algorithm 1.
Revised: The whole process of vanilla DGR-based continual learning is illustrated in Algorithm 1.

Original: The DRG models can be simplified into a single replayer G that is updated over time, as opposedto one replayer per task.
Revised: The DRG models can be simplified into a single replayer G that is updated over time, as opposed to one replayer per task.

Original: However, our contribution is the poisoner , not the replayer.
Revised: However, our contribution is the poisoner , not the replayer.

Original: Employingmultiple replayers only makes attacks even more challenging, because instead of just poisoningone running replayer, we are now missioned to poison many of them, each of which can only bepoisoned once at its current task.
Revised: Employing multiple replayers only makes attacks even more challenging, because instead of just poisoning one running replayer, we are now missioned to poison many of them, each of which can only be poisoned once at its current task.

Original: If we simply keep a single running replayer, then we enjoy manyopportunities of poisoning it at any time and our objectives will become much easier to achieve.
Revised: If we simply keep a single running replayer, then we enjoy many opportunities of poisoning it at any time and our objectives will become much easier to achieve.

Original: Toconclude, multiple replayers set up a more stringent benchmark for testing our poisoner.
Revised: To conclude, multiple replayers set up a more stringent benchmark for testing our poisoner.

Original: A naturalchoice of the replayer is a conditional generative model such as conditional GAN (cGAN), whichfirst samples the label Y from a discrete distribution, and then generates the feature X via the cGAN.
Revised: A natural choice of the replayer is a conditional generative model such as conditional GAN (cGAN), which first samples the label Y from a discrete distribution, and then generates the feature X via the cGAN.

Original: 2.1 Attackers and Learners
Revised: 2.1 Attackers and Learners

Original: Input: Tasks 1 , 2 , . .
Revised: Input: Tasks 1 , 2 , . .

Original: represented as P 1 , P 2 , . . .
Revised: represented as P 1 , P 2 , . . .

Original: [Table] C2 for t = 1 , 2 , . . .
Revised: 1 Initialize classifier C2 for t = 1 , 2 , . . .

Original: do for i ∈ [ t − 1] := { 1 , 2 , . . .
Revised: do for i ∈ [ t − 1] := { 1 , 2 , . . .

Original: , t − 1 } do
Revised: , t − 1 } do

Original: S i ← SampleFromDGR ( G i )
Revised: S i ← SampleFromDGR ( G i )

Original: ( X, Y pairs)
Revised: ( X, Y pairs)

Original: Sample D t from task P t ← to be poisoned
Revised: Sample D t from task P t ← to be poisoned

Original: C t ← TrainClassifier ( C t − 1 , D t ∪ S 1: t − 1 )
Revised: C t ← TrainClassifier ( C t − 1 , D t ∪ S 1: t − 1 )

Original: G t ← TrainReplayer ( D t ) , e.g. conditional GAN
Revised: G t ← TrainReplayer ( D t ) , e.g. conditional GAN

Original: Two parties participate in the process, andwe first set forth our assumptions on them.
Revised: Two parties participate in the process, and we first set forth our assumptions on them.

Original: The victim learner/user consists of a classifier , a replayer , and a defender .
Revised: The victim learner/user consists of a classifier , a replayer , and a defender .

Original: We assume none of them has access to the original clean data , and can only access thepoisoned data.
Revised: We assume none of them has access to the original clean data , and can only access the poisoned data.

Original: Further, the replayer mustperform well, i.e., the generated samplesmatch the distribution of data presented toit for training.
Revised: Further, the replayer must perform well, i.e., the generated samples match the distribution of data presented to it for training.

Original: Otherwise, the replayer would not be adopted by the user in the first place, and canspare any need of attack by, e.g., generating random images with random labels.
Revised: Otherwise, the replayer would not be adopted by the user in the first place, and can spare any need of attack by, e.g., generating random images with random labels.

Original: We also assume thatthe learner cannot store any data beyond its current task, which is standard in continual learning.
Revised: We also assume that the learner cannot store any data beyond its current task, which is standard in continual learning.

Original: A defender is an algorithm that the learner employs to scrutinize and prune the possible poisons inthe training data.
Revised: A defender is an algorithm that the learner employs to scrutinize and prune the possible poisons in the training data.

Original: In DGR, it means examining the data collected from the current task t , as wellas the replayed samples from previous G i in step 4.
Revised: In DGR, it means examining the data collected from the current task t , as well as the replayed samples from previous G i in step 4.

Original: This is known as pre-training defense, whosecounterpart— post-training defense—patches up the learned model [35].
Revised: This is known as pre-training defense, whose counterpart— post-training defense—patches up the learned model [35].

Original: The attacker (our threat model) is only allowed to poison (modify) the samples D t in step 5 of
Revised: The attacker (our threat model) is only allowed to poison (modify) the samples D t in step 5 of

Original: Algorithm 1 .
Revised: Algorithm 1 .

Original: The attacker has no access to the internal mechanism of classifier, replayer, or defender.
Revised: The attacker has no access to the internal mechanism of classifier, replayer, or defender.

Original: It can read the gradient of the classifier’s training objective with respect to the input, but not itsstructure or weights.
Revised: It can read the gradient of the classifier’s training objective with respect to the input, but not its structure or weights.

Original: This is a moderate mid-ground between full access (e.g., training by the attackeritself on the cloud) and no access (independently manipulating the training examples).
Revised: This is a moderate mid-ground between full access (e.g., training by the attacker itself on the cloud) and no access (independently manipulating the training examples).

Original: Such anassumption has been commonly adopted, e.g., by dynamic backdoors [36], Witches’ brew [37], andby implicit differentiation based poisoning [38].
Revised: Such an assumption has been commonly adopted, e.g., by dynamic backdoors [36], Witches’ brew [37], and by implicit differentiation based poisoning [38].

Original: We will pursue a dirty-label backdoor attack, i.e., asmall portion ρ b of D t will flip their label, along with a trigger of size ρ a planted to their input image(more details in Section 3.2).
Revised: We will pursue a dirty-label backdoor attack, i.e., a small portion ρ b of D t will flip their label, along with a trigger of size ρ a planted to their input image (more details in Section 3.2).

Original: Achieving poisoning through backdoor.
Revised: Achieving poisoning through backdoor.

Original: Although our method will leverage backdoor attacks, ouroverall goal is poisoning attack, not backdoor.
Revised: Although our method will leverage backdoor attacks, our overall goal is poisoning attack, not backdoor.

Original: In backdoor attacks, a classifier predicts poorly onlyon backdoored examples with triggers, while remaining well on clean test examples.
Revised: In backdoor attacks, a classifier predicts poorly only on backdoored examples with triggers, while remaining well on clean test examples.

Original: In contrast,poisoning attacks (excluding backdoor) aim to predict poorly on all examples, irrelevant of “trigger”.
Revised: In contrast, poisoning attacks (excluding backdoor) aim to predict poorly on all examples, irrelevant of “trigger”.

Original: In continual learning, an attacker generally does not have the liberty of planting a trigger on testexamples.
Revised: In continual learning, an attacker generally does not have the liberty of planting a trigger on test examples.

Original: So we will address in O1 and O2 a much more challenging setting (from an attacker’sperspective) where such an access is not available.
Revised: So we will address in O1 and O2 a much more challenging setting (from an attacker’s perspective) where such an access is not available.

Original: It is important to note that our approach onlyutilizes backdoor attacks as a means of achieving the goal of poisoning the generator/replayer.
Revised: It is important to note that our approach only utilizes backdoor attacks as a means of achieving the goal of poisoning the generator/replayer.

Original: 2.2 Difficulties in the attack
Revised: 2.2 Difficulties in the attack

Original: Since most of the existing poisoning algorithms are in the batch setting, the extension to continuallearning brings about new and significant challenges.
Revised: Since most of the existing poisoning algorithms are in the batch setting, the extension to continual learning brings about new and significant challenges.

Original: Firstly, the poison cannot compromise the current classifier, but should sufficiently poison the DGRso that the samples drawn from it during the later tasks will be detrimental enough to forget thepreviously learned tasks.
Revised: Firstly, the poison cannot compromise the current classifier, but should sufficiently poison the DGR so that the samples drawn from it during the later tasks will be detrimental enough to forget the previously learned tasks.

Original: This rules out simply flipping the label of some examples in D t , because itis easy to detect [3] and the resulting classifier C t will perform poorly on task t .
Revised: This rules out simply flipping the label of some examples in D t , because it is easy to detect [3] and the resulting classifier C t will perform poorly on task t .

Original: Secondly, since the future tasks have not been witnessed yet at task t and the training of C t +1 has notstarted, it is infeasible to optimize the forget-inducing distortion on D t via back-propagation basedoptimization – the context and objective are not yet available for future forgetting.
Revised: Secondly, since the future tasks have not been witnessed yet at task t and the training of C t +1 has not started, it is infeasible to optimize the forget-inducing distortion on D t via back-propagation based optimization – the context and objective are not yet available for future forgetting.

Original:  The Attack Algorithm
Revised:  The Attack Algorithm

Original: Our poisoning attack proposes evading the defense by leveraging the input-aware backdoor attack[20], so that mislabeled data points carrying a trigger can be injected to D t in a small amount.
Revised: Our poisoning attack proposes evading the defense by leveraging the input-aware backdoor attack [20], so that mislabeled data points carrying a trigger can be injected to D t in a small amount.

Original: Inparticular, our approach achieves O1 to O3 through the following effects:
Revised: In particular, our approach achieves O1 to O3 through the following effects:

Original: Since the triggers depend on (hence vary across) the input data, the defender can hardly detect it.
Revised: Since the triggers depend on (hence vary across) the input data, the defender can hardly detect it.

Original: Since the mislabeled examples for the current task all carry an input-aware trigger, the learned backdoored classifier for task t makes mistakes only for backdoored examples.
Revised: Since the mislabeled examples for the current task all carry an input-aware trigger, the learned backdoored classifier for task t makes mistakes only for backdoored examples.

Original: As such, it predicts accurately on pristine test examples for task t which carry no trigger.
Revised: As such, it predicts accurately on pristine test examples for task t which carry no trigger.

Original: As generative models essentially represent a lossy compression, it is generally unable to capture the triggers that change with the input.
Revised: As generative models essentially represent a lossy compression, it is generally unable to capture the triggers that change with the input.

Original: When replayed later for a future classifier at task t +1 , t +2 , . ..
Revised: When replayed later for a future classifier at task t +1 , t +2 , . .

Original:  the triggers go absent while the incorrect label is retained.
Revised: . , the triggers go absent while the incorrect label is retained.

Original: So the classifier will be trained on mislabeled examples of task t with no backdoor, hence misclassifying clean test examples.
Revised: So the classifier will be trained on mislabeled examples of task t with no backdoor, hence misclassifying clean test examples.

Original: 3.1 Backdoor attacks
Revised: 3.1 Backdoor attacks

Original: Our solution is based on backdoor attacks, which despite the marked resemblance to poisoningattacks, do not misclassify a test example unless a pre-designed trigger is inserted to it.
Revised: Our solution is based on backdoor attacks, which despite the marked resemblance to poisoning attacks, do not misclassify a test example unless a pre-designed trigger is inserted to it.

Original: Such aflexibility of modulation proves essential.
Revised: Such a flexibility of modulation proves essential.

Original: Backdoor attacks such as BadNets [39] plant a pre-selectedor learned trigger into some training images at a pre-selected or varying location.
Revised: Backdoor attacks such as BadNets [39] plant a pre-selected or learned trigger into some training images at a pre-selected or varying location.

Original: A number ofvariations are available such as soft blending and multi-channel.
Revised: A number of variations are available such as soft blending and multi-channel.

Original: Adding the resulting image to thetraining data along with a flipped label (randomly selected for an untargeted attack, and pre-specifiedfor a targeted attack), a classifier can be trained that enjoys two important properties:
Revised: Adding the resulting image to the training data along with a flipped label (randomly selected for an untargeted attack, and pre-specified for a targeted attack), a classifier can be trained that enjoys two important properties:

Original: P1 Once a test image is also backdoored with a trigger, the predicted label will change to the pre-specified (or random) one in a targeted (or untargetted) attack.
Revised: P1 Once a test image is also backdoored with a trigger, the predicted label will change to the pre-specified (or random) one in a targeted (or untargetted) attack.

Original: P

Revised: P

Original: However, the test accuracy 
Revised: However, the test accuracy on

Original: on pristine images (without a trigger) can remain very
Revised: pristine images (without a trigger

Original: high
Revised: can remain very high.

Original: Our inspiration originates from this trigger-based modulation.
Revised: Our inspiration originates from this trigger-based modulation.

Original: Suppose we plant the trigger on D t instep 5 of Algorithm 1, and denote the resulting training set as ˜ D t .
Revised: Suppose we plant the trigger on D t in step 5 of Algorithm 1, and denote the resulting training set as ˜ D t .

Original: Then the resulting C t will performwell on pristine test examples of task t , because they do not carry the trigger.
Revised: Then the resulting C t will perform well on pristine test examples of task t , because they do not carry the trigger.

Original: After moving to taskt + 1 , the replayer will (approximately) reproduce ˜ D t , at which point two cases can be considered:
Revised: After moving to task t + 1 , the replayer will (approximately) reproduce ˜ D t , at which point two cases can be considered:

Original: • If the replayer works purely by rote, then ˜ D t will be exactly replayed and the resulting classifier C t +1 will be backdoored in the same way as C t .
Revised: • If the replayer works purely by rote, then ˜ D t will be exactly replayed and the resulting classifier C t +1 will be backdoored in the same way as C t .

Original: As a result, it will still predict accurately on clean samples from task t , i.e., the attack fails in promoting forgetting.
Revised: As a result, it will still predict accurately on clean samples from task t , i.e., the attack fails in promoting forgetting.

Original: • If the replayer is lossy and is unable to capture or reproduce the trigger, then the replayed examples might no longer carry the trigger.
Revised: • If the replayer is lossy and is unable to capture or reproduce the trigger, then the replayed examples might no longer carry the trigger.

Original: However, they still carry the flipped label.
Revised: However, they still carry the flipped label.

Original: As a result, C t +1 will now be trained on mislabeled examples without a trigger, and will therefore perform poorly on taskt .
Revised: As a result, C t +1 will now be trained on mislabeled examples without a trigger, and will therefore perform poorly on task t .

Original: In this case, the attacker successfully promoted forgetting.
Revised: In this case, the attacker successfully promoted forgetting.

Original: The requirement on the replayer in the second case may appear unrealistic, because firstly thereplayer is supposed to faithfully preserve the salient information in the inputs to address catastrophicforgetting.
Revised: The requirement on the replayer in the second case may appear unrealistic, because firstly the replayer is supposed to faithfully preserve the salient information in the inputs to address catastrophic forgetting.

Original: Secondly, a user (who constructs and trains the replayer) obviously has no motivationto collaborate with an attacker.
Revised: Secondly, a user (who constructs and trains the replayer) obviously has no motivation to collaborate with an attacker.

Original: Therefore, the key challenge for the attacker is to design delicatetriggers that are as likely to be overlooked and disregarded as possible under generative modeling,while retaining the good performance on clean test data during task t (property P2 ).
Revised: Therefore, the key challenge for the attacker is to design delicate triggers that are as likely to be overlooked and disregarded as possible under generative modeling, while retaining the good performance on clean test data during task t (property P2 ).

Original: Input: Data generation distribution P ( X, Y ) , which will the invoked with P =
Revised: Input: Data generation distribution P ( X, Y ) , which will the invoked with P =

Original: P t at task t1 ( ˜ D , L div ) ← InputAwareBackdoor–Obj ( P, B
Revised: P t at task t

Original: [ Y ] ) 2 return arg min C,B
Revised: ) 2 return arg min C,B

Original: [ Y ] {L cl + λ div L div } , where L cl =
Revised: [ Y ] {L cl + λ div L div } , where L cl =

Original: (cid:80) ( x,y ) ∈
Revised: (cid:80) ( x,y ) ∈

Original: ˜ D ℓ ( C ( x ) , y ) is the classif.
Revised: ˜ D ℓ ( C ( x ) , y ) is the classif.

Original: risk
Revised: risk

Original: Input: P as in InputAwareBackdoor, and backdoorgenerators B[ Y ] := { B y : y ∈ Y} 1 Initialize ˜ D
Revised: Input InputAwareBackdoor backdoor B

Original: = ∅ which will contain clean and poisoned examples.
Revised: Initialize poisoned examples

Original: Set L div = 0 (diversity loss).
Revised: Set L do

Original: 2 for ( x, y ) sampled from P for task t do Sample d ∼ U(0 , 1) , sample (ˆ x, ˆ y ) from P excluding ( x, y ) , sample y ′ from Y\{ y } , L div += ∥ x − ˆ x∥ /
Revised: Sample d ∼ U ) , sample (ˆ x

Original: ∥ B y ′
Revised: ˆ y )

Original: ( x ) − B y ′
Revised: P y ) , sample y ′

Original: (ˆ x ) ∥ //
Revised: from Y\{ y } ,

Original:  Accrue the diversity lossif b then make a backdoorexample else if / make a cross example / clean example
Revised: / Accrue the diversity loss backdoor

Original: This is indeed challenging as we experimented.
Revised: This is indeed challenging as we experimented.

Original: Static backdoor (BadNet) can be easily replayed by agenerative model.
Revised: Static backdoor (BadNet) can be easily replayed by a generative model.

Original: Trojan attack requires access to the victim model’s structure and weights.
Revised: Trojan attack requires access to the victim model’s structure and weights.

Original: Neithercan it survive the defense of neural cleansing.
Revised: Neither can it survive the defense of neural cleansing.

Original: We also tested static backdoor with changing location,which again, turned out easily detected and fixed by neural cleansing.
Revised: We also tested static backdoor with changing location, which again, turned out easily detected and fixed by neural cleansing.

Original: Witches’ Brew [37] and othergradient matching based methods need to know the target before deploying the attack, while futuretasks are unknown in continual learning.
Revised: Witches’ Brew [37] and other gradient matching based methods need to know the target before deploying the attack, while future tasks are unknown in continual learning.

Original: Eventually, it turns out the input-aware backdoor satisfiesour need, where a trigger is customized for each example through a learnable generative model, henceexhibiting much less regularity for the replayer to capture.
Revised: Eventually, it turns out the input-aware backdoor satisfies our need, where a trigger is customized for each example through a learnable generative model, hence exhibiting much less regularity for the replayer to capture.

Original: 3.2 Input-aware backdoor
Revised: 3.2 Input-aware backdoor

Original: We first recap the input-aware backdoor [IAB, 20] as shown in Algorithm 2 under a given datadistribution P , and then detail how to utilize it for our purpose.
Revised: We first recap the input-aware backdoor [IAB, 20] as shown in Algorithm 2 under a given data distribution P , and then detail how to utilize it for our purpose.

Original: In line 2, the classifier C andclass-wise backdoor generating networks B y are jointly optimized over an objective constructed in
Revised: In line 2, the classifier C and class-wise backdoor generating networks B y are jointly optimized over an objective constructed in

Original: Algorithm 3, where each ( x, y ) sampled from P contributes in one of the following modes:
Revised: Algorithm 3, where each ( x, y ) sampled from P contributes in one of the following modes:

Original: • As a backdoor example with probability ρ b : a wrong label y ′ is randomly picked, and then a trigger that depends on x is generated by B y ′
Revised: • As a backdoor example with probability ρ b : a wrong label y ′ is randomly picked, and then a trigger that depends on x is generated by B y ′

Original: ( x ) and injected to x in line 5 via elementwise product ⊙ .•
Revised: ( x ) and injected to x in line 5 via elementwise product ⊙ .

Original:  As a cross-trigger example with probability ρ c .
Revised: • As a cross-trigger example with probability ρ c .

Original: To ensure that a trigger synthesized for one example is not effective for another, another ˆ x is sampled from P with label ˆ y .
Revised: To ensure that a trigger synthesized for one example is not effective for another, another ˆ x is sampled from P with label ˆ y .

Original: Then x is injected with thetrigger generated from ˆ x for a wrong label y ′ , and the result is paired with the clean label
Revised: Then x is injected with the trigger generated from ˆ x for a wrong label y ′ , and the result is paired with the clean label

Original: y
Revised: y

Original: (line 6).• As a clean example otherwise (line 7).
Revised: (line 6). • As a clean example otherwise (line 7).

Original: In Algorithm 2, B [ Y ] is explicitized in line 1 to stress that both the diversity loss L div and classificationrisk L cl (through ˜ D ) are functions of B [ Y ] , which is then optimized in line 2 along with the classifier
Revised: In Algorithm 2, B [ Y ] is explicitized in line 1 to stress that both the diversity loss L div and classification risk L cl (through ˜ D ) are functions of B [ Y ] , which is then optimized in line 2 along with the classifier

Original: C .
Revised: C .

Original: To see that the attacker fits in our threat model, note C and B are jointly optimized in line 2 of
Revised: To see that the attacker fits in our threat model, note C and B are jointly optimized in line 2 of

Original: Algorithm 2, and the attacker only requires reading the gradient with respect to the input of C .
Revised: Algorithm 2, and the attacker only requires reading the gradient with respect to the input of C .

Original: Asobserved in our experiment, IAB proffers the following property [20]:
Revised: As observed in our experiment, IAB proffers the following property [20]:

Original: P3 The backdoor in IAB can be hardly detected by state-of-the-art methods, e.g., neural cleansing.
Revised: P3 The backdoor in IAB can be hardly detected by state-of-the-art methods, e.g., neural cleansing.

Original: To summarize, we fulfilled O2 by P2 , and O3 by P3 . 
Revised: To summarize, we fulfilled O2 by P2 , and O3 by P3 . To meet O1 , we require

Original: To meet O1 , we require a trigger-discardingproperty as follows, which plays a key role in our method and will be discussed in the next subsection:
Revised: a trigger -discarding property as follows, which plays a key role in our method and will be discussed in the next subsection:

Original: P4 The replayer cannot well capture the trigger generation network of IAB, in the sense that thereplayed examples do not well preserve the triggers.
Revised: P4 The replayer cannot well capture the trigger generation network of IAB, in the sense that the replayed examples do not well preserve the triggers.

Original: 3.3 Trigger-discarding generative models
Revised: 3.3 Trigger-discarding generative models

Original: Property P4 depends on both the replayer and the trigger.
Revised: Property P4 depends on both the replayer and the trigger.

Original: If the trigger is a constant small whitesquare at the image center, most generative models will preserve it.
Revised: If the trigger is a constant small white square at the image center, most generative models will preserve it.

Original: Same is true if the replayer onlyreplicates the training set.
Revised: Same is true if the replayer only replicates the training set.

Original: In general, it is supposed to capture the salient features of the input, andone might presume that triggers are likely to be discarded if they vary a lot across examples.
Revised: In general, it is supposed to capture the salient features of the input, and one might presume that triggers are likely to be discarded if they vary a lot across examples.

Original: It turns out not true.
Revised: It turns out not true.

Original: For example, we placed a square/triangle/round in random colors at random positions ofthe images, and a WGAN easily reproduced them with these (interpolated) color, shape, and position.
Revised: For example, we placed a square/triangle/round in random colors at random positions of the images, and a WGAN easily reproduced them with these (interpolated) color, shape, and position.

Original: Formally, let P x be the data distribution, and suppose given an input x , the trigger is generated bya learnable network f θ ( x ) and is added to x by a pre-specified operation g ( x, f θ ( x )) .
Revised: Formally, let P x be the data distribution, and suppose given an input x , the trigger is generated by a learnable network f θ ( x ) and is added to x by a pre-specified operation g ( x, f θ ( x )) .

Original: It induces adistribution of backdoored examples as a push-forward of P x : Q θx :=
Revised: It induces a distribution of backdoored examples as a push-forward of P x : Q θx :=

Original: ( x (cid:55)→ g ( x, f θ ( x )))
Revised: ( x (cid:55)→ g ( x, f θ ( x )))

Original: # P x .
Revised: # P x .

Original: Leta generative learning algorithm A map a set of examples { x i } i to a distribution.
Revised: Let a generative learning algorithm A map a set of examples { x i } i to a distribution.

Original: Then the trigger isintended to demote some divergence (e.g., KL and Wasserstein) between P x and E x i ∼ Q θx A ( { x i } i ) .
Revised: Then the trigger is intended to demote some divergence (e.g., KL and Wasserstein) between P x and E x i ∼ Q θx A ( { x i } i ) .

Original: Directly computing the gradient in θ is both expensive and infeasible, because A is assumed inaccessible in Section 2.1.
Revised: Directly computing the gradient in θ is both expensive and infeasible, because A is assumed inaccessible in Section 2.1.

Original: Fortunately, our experiments show that P4 is well (but not perfectly ) achievedwhen IAB is applied in conjunction with several SOTA generative models such as conditional Wasserstein GAN [cWGAN, 40] and conditional VAE [41].
Revised: Fortunately, our experiments show that P4 is well (but not perfectly) achieved when IAB is applied in conjunction with several SOTA generative models such as conditional Wasserstein GAN [cWGAN, 40] and conditional VAE [41].

Original: This is no surprise because these models havelimited capacity, and the trigger’s dependency on the input, which is more involved than just random,significantly raises the sample complexity for generative learning.
Revised: This is no surprise because these models have limited capacity, and the trigger’s dependency on the input, which is more involved than just random, significantly raises the sample complexity for generative learning.

Original: A theoretical analysis is left forfuture work, and Appendix B empirically illustrates this intriguing property.
Revised: A theoretical analysis is left for future work, and Appendix B empirically illustrates this intriguing property.

Original: 3.4 Poisoning the replayer via IAB
Revised: 3.4 Poisoning the replayer via IAB

Original: We are now ready to apply IAB to poison the DGR used by continual learning against forgetting.
Revised: We are now ready to apply IAB to poison the DGR used by continual learning against forgetting.

Original: Wewill call our method Continual Input-Aware Poisoning (CIAP).
Revised: We will call our method Continual Input-Aware Poisoning (CIAP).

Original: It does not backdoor test images.
Revised: It does not backdoor test images.

Original: place of

Original: Algorithm 1)

Original: Initialize for i ∈ [ t − 1] do
Revised: Input: P for task t Input:

Original: S
Revised: C

Original:  i← SampleFromReplayer (G i )
Revised: and backdoor generators B ] do i SampleFromReplayer (

Original: ( X, Y pairs) 4 Defender:
Revised: ) 4 Defender

Original: Apply ν -SVM on the replayed data S1: t −5 for number of iteration ( run in mini-batches ) do ˜ D t , L div ← InputAwareBackdoor–Obj ( P t , B
Revised: Apply ν SVM S do ˜ D t , L div ← InputAwareBackdoor–Obj ( P t , B

Original: [ Y ] ) //
Revised: [ Y ] ) //

Original: Both ˜ D t and L div are functions of B [ Y ]
Revised: Both ˜ D t and L div are functions of B [ Y ]

Original: L cl ← sum of ℓ ( C ( x ) , y ) over ( x, y ) ∈
Revised: L cl ← sum of ℓ ( C ( x ) , y ) over ( x, y ) ∈

Original: ˜ D t ∪ S 1: t −// L cl is a function of C and B [ Y ]
Revised: ˜ D t ∪ S 1: t −// L cl is a function of C and B [ Y ]

Original: User : update C to reduce L cl
Revised: User : update C to reduce L cl

Original: Attacker : update B [ Y ] to reduce L cl + λ div L div
Revised: Attacker : update B [ Y ] to reduce L cl + λ div L div

Original: Update the replayer G t using ˜ D t and the latest B [ Y ] 11 Defender :
Revised: Update the replayer G t using ˜ D t and the latest B [ Y ] 11 Defender :

Original: Neural Cleansing on C t ← C 12 return
Revised: Neural Cleansing on C t ← C 12 return

Original: G t , B
Revised: G t , B

Original: [ Y ] , and C t
Revised: [ Y ] , and C t

Original: Algorithm 4 demonstrates the operation of all participants (user, attacker,and defender) during task t , which corresponds to the loop under a given t in
Revised: Algorithm 4 demonstrates the operation of all participants (user, attacker, and defender) during task t , which corresponds to the loop under a given t in

Original: Algorithm 1 (line 3 to 7 therein).
Revised: Algorithm 1 (line 3 to 7 therein).

Original: Thered colored steps are reserved for defense, which will be detailed in Appendix C.
Revised: The red colored steps are reserved for defense, which will be detailed in Appendix C.

Original: In line 7, the poisoned data˜ D t for task t is joined with the replayeddata S 1: t − 1 to construct the classifica- 254 tion risk L cl .
Revised: In line 7, the poisoned data ˜ D t for task t is joined with the replayed data S 1: t − 1 to construct the classification risk L cl .

Original: Both the attacker and theuser are trained in the same way as in
Revised: Both the attacker and the user are trained in the same way as in

Original: Algorithm 2, while the only differenceis that our optimization here is basedon mini-batches, and the replayer G t isadditionally learned in line 10.
Revised: Algorithm 2, while the only difference is that our optimization here is based on mini-batches, and the replayer G t is additionally learned in line 10.

Original:  Experimental Results
Revised:  Experimental Results

Original: We next conduct experiments on CIAPand verified: i) it attains the two objectives O1 and O2 , details in Appendix A; ii) the triggerdiscarding property introduced in Section 3.3 holds true for commonly used generative models,details in Appendix B; iii) CIAP remains effective under strong defenders ( O3 ), details in Appendix
Revised: We next conduct experiments on CIAP and verified: i) it attains the two objectives O1 and O2 , details in Appendix A; ii) the triggerdiscarding property introduced in Section 3.3 holds true for commonly used generative models, details in Appendix B; iii) CIAP remains effective under strong defenders ( O3 ), details in Appendix

Original: D. The code is available at [42].
Revised: D. The code is available at [42].

Original:  Conclusion and Future Work
Revised:  Conclusion and Future Work

Original: We proposed a novel poisoning attack on the generative replayer in continual learning, so thatcatastrophic forgetting can be promoted while the accuracy at the current task is not hurt.
Revised: We proposed a novel poisoning attack on the generative replayer in continual learning, so that catastrophic forgetting can be promoted while the accuracy at the current task is not hurt.

Original: Ourapproach takes advantage of input-aware backdoor attacks, whose triggers can not be well capturedby normal generative models thanks to their input dependency.
Revised: Our approach takes advantage of input-aware backdoor attacks, whose triggers can not be well captured by normal generative models thanks to their input dependency.

Original: In future work, we will delve moreinto the theoretical analysis of the trigger-discarding property.
Revised: In future work, we will delve more into the theoretical analysis of the trigger-discarding property.

Original: We will also extend the approach tocontinual learning without known task boundaries, which is promising because the generators G t canbe aggregated into a single one.
Revised: We will also extend the approach to continual learning without known task boundaries, which is promising because the generators G t can be aggregated into a single one.

Original: Societal impact.
Revised: Societal impact.

Original: We revealed important vulnerabilities in continual learning methods that arebased on generative rehearsal.
Revised: We revealed important vulnerabilities in continual learning methods that are based on generative rehearsal.

Original: Similar to many works that identify an attack without effectivecountermeasures available in the existing literature, we will address this defense issue in the future.
Revised: Similar to many works that identify an attack without effective countermeasures available in the existing literature, we will address this defense issue in the future.

Original: Supplementary Material  
Revised: Supplementary Material A Effectiveness of the attack for objectives O1 and O

Original: [Figure] A Effectiveness of the attack for objectives O1 and O2

Original: We tested CIAP on four datasets: Split-MNIST [43], Permuted-MNIST [44], FashionMNIST-MNIST[45], and Split-CIFAR-10 [46].
Revised: We tested CIAP on four datasets: Split-MNIST [43], Permuted-MNIST [44], FashionMNIST-MNIST [45], and Split-CIFAR-10 [46].

Original: We used SpinalVGG as the victim classifier [47] for the three MNISTdatasets, and ResNet [48] for the Split-CIFAR-10 dataset.
Revised: We used SpinalVGG as the victim classifier [47] for the three MNIST datasets, and ResNet [48] for the Split-CIFAR-10 dataset.

Original: The results shown here use cWGANwith gradient penalty as the replayer.
Revised: The results shown here use cWGAN with gradient penalty as the replayer.

Original: The poison ratio was set to ρ b = 0 .
Revised: The poison ratio was set to ρ b = 0 .

Original: 25 , and the cross ratio toρ c = 0 .
Revised: 25 , and the cross ratio to ρ c = 0 .

Original: Each trigger was allowed to change 10% of the pixels of a selected image (mask density),and Figure 2 will show that the triggers are quite inconspicuous.
Revised: Each trigger was allowed to change 10% of the pixels of a selected image (mask density), and Figure 2 will show that the triggers are quite inconspicuous.

Original: Split-MNIST We split the entire dataset of MNIST into five tasks, each consisting of images fromtwo disjoint classes in MNIST – the first task includes classes 0 and 1; the second task includes 2 and3; and so on.
Revised: Split-MNIST We split the entire dataset of MNIST into five tasks, each consisting of images from two disjoint classes in MNIST – the first task includes classes 0 and 1; the second task includes 2 and 3; and so on.

Original: The victim model was trained for 100 epoch in each task.
Revised: The victim model was trained for 100 epoch in each task.

Original: [Figure]
Revised: [Figure]

Original: Figure 1a shows the baseline result without a replayer, where the blue line represents the test accuracyof the first task, orange line for the second, etc.
Revised: Figure 1a shows the baseline result without a replayer, where the blue line represents the test accuracy of the first task, orange line for the second, etc.

Original: As expected, the test accuracy for each task dropsrapidly to 0% after the victim model proceeds to a new task.
Revised: As expected, the test accuracy for each task drops rapidly to 0% after the victim model proceeds to a new task.

Original: It is 0% because the new task has nooverlap with the previous ones in the label space.
Revised: It is 0% because the new task has no overlap with the previous ones in the label space.

Original: Figure 1b shows the result of DGR-facilitatedtraining, where the forgetting is significantly mitigated, and the test accuracy remains high on alltrained tasks.
Revised: Figure 1b shows the result of DGR-facilitated training, where the forgetting is significantly mitigated, and the test accuracy remains high on all trained tasks.

Original: This confirms the effectiveness of DGR and the sufficient capacity of the cWGAN.
Revised: This confirms the effectiveness of DGR and the sufficient capacity of the cWGAN.

Original: Figure 1c shows the result after our attack CIAP is enacted.
Revised: Figure 1c shows the result after our attack CIAP is enacted.

Original: The test accuracy of each current task canstill achieve nearly 100%, corroborating the achievement of objective O2 .
Revised: The test accuracy of each current task can still achieve nearly 100%, corroborating the achievement of objective O2 .

Original: When the learner movesto the next task, the accuracy on the previous tasks falls significantly to around 20% despite somefluctuations.
Revised: When the learner moves to the next task, the accuracy on the previous tasks falls significantly to around 20% despite some fluctuations.

Original: This confirms that the objective O1 (forgetting) has also been attained.
Revised: This confirms that the objective O1 (forgetting) has also been attained.

Original: [Figure]
Revised: [Figure]

Original: Finally, we plot in Figure 2 some exampleclean images of class 0 and 1 from the firsttask of Split-MNIST, along with their corresponding poisoned images constructed by
Revised: Finally, we plot in Figure 2 some example clean images of class 0 and 1 from the first task of Split-MNIST, along with their corresponding poisoned images constructed by

Original: IAB (before label flipping).
Revised: IAB (before label flipping).

Original: Clearly the poisons are quite inconspicuous.
Revised: Clearly the poisons are quite inconspicuous.

Original: Permuted-MNIST Here, each task consists of images from all the 10 classes of
Revised: Permuted-MNIST Here, each task consists of images from all the 10 classes of

Original: MNIST.
Revised: MNIST.

Original: However, each task also employsa unique pixel-level permutation, applied toall the images.
Revised: However, each task also employs a unique pixel-level permutation, applied to all the images.

Original: The performance is similar to
Revised: The performance is similar to

Original: Split-MNIST.
Revised: Split-MNIST.

Original: In Figure 3a where no replayer is used, the test accuracy drops to 20% after new tasks start.
Revised: In Figure 3a where no replayer is used, the test accuracy drops to 20% after new tasks start.

Original: Since allthe tasks in permuted-MNIST share the same label space, even random guessing would give 10%accuracy.
Revised: Since all the tasks in permuted-MNIST share the same label space, even random guessing would give 10% accuracy.

Original: So this 20% is already very close to complete forgetting.
Revised: So this 20% is already very close to complete forgetting.

Original: Replay ameliorated the problemas in Figure 3b, but the gain is much obliterated by the CIAP attack in Figure 3c.
Revised: Replay ameliorated the problem as in Figure 3b, but the gain is much obliterated by the CIAP attack in Figure 3c.

Original: [Figure]
Revised: [Figure]

Original: FashionMNIST-MNIST There are two tasks in this dataset: FashionMNIST followed by MNIST.
Revised: FashionMNIST-MNIST There are two tasks in this dataset: FashionMNIST followed by MNIST.

Original: Both tasks have 10 classes, and the victim model’s goal is to classify images from them into a sharedset of labels – "T-shirt" and "hand written 0" are both associated with "class 0".
Revised: Both tasks have 10 classes, and the victim model’s goal is to classify images from them into a shared set of labels – "T-shirt" and "hand written 0" are both associated with "class 0".

Original: The victim modelwas trained for 10 epochs with samples from the first task, followed by 10 epochs on the second task.
Revised: The victim model was trained for 10 epochs with samples from the first task, followed by 10 epochs on the second task.

Original: [Figure]
Revised: [Figure]

Original: Figure 4a shows the baseline result without a replayer, where the blue and orange lines representthe test accuracy of the first and second tasks, respectively.
Revised: Figure 4a shows the baseline result without a replayer, where the blue and orange lines represent the test accuracy of the first and second tasks, respectively.

Original: As expected, the test accuracy for
Revised: As expected, the test accuracy for

Original: FashionMNIST declines rapidly from 85% to 48% as the training on MNIST progresses to 10 epochs.
Revised: FashionMNIST declines rapidly from 85% to 48% as the training on MNIST progresses to 10 epochs.

Original: Figure 4b shows the result of DGR-facilitated training.
Revised: Figure 4b shows the result of DGR-facilitated training.

Original: Although the test accuracy of FashionMNISTstill drops a bit after MNIST starts to be trained on, it stabilizes at 79%, which illustrates theeffectiveness of DGR and the sufficient capacity of the cWGAN.
Revised: Although the test accuracy of FashionMNIST still drops a bit after MNIST starts to be trained on, it stabilizes at 79%, which illustrates the effectiveness of DGR and the sufficient capacity of the cWGAN.

Original: Figure 4c shows the result after ourattack is introduced.
Revised: Figure 4c shows the result after our attack is introduced.

Original: In the first task (epochs 0-10), the test accuracy remains very similar to that of
Revised: In the first task (epochs 0-10), the test accuracy remains very similar to that of

Original: Figure 4a and 4b, corroborating the achievement of objective O2 .
Revised: Figure 4a and 4b, corroborating the achievement of objective O2 .

Original: When the continual learning moveson to the second task, the test accuracy on the earlier task falls significantly to 61%, much lower thanthe 79% achieved at the end of the first task.
Revised: When the continual learning moves on to the second task, the test accuracy on the earlier task falls significantly to 61%, much lower than the 79% achieved at the end of the first task.

Original: This confirms that the objective O1 (forgetting) has alsobeen attained.
Revised: This confirms that the objective O1 (forgetting) has also been attained.

Original: Split-CIFAR-10 To illustrate the effectiveness of CIAP in colored space, we repeated the experiment on CIFAR-10, with the same setup of five disjoint tasks.
Revised: Split-CIFAR-10 To illustrate the effectiveness of CIAP in colored space, we repeated the experiment on CIFAR-10, with the same setup of five disjoint tasks.

Original: Here the victim model was trained for50 epochs on each task.
Revised: Here the victim model was trained for 50 epochs on each task.

Original: [Figure]
Revised: [Figure]

Original: Similar to the Split-MNIST, the victim model completely forgets the earlier trained tasks after a newtask starts, as shown in Figure 5a.
Revised: Similar to the Split-MNIST, the victim model completely forgets the earlier trained tasks after a new task starts, as shown in Figure 5a.

Original: After DGR is introduced in Figure 5b, although forgetting is not aswell mitigated as in Split-MNIST, solid improvement is still made on the test accuracy for all trainedtasks.
Revised: After DGR is introduced in Figure 5b, although forgetting is not as well mitigated as in Split-MNIST, solid improvement is still made on the test accuracy for all trained tasks.

Original: However, the improvements brought by DGR were completely eliminated by the CIAP attack.
Revised: However, the improvements brought by DGR were completely eliminated by the CIAP attack.

Original: As Figure 5c shows, the test accuracy on past tasks drops down to 0% after being poisoned.
Revised: As Figure 5c shows, the test accuracy on past tasks drops down to 0% after being poisoned.

Original: Duringthe current task, however, the accuracy can still achieve the same level as in Figure 5a.
Revised: During the current task, however, the accuracy can still achieve the same level as in Figure 5a.

Original: B Investigation of the trigger-discarding property
Revised: B Investigation of the trigger-discarding property

Original: To better illustrate property P4 , we set up two experiments on Split-MNIST using cWGAN.
Revised: To better illustrate property P4 , we set up two experiments on Split-MNIST using cWGAN.

Original: The first experiment studies the percentage of backdoored images (images with a trigger) generatedby the replayer, when a varying portion of the training images are backdoored.
Revised: The first experiment studies the percentage of backdoored images (images with a trigger) generated by the replayer, when a varying portion of the training images are backdoored.

Original: Our goal is to showthat such a percentage is much lower for IAB than for a static backdoor, i.e., the triggers of IAB aremuch less likely to survive the generative learning.
Revised: Our goal is to show that such a percentage is much lower for IAB than for a static backdoor, i.e., the triggers of IAB are much less likely to survive the generative learning.

Original: A static backdoor refers to a white square onthe image’s top left corner.
Revised: A static backdoor refers to a white square on the image’s top left corner.

Original: To this end, we trained a binary poison detector based on a training setthat is backdoored with the IAB network learned from the first task.
Revised: To this end, we trained a binary poison detector based on a training set that is backdoored with the IAB network learned from the first task.

Original: Another detector was trainedanalogously for static backdoor.
Revised: Another detector was trained analogously for static backdoor.

Original: This allows us to measure the percentage of backdoored imagesfrom the replayer.
Revised: This allows us to measure the percentage of backdoored images from the replayer.

Original: The two detectors achieve 97% accuracy, and similar ideas have been used inevaluating generative models such as inception score [49].
Revised: The two detectors achieve 97% accuracy, and similar ideas have been used in evaluating generative models such as inception score [49].

Original: As shown in Figure 6, the static backdoor maintains almost the same percentage of backdoored imagesused for training, while that for IAB grows much more slowly, producing only 40% backdooredimages when 90% of the training images are backdoored.
Revised: As shown in Figure 6, the static backdoor maintains almost the same percentage of backdoored images used for training, while that for IAB grows much more slowly, producing only 40% backdoored images when 90% of the training images are backdoored.

Original: This shows that IAB is far more likely tobe discarded by the replayer.
Revised: This shows that IAB is far more likely to be discarded by the replayer.

Original: [Figure]
Revised: [Figure]

Original: backdoor and IAB

Original: Our second experiment examines the chance of replaying a backdoored image by using the attacksuccess rate (ASR).
Revised: Our second experiment examines the chance of replaying a backdoored image by using the attack success rate (ASR).

Original: In the same setting as the above experiment, we backdoored 25% images in thefirst 30 epochs with flipped labels, and used them to train a replayer.
Revised: In the same setting as the above experiment, we backdoored 25% images in the first 30 epochs with flipped labels, and used them to train a replayer.

Original: Then we generated examplesfrom it in the second 30 epochs, and used them to incrementally train a new classifier.
Revised: Then we generated examples from it in the second 30 epochs, and used them to incrementally train a new classifier.

Original: This classifieris tested on backdoored images, and the proportion of misclassified images is calculated as the ASR.
Revised: This classifier is tested on backdoored images, and the proportion of misclassified images is calculated as the ASR.

Original: If the triggers are preserved by the replayer, then a classifier learned from the replayed images shouldpermit a high ASR.
Revised: If the triggers are preserved by the replayer, then a classifier learned from the replayed images should permit a high ASR.

Original: This is confirmed in Figure 7a where the ASR remains at 100% for static backdoor.
Revised: This is confirmed in Figure 7a where the ASR remains at 100% for static backdoor.

Original: The drop in the middle is because the new classifier was trained from scratch.
Revised: The drop in the middle is because the new classifier was trained from scratch.

Original: In contrast, the ASRdrops to zero for IAB in Figure 7b, confirming that the newly trained classifier is not backdoored, i.e.,the replayed images do not preserve triggers sufficiently well for training a backdoored classifier.
Revised: In contrast, the ASR drops to zero for IAB in Figure 7b, confirming that the newly trained classifier is not backdoored, i.e., the replayed images do not preserve triggers sufficiently well for training a backdoored classifier.

Original: C The Defender
Revised: C The Defender

Original: We consider two defenses against the CIAP attack.
Revised: We consider two defenses against the CIAP attack.

Original: The first is neural cleansing [35], which has beeninserted in line 11 of Algorithm 4.
Revised: The first is neural cleansing [35], which has been inserted in line 11 of Algorithm 4.

Original: If it were successful, then the backdoor planted in C t would bedetected and removed, thereby defeating objective O2 with immediate poor accuracy on clean testexamples at task t .
Revised: If it were successful, then the backdoor planted in C t would be detected and removed, thereby defeating objective O2 with immediate poor accuracy on clean test examples at task t .

Original: Our second defense is aimed at objective O1 .
Revised: Our second defense is aimed at objective O1 .

Original: To this end, we apply an outlier detector ν -SVM to
Revised: To this end, we apply an outlier detector ν -SVM to

Original: S 1: t − 1 , which is in line 4 of Algorithm 4.
Revised: S 1: t − 1 , which is in line 4 of Algorithm 4.

Original: If it managed to filter out mislabeled replayed samples,then the attacker would fail to bolster catastrophic forgetting.
Revised: If it managed to filter out mislabeled replayed samples, then the attacker would fail to bolster catastrophic forgetting.

Original: Here ν is a hyperparameter controlling the fraction of outliers.
Revised: Here ν is a hyperparameter controlling the fraction of outliers.

Original: Since its value is unknown in practice, our experiment will enumerate a rangeof ν values, and demonstrate the extent to which the learner’s performance can be saved respectively.
Revised: Since its value is unknown in practice, our experiment will enumerate a range of ν values, and demonstrate the extent to which the learner’s performance can be saved respectively.

Original: It is crucial to recognize that the replayed examples are not simply label-flipped poisons (i.e.,clean images with a wrong label), although the replayer is poisoned with label-flipped and backdooredexamples.
Revised: It is crucial to recognize that the replayed examples are not simply label-flipped poisons (i.e., clean images with a wrong label), although the replayer is poisoned with label-flipped and backdoored examples.

Original: This is for two reasons.
Revised: This is for two reasons.

Original: Firstly, since the examples of a class y ′ is fed to the replayer totrain for the class y , the generation of the features/images for class y is contaminated.
Revised: Firstly, since the examples of a class y ′ is fed to the replayer to train for the class y , the generation of the features/images for class y is contaminated.

Original: Secondly, theinput-dependent triggers introduce additional complications to the generative model.
Revised: Secondly, the input-dependent triggers introduce additional complications to the generative model.

Original: Indeed, wetested by directly generating label-flipped examples based on clean images, and ν -SVM easily filteredthem out.
Revised: Indeed, we tested by directly generating label-flipped examples based on clean images, and ν -SVM easily filtered them out.

Original: However, this is not the case when ν -SVM is applied to our replayed images (Appendix D).[Figure]
Revised: However, this is not the case when ν -SVM is applied to our replayed images (Appendix D).

Original: D Assessing CIAP attack under defense (objective O3)
Revised: D Assessing CIAP attack under defense (objective O3)

Original: [Figure]

Original: We next study how well our attack withstands defenses.
Revised: We next study how well our attack withstands defenses.

Original: To this end, a ν -SVM with a radial basis[Figure] kernel was applied to the output of the convolutional layer of SpinalVGG.
Revised: To this end, a ν -SVM with a radial basis kernel was applied to the output of the convolutional layer of SpinalVGG.

Original: This allows a portion ofreplayed samples to be filtered out, and the proportion is controlled by ν ∈ (0 , 1) .
Revised: This allows a portion of replayed samples to be filtered out, and the proportion is controlled by ν ∈ (0 , 1) .

Original: [Figure]
Revised: [Figure]

Original: As shown in Figure 8 where ν is varied in { 0 . 15 , 0 . 25 , 0 . 35 } on the Split-MNIST dataset, the filteringby ν -SVM does help a little, especially when ν is set around the poison ratio ( ρ b = 0 . 25 ).
Revised: As shown in Figure 8 where ν is varied in { 0 . 15 , 0 . 25 , 0 . 35 } on the Split-MNIST dataset, the filtering by ν -SVM does help a little, especially when ν is set around the poison ratio ( ρ b = 0 . 25 ).

Original: However,it remains unable to well remove the impact of the attack, and the test accuracy on past tasks stillfalls below 50%.
Revised: However, it remains unable to well remove the impact of the attack, and the test accuracy on past tasks still falls below 50%.

Original: [Figure]

Original: The limited improvement could be partially ascribed to the compromised image quality due to thebackdoors.
Revised: The limited improvement could be partially ascribed to the compromised image quality due to the backdoors.

Original: To better visualize the consequence of poisoning on the replayer, we compare in Figure 9the replayed images before and after the attack.
Revised: To better visualize the consequence of poisoning on the replayer, we compare in Figurethe replayed images before and after the attack.

Original: Figure 9a presents example images generated by areplayer that is trained on clean images only.
Revised: Figure 9a presents example images generated by a replayer that is trained on clean images only.

Original: In contrast, Figure 9b shows that the poisoned replayercan often generate images from an incorrect class, i.e., another class that is incorrectly labeled as"1".
Revised: In contrast, Figure 9b shows that the poisoned replayer can often generate images from an incorrect class, i.e., another class that is incorrectly labeled as "1".

Original: Although the replayer cannot reproduce the input-aware backdoor, it tends to turn the backdoorsinto some random noise, making it harder for the filter to identify those poisons.
Revised: Although the replayer cannot reproduce the input-aware backdoor, it tends to turn the backdoors into some random noise, making it harder for the filter to identify those poisons.

Original: As a result, theremaining replayed images are only slightly improved by the ν -SVM filtering as shown in Figure 9c.
Revised: As a result, the remaining replayed images are only slightly improved by the ν -SVM filtering as shown in Figure 9c.

Original: [Figure]
Revised: [Figure]

Original: Similar to the experiment on Split-MNIST, we tested the attack with ν -SVM defense on CIFAR-10.
Revised: Similar to the experiment on Split-MNIST, we tested the attack with ν -SVM defense on CIFAR-10.

Original: As shown in Figure 10, with three different values of ν , the test accuracy on past tasks dropped toalmost 0 after switching to a new task.
Revised: As shown in Figure 10, with three different values of ν , the test accuracy on past tasks dropped to almost 0 after switching to a new task.

Original: This confirms that our CIAP remains effective under thedefense of ν -SVM.
Revised: This confirms that our CIAP remains effective under the defense of ν -SVM.

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: E Generalizability of CIAP
Revised: E Generalizability of CIAP

Original: Attack CVAE Replayer Similar to the experiment with cWGAN in Appendix A, we tested theattack with cVAE replayer on FashionMNIST-MNIST.
Revised: Attack CVAE Replayer Similar to the experiment with cWGAN in Appendix A, we tested the attack with cVAE replayer on FashionMNIST-MNIST.

Original: As shown in figure 11, the testing accuracy onthe earlier task dropped from 82% to 24% after attack.
Revised: As shown in figure 11, the testing accuracy on the earlier task dropped from 82% to 24% after attack.

Original: This indicates that cVAE is also vulnerable tothe proposed attack.
Revised: This indicates that cVAE is also vulnerable to the proposed attack.

Original: [Figure]
Revised: [Figure]

Original: Attack
Revised: Attack

Original: More Tasks We also tested proposed model on EMNIST [50] dataset, an extended versionof MNIST, consists of handwritten digits and letters.
Revised: More Tasks We also tested proposed model on EMNIST [50] dataset, an extended version of MNIST, consists of handwritten digits and letters.

Original: Since some classes looks alike (i.e. "C" and "c","S" and "s"), [50] merged those similar classes, and balanced the merged classes.
Revised: Since some classes looks alike (i.e. "C" and "c", "S" and "s"), [50] merged those similar classes, and balanced the merged classes.

Original: Figure 12 shows the test accuracy on clean testing images for 10 disjoint tasks in EMNIST, each taskconsists of 2 classes.
Revised: Figure 12 shows the test accuracy on clean testing images for 10 disjoint tasks in EMNIST, each task consists of 2 classes.

Original: The results again confirms the effectiveness of CIAP over an increased numberof tasks.
Revised: The results again confirms the effectiveness of CIAP over an increased number of tasks.

Original: [Figure]
Revised: [Figure]