  Continual (or "incremental") learning approaches are employed when additional
knowledge or tasks need to be learned from subsequent batches or from streaming
data. However these approaches are typically adversary agnostic, i.e., they do
not consider the possibility of a malicious attack. In our prior work, we
explored the vulnerabilities of Elastic Weight Consolidation (EWC) to the
perceptible misinformation. We now explore the vulnerabilities of other
regularization-based as well as generative replay-based continual learning
algorithms, and also extend the attack to imperceptible misinformation. We show
that an intelligent adversary can take advantage of a continual learning
algorithm's capabilities of retaining existing knowledge over time, and force
it to learn and retain deliberately introduced misinformation. To demonstrate
this vulnerability, we inject backdoor attack samples into the training data.
These attack samples constitute the misinformation, allowing the attacker to
capture control of the model at test time. We evaluate the extent of this
vulnerability on both rotated and split benchmark variants of the MNIST dataset
under two important domain and class incremental learning scenarios. We show
that the adversary can create a "false memory" about any task by inserting
carefully-designed backdoor samples to the test instances of that task thereby
controlling the amount of forgetting of any task of its choosing. Perhaps most
importantly, we show this vulnerability to be very acute and damaging: the
model memory can be easily compromised with the addition of backdoor samples
into as little as 1\% of the training data, even when the misinformation is
imperceptible to human eye.
