  Rehearsal, seeking to remind the model by storing old knowledge in lifelong
learning, is one of the most effective ways to mitigate catastrophic
forgetting, i.e., biased forgetting of previous knowledge when moving to new
tasks. However, the old tasks of the most previous rehearsal-based methods
suffer from the unpredictable domain shift when training the new task. This is
because these methods always ignore two significant factors. First, the Data
Imbalance between the new task and old tasks that makes the domain of old tasks
prone to shift. Second, the Task Isolation among all tasks will make the domain
shift toward unpredictable directions; To address the unpredictable domain
shift, in this paper, we propose Multi-Domain Multi-Task (MDMT) rehearsal to
train the old tasks and new task parallelly and equally to break the isolation
among tasks. Specifically, a two-level angular margin loss is proposed to
encourage the intra-class/task compactness and inter-class/task discrepancy,
which keeps the model from domain chaos. In addition, to further address domain
shift of the old tasks, we propose an optional episodic distillation loss on
the memory to anchor the knowledge for each old task. Experiments on benchmark
datasets validate the proposed approach can effectively mitigate the
unpredictable domain shift.
