  We present Gaussian Mixture Replay (GMR), a rehearsal-based approach for
continual learning (CL) based on Gaussian Mixture Models (GMM). CL approaches
are intended to tackle the problem of catastrophic forgetting (CF), which
occurs for Deep Neural Networks (DNNs) when sequentially training them on
successive sub-tasks. GMR mitigates CF by generating samples from previous
tasks and merging them with current training data. GMMs serve several purposes
here: sample generation, density estimation (e.g., for detecting outliers or
recognizing task boundaries) and providing a high-level feature representation
for classification. GMR has several conceptual advantages over existing
replay-based CL approaches. First of all, GMR achieves sample generation,
classification and density estimation in a single network structure with
strongly reduced memory requirements. Secondly, it can be trained at constant
time complexity w.r.t. the number of sub-tasks, making it particularly suitable
for life-long learning. Furthermore, GMR minimizes a differentiable loss
function and seems to avoid mode collapse. In addition, task boundaries can be
detected by applying GMM density estimation. Lastly, GMR does not require
access to sub-tasks lying in the future for hyper-parameter tuning, allowing CL
under real-world constraints. We evaluate GMR on multiple image datasets, which
are divided into class-disjoint sub-tasks.
