Paper:
Memory Replay GANs: learning to generate images from new categories without forgetting
Previous works on sequential learning address the problem of forgetting in discriminative models. In this paper we consider the case of generative models. In particular, we investigate generative adversarial networks (GANs) in the task of learning new categories in a sequential fashion. We first show that sequential fine tuning renders the network unable to properly generate images from previous categories (i.e. forgetting). Addressing this problem, we propose Memory Replay GANs (MeRGANs), a conditional GAN framework that integrates a memory replay generator. We study two methods to prevent forgetting by leveraging these replays, namely joint training with replay and replay alignment. Qualitative and quantitative experimental results in MNIST, SVHN and LSUN datasets show that our memory replay approach can generate competitive images while significantly mitigating the forgetting of previous categories.
While previous works study forgetting in discriminative tasks, in this paper we focus on forgetting in generative models (GANs in particular) through the problem of generating images when categories are presented sequentially as disjoint tasks. The closest related work is [24], that adapts elastic weight consolidation (EWC) [4] to GANs. In contrast, our method relies on memory replay and we describe two approaches to prevent forgetting by joint retraining and by aligning replays. The former includes replayed samples in the training process, while the latter forces to synchronize the replays of the current generator with those generated by an auxiliary generator (a snapshot taken before starting to learn the new task). An advantage of studying forgetting in image generation is that the dynamics of forgetting and consolidation can be observed visually through the generated images themselves.

Review:
Update following the author rebuttal:  I would like to thank the authors for their thoughtful rebuttal. I feel like they appropriately addressed the main points I raised, namely the incomplete evaluation and the choice of GANs over other generative model families, and I'm inclined to recommend the paper's acceptance. I updated my review score accordingly.  -----  The paper presents a conditional GAN framework to combat catastrophic forgetting when learning new image categories in a sequential fashion. The paper is well-written and its exposition of the problem, proposed solution, and related work is clear.  Starting from the AC-GAN conditional generative modeling formulation, the authors introduce the notion of a sequence of tasks by modeling image classes (for MNIST, SVHN, and LSUN) in sequence, where the model for each class in the sequence is initialized with the model parameters for the previous class in the sequence. They note that in this setup the AC-GAN’s classifier cannot be used, as for any given task there is only one class being trained on.  The idea put forward is to use the generator as a memory replay mechanism to prevent forgetting when training for the current task. Two approaches are considered:  - A joint replay approach (MerGAN-JTR), where upon completing a training task the model is sampled from, and the samples for previous tasks are combined with the dataset for the current task to form a hybrid real/synthetic class-conditional dataset. In this setup the AC-GAN formulation is used. - A replay alignment approach (MeRGAN-RA), where class-conditional samples for previous tasks are drawn continuously throughout training using the current value of the generator’s class-independent parameters and are compared with the class-conditional samples of the hybrid dataset using an L2 pixelwise loss (using the same latent samples z for both). In this setup the AC-GAN’s classifier is not used.  The two approaches, along with several baselines -- naive sequential fine-tuning (SFT), elastic weight consolidation (EWC), deep generative replay (DGR), and joint training (JT) -- are trained on sequences of classes from MNIST, SVHN, and LSUN and evaluated qualitatively and quantitatively.  For MNIST and SVHN, qualitative evaluation consists of examining class-conditional model samples after completing the last training task in the sequence and shows that both proposed approaches perform better than the competing EWC and DGR approaches (especially for SVHN). For LSUN, qualitative evaluation compares both proposed approaches with the naive SFT baseline and shows samples drawn after training on each task in the sequence. I would have liked to see a comparison with EWC and DGR here as well; can the authors tell how they compare?  In terms of quantitative evaluation, the accuracy of an external classifier on class-conditional samples is presented for MNIST, SVHN, and LSUN, showing that the proposed approaches outperform the naive SFT baseline as well as the competing approaches. Once again, I would have liked to see a comparison with EWC and DGR for LSUN. I find it interesting that for MNIST the proposed approaches do better than the upper-bound JT baseline; do the authors have an idea why? I would like to see confidence intervals on the accuracies, as it may help decide whether the improved performance over JT is statistically significant or not. Models trained on LSUN are also evaluated in terms of Fréchet Inception Distance, showing again that the two proposed approaches perform much better than the naive SFT baseline.  One criticism I would have to voice about the evaluation procedure is that the sequence of class-conditional tasks setup is a little contrived. I agree with the premise that in real-world scenarios the full training data is often not immediately available, but at the same time the sort of input distribution drifts we are likely to encounter is probably smoother than the discrete shifts in input distribution corresponding to switching the object class (an example would be the day/night cycle a camera pointed at a highway experiences). Moreover, information about the input distribution -- be it a discrete class label or some continuous attribute representation -- probably won’t be available in real-world scenarios.  Having said that, I am aware this is an issue not unique to the proposed method, and I am okay with the use of the proposed evaluation setup given that most previous work on the topic also uses a similar evaluation setup -- and also given that the search for a better setup is still an open question to me and I don’t have a better idea to propose.  Another criticism I have is that the choice of GANs as the training framework makes it unnecessarily difficult to compare and evaluate different methods: the right way of evaluating GANs is still very much an open problem, which means that any choice of GAN evaluation metric comes with its own set of caveats. For instance, one problem with the classifier accuracy metric is that the model could produce class-conditional samples achieving a high accuracy by collapsing the generator to only a few exemplars per class.  Given that the proposed method appears to be applicable to any class-conditional generative model, I think the choice of an explicit likelihood model family along with a log-likelihood metric would have been easier to defend from an evaluation point of view.  In summary, the paper presents a clear exposition of an idea to solve a very relevant research problem, namely catastrophic forgetting in generative models. I am a bit less enthusiastic about the empirical evaluation, mostly because I feel it is incomplete (as I said, I would like to see comparisons with competing methods for LSUN), and because the choice of model family and evaluation metric makes it harder to compare and evaluate competing methods.