Paper:
Training a neural network for a classification task typically assumes that the data to train are given from the beginning. However, in the real world, additional data accumulate gradually and the model requires additional training without accessing the old training data. This usually leads to the catastrophic forgetting problem which is inevitable for the traditional training methodology of neural networks. In this paper, we propose a memory-based continual learning method that is able to learn additional tasks while retaining the performance of previously learned tasks. Composed of two complementary networks, the Hippocampus-Net (H-Net) and the Cortex-Net (C-Net), our model estimates the index of the corresponding task for an input sample and utilizes a particular portion of itself with the estimated index. The C-Net guarantees no degradation in the performance of the previously learned tasks and the H-Net shows high confidence in finding the origin of an input sample.

Review:
The key idea of this paper is to expand the network for training on new tasks which is termed as C-Net, and train an additional generative model which is used for predicting task id (which is called H-Net), and use the task id for selecting weights from the C-Net.
Pros:
1. It is relatively easy to understand the paper. 
2. The originality of this paper lies in the usage of generative model to predict task id (H-Net). To my knowledge has not been proposed before.
3. In contrast to previous works in multi-task learning, which assumes task id is available both during training and inference, this work tries to remove the need of task id during inference, which makes it closer to the general definition of continual learning.
Cons:
1. Expanding the network for new tasks is not a novel contribution of this paper, it has already been proposed in previous works on multi-task learning. Doing expansion on all of the layers does not qualify for a major contribution in my opinion.
2. The experimental comparison is not very fair in my opinion, 
     a. Comparing accuracy of C-Net to other methods is not very useful. Because this methods expands the network for every new task, while other methods (EWC, LwF) has limited to no expansion in the network. Given that the single network result is far from state of the art (table 3), I suppose model size could contribute to the accuracy boost.
     b. It is not explicitly stated in the paper whether the output neurons are shared between tasks or an individual set of output neurons are used for different tasks, but from the rest of the paper I suppose disjoint neurons are used. Then the comparison between EWC and this work is not fair because EWC shares the output neurons among tasks.
This is not to blame this paper for not making fair comparison, since given different assumptions between methods (availability of task id, shared output neurons etc.), it is usually difficult to fairly compare between continual learning methods.  This problem is raised in another submission https://openreview.net/forum?id=ByGVui0ctm. The point here is that the accuracy of C-Net is not a good measure of how good this method is.
3. I disagree with the point that MNIST and SVHN are similar, they have very different distributions and are very easy to tell apart with a model. One concern is that the generative H-Net may fail to work once the distributions of the tasks overlap to some extent. e.g. cifar10 vs cifar100.
As a conclusion, the key contribution of this work is using generative model to determine task id which removes the need for task id during inference. It is relatively insufficient for publication on ICLR.