Paper:
In the field of Continual Learning, the objective is to learn several tasks one after the other without access to the data from previous tasks. Several solutions have been proposed to tackle this problem but they usually assume that the user knows which of the tasks to perform at test time on a particular sample, or rely on small samples from previous data and most of them suffer of a substantial drop in accuracy when updated with batches of only one class at a time. In this article, we propose a new method, OvA-INN, which is able to learn one class at a time and without storing any of the previous data. To achieve this, for each class, we train a specific Invertible Neural Network to extract the relevant features to compute the likelihood on this class. At test time, we can predict the class of a sample by identifying the network which predicted the highest likelihood. With this method, we show that we can take advantage of pretrained models by stacking an Invertible Network on top of a features extractor. This way, we are able to outperform stateof-the-art approaches that rely on features learning for the Continual Learning of MNIST and CIFAR-100 datasets. In our experiments, we reach 72% accuracy on CIFAR-100 after training our model one class at a time.

Review:
[update after rebuttal]
I thank the authors for their detailed reply, answers to my questions, and updates to the paper. (I especially appreciate the substantial effort to address all points and improve the manuscript even given the strongly negative rating. I am not personally in favour of "extremifying" the rating system the way ICLR did this year; I think it discouraged many authors from working on their rebuttals, and caused many reviewers to have a sense of inertia to keep their bad scores. I'm glad the authors didn't fall into the first camp, and I'm doing my best not to fall into the second! I think you did a very good job of improving the paper).
I think the manuscript has improved substantially; especially the clarifications about (not) using negative samples and additional detail on INNs with citations. 
I find some of the sentences still unclear, and strongly suggest having the text read over for grammar/clarity for a camera-ready version.
But overall I recommend acceptance of this work. 
------------
Paper summary:
This paper does continual learning using an invertible neural network (INN) trained to recognize each class. At test time a new example is presented to all of the INNs and the INN whose prediction has the smallest norm is used to predict the class of the test example. They do experiments on MNIST and CIFAR-100
Paper contributions: 
- Review of methods and evaluation settings for continual learning
- Review of invertible neural networks 
- Experiments comparing the proposed method to several other continual learning methods 
- Examination of memory cost
- Exploration of feature space of trained INNs
Review summary & decision: 
The proposed idea of using invertible networks for continual learning sounds interesting, and I think that this could be a good paper. The experimental evaluation is not consistent or complete enough for me to to tell, however, and the core motivation for the idea is not clearly explained. There are also some less critical, but still important aspects of the paper (related work, clarity of explanations, repetitiveness) which lead me to decide this paper is not currently ready for publication.
Reasons for decision: 
1. A lot of statements and decisions are made without being explained, or are unclear / innacurate
      - In CL, the objective is to learn several tasks one after the other" this is not the objective, it's the problem setting. The objective is classification accuracy (or some other metric) in this setting.
      - The choice of datasets seems a little odd; why not CIFAR-10 as well?
      - It's very misleading to say your method is able to learn one class at a time without storing data; unless I've misunderstood this is only after pretraining all of the invertible networks with labelled data.
       - Why 500 examples per class for CIFAR-100, and how many for MNIST? What criteria was used to arrive at this number?
      - in the abstract you state the definition of continual learning says you can't have access to the data from previous tasks, but then mention that previous works use samples from previous data. This seems contradictory
      - I'm not extremely familiar with Li & Hoeim (learning without forgetting), but I don't think it's very accurate to call it distillation regularization
      - In 4.1 I don't think it's accurate to call this compression; it's just having smaller size layers in between (unless I misunderstood).
      - I'm not familiar with model superposition; it should be more explained since your claims of having lower memory cost rely on this, as far as I can tell.
      - The experimental baselines seem inconsistent / not comparable, making it difficult to evaluate what is going on. E.g. some update pretrained features and some do not. Some (including yours) do offline pretraining on a subset of examples, with inconsistent amounts of data, and sometimes this is considered part of training (Fearnet) and sometimes not (yours). 
       - I don't understand the central motivation for using INNs. The only sentence that seems to talk about it is "this way ... network won't be able to have an output similar to the outputs on data from its training set", but this is not clear, not enough explanation, and seems somewhat innaccurate. It's not that it's not able, it's just not likely (I think; please correct me if I'm wrong!), and I don't see why this isn't also true of e.g. MLPs (i.e. for MNIST, train 10 MLPs each to classify [class i] vs [all classes except i] - In order to justify the proposed approach, the baseline described above (using MLPs instead of invertible NNs) would be very useful
 2. The training regime for the INNs is not clear (are they shown negative (other class) examples? How many, how are they sampled? I didn't find any hyperparameters for the INN training in the appendix, even though it says they should be there.    
 3. The single head setting seems very similar to (or maybe the same as?) open set learning; I think this should be mentioned and works in this area should be reviewed.
 4. Section 3 is repetitive and unclear. It could be greatly shortened to make space for more experiments.
 5. It would be nice to see computational cost as well as memory cost; this is important in many settings where continual learning would be deployed.
Feedback/suggestions/nits (not necessarily part of decision assessment): 
1. Cite the definition of continual learning (e.g. with a reference to a textbook or review) 
2. A lot of the writing is unclear, wordy, and/or grammatically incorrect
    - inconsistent verb tense
         -  e.g. "if one would need .... it is required" should be "if one would need .... it would require" I'd suggest rewording this sentence entirely, because it's misleading - it says "retrain on this new dataset (which sounds like train just on the new data), but I guess you mean retrain on all data including the new data. e.g. "Updating the model with new data requires retraining on the full dataset (old + new data). However, there are.... method may not be applicable." 
         - "we are reaching" -> we reach    - frequent use of "indeed" when it doesn't make sense
    - Section 3 repeats the intro, 3.1 and 3.2 are sort of saying the same thing, and are also sort of repetitive of the related work. The related work should all be in the related work section, and this section should just be about what _you_ are doing. Invertible neural networks should be reviewed in something like a 'background' section, which possibly could be in appendix. Right now that section has even more related work in it
3. Include tsne hyperparameters in appendix 
4. "appendix" not "annexe" (annexes are on buildings :) ) 
5. when you define continual learning, bold it (not italicize) 
6. "Y is then Y_i" -> Y = Y_i" 
7. Caption for Figure 3 is unclear; I don't understand what I'm supposed to take away from these images. I also don't see a black cross.
Questions:
1. What is the motivation for using INNs specifically? (rather than e.g. normal MLPs). 
2. What is the training regime for the INNs?
3. What is model superposition, is it expensive in computation or something other than memory, and if you don't do that what is the memory cost of your method?