Paper:
Continual learning is a critical ability of continually acquiring and transferring knowledge without catastrophically forgetting previously learned knowledge. However, enabling continual learning for AI remains a long-standing challenge. In this work, we propose a novel method, Prototype Recalls, that efficiently embeds and recalls previously learnt knowledge to tackle catastrophic forgetting issue. In particular, we consider continual learning in classification tasks. For each classification task, our method learns a metric space containing a set of prototypes where embedding of the samples from the same class cluster around prototypes and class-representative prototypes are separated apart. To alleviate catastrophic forgetting, our method preserves the embedding function from the samples to the previous metric space, through our proposed prototype recalls from previous tasks. Specifically, the recalling process is implemented by replaying a small number of samples from previous tasks and correspondingly matching their embedding to their nearest class-representative prototypes. Compared with recent continual learning methods, our contributions are fourfold: first, our method achieves the best memory retention capability while adapting quickly to new tasks. Second, our method uses metric learning for classification, and does not require adding in new neurons given new object classes. Third, our method is more memory efficient since only class-representative prototypes need to be recalled. Fourth, our method suggests a promising solution for few-shot continual learning. Without tampering with the performance on initial tasks, our method learns novel concepts given a few training examples of each class in new tasks.

Review:
The proposed method addresses continual learning, by learning a mapping from the input space to an embedding space, and employing a loss that encourages clustering the embeddings by class (and task?) around some centroids called prototypes. Catastrophic forgetting is mitigated by adding a penalty term that is proportional to the distance of the embeddings under the current network of some samples from the past tasks, and the centroids previously associated to each of them.
The idea might have some merit, but the paper in its current state fails to convey it clearly and is too vague on the experimental settings for the results to be considered in any way. The notation is at times unnecessary heavy (see specific comments below), and many details on the method and experimental setup are missing to the point that it’s difficult to judge it and it would be impossible to reproduce it. In addition, it’s unclear whether the baselines have been properly tuned since the reported performance is incompatible with that of the original paper.
The submitted paper is clearly not ready for publication. I can consider modifying my vote if the paper undergoes a major review, fixing the notation, explaining the method and the experimental settings extensively and clearly, with proper comparison in terms of capacity/performance tradeoff w.r.t. the state of the art.
Detailed feedback:
1) Notation
- I believe the exposition would be more fluent and easier to follow if you described the method under the assumption that all tasks have the same number of classes and a shared dataset, and mentioning that the method can be easily extended to work in the more general setting. Under this assumption, e.g., c_{kT} would simplify to c_{k}.
- When you refer to a generic task, please use a lower case letter (e.g., t) instead of T (e.g., first paragraph of page 3, Eqn. 1, Eqn. 2, ..). Use capital letters for constants only (e.g., use T for the total number of tasks). When you need to refer to two tasks that are in a particular relative ordering. For instance, Eqn. 1 should be the argmin of “\sum_{s=1}^{t} [...], with s < t”. Similarly, Sec 2.2 should refer to tasks s and t, rather than t and T (unless T is specifically referring to the last task alone, which doesn’t seem the case here as from what I gather the same should hold for all tasks, i.e., for a specific task t and any previous task s, with s < t.).
- If I am not mistaken, L_T is defined just before Eqn. 1 and only used there. It is not very informative and it’s based on some undefined loss L. I would suggest to drop it entirely, and simplify Eqn. 1 accordingly.
- Please avoid subscript with commas as much as possible, as they are difficult to parse. E.g., L_{classi, Dt} can become L_{classi_t} (or even L_{c_t} or L_{c}^{(t)}), remaining equally informative. All other “D_t” subscripts can be similarly replaced by “t” throughout the text. Generally speaking, I would recommend the ^{(t)} notation to denote task t (which you already adopted when defining \tilde{D}_t
- I believe Eqn. 1 can be simplified by merging the two summations into a single expression. Also, I think it would be clearer to define the argmin to be over “f \in \mathcal{F}”, since it is a search over the space of all possible functions. Finally, I also find a bit confusing the use of f and f_t to denote an evolving (currently evaluated) function and a fixed (best function from previous search) one respectively, but I am not sure I have a suggestion to improve this notation.
- Please avoid unnecessary repetitions. In a number of cases a big part of a formula is defined inline in the text just before the equation. This doesn’t add to the conversation and makes it unnecessarily hard to parse the text (e.g., inline formulas just before Eqns. 4 and 6. Also prefer a textual description instead of the variable when introducing a definition (e.g., before Eqn. 7, “we define the distance as” or “we define the distance between the functions f and f_t embeddings as”)
- Please fix all citations according to the ICLR style guidelines: When the authors or the publication are included in the sentence, the citation should not be in parenthesis (as in ``See \citet{Hinton06} for more information.''). Otherwise, the citation should be in parenthesis (as in ``Deep learning shows promise to make progress towards AI~\citep{Bengio+chapter2007}.''). For instance, all citations in Sec 2.2 should use \citep.
2) To my best understanding classification in the embedding space is performed with some variation of k-means, with k being the number of “prototypes” for the task at hand (i.e., the number of classes for the specific task). If this is correct, referring to k-means in the paper and highlighting the similarities and differences to it, would have been a more effective way to describe your method.
3) The authors claim not to use a final classification, as opposed to competing methods, hence not being subject to memory usage increases when the number of classes increase (sec 2.1). Surprisingly though, they then employ a softmax over the prototypes (Eqn. 3) which to the best of my knowledge would require an incoming layer whose dimensionality would need to grow with the increase of the number of prototypes, (i.e., classes). Please update the text to either correct the claim or explain exactly how your method can drive the softmax in the case of an increasing number of classes without increasing the dimensionality of the layer before the softmax.
4) Similarly, the authors claim that “Unlike parameter regularization methods or iCARL or FSR, our approach further reduces the memory storage by replacing logits of each data or network parameters with one prototype of each class in the episodic memory”. This is not entirely true, as they also need to store a (unspecified) number of samples from all tasks to prevent forgetting. It is unclear that replacing a snapshot of the network parameters is better (in terms of memory consumption) than storing some data from each task, and a proper discussion on this crucial point of the paper is missing.
5) Apart from the losses, all the details on the architecture (type and number of layers, activation functions, ..) and on the hyperparameters are missing. The paper only mentions that the same feedforward architecture as EWC is used, with similar memory usage, but also mentions that unlike all other methods (including EWC I assume), theirs doesn’t need to vary the architecture if the number of tasks is increased. It is then completely unclear to me how this is possible if the architecture is the same. If it’s not, which parts of the network are shared and which are task specific. I assume most of the network to be shared among tasks and only the last layer (that drives the softmax that determines the probabilities of each class (prototype) for the current task) to be task-specific. This details are crucial to the understanding of the method and to be able to reproduce the results.
6) Crucially, the reported Permuted MNIST performance on EWC is much worse than that on the original paper. This suggests that the baselines have not been properly tuned or there are issues in the implementation.
7) Unless I am mistaken, in Sec 3.3 there seems to be a flaw in the calculation of the number of examples that can be stored by the proposed method in order to use approximately the same memory as the baselines. Indeed, the memory allocation for weight regularization is independent of the number of classes and so should be the equivalent capacity allocated for samples retention, i.e., in order to be comparable with EWC (and similar methods), the proposed model should be allowed to store 530 examples in total, rather than per class.
8) I suggest to drop the definitions of the obvious functions, such as the softmax and the NLL (that is even defined twice!), and use that space to improve the description of the model and algorithm instead.
9) Can you clarify what you mean by “The pairwise distance of one embedding and one prototype within the same class should be smaller than the intra-class ones.”? Should it be inter-class?
10) In my opinion the narrative would be clearer backwards: rather than first defining the loss as a sum of three sub-losses, I’d suggest to define the sub-losses first and finally aggregate them. In its current state, Eqn. 1 is not informative until \delta_{D_t} is defined, a full page later.
11) Eqns. 3 and 5 are not “distance distributions” but rather “class probabilities distributions”, if I understood correctly. Can you confirm and amend the text?
12) Eqn. 5 is more confusing than informative, I recommend to remove it and replace it with a better textual description. There is no need to redefine c_{kt}, as it is already defined in Eqn. 2 exactly in the same way. Secondly, there is no need to define the softmax over the distances a second time, as it’s already defined in Eqn. 3 and only the arguments change. 
13) Re Sec 2.3, to my best understanding there is no “emphasis on reviewing earlier tasks”, as the number of rehearsal samples is the same for all tasks. In other words, \tilde{D}_t is sampled uniformly across tasks I believe. Is this right?
14) Sec 3.1, “typical continual learning schemes assume that a large amount of training data over all tasks is always available for fine-tuning”. This is not true, and it’s actually usually not the case  (see e.g., generative replay approaches, or those that rely on pruning or other forms of architecture modification.)
15) The introduction fails to mention the category of approaches that try to combat forgetting by being smart about the architecture, e.g., Progressive nets, Progress & compress, etc, .. Please amend this.
Minor:
- Add more space after Figure 1’s caption, so that it’s clear where the caption ends and the text begins.
- Eqn. 2, no need to define yiT since it’s not used in the equation.
- The sentence at the end of Sec 2.1 (“In practice … estimating the distance distribution” ..) should go after Eqn. 2. Similarly, the sentence “Our primary choice of  the distance function” would be better placed right after the definition of the distance function IMO.
- The MNIST citation is wrong. Please correct it: LeCun, Yann and Cortes, Corinna. "MNIST handwritten digit database." (1998)
- The introduction is a bit long and a “related work” section is missing. You might want to split the introduction into two chapters.
- Do not use D to denote the datasets and the dimensionality of the datasets.
- The sentence after Eqn. 1 “learning f_T requires minimizing both terms ..” is incorrect, as it goes against Eqn. 1 in which there are 3 terms rather than 2.
- End of page 3: “without loss of generality”...w.r.t. what? The phrase doesn’t seem to fit the context, consider changing it.
- End of page 7, the detailed description of Figure 3 would be better placed in the caption of the figure itself.
Typos: 
- Page 2: state-of-the-arts -> state-of-the-art
- “There have been some attempts [citations] selecting”, “of” is missing
- End of page 4: which is pre-computed -> are
- End of page 5: public -> publicly