Paper:
Machine learning (ML) research has investigated prototypes: examples that are representative of the behavior to be learned. We systematically evaluate five methods for identifying prototypes, both ones previously introduced as well as new ones we propose, finding all of them to provide meaningful but different interpretations. Through a human study, we confirm that all five metrics are well matched to human intuition. Examining cases where the metrics disagree offers an informative perspective on the properties of data and algorithms used in learning, with implications for data-corpus construction, efficiency, adversarial robustness, interpretability, and other ML aspects. In particular, we confirm that the "train on hard" curriculum approach can improve accuracy on many datasets and tasks, but that it is strictly worse when there are many mislabeled or ambiguous examples.

Review:
Summary: The paper proposes methods for identifying prototypes. Unfortunately, a formal definition of a prototype is lacking, and the authors instead present a set of heuristics for sorting data points that purport to measure 'prototypicality', although different heuristics have different (and possibly conflicting) notions of what this means. The experiments are not very convincing, and often present results that are either inconclusive or negative, i.e. seem to demonstrate that prototypes are not very useful. 
Pros:
- The notion of prototypes is used in various papers, but a formal definition is lacking, and the usefulness of prototypes is not demonstrated. The fact that this paper sets out to do both is laudable, although the paper needs work before it can be accepted for publications.
Detailed comments / cons:
*Defining prototypes: 
  - The authors list desirable properties before defining (even informally) what a prototype is, and what its purposes are. Taking the first property as an example, is it reasonable to expect a metric for prototypes to be useful for image classification AND image generation? The answer completely depends on what one expects from a prototype, what its purpose is, etc.
  - The second property seems to indicate that prototypes are model-independent, i.e. two models trained on the same dataset will have the same prototypes. This is confusing as the metrics proposed are clearly model-dependent (e.g. adv completely depends on the trained model's decision boundary, conf obviously depends on the model providing the confidence score)
- The third and fourth property are poorly defined. Human intuition presupposes that humans agree on what a prototype means. Using 'modes of prototypical examples' in trying to define a metric for prototypes is circular, as a mode of prototypical example depends on a working notion of prototypical examples.
- The last property is completely dependent on which models are trained, and how they are trained. If a model has high label complexity, maybe it does not achieve high accuracy even when trained on high quality prototypes. In any case, this property is at odds with the first two properties.
In sum: it's not clear what prototypes are, so it becomes hard to judge if the list of desiderata is reasonable. The list is in any case ill-defined, and contains contradictions.
* Metrics for prototypicality
- The second paragraph in this section is unnecessary
- All of the metrics proposed are heuristics with little to no justification. Specific comments below.
- Adversarial robustness is a property of a trained model, not of prototypical examples, unless prototypes are supposed to be model dependent (contra property 1). In any case, it is not clear why examples that are robust to adversarial noise are good 'prototypes'.  Using facial recognition as an example, a 'mean face' may be very robust to adversarial noise but not prototypical at all under common definitions. A face with a particular type of facial hair (e.g. nose hair) may be very representative of a class of faces (i.e. a prototype), but very susceptible to adversarial noise. In fact, any examples in the boundary of the decision function will be more susceptible to adversaries, but that does not make them 'less prototypical'.
- Holdout retraining is again completely model dependent. Why should we expect a model to treat a prototype the same regardless of whether or not it is trained on it? This basically means that we expect the model to always be accurate on prototypes.
- Ensemble agreement proposes a notion of prototypes that is based on prediction 'hardness'. It is clear that such a notion depends completely on which models are being considered, which features are being used, and etc, much more than on notions of prototypicality inherent in the data. The same criticism applies to model confidence.
- Privacy preserving training assumes prototypicality has to do with the model being able to learn with some robustness to noise (related to Adversarial Robustness, but different). This assumes a definition of prototypes that is not congruent with the other metrics.
In sum: the proposed metrics are basically heuristics with little justification, and different metrics assume different notions of what a prototype is.
* Evaluation
- Section 3.1 claims that the metrics are strongly correlated, but that is not true for MNIST or CIFAR, and is somewhat true for fashion-mnist. In any case, since the metrics are so model-dependent, it is not clear if these results would hold if other models were used.
- Section 3.2 - The question asked of turkers in the study is too vague, and borderline irrelevant for the task at hand - what does the 'best image' of an airplane mean, and how does this translate to it being a prototype? All that the study demonstrates is that the proposed metrics score malformed images with low score. The results in Table 1 are very spread out, and seem to indicate a low agreement between the metrics and human evaluation - although Table 1 is almost irrelevant given the question that was asked of users.
- The results in Section 4 are very discouraging: sometimes it is better to train on most prototypical examples according to the metrics, sometimes it is worse, sometimes it's better to take examples in the middle. That is, prototypes don't seem to help at all. 'Prototype percentile' is uncorrelated with robustness for MNIST in Appendix E, while being correlated for other datasets. It is clear why this would be the case for metrics such as confidence, but in general models trained on less examples are less robust than models trained on the whole dataset (again, as expected). As a whole, the results do not provide any help for a user who wants to produce a more robust model, other than 'ignore prototypes and use the whole dataset'.