Paper:
EFFICIENT AUGMENTATION VIA DATA SUBSAMPLING
Data augmentation is commonly used to encode invariances in learning methods. However, this process is often performed in an inefficient manner, as artificial examples are created by applying a number of transformations to all points in the training set. The resulting explosion of the dataset size can be an issue in terms of storage and training costs, as well as in selecting and tuning the optimal set of transformations to apply. In this work, we demonstrate that it is possible to significantly reduce the number of data points included in data augmentation while realizing the same accuracy and invariance benefits of augmenting the entire dataset. We propose a novel set of subsampling policies, based on model influence and loss, that can achieve a 90% reduction in augmentation set size while maintaining the accuracy gains of standard data augmentation.

Review:
Summary: The authors study the problem of identifying subsampling strategies for data augmentation, primarily for encoding invariances in learning methods. The problem seems relevant with applications to learning invariances as well as close connections with the covariate shift problem. 
Contributions: The key contributions include the proposal of strategies based on model influence and loss as well as empirical benchmarking of the proposed methods on vision datasets. 
Clarity: While the paper is written well and is easily accessible, the plots and the numbers in the tables were a bit small and thereby hard to read. I would suggest the authors to have bigger plots and tables in future revisions to ensure readability. 
>> The authors mention in Section 4.1 that "support vector are points with non-zero loss": In all generality, this statement seems to be incorrect. For example, even for linearly separable data, a linear SVM would have support vectors which are correctly classified. 
>> The experiment section seems to be missing a table on the statistics of the datasets used: This is important to understand the class distribution in the datasets used and if at all there was label imbalance in any of them. It looks like all the datasets used for experimentation had almost balanced class labels and in order to fully understand the scope of these sampling strategies, I would suggest the authors to also provide results on class imbalanced datasets where the distribution over labels is non-uniform. 
>> Incomprehensive comparison with benchmarks: 
a) The comparison of their methods with VSV benchmark seems incomplete. While the authors used the obtained support vectors as the augmentation set and argued that it is of fixed size, a natural way to extend these to any support size is to instead use margin based sampling where the margins are obtained from the trained SVM since these are inherently margin maximizing classifiers. Low margin points are likely to be more influential than high margin points.
b) In Section 5.3, a key takeaway is "diversity and removing redundancy is key in learning invariances". This leads to possibly other benchmarks to which the proposed policies could be compared, for example those based on Determinantal point processes (DPP) which are known for inducing diversity in subset selection. There is a large literature on sampling diverse subsets (based on submodular notions of diversity) which seems to be missing from comparisons. Another possible way to overcome this would be to use stratified sampling to promote equal representation amongst all classes. 
c) In Section 2, it is mentioned that general methods for dataset reduction are orthogonal to the class of methods considered in this paper. However, on looking at the data augmentation problem as that of using fewest samples possible to learn a new invariance, it can be reduced to a dataset reduction problem. One way of using these reduction methods is to use the selected set of datapoints as the augmentation set and compare their performance. This would provide another set of benchmarks to which proposed methods should be compared.
>> Accuracy Metrics: While the authors look at the overall accuracy of the learnt classifiers, in order to understand the efficacy of the proposed sampling methods at learning invariances, it would be helpful to see the performance numbers separately on the original dataset as well as the transformed dataset using the various transformations. 
>> Experiments in other domains: The proposed schemes seem to be general enough to be applicable to domains other than computer vision. Since the focus of the paper is the proposal of general sampling strategies, it would be good to compare them to baselines on other domains possibly text datasets or audio datasets.