Paper:
Modern deep artificial neural networks have achieved impressive results through models with very large capacity-compared to the number of training examplesthat control overfitting with the help of different forms of regularization. Regularization can be implicit, as is the case of stochastic gradient descent or parameter sharing in convolutional layers, or explicit. Most common explicit regularization techniques, such as dropout and weight decay, reduce the effective capacity of the model and typically require the use of deeper and wider architectures to compensate for the reduced capacity. Although these techniques have been proven successful in terms of results, they seem to waste capacity. In contrast, data augmentation techniques reduce the generalization error by increasing the number of training examples and without reducing the effective capacity. In this paper we systematically analyze the effect of data augmentation on some popular architectures and conclude that data augmentation alone-without any other explicit regularization techniques-can achieve the same performance or higher as regularized models, especially when training with fewer examples.

Review:
The paper proposes data augmentation as an alternative to commonly used regularisation techniques like weight decay and dropout, and shows for a few reference models / tasks that the same generalization performance can be achieved using only data augmentation.
I think it's a great idea to investigate the effects of data augmentation more thoroughly. While it is a technique that is often used in literature, there hasn't really been any work that provides rigorous comparisons with alternative approaches and insights into its inner workings. Unfortunately I feel that this paper falls short of achieving this.
Experiments are conducted on two fairly similar tasks (image classification on CIFAR-10 and CIFAR-100), with two different network architectures. This is a bit meager to be able to draw general conclusions about the properties of data augmentation. Given that this work tries to provide insight into an existing common practice, I think it is fair to expect a much stronger experimental section. In section 2.1.1 it is stated that this was a conscious choice because simplicity would lead to clearer conclusions, but I think the conclusions would be much more valuable if variety was the objective instead of simplicity, and if larger-scale tasks were also considered.
Another concern is that the narrative of the paper pits augmentation against all other regularisation techniques, whereas more typically these will be used in conjunction. It is however very interesting that some of the results show that augmentation alone can sometimes be enough.
I think extending the analysis to larger datasets such as ImageNet, as is suggested at the end of section 3, and probably also to different problems than image classification, is going to be essential to ensure that the conclusions drawn hold weight.
Comments:
- The distinction between "explicit" and "implicit" regularisation is never clearly enunciated. A bunch of examples are given for both, but I found it tricky to understand the difference from those. Initially I thought it reflected the intention behind the use of a given technique; i.e. weight decay is explicit because clearly regularisation is its primary purpose -- whereas batch normalisation is implicit because its regularisation properties are actually a side effect. However, the paper then goes on to treat data augmentation as distinct from other explicit regularisation techniques, so I guess this is not the intended meaning. Please clarify this, as the terms crop up quite often throughout the paper. I suspect that the distinction is somewhat arbitrary and not that meaningful.
- In the abstract, it is already implied that data augmentation is superior to certain other regularisation techniques because it doesn't actually reduce the capacity of the model. But this ignores the fact that some of the model's excess capacity will be used to model out-of-distribution data (w.r.t. the original training distribution) instead. Data augmentation always modifies the distribution of the training data. I don't think it makes sense to imply that this is always preferable over reducing model capacity explicitly. This claim is referred to a few times throughout the work.
- It could be more clearly stated that the reason for the regularising effect of batch normalisation is the noise in the batch estimates for mean and variance.
- Some parts of the introduction could be removed because they are obvious, at least to an ICLR audience (like "the model would not be regularised if alpha (the regularisation parameter) equals 0").
- The experiments with smaller dataset sizes would be more interesting if smaller percentages were used. 50% / 80% / 100% are all on the same order of magnitude and this setting is not very realistic. In practice, when a dataset is "too small" to be able to train a network that solves a problem reliably, it will generally be one or more orders of magnitude too small, not 2x too small.
- The choices of hyperparameters for "light" and "heavy" motivation seem somewhat arbitrary and are not well motivated. Some parameters which are sampled uniformly at random should be probably be sampled log-uniformly instead, because they represent scale factors. It should also be noted that much more extreme augmentation strategies have been used for this particular task in literature, in combination with padding (for example by Graham). It would be interesting to include this setting in the experiments as well.
- On page 7 it is stated that "when combined with explicit regularization, the results are much worse than without it", but these results are omitted from the table. This is unfortunate because it is a very interesting observation, that runs counter to the common practice of combining all these regularisation techniques together (e.g. L2 + dropout + data augmentation is a common combination). Delving deeper into this could make the paper a lot stronger.
- It is not entirely true that augmentation parameters depend only on the training data and not the architecture (last paragraph of section 2.4). Clearly more elaborate architectures benefit more from data augmentation, and might need heavier augmentation to perform optimally because they are more prone to overfitting (this is in fact stated earlier on in the paper as well). It is of course true that these hyperparameters tend to be much more robust to architecture changes than those of other regularisation techniques such as dropout and weight decay. This increased robustness is definitely useful and I think this is also adequately demonstrated in the experiments.
- Phrases like "implicit regularization operates more effectively at capturing reality" are too vague to be meaningful.
- Note that weight decay has also been found to have side effects related to optimization (e.g. in "Imagenet classification with deep convolutional neural networks", Krizhevsky et al.)
REVISION: I applaud the effort the authors have put in to address many of my and the other reviewers' comments. I think they have done so adequately for the most part, so I've decided to raise the rating from 3 to 5, for what it's worth.
The reason I have decided not to raise it beyond that, is that I still feel that for a paper like this, which studies an existing technique in detail, the experimental side needs to be significantly stronger. While ImageNet experiments may be a lot of work, some other (smaller) additional datasets would also have provided more interesting evidence. CIFAR-10 and CIFAR-100 are so similar that they may as well be considered variants of the same dataset, at least in the setting where they are used here.
I do really appreciate the variety in the experiments in terms of network architectures, regularisation techniques, etc. but I think for real-world relevance, variety in problem settings (i.e. datasets) is simply much more important. I think it would be fine if additional experiments on other datasets were not varied along all these other axes, to cut down on the amount of work this would involve. But not including them at all unfortunately makes the results much less impactful.