Paper:
Image classification problems today are often solved by first collecting examples along with candidate labels, second obtaining clean labels from workers, and third training a large, overparameterized deep neural network on the clean examples. The second, manual labeling step is often the most expensive one as it requires manually going through all examples. In this paper we propose to i) skip the manual labeling step entirely, ii) directly train the deep neural network on the noisy candidate labels, and iii) early stop the training to avoid overfitting. With this procedure we exploit an intriguing property of overparameterized neural networks: While they are capable of perfectly fitting the noisy data, gradient descent fits clean labels faster than noisy ones. Thus, training and early stopping on noisy labels resembles training on clean labels only. Our results show that early stopping the training of standard deep networks (such as ResNet-18) on a subset of the Tiny Images dataset (which is obtained without any explicit human labels and only about half of the labels are correct), gives a significantly higher test performance than when trained on the clean CIFAR-10 training dataset (which is obtained by labeling a subset of the Tiny Images dataset). In addition, our results show that the noise generated through the label collection process is not nearly as adversarial for learning as the noise generated by randomly flipping labels, which is the noise most prevalent in works demonstrating noise robustness of neural networks.

Review:
The paper reads well, the topic is interesting, and the connection between early stopping and fitting true label distributions before noisy ones, if generally true, is of potential impact. I do not feel the paper is ready for publication, though: 
Missing context/references:
	⁃	Noisily collected labels are standard elsewhere, e.g., in sentiment analysis (self-ratings), in discourse parsing (explicit discourse markers), in weakly supervised POS tagging (crowdsourced dictionaries), in NER (Wikidata links), and in machine translation from mined parallel corpora or comparable corpora. 
	⁃	Learning a ground truth from a population of turkers (MACE and subsequent work). It is well-known that the noise in such silver standard or non-adjudicated annotations is also “more structured” and to a large extent predictable (Plank et al., EACL 2014 and subsequent work on learning/predicting inter-annotator disagreements). 
	⁃	Connection between L2 and early stopping. Can you replicate your results with simple L2 regularisation?
	⁃	Cost-sensitive learning of systematic disagreements (use class coherence scores for cost-sensitive learning). 
	⁃	The observation that “clean examples are fitted faster than noise” is obviously related to baby steps training regimes (training on easy examples first), including active learning. 
Experimental details and flaws: 
	⁃	Using a black box search engine to collect labels is problematic for a few reasons: (a) Search engines change, so results are hard to reproduce. (b) Search engines are biased toward certain types of categories. Such biases will be reinforced by any model trained on this data. 
	⁃	I’m always a little worried about papers that only report results for a single dataset. 
	⁃	I would have liked to see some more error analysis. Are the improvements on classes with more or less support in the original dataset, for example?
	⁃	It seems to be that it would have been relatively straight-forward to construct synthetic datasets that would more directly evaluate the hypothesis that the true labels are learned first. I realize you’re interested in “real” noise rather than “random” noise - but synthetic noise doesn’t have to be random. 
	⁃	That “the noise generated through the label collection process is not nearly as adversarial for learning as the noise generated by randomly flipping labels” is no surprise, and I would not present this as a finding. It would be interesting to describe the bias, e.g., by presenting confusion matrices (see Plank et al., EACL 2014).