Paper:
As deep CNN classifier performance using ground-truth labels has begun to asymptote at near-perfect levels, a key aim for the field is to extend training paradigms to capture further useful structure in natural image data and improve model robustness and generalization. In this paper, we present a novel natural image benchmark for making this extension, which we call CIFAR10H. This new dataset comprises a human-derived, full distribution over labels for each image of the CIFAR10 test set, offering the ability to assess the generalization of stateof-the-art CIFAR10 models, as well as investigate the effects of including this information in model training. We show that classification models trained on CIFAR10 do not generalize as well to our dataset as it does to traditional extensions, and that models fine-tuned using our label information are able to generalize better to related datasets, complement popular data augmentation schemes, and provide robustness to adversarial attacks. We explain these improvements in terms of better empirical approximations to the expected loss function over natural images and their categories in the visual world.

Review:
The authors create a new dataset with label distributions (rather than one-hot annotations) for the CIFAR-10 test set. They then study the effect of fine-tuning using this dataset on the generalization performance of SOTA deep networks. They also study the effects on adversarial robustness.
I think that datasets such as the one generated in this paper could indeed be a valuable testbed to study deep network generalization and robustness. There are many nice benefits of label distributions over one hot labels (that the authors summarize in Section 2.) The paper is also clear and well-written. 
That being said, I do not find the investigation of this paper completely satisfactory. For instance in the generalization experiments, the numbers presented seem to show some interesting (and somewhat surprising) trends, however the authors do not really pursue these or provide any insight as to why this is the case. I also find the section on robustness very weak.
Detailed comments:
- The theoretical contribution mentioned in the appendix does not really seem to be a contribution - it is just a simple derivation of the loss under label distributions. Theoretical contributions are not necessary for a paper to have merit - the authors should remove this statement from the introduction as it detracts from the value of the paper.
- I find it somewhat surprising that the accuracy of the models does not change on training with Cifar10H. Do the authors have any intuition as to why this is the case? The model cross entropy seems to go down, indicating that probability assigned to the correct class increases. I would think that training with label distributions would actually reduce misclassification on confusing instances. It would be interesting to see how the logit distributions change for different examples. For instance, how does the model confidence change on correctly vs wrongly classified examples?
- The authors mention that they run each hyperparameter configuration for three random seeds. It would be nice then to see error bars for the results reported Tables 1 and 2, particularly because the differences in accuracy are small. Did the authors try different train-test splits of the test set? It would also be helpful if the authors could make plots for the results in these tables (at least in the appendix). It is hard to compare numbers across different tables.
-I find the results in Table 2 confusing. Comparing the numbers to Table 1, it seems that mixup does not really change accuracies/loss. The model names in Table 2 do not exactly match Table 1 so it is hard to identify the additional gain from using mixup that the authors mention. The authors should add plots for these results to illustrate the effect of adding mixup more clearly.
-I am not convinced by the section on robustness. Firstly, it is not clear to me why the authors chose FGSM which is known to be a somewhat simple attack to illustrate improved robustness of their model. To perform a useful study of robustness, the authors should study SOTA attacks such as PGD [Madry et al., 2017]. I also do not understand the claim that the top-1 choice becomes less confident after training with CIFAR10H -- this seems to be contradicted by the fact that the cross entropy loss goes down. The authors should provide supporting evidence for this claim by looking at changes in confidence (see point 3 above). Also, the comment about the trade-off between accuracy and robustness seems vague - could the authors clarify what they mean?
Overall, I like the premise of this paper and agree that with the potential benefits of the dataset generated. However, I think that the current experiments are not strong enough to corroborate this.