Paper:
Deep neural networks represent data as projections on trained weights in a high dimensional manifold. This is a first-order based absolute representation that is widely used due to its interpretable nature and simple mathematical functionality. However, in the application of visual recognition, first-order representations trained on pristine images have shown a vulnerability to distortions. Visual distortions including imaging acquisition errors and challenging environmental conditions like blur, exposure, snow and frost cause incorrect classification in first-order neural nets. To eliminate vulnerabilities under such distortions, we propose representing data points by their relative positioning in a high dimensional manifold instead of their absolute positions. Such a positioning scheme is based on a data point's second-order property. We obtain a data point's second-order representation by creating adversarial examples to all possible decision boundaries and tracking the movement of corresponding boundaries. We compare our representation against first-order methods and show that there is an increase of more than 14% under severe distortions for . We test the generalizability of the proposed representation on larger networks and on 19 complex and real-world distortions from CIFAR-10-C. Furthermore, we show how our proposed representation can be used as a plug-in approach on top of any network. We also provide methodologies to scale our proposed representation to larger datasets.

Review:
The authors propose an approach to reduce the sensitivity of neural networks to visual distortions. To do so, they modify the representation of a data point within a network, using its relative position (to other points) in representation space rather than its absolute position  (which is measured as distance of a point to the decision boundary of each class). They then evaluate their approach on common corruptions from the CIFAR-C dataset.
The paper addresses an important problem, however I do not find the high-level motivation behind the proposed approach or the experimental results sufficiently convincing. 
In particular, conceptually, it is completely unclear why the second order representations should be more resilient to visual distortions. Even in the sample illustration provided in Figure 2, it is evident that the clusters in this new representation space are not invariant to visual distortion, or even are significantly more invariant than the first-order representation. At a more fundamental level, given that the accuracy of the original network does drop, it is tautological that the distance of a point to the decision boundaries is decreasing under distortion (and thus is sensitive to it).
Experimentally, my chief concerns are:
1. When the authors evaluate the proposed second-order representations, they use networks with additional layers which do not seem to be present in the original baseline. Prior work [Hendrycks and Dietterich, 2019] has shown that model capacity has a marked influence on the corruption robustness of a network. Thus, it is unclear where the improvement here is coming solely from the additional capacity compared to the original model. The baseline network that the authors compare to should also include the additional layers.
2. The results reported for some of the baselines seem inconsistent with prior work. 
a) For example, the authors state that the results for the Hossain et al. [2018] baseline are similar to NR1, which is worse than the original network. However, Hossain et al. report an improvement for the same corruptions and the same dataset in their paper. Where is this inconsistency coming from? Do the authors train with the DCT filtering or is it only applied at test time? If it is the latter, it could explain why the authors fail to reproduce the baseline correctly.
b) For the adversarially robust network baseline, why did the authors choose an FGSM adversary? FGSM is not typically used to train state-of-the-art robust models because of the existence of much stronger attacks (such as PGD). How was the eps used to train the network chosen? In prior work, Kang et al. [2019; arxiv:1908.0801] evaluate both L2 and Linf robust models and show improvements over the baseline for several common corruptions. This seems to suggest that the robust model baseline reported in this paper is not accurate/representative.
3. Moreover, given that the representation size scales with number of classes, the proposed method should be evaluated on datasets with more classes such as CIFAR-100 or ImageNet. Improvements demonstrated in these settings with dimensionality reduction would be more convincing.
Overall, my main reservations are: a) the lack of a conceptual justification for the proposed approach, and b) issues with the experimental evaluation, particularly in the reported baselines and how they seem to contradict prior work. Thus, I recommend rejection.
Other comments:
- The right-side of Figure 1 is essentially the same map flipped.
- Why was the MSE loss chosen for J? Given that the network was presumably trained with cross-entropy, this choice seems somewhat arbitrary. Are the results consistent for cross entropy loss as well? The authors should include these results in the appendix.
- Figure 4 is very hard to read---the authors should change the plotting style to make the results more legible. 
- For the results in Figure 5, the authors should once again compare to adding extra layers to the baseline networks as well.
- In Table 1, why does the performance of NR improve when the features are based only on gradients for the last layer?