Paper:
Batch normalization (BN) is often used in an attempt to stabilize and accelerate training in deep neural networks. In many cases it indeed decreases the number of parameter updates required to achieve low training error. However, it also reduces robustness to small adversarial input perturbations and common corruptions by double-digit percentages, as we show on five standard datasets. Furthermore, we find that substituting weight decay for BN is sufficient to nullify a relationship between adversarial vulnerability and the input dimension. A recent mean-field analysis found that BN induces gradient explosion when used on multiple layers, but this does not fully explain the vulnerability we observe, given that it occurs for a single BN layer. We argue that the main cause is the tilting of the decision boundary with respect to the nearest-centroid classifier along input dimensions of low variance. As a result, the numerical stability constant of BN acts as an important meta-parameter that can be tuned to recover some robustness at the cost of standard test accuracy. We explain this mechanism explicitly for linear models but find that it still holds for nonlinear models.

Review:
This paper identifies an important weakness of batch normalization: it increases adversarial vulnerability. It is very well written and the claims are theoretically sound. In the experiments, the authors demonstrated a significant difference in robustness between networks with or without batch normalization layers, in varies settings against both random input noise and adversarial noise. This weakness of batch norm was explained due to the "decision boundary tilting" effect caused by the normalization. Overall, this paper has done solid work to reveal an interesting phenomenon. If it is true, this finding will impact almost all DNN models. 
My concern is that this phenomenon is just another effect of "gradient masking " (as pointed out by Athalye, et al.). Batch norm is a well-known technique to avoid overfitting, without batch norm the network can be easily trained to be saturated with almost zero gradients, demonstrating a false signal of "robustness" to noise. The random noise and real-world corruption experiments are definitely helpful to clear this doubt, but only partially. My concern remains because of two obvious signs of  gradient masking: 
1. The accuracy on PGD-li (epsilon=0.031) attacks are suspiciously too high (20% - 40% Table 3/4). For this level of attack, the acc should be nearly zero. This is likely caused by the gradient masking effect, considering the cifar-10 networks were trained for longer time with larger learning rate (150 epochs, fixed lr 0.01). Training on MNIST is much easier to get zero gradients.  
2. The weight decay discussion is not helpful at all, on the contrary, it confirms my concern on the gradient masking effect. In Table 8, the robustness was increased ~40% by just using large weight decay. This is not the "real robustness", and can be easily evaded by adaptive attack (see Athalye's paper).
With the above two concerns in mind, I doubt the phenomenon revealed in this paper is just "one can easily train a saturated model without batch norm" or equivalently "it's hard to train a saturated model with batch norm". It is hard to say if this is a bad thing for batch norm.
I am quite surprised that the authors ignore this completely. Here are a few things that can be done to rule out the possibility of gradient masking. The masked gradient can be identified by: 1) One-step attacks perform better than iterative attacks; 2) Unbounded attacks do not reach 100% success., etc (see Section 3.1 of Athalye's paper).
1. Including FGSM in the experiments and show the same trends as PGD-li. 
2. Show two networks have similar gradient norms.
3. Apply cw-l2 attack, and show batch norm has forced large perturbation.
Two other suggestions:
1. Summarize the different angles/steps taken to verify the phenomenon, somewhere before the experiments.
2. Cannot see why the input dimension discussion contribute to explanations of the batch norm weakness.
============
My rating stays the same after rebuttal. 
My original concerns are like the other reviewers: why BN, not other techniques such as structure of DNNs MLP vs CNN vs ResNet, activation functions, weight decay, learning rates, softmax etc. My initial suspect was that it is caused by gradient masking likely caused by the l2 weight regularization, so asked the authors to look at the gradient norms and run some testes to rule this out. Yes, the weight norm is directly related to the Lipschitz continuity of the function represented by the network, but it often becomes more complicated on complex nonlinear neural networks. 
According to the new experiment results, the vulnerability is indeed not an effect of gradient masking, thanks for the clarification. However, the new results also indicate that the finding is susceptible to both weight decay and learning rate: in Figure 16 (a): "Un PGD" < "BN PGD" before learning rate decay, andFigure 17 (a) vs (b), doubling the weight decay penalty to 1e-3 also increases the vulnerability of BN. Overall, I believe the phenomenon exists, but the reasons behind requires more explanations, at least not just the batch norm.