Paper:
Adversarial input poses a critical problem to deep neural networks (DNN). This problem is more severe in the "black box" setting where an adversary only needs to repeatedly query a DNN to estimate the gradients required to create adversarial examples. Current defense techniques against attacks in this setting are not effective. Thus, in this paper, we present a novel defense technique based on randomization applied to a DNN's output layer. While effective as a defense technique, this approach introduces a trade off between accuracy and robustness. We show that for certain types of randomization, we can bound the probability of introducing errors by carefully setting distributional parameters. For the particular case of finite difference black box attacks, we quantify the error introduced by the defense in the finite difference estimate of the gradient. Lastly, we show empirically that the defense can thwart three adaptive black box adversarial attack algorithms.

Review:
This paper proposes applying randomization to the output layer of a DNN to defend against query-based attacks based on finite difference estimates. Then some theoretical analysis is provided, showing that with perturbation of a suitable scale, the randomization layer will not affect the accuracy of the model, while causing a large estimation error of finite difference methods that prevents finite-difference based attacks. Empirical results verify that the proposed defense is still effective against adaptive attacks where the randomness is averaged.
Pros:
The proposed method is simple, straightforward, yet novel. Its working mechanism is easy to understand and analyze, so it should be useful against finite-difference based attacks.
Limitation:
The proposed method is not useful to defense against white-box attacks and transfer-based attacks, since basically it does not change the predictive model. Some other randomization methods like [26], by contrast, change the predictive model, hence they may be useful against white-box attacks and transfer-based attacks.
Questions and suggestions:
This part is my main concern.
It seems that the experimental results are very good. For example, in Figure 3a, the defense is effective even if \sigma^2<1e-6. However, by the analysis in Section 4.2 (the formula below Line 3, Page 6), when \sigma^2=1e-6, |E[g_i-\gamma_i]| should be rather small, hence it should not block finite-difference based attacks. I think more explanation is needed for the good performance in the experiments.
Finite differences are extremely sensitive to small random perturbation of the function value when the spacing (step size) h is small. For example, g_i=\frac{L(f(x+he_i))-L(f(x-he_i))}{2h}, when h is very small, f(x+he_i) and f(x-he_i) is very close, hence adding perturbation to them will change g_i a lot. To present stronger adaptive attacks to output randomization, my suggestion is that a larger h can be adopted. It will be better if the results are investigated against attacks with different values of h.
A mistake:
In Section 4.1 on Page 4: "we can express the probability that x is misclassified in the vector d(p) as: \sum_{i=2}^C P(d(p_i)>d(p_m))". I think this is wrong, since P(A or B happens)=P(A happens)+P(B happens) only when A and B are mutually exclusive. However, "d(p_i)>d(p_m)" and "d(p_j)>d(p_m)" are not mutually exclusive. Hence, the probability that x is misclassified in the vector should be less than or equal to that sum of probabilities.
By the way, the writing in Section 4.1 is not clear:
- The overall misclassification probability is presented first, but after that K only represents the misclassification probability into a specific class. The connection between them is unclear.
- At the beginning of Section 4.1, the distribution of \epsilon is \epsilon\sim\mathcal{N}(\mu,\sigma^2\cdot\mathbf{I}_C): a unique \sigma is used. But after that, the variance of \epsilon_i becomes \sigma_i^2 instead of \sigma^2.
- In the second to the last line on Page 4, "level of noise (\sigma^2) can be set for each class separately", but the authors did not explain how to set them, and in the experiments \sigma^2 is set as the same scalar.
- In Figure 1b, the line style of "K=5.0e-3" and "K=1.0e-1" in the legend is very similar. The line style of "K=2.0e-01" in the legend is not clear: I do not know whether it refers to "-.-.-." or "------".
Typos:
Section 6, Page 8: "unintentionlly" => "unintentionally"
Some missing spaces after punctuation:
- Section 4.2, Page 5, "... gradient estimate.When the ..." => should add a space before "When"
- Section 5, Page 7, "In addition,input randomization ..." => should add a space before "input"