Original: E XPLORING THE S PACE OF B LACK - BOX A TTACKS ON D EEP N EURAL N ETWORKS
Revised: E XPLORING THE S PACE OF B LACK - BOX A TTACKS ON D EEP N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Existing black-box attacks on deep neural networks (DNNs) so far have largely focused on transferability, where an adversarial instance generated for a locally trained model can “transfer" to attack other learning models.
Revised: Existing black-box attacks on deep neural networks (DNNs) so far have largely focused on transferability, where an adversarial instance generated for a locally trained model can “transfer" to attack other learning models.

Original: In this paper, we propose novel Gradient Estimation black-box attacks for adversaries with query access to the target model, which do not rely on transferability.
Revised: In this paper, we propose novel Gradient Estimation black-box attacks for adversaries with query access to the target model, which do not rely on transferability.

Original: We also propose strategies to reduce the number of queries required to generate each adversarial sample to a constant.
Revised: We also propose strategies to reduce the number of queries required to generate each adversarial sample to a constant.

Original: An iterative variant of our attack achieves close to 100% adversarial success rates for both targeted and untargeted attacks on DNNs.
Revised: An iterative variant of our attack achieves close to 100% adversarial success rates for both targeted and untargeted attacks on DNNs.

Original: We carry out extensive experiments for a thorough comparative evaluation of black-box attacks and show that the proposed Gradient Estimation attacks outperform all transferability based black-box attacks on both MNIST and CIFAR-10 datasets, achieving adversarial success rates similar to white-box attacks.
Revised: We carry out extensive experiments for a thorough comparative evaluation of black-box attacks, and show that the proposed Gradient Estimation attacks outperform all transferability based black-box attacks on both MNIST and CIFAR-10 datasets achieving adversarial success rates similar to white-box attacks.

Original: We also apply the Gradient Estimation attacks against a real-world content moderation classiﬁer hosted by Clarifai.
Revised: We also apply the Gradient Estimation attacks against a real-world content moderation classiﬁer hosted by Clarifai.

Original: Furthermore, we evaluate black-box attacks against state-of-theart defenses.
Revised: Furthermore, we evaluate black-box attacks against state-of-theart defenses

Original: We show that the Gradient Estimation attacks are very effective even against these defenses.
Revised: We show that the Gradient Estimation attacks are very effective even against these defenses.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The ubiquity of machine learning provides adversaries with both opportunities and incentives to develop strategic approaches to fool learning systems and achieve their malicious goals.
Revised: The ubiquity of machine learning provides adversaries with both opportunities and incentives to develop strategic approaches to fool learning systems and achieve their malicious goals.

Original: Many attack strategies devised so far to generate adversarial examples to fool learning systems have been in the white-box setting, where adversaries are assumed to have access to the learning model (Szegedy et al.
Revised: Many attack strategies devised so far to generate adversarial examples to fool learning systems have been in the white-box setting, where adversaries are assumed to have access to the learning model (Szegedy et al.

Original: ; Goodfellow et al.
Revised: ; Goodfellow et al.

Original: ; Carlini & Wagner (2017); Moosavi-Dezfooli et al.
Revised: ; Carlini & Wagner (2017); Moosavi-Dezfooli et al.

Original: However, in many realistic settings, adversaries may only have black-box access to the model, i.e. they have no knowledge about the details of the learning system such as its parameters, but they may have query access to the model’s predictions on input samples, including conﬁdence scores.
Revised: However, in many realistic settings, adversaries may only have black-box access to the model, i.e. they have no knowledge about the details of the learning system such as its parameters, but they may have query access to the model’s predictions on input samples, including conﬁdence scores.

Original: For example, we ﬁnd this to be the case in some popular commercial AI offerings, such as those from
Revised: For example, we ﬁnd this to be the case in some popular commercial AI offerings, such as those from

Original: IBM, Google and Clarifai.
Revised: IBM, Google and Clarifai.

Original: Most existing black-box attacks on DNNs have focused on transferability based attacks (Papernot et al.
Revised: Most existing black-box attacks on DNNs have focused on transferability based attacks (Papernot et al.

Original: (2016); Moosavi-Dezfooli et al.
Revised: (2016); Moosavi-Dezfooli et al.

Original: Papernot et al.
Revised: Papernot et al.

Original: (2017)), where adversarial examples crafted for a local surrogate model can be used to attack the target model to which the adversary has no direct access.
Revised: (2017)), where adversarial examples crafted for a local surrogate model can be used to attack the target model to which the adversary has no direct access.

Original: The exploration of other black-box attack strategies is thus somewhat lacking so far in the literature.
Revised: The exploration of other black-box attack strategies is thus somewhat lacking so far in the literature.

Original: In this paper, we design powerful new black-box attacks using limited query access to learning systems which achieve adversarial success rates close to that of white-box attacks.
Revised: In this paper, we design powerful new black-box attacks using limited query access to learning systems which achieve adversarial success rates close to that of white-box attacks.

Original: These black-box attacks help us understand the extent of the threat posed to deployed systems by adversarial samples.
Revised: These black-box attacks help us understand the extent of the threat posed to deployed systems by adversarial samples.

Original: The code to reproduce our results can be found at https://github.com/anonymous 1 .

Original: New black-box attack strategies.
Revised: New black-box attack strategies.

Original: We propose novel Gradient Estimation attacks on DNNs, where the adversary is only assumed to have query access to the target model.
Revised: We propose novel Gradient Estimation attacks on DNNs, where the adversary is only assumed to have query access to the target model.

Original: These attacks do not need any access to a representative dataset or any knowledge of the target model architecture.
Revised: These attacks do not need any access to a representative dataset or any knowledge of the target model architecture.

Original: In the Gradient
Revised: In the Gradient

Original: Estimation attacks, the adversary adds perturbations proportional to the estimated gradient , instead of the true gradient as in white-box attacks (Goodfellow et al.
Revised: Estimation attacks, the adversary adds perturbations proportional to the estimated gradient , instead of the true gradient as in white-box attacks (Goodfellow et al.

Original: ; Kurakin et al.
Revised: ; Kurakin et al.

Original: Since the direct Gradient Estimation attack requires a number of queries on the order of the dimension of the input, we explore strategies for reducing the number of queries to the target model.
Revised: Since the direct Gradient Estimation attack requires a number of queries on the order of the dimension of the input, we explore strategies for reducing the number of queries to the target model.

Original: We propose two strategies: random feature grouping and principal component analysis (PCA) based query reduction .
Revised: We propose two strategies: random feature grouping and principal component analysis (PCA) based query reduction .

Original: In our experiments with the Gradient Estimation attacks on state-of-the-art models on MNIST ( dimensions) and CIFAR-10 (3072 dimensions) datasets, we ﬁnd that they match white-box attack performance, achieving attack success rates up to 90% for single-step attacks in the untargeted case and up to 100% for iterative attacks in both targeted and untargeted cases.
Revised: In our experiments with the Gradient Estimation attacks on state-of-the-art models on MNIST ( dimensions) and CIFAR-10 (3072 dimensions) datasets, we ﬁnd that they match white-box attack performance, achieving attack success rates up to 90% for single-step attacks in the untargeted case and up to 100% for iterative attacks in both targeted and untargeted cases.

Original: We achieve this performance with just 100 to 400 queries per sample for single-step attacks and around 4000 queries for iterative attacks.
Revised: We achieve this performance with just 100 to 400 queries per sample for single-step attacks and around 4,000 queries for iterative attacks.

Original: A further advantage of the Gradient Estimation attack is that it does not require the adversary to train a local model, which could be an expensive and complex process on real-world datasets.
Revised: A further advantage of the Gradient Estimation attack is that it does not require the adversary to train a local model, which could be an expensive and complex process on real-world datasets, and training such a local model may require even more queries based on the training data.

Original: Attacking real-world systems .
Revised: Attacking real-world systems .

Original: To demonstrate the effectiveness of our Gradient Estimation attacks in the real world, we also carry out a practical black-box attack using these methods against the
Revised: To demonstrate the effectiveness of our Gradient Estimation attacks in the real world, we also carry out a practical black-box attack using these methods against the

Original: Not Safe For Work (NSFW) classiﬁcation and Content Moderation models deployed by Clarifai, which we choose due to their socially relevant application.
Revised: Not Safe For Work (NSFW) classiﬁcation and Content Moderation models deployed by Clarifai, which we choose due to their socially relevant application.

Original: These models have begun to be developed for real-world moderation (Clarifai-Disqus-2017), which makes such black-box attacks especially pernicious.
Revised: These models have begun to be developed for real-world moderation (Clarifai-Disqus-2017), which makes such black-box attacks especially pernicious.

Original: We carry out these attacks with no knowledge of the training set .
Revised: We carry out these attacks with no knowledge of the training set .

Original: We have demonstrated successful attacks (Figure 1) with just around 200 queries per image, taking around a minute per image.
Revised: We have demonstrated successful attacks (Figure 1) with just around 200 queries per image, taking around a minute per image.

Original: In Figure 1, the target model classiﬁes the adversarial image as ‘safe’ with high conﬁdence, in spite of the content that had to be moderated still being clearly visible.
Revised: In Figure 1, the target model classiﬁes the adversarial image as ‘safe’ with high conﬁdence, in spite of the content that had to be moderated still being clearly visible.

Original: We note here that due to the nature of the images we experiment with, we only show one example here, as the others may be offensive to readers.
Revised: We note here that due to the nature of the images we experiment with, we only show one example here, as the others may be offensive to readers.

Original: The full set of images and their classiﬁcation can be found at https://anonymous.github.io/blackbox-attacks/ 2 .
Revised: The full set of images and their classiﬁcation can be found at https:// anonymous.github.io/blackbox-attacks/ .

Original:  We hope that our work encourages further development of effective black-box defenses for real-world ML systems.
Revised: 1 We hope that our work encourages further development of effective black -box defenses for real-world ML systems.

Original: Comparative evaluation of black-box attacks.
Revised: Comparative evaluation of black-box attacks.

Original: We carry out a thorough empirical comparison of various black-box attacks (given in Table 6) on both MNIST and CIFAR-10 datasets.
Revised: We carry out a thorough empirical comparison of various black-box attacks (given in Table 6) on both MNIST and CIFAR-10 datasets.

Original: We study attacks that require zero queries to the learning model, including the addition of perturbations that are either random or proportional to the difference of means of the original and targeted classes, as well as various transferability based black-box attacks.
Revised: We study attacks that require zero queries to the learning model, including the addition of perturbations that are either random or proportional to the difference of means of the original and targeted classes, as well as various transferability based black-box attacks.

Original: We show that the proposed Gradient
Revised: We show that the proposed Gradient

Original: Estimation attacks outperform other black-box attacks in terms of attack success rate, and achieve results comparable with white-box attacks.
Revised: Estimation attacks outperform other black-box attacks in terms of attack success rate, and achieve results comparable with white-box attacks.

Original: In addition, we also evaluate the effectiveness of these attacks on DNNs made more robust using adversarial training (Goodfellow et al.
Revised: In addition, we also evaluate the effectiveness of these attacks on DNNs made more robust using adversarial training (Goodfellow et al.

Original: ; Szegedy et al.
Revised: ; Szegedy et al.

Original: (2014)), and its recent variants including ensemble adversarial training (Tramèr et al. (2017a))
Revised: (2014)), and its recent variants including ensemble adversarial training (Tramèr et al. (2017a))

Original: and iterative adversarial training (M ˛adry et al.
Revised: and iterative adversarial training (M ˛adry et al.

Original: We ﬁnd that although standard and ensemble adversarial training confer some robustness against single-step attacks, they are vulnerable to iterative Gradient Estimation attacks, with adversarial success rates in excess of 70% for both targeted and untargeted attacks.
Revised: We ﬁnd that although standard and ensemble adversarial training confer some robustness against single-step attacks, they are vulnerable to iterative Gradient Estimation attacks, with adversarial success rates in excess of 70% for both targeted and untargeted attacks.

Original: We ﬁnd that our methods outperform other black-box attacks and achieve performance comparable to white-box attacks.
Revised: We ﬁnd that our methods outperform other black-box attacks and achieve performance comparable to white-box attacks.

Original: In summary, our contributions include:
Revised: In summary, our contributions include:

Original: We propose new Gradient Estimation black-box attacks using queries to the target model.
Revised: We propose new Gradient Estimation black-box attacks using queries to the target model.

Original: We also investigate two methods to reduce the number of queries needed to a small constant for every sample.
Revised: We also investigate two methods to reduce the number of queries needed to a small constant for every sample.

Original: We conduct a thorough evaluation of 10 different black-box attack strategies on state-ofthe-art classiﬁers on the MNIST and CIFAR-10 datasets, and we ﬁnd that with a small number of queries, our proposed Gradient Estimation attacks outperform transferability based attacks and achieve attack success rates matching those of white-box attacks.
Revised: We conduct a thorough evaluation of 10 different black-box attack strategies on state-ofthe-art classiﬁers on the MNIST and CIFAR-10 datasets, and we ﬁnd that with a small number of queries, our proposed Gradient Estimation attacks outperform transferability based attacks and achieve attack success rates matching those of white-box attacks.

Original: We carry out practical black-box attacks on Clarifai’s Not Safe
Revised: We carry out practical black-box attacks on Clarifai’s Not Safe

Original: For Work (NSFW) classiﬁcation and Content Moderation models through public APIs, and show that the adversarial examples can mislead these models with high conﬁdence.
Revised: For Work (NSFW) classiﬁcation and Content Moderation models through public APIs, and show that the adversarial examples can mislead these models with high conﬁdence.

Original: Finally, we evaluate these black-box attacks against state-of-the-art adversarial training based defenses on DNNs, and we ﬁnd that both standard and ensemble adversarial training are not robust against Gradient Estimation attacks.
Revised: Finally, we evaluate these black-box attacks against state-of-the-art adversarial training based defenses on DNNs, and we ﬁnd that both standard and ensemble adversarial training are not robust against Gradient Estimation attacks.

Original: Related Work.
Revised: Related Work.

Original: Existing black-box attacks that do not use a local model were ﬁrst proposed for convex inducing two-class classiﬁers by Nelson et al.
Revised: Existing black-box attacks that do not use a local model were ﬁrst proposed for convex inducing two-class classiﬁers by Nelson et al.

Original: For malware data, Xu et al.
Revised: For malware data, Xu et al.

Original: (2016) use genetic algorithms to craft adversarial samples, while Dang et al.
Revised: (2016) use genetic algorithms to craft adversarial samples, while Dang et al.

Original: (2017) use hill climbing algorithms.
Revised: (2017) use hill climbing algorithms.

Original: These methods are prohibitively expensive for non-categorical and high-dimensional data such as images.
Revised: These methods are prohibitively expensive for non-categorical and high-dimensional data such as images.

Original: Papernot et al.
Revised: Papernot et al.

Original: (2017) proposed using queries to a target model to train a local surrogate model, which was then used to to generate adversarial samples.
Revised: (2017) proposed using queries to a target model to train a local surrogate model, which was then used to to generate adversarial samples.

Original: This attack relies on transferability.
Revised: This attack relies on transferability.

Original: To the best of our knowledge, the only previous literature on query-based black-box attacks in the deep learning setting are independent works by Narodytska & Kasiviswanathan (2016) and Chen et al.
Revised: To the best of our knowledge, the only previous literature on query-based black-box attacks in the deep learning setting are independent works by Narodytska & Kasiviswanathan (2016) and Chen et al.

Original: Narodytska & Kasiviswanathan (2016) propose a greedy local search to generate adversarial samples by perturbing randomly chosen pixels and using those which have a large impact on the output probabilities.
Revised: Narodytska & Kasiviswanathan (2016) propose a greedy local search to generate adversarial samples by perturbing randomly chosen pixels and using those which have a large impact on the output probabilities.

Original: Their method uses 500 queries per iteration, and the greedy local search is run for around 150 iterations for each image, resulting in a total of 75,000 queries per image, which is much higher than any of our attacks.
Revised: Their method uses 500 queries per iteration, and the greedy local search is run for around 150 iterations for each image, resulting in a total of 75,000 queries per image, which is much higher than any of our attacks.

Original: Further, we ﬁnd that our methods achieve higher targeted and untargeted attack success rates on both MNIST and CIFAR-10 as compared to their method.
Revised: Further, we ﬁnd that our methods achieve higher targeted and untargeted attack success rates on both MNIST and CIFAR-10 as compared to their method.

Original: Chen et al.
Revised: Chen et al.

Original: (2017) use zero order optimization methods which do not require knowledge of model gradients to ﬁnd adversarial samples using the loss function proposed by Carlini & Wagner (2017).
Revised: (2017) use zero order optimization methods which do not require knowledge of model gradients to ﬁnd adversarial samples using the loss function proposed by Carlini & Wagner (2017).

Original: While they achieve similar success rates as our attack, the running time of their attack is 40 × larger for each adversarial image on both MNIST and CIFAR-10.
Revised: While they achieve similar success rates as our attack, the running time of their attack is 40 × larger for each adversarial image on both MNIST and CIFAR-10.

Original: Neither of these works demonstrates the effectiveness of their attacks on real-world systems or on state-of-the-art defenses.
Revised: Neither of these works demonstrates the effectiveness of their attacks on real-world systems or on state-of-the-art defenses.

Original:  B ACKGROUND AND E VALUATION SETUP
Revised:  B ACKGROUND AND E VALUATION SETUP

Original: In this section, we will ﬁrst introduce the notation we use throughout the paper and then describe the evaluation setup and metrics used in the remainder of the paper.
Revised: In this section, we will ﬁrst introduce the notation we use throughout the paper and then describe the evaluation setup and metrics used in the remainder of the paper.

Original: 2. N OTATION
Revised: 2. N OTATION

Original: A classiﬁer f ( · ; θ ) :
Revised: A classiﬁer f ( · ; θ ) :

Original: X → Y is a function mapping from the domain X to the set of classiﬁcation outputs Y .
Revised: X → Y is a function mapping from the domain X to the set of classiﬁcation outputs Y .

Original: ( Y = { 0 , 1 } in the case of binary classiﬁcation, i.e. Y is the set of class labels.)
Revised: ( Y = { 0 , 1 } in the case of binary classiﬁcation, i.e. Y is the set of class labels.)

Original: The number of possible classiﬁcation outputs is then |Y| .
Revised: The number of possible classiﬁcation outputs is then |Y| .

Original: θ is the set of parameters associated with a classiﬁer.
Revised: θ is the set of parameters associated with a classiﬁer.

Original: Throughout, the target classiﬁer is denoted as f ( · ; θ ) , but the dependence on θ is dropped if it is clear from the context.
Revised: Throughout, the target classiﬁer is denoted as f ( · ; θ ) , but the dependence on θ is dropped if it is clear from the context.

Original: H denotes the constraint set within which an adversarial sample must satisfy.
Revised: H denotes the constraint set within which an adversarial sample must satisfy.

Original: (cid:96) f
Revised: (cid:96) f

Original: ( x , y ) is used to represent the loss function for the classiﬁer f with respect to inputs x ∈ X and their true labels y ∈ Y .
Revised: ( x , y ) is used to represent the loss function for the classiﬁer f with respect to inputs x ∈ X and their true labels y ∈ Y .

Original: Since the black-box attacks we analyze focus on neural networks in particular, we also deﬁne some notation speciﬁcally for neural networks.
Revised: Since the black-box attacks we analyze focus on neural networks in particular, we also deﬁne some notation speciﬁcally for neural networks.

Original: The outputs of the penultimate layer of a neural network f , representing the output of the network computed sequentially over all preceding layers, are known as the logits.
Revised: The outputs of the penultimate layer of a neural network f , representing the output of the network computed sequentially over all preceding layers, are known as the logits.

Original: We represent the logits as a vector φ f ( x ) ∈ R |Y| .
Revised: We represent the logits as a vector φ f ( x ) ∈ R |Y| .

Original: The ﬁnal layer of a neural network f used for classiﬁcation is a softmax layer represented as a vector of probabilities p f ( x ) =
Revised: The ﬁnal layer of a neural network f used for classiﬁcation is a softmax layer represented as a vector of probabilities p f ( x ) =

Original: [ p f 1 ( x ) , . . . , p f |Y| ( x )] , with (cid:80) |Y| i =1 p f i ( x ) = 1 and p f i ( x ) = e φ fi ( x ) (cid:80) |Y| j =1 e φ fj ( x ) .
Revised: [ p f 1 ( x ) , . . . , p f |Y| ( x )] , with (cid:80) |Y| i =1 p fi ( x ) = 1 and p fi ( x ) = e φ fi ( x ) (cid:80) [Equation]

Original: 2. E VALUATION SETUP FOR MNIST AND CIFAR- 
Revised: 2. E VALUATION SETUP FOR MNIST AND CIFAR-

Original: The empirical evaluation carried out in Section 3 is on state-of-the-art neural networks on the MNIST (LeCun & Cortes (1998)) and CIFAR-10 (Krizhevsky & Hinton (2009))
Revised: The empirical evaluation carried out in Section 3 is on state-of-the-art neural networks on the MNIST (LeCun & Cortes (1998)) and CIFAR-10 (Krizhevsky & Hinton (2009))

Original: datasets.
Revised: datasets.

Original: The details of the datasets are given in B.1 and the architecture and training details for all models are given in
Revised: The details of the datasets are given in B.1 and the architecture and training details for all models are given in

Original: Appendix B.2.. Only results for untargeted attacks are given in the main body of the paper.
Revised: Appendix B.2.. Only results for untargeted attacks are given in the main body of the paper.

Original: All results for targeted attacks are contained in
Revised: All results for targeted attacks are contained in

Original: Appendix E. We use two different loss functions in our evaluation, the standard cross-entropy loss (abbreviated as xent ) and the logit-based loss (ref.
Revised: Appendix E. We use two different loss functions in our evaluation, the standard cross-entropy loss (abbreviated as xent ) and the logit-based loss (ref.

Original: Section A.1.3) (abbreviated as logit ).
Revised: Section A.1.3) (abbreviated as logit ).

Original: In all of these attacks, the adversary’s perturbation is constrained using the
Revised: In all of these attacks, the adversary’s perturbation is constrained using the

Original: L ∞ distance.
Revised: L ∞ distance.

Original: The details of baseline black-box attacks and can be found in Appendix A.1.
Revised: The details of baseline black-box attacks and can be found in Appendix A.1.

Original: Similary, detailed descriptions and results for transferability-based attacks are in Appendix G.
Revised: Similary, detailed descriptions and results for transferability-based attacks are in Appendix G.

Original: MNIST.
Revised: MNIST.

Original: Each pixel of the MNIST image data is scaled to [0,1].
Revised: Each pixel of the MNIST image data is scaled to [0,1].

Original: We trained four different models on the MNIST dataset, denoted Models A to D , which are used by Tramèr et al.
Revised: We trained four different models on the MNIST dataset, denoted Models A to D , which are used by Tramèr et al.

Original: (2017a)
Revised: (2017a)

Original: and represent a good variety of architectures.
Revised: and represent a good variety of architectures.

Original: For the attacks constrained with the L ∞ distance, we vary the adversary’s perturbation budget (cid:15) from 0 to 0.4, since at a perturbation budget of 0.5, any image can be made solid gray.
Revised: For the attacks constrained with the L ∞ distance, we vary the adversary’s perturbation budget (cid:15) from 0 to 0.4, since at a perturbation budget of 0.5, any image can be made solid gray.

Original: CIFAR-10.
Revised: CIFAR-10.

Original: Each pixel of the CIFAR-10 image data is in [0,255].
Revised: Each pixel of the CIFAR-10 image data is in [0,255].

Original: We choose 3 model architectures for this dataset, which we denote as Resnet-32 , Resnet-28-10 (ResNet variants He et al.
Revised: We choose 3 model architectures for this dataset, which we denote as Resnet-32 , Resnet-28-10 (ResNet variants He et al.

Original: Zagoruyko & Komodakis (2016)) and Std-CNN (a standard CNN 3 from Tensorﬂow (Abadi et al.
Revised: Zagoruyko & Komodakis (2016)) and Std-CNN (a standard CNN 2 from Tensorﬂow (Abadi et al.

Original: For the attacks constrained with the L ∞ distance, we vary the adversary’s perturbation budget (cid:15) from 0 to 28.
Revised: For the attacks constrained with the L ∞ distance, we vary the adversary’s perturbation budget (cid:15) from 0 to 28.

Original: 2. M ETRICS
Revised: 2. M ETRICS

Original: Throughout the paper, we use standard metrics to characterize the effectiveness of various attack strategies.
Revised: Throughout the paper, we use standard metrics to characterize the effectiveness of various attack strategies.

Original: For MNIST , all metrics for single-step attacks are computed with respect to the test set consisting of 10,000 samples, while metrics for iterative attacks are computed with respect to the ﬁrst 1,000 samples from the test set.
Revised: For MNIST , all metrics for single-step attacks are computed with respect to the test set consisting of 10,000 samples, while metrics for iterative attacks are computed with respect to the ﬁrst 1,000 samples from the test set.

Original: For the CIFAR-10 data, we choose 1,000 random samples from the test set for single-step attacks and a 100 random samples for iterative attacks.
Revised: For the CIFAR-10 data, we choose 1,000 random samples from the test set for single-step attacks and a 100 random samples for iterative attacks.

Original: In our evaluations of targeted attacks, we choose target T for each sample uniformly at random from the set of classiﬁcation outputs, except the true class y of that sample.
Revised: In our evaluations of targeted attacks, we choose target T for each sample uniformly at random from the set of classiﬁcation outputs, except the true class y of that sample.

Original: Attack success rate.
Revised: Attack success rate.

Original: The main metric, the attack success rate, is the fraction of samples that meets the adversary’s goal: f ( x adv ) (cid:54) =
Revised: The main metric, the attack success rate, is the fraction of samples that meets the adversary’s goal: f ( x adv ) (cid:54) =

Original: y for untargeted attacks and f ( x adv )
Revised: y for untargeted attacks and f ( x adv )

Original: =
Revised: =

Original: T for targeted attacks with target T (Szegedy et al.
Revised: T for targeted attacks with target T (Szegedy et al.

Original: ; Tramèr et al. (2017a)).
Revised: ; Tramèr et al. (2017a)).

Original: Alternative evaluation metrics are discussed in Appendix B.3.
Revised: Alternative evaluation metrics are discussed in Appendix B.3.

Original: Average distortion.
Revised: Average distortion.

Original: We also evaluate the average distortion for adversarial examples using average
Revised: We also evaluate the average distortion for adversarial examples using average

Original: L 2 distance between the benign samples and the adversarial ones as suggested by Gu & Rigazio (2014): ∆( X adv , X ) = 1 N (cid:80) Ni =1 (cid:107) (
Revised: L 2 distance between the benign samples and the adversarial ones as suggested by Gu & Rigazio (2014): ∆( X adv , X ) = 1 N (cid:80) Ni =1 (cid:107) (

Original: X adv ) i − ( X ) i (cid:107) 2 where N is the number of samples.
Revised: X adv ) i − ( X ) i (cid:107) 2 where N is the number of samples.

Original: This metric allows us to compare the average distortion for attacks which achieve similar attack success rates, and therefore infer which one is stealthier.
Revised: This metric allows us to compare the average distortion for attacks which achieve similar attack success rates, and therefore infer which one is stealthier.

Original: Number of queries.
Revised: Number of queries.

Original: Query based black-box attacks make queries to the target model and this metric may affect the cost of mounting the attack.
Revised: Query based black-box attacks make queries to the target model and this metric may affect the cost of mounting the attack.

Original: This is an important consideration when attacking real-world systems which have costs associated with the number of queries made.
Revised: This is an important consideration when attacking real-world systems which have costs associated with the number of queries made.

Original:  Q UERY BASED ATTACKS : G RADIENT E STIMATION ATTACK
Revised:  Q UERY BASED ATTACKS : G RADIENT E STIMATION ATTACK

Original: Deployed learning systems often provide feedback for input samples provided by the user.
Revised: Deployed learning systems often provide feedback for input samples provided by the user.

Original: Given query feedback, different adaptive, query-based algorithms can be applied by adversaries to understand the system and iteratively generate effective adversarial examples to attack the system.
Revised: Given query feedback, different adaptive, query-based algorithms can be applied by adversaries to understand the system and iteratively generate effective adversarial examples to attack the system.

Original: Given the fact that many white-box attacks for generating adversarial examples are based on gradient information, we tried directly estimating the gradient to carry out black-box attacks, and found it to be very effective in a range of conditions.
Revised: Given the fact that many white-box attacks for generating adversarial examples are based on gradient information, we tried directly estimating the gradient to carry out black-box attacks, and found it to be very effective in a range of conditions.

Original: In other words, the adversary can approximate white-box
Revised: In other words, the adversary can approximate white-box

Original: Single-step and Iterative FGSM attacks using estimates of the losses that are needed to carry out those attacks.
Revised: Single-step and Iterative FGSM attacks using estimates of the losses that are needed to carry out those attacks.

Original: We ﬁrst propose a Gradient Estimation black-box attack based on the method of ﬁnite differences (Spall (2005)).
Revised: We ﬁrst propose a Gradient Estimation black-box attack based on the method of ﬁnite differences (Spall (2005)).

Original: The drawback of a naive implementation of the ﬁnite difference method, however, is that it requires O ( d ) queries per input, where d is the dimension of the input.
Revised: The drawback of a naive implementation of the ﬁnite difference method, however, is that it requires O ( d ) queries per input, where d is the dimension of the input.

Original: This leads us to explore methods such as random grouping of features and feature combination using components obtained from Principal Component Analysis (PCA) to reduce the number of queries.
Revised: This leads us to explore methods such as random grouping of features and feature combination using components obtained from Principal Component Analysis (PCA) to reduce the number of queries.

Original: Threat model and justiﬁcation.
Revised: Threat model and justiﬁcation.

Original: We assume that the adversary can obtain the vector of output probabilities for any input x .
Revised: We assume that the adversary can obtain the vector of output probabilities for any input x .

Original: The set of queries the adversary can make is then Q f = { p f ( x ) , ∀ x } .
Revised: The set of queries the adversary can make is then Q f = { p f ( x ) , ∀ x } .

Original: Note that an adversary with access to the softmax probabilities will be able to recover the logits up to an additive constant, by taking the logarithm of the softmax probabilities.
Revised: Note that an adversary with access to the softmax probabilities will be able to recover the logits up to an additive constant, by taking the logarithm of the softmax probabilities.

Original: For untargeted attacks, the adversary only needs access to the output probabilities for the two most likely classes.
Revised: For untargeted attacks, the adversary only needs access to the output probabilities for the two most likely classes.

Original: A compelling reason for assuming this threat model for the adversary is that many existing cloudbased ML services allow users to query trained models (Watson Visual Recognition, Clarifai, Google
Revised: A compelling reason for assuming this threat model for the adversary is that many existing cloudbased ML services allow users to query trained models (Watson Visual Recognition, Clarifai, Google

Original: Vision API).
Revised: Vision API).

Original: The results of these queries are conﬁdence scores which can be used to carry out
Revised: The results of these queries are conﬁdence scores which can be used to carry out

Original: Gradient Estimation attacks.
Revised: Gradient Estimation attacks.

Original: These trained models are often deployed by the clients of these ML as a service (MLaaS) providers (Clarifai-Disqus-2017).
Revised: These trained models are often deployed by the clients of these ML as a service (MLaaS) providers (Clarifai-Disqus-2017).

Original: Thus, an adversary can pose as a user for a
Revised: Thus, an adversary can pose as a user for a

Original: MLaaS provider and create adversarial examples using our attack, which can then be used against any client of that provider.
Revised: MLaaS provider and create adversarial examples using our attack, which can then be used against any client of that provider.

Original: 3. F INITE DIFFERENCE METHOD FOR GRADIENT ESTIMATION
Revised: 3. F INITE DIFFERENCE METHOD FOR GRADIENT ESTIMATION

Original: In this section, we focus on the method of ﬁnite differences to carry out gradient approximation based attacks.
Revised: In this section, we focus on the method of ﬁnite differences to carry out gradient approximation based attacks.

Original: All the analysis and results are presented for untargeted attacks, but can be easily extended to targeted attacks (Appendix E).
Revised: All the analysis and results are presented for untargeted attacks, but can be easily extended to targeted attacks (Appendix E).

Original: Let the function whose gradient is being approximated be g ( x ) .
Revised: Let the function whose gradient is being approximated be g ( x ) .

Original: The input to the function is a d -dimensional vector x , whose elements are represented as x i , where i ∈ [1 , . . .
Revised: The input to the function is a d -dimensional vector x , whose elements are represented as x i , where i ∈ [1 , . . .

Original: , d ] .
Revised: , d ] .

Original: The canonical basis vectors are represented as e i , where e i is 1 only in the i th component and 0 everywhere else.
Revised: The canonical basis vectors are represented as e i , where e i is 1 only in the i th component and 0 everywhere else.

Original: Then, a two-sided approximation of the gradient of g with respect to x is given by
Revised: Then, a two-sided approximation of the gradient of g with respect to x is given by

Original: [Equation]
Revised: [Equation]

Original: δ is a free parameter that controls the accuracy of the approximation.
Revised: δ is a free parameter that controls the accuracy of the approximation.

Original: A one-sided approximation can also be used.
Revised: A one-sided approximation can also be used.

Original: If the gradient of the function g exists, then lim δ → 0 FD x ( g ( x ) , δ ) = ∇ x g ( x ) .
Revised: If the gradient of the function g exists, then lim δ → 0 FD x ( g ( x ) , δ ) = ∇ x g ( x ) .

Original: The ﬁnite difference method is useful for a black-box adversary aiming to approximate a gradient based attack, since the gradient can be directly approximated with access to only the function values.
Revised: The ﬁnite difference method is useful for a black-box adversary aiming to approximate a gradient based attack, since the gradient can be directly approximated with access to only the function values.

Original: 3.1. A PPROXIMATE FGS WITH FINITE DIFFERENCES
Revised: 3.1. A PPROXIMATE FGS WITH FINITE DIFFERENCES

Original: In the untargeted FGS method, the gradient is usually taken with respect to the cross-entropy loss between the true label of the input and the softmax probability vector.
Revised: In the untargeted FGS method, the gradient is usually taken with respect to the cross-entropy loss between the true label of the input and the softmax probability vector.

Original: The cross-entropy loss of a network f at an input x is then (cid:96) f
Revised: The cross-entropy loss of a network f at an input x is then (cid:96) f

Original: ( x , y )
Revised: ( x , y )

Original: = − (cid:80) |Y| j =1 

Revised: = − (cid:80) |Y| j =1

Original: [ j = y ] log p fj ( x ) = − log p fy ( x ) , where y is the index of the original class of the input.
Revised: [ j = y ] log p fj ( x ) = − log p fy ( x ) , where y is the index of the original class of the input.

Original: The gradient of (cid:96) f
Revised: The gradient of (cid:96) f

Original: ( x , y ) is
Revised: ( x , y ) is

Original: [Equation]
Revised: [Equation]

Original: An adversary with query access to the softmax probabilities has two options: i) Estimate the gradient of the loss function (cid:96) f ( x , y ) directly, in which case the adversarial sample will be
Revised: An adversary with query access to the softmax probabilities has two options: i) Estimate the gradient of the loss function (cid:96) f ( x , y ) directly, in which case the adversarial sample will be

Original: [Equation]
Revised: [Equation]

Original: ii) Estimate

Original:  the gradient of p fy ( x ) and plug it into Eq. 2, resulting 
Revised: ii) Estimate the gradient of p fy ( x ) and plug it into Eq. 2, resulting in the adversarial sample

Original: in

Original: the adversarial sample

Original: [Equation]
Revised: [Equation]

Original: In our evaluation, we use the latter method for generating adversarial samples as we ﬁnd that it 
Revised: In our evaluation, we use the latter method for generating adversarial samples as we ﬁnd that it performs better.

Revised: This is denoted as FD-xent .

Original: 3.1. A PPROXIMATING THE LOGIT - BASED LOSS
Revised: 3.1. A PPROXIMATING THE LOGIT - BASED LOSS

Original: The gradient of the logit based loss described earlier in the context of the optimization based attack can be used to carry out a one-step gradient based attack starting from the original sample, instead of using the cross-entropy loss.
Revised: The gradient of the logit based loss described earlier in the context of the optimization based attack can be used to carry out a one-step gradient based attack starting from the original sample, instead of using the cross-entropy loss.

Original: If the conﬁdence parameter κ is set to 0, the logit loss is max( φ ( x + δ ) y − max { φ ( x + δ ) i : i (cid:54) =
Revised: If the conﬁdence parameter κ is set to 0, the logit loss is max( φ ( x + δ ) y − max { φ ( x + δ ) i : i (cid:54) =

Original: y } , 0) .
Revised: y } , 0) .

Original: For an input that is correctly classiﬁed, the ﬁrst term is always greater than 0, and for an incorrectly classiﬁed input, an untargeted attack is not meaningful to carry out.
Revised: For an input that is correctly classiﬁed, the ﬁrst term is always greater than 0, and for an incorrectly classiﬁed input, an untargeted attack is not meaningful to carry out.

Original: Thus, the loss term reduces to φ ( x + δ ) y − max { φ ( x + δ ) i : i (cid:54) =
Revised: Thus, the loss term reduces to φ ( x + δ ) y − max { φ ( x + δ ) i : i (cid:54) =

Original: y } for relevant inputs.
Revised: y } for relevant inputs.

Original: An adversary with access to the logits can use the ﬁnite differences method to approximate the logit values for the original class y , and the second most likely class y (cid:48) , i.e., the one given by y (cid:48)
Revised: An adversary with access to the logits can use the ﬁnite differences method to approximate the logit values for the original class y , and the second most likely class y (cid:48) , i.e., the one given by y (cid:48)

Original: =
Revised: =

Original: argmax i (cid:54) =
Revised: argmax i (cid:54) =

Original: y φ ( x ) i .
Revised: y φ ( x ) i .

Original: The untargeted adversarial sample generated for this loss in the white-box case is x adv = x + (cid:15) · sign ( ∇ x ( φ ( x ) y (cid:48) − φ ( x ) y )) .
Revised: The untargeted adversarial sample generated for this loss in the white-box case is x adv = x + (cid:15) · sign ( ∇ x ( φ ( x ) y (cid:48) − φ ( x ) y )) .

Original: Similarly, in the case of a black-box adversary with query-access to the logits, the adversarial sample is
Revised: Similarly, in the case of a black-box adversary with query-access to the logits, the adversarial sample is

Original: [Equation]
Revised: [Equation]

Original: We note that an adversary with access to the softmax probabilities will also be able to carry out this attack, since the difference of logits leads to the constant term being canceled out.
Revised: We note that an adversary with access to the softmax probabilities will also be able to carry out this attack, since the difference of logits leads to the constant term being canceled out.

Original: This attack is denoted as FD-logit .
Revised: This attack is denoted as FD-logit .

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: 3.1. I TERATIVE ATTACKS WITH ESTIMATED GRADIENTS
Revised: . I TERATIVE ATTACKS WITH ESTIMATED GRADIENTS

Original: The iterative variant of the gradient based attack described in Section A.1.2 is a powerful attack that often achieves much higher attack success rates in the white-box setting than the simple single-step gradient based attacks.
Revised: The iterative variant of the gradient based attack described in Section A.1.2 is a powerful attack that often achieves much higher attack success rates in the white-box setting than the simple single-step gradient based attacks.

Original: Thus, it stands to reason that a version of the iterative attack with estimated gradients will also perform better than the single-step attacks described until now.
Revised: Thus, it stands to reason that a version of the iterative attack with estimated gradients will also perform better than the single-step attacks described until now.

Original: An iterative attack with t + 1 iterations using the cross-entropy loss is:
Revised: An iterative attack with t + 1 iterations using the cross-entropy loss is:

Original: [Equation]
Revised: [Equation]

Original: where α is the step size and H is the constraint set for the adversarial sample.
Revised: where α is the step size and H is the constraint set for the adversarial sample.

Original: This attack is denoted as IFD-xent .
Revised: This attack is denoted as IFD-xent .

Original: If the logit loss is used instead, it is denoted as IFD-logit .
Revised: If the logit loss is used instead, it is denoted as IFD-logit .

Original: 3.1. E VALUATION OF G RADIENT E STIMATION USING F INITE D IFFERENCES
Revised: 3.1. E VALUATION OF G RADIENT E STIMATION USING F INITE D IFFERENCES

Original: In this section, we summarize the results obtained using Gradient Estimation attacks with Finite
Revised: In this section, we summarize the results obtained using Gradient Estimation attacks with Finite

Original: Differences and describe the parameter choices made.
Revised: Differences and describe the parameter choices made.

Original: FD-logit and IFD-logit match white-box attack adversarial success rates : The Gradient Estimation attack with Finite Differences ( FD-logit ) is the most successful untargeted single-step black-box attack for MNIST and CIFAR-10 models.
Revised: FD-logit and IFD-logit match white-box attack adversarial success rates : The Gradient Estimation attack with Finite Differences ( FD-logit ) is the most successful untargeted single-step black-box attack for MNIST and CIFAR-10 models.

Original: It signiﬁcantly outperforms transferability-based attacks (Table 1) and closely tracks white-box FGS with a logit loss ( WB FGS-logit ) on MNIST and CIFAR

Revised: It signiﬁcantly outperforms transferability-based attacks (Table 1) and closely tracks white-box FGS with a logit loss ( WB FGS-logit ) on MNIST and CIFAR

Original: (Figure 2).
Revised: (Figure 2).

Original: For adversarial samples generated iteratively, the Iterative Gradient Estimation attack 
Revised: For adversarial samples generated iteratively, the Iterative Gradient Estimation attack with Finite Differences ( IFD-logit ) achieves 100% adversarial success rate across all models on both datasets (Table 1). The average distortion for both FD-logit and IFD-logit closely matches their white-box counterparts, FGS-logit and IFGS-logit as given in Table 7.

Original: [Figure]
Revised: [Figure]

Original: The most successful strategy is the Gradient Estimation using Finite Differences with the loss
Revised: FD T and IFD T achieve the highest adversarial success rates in the targeted setting

Original: ), which coincides almost exactly with with

Original: Also, the Gradient Estimation attack with query reduction using PCA ( GE-QR (PCA- k , logit) ) performs well for both datasets as well. with Finite Differences ( IFD-logit ) achieves 100% adversarial success rate across all models on both datasets (Table 1). The average distortion for both FD-logit and IFD-logit closely matches their white-box counterparts, FGS-logit and IFGS-logit as given in Table 7. FD-T and IFD-T achieve the highest adversarial success rates in the targeted setting : Fortargeted black-box attacks, IFD-xent-T achieves100% adversarial success rateson almost all models as shown by the results in Table 4.
Revised: For achieves rates by the results in Table 4.

Original: While FD-xent-T only achieves about 30% adversarial success rates, this matches the performance of single-step white-box attacks such as FGS-xent-T and FGS-logit-T (Table 8).
Revised: While FD-xent-T only achieves about 30% adversarial success rates, this matches the performance of single-step white-box attacks such as FGS-xent-T and FGS-logit-T (Table 8).

Original: The average distortion for samples generated using gradient estimation methods is similar with that of white-box attacks.
Revised: The average distortion for samples generated using gradient estimation methods is similar with that of white-box attacks.

Original: Parameter choices : We use δ = 1 . 0 for FD-xent and IFD-xent for both datasets, while using δ = 0 .
Revised: Parameter choices : We use δ = 1 . 0 for FD-xent and IFD-xent for both datasets, while using δ = 0 .

Original: 01 for FD-logit and IFD-logit .
Revised: 01 for FD-logit and IFD-logit .

Original: We ﬁnd that a larger value of δ is needed for xent loss based attacks to work.
Revised: We ﬁnd that a larger value of δ is needed for xent loss based attacks to work.

Original: The reason for this is that the probability values used in the xent loss are not as sensitive to changes as in the logit loss, and thus the gradient cannot be estimated since the function value does not change at all when a single pixel is perturbed.
Revised: The reason for this is that the probability values used in the xent loss are not as sensitive to changes as in the logit loss, and thus the gradient cannot be estimated since the function value does not change at all when a single pixel is perturbed.

Original: For the Iterative Gradient Estimation attacks using Finite Differences, we use α = 0 .
Revised: For the Iterative Gradient Estimation attacks using Finite Differences, we use α = 0 .

Original: 01 and t = 40 for the MNIST results and α = 1 .and
Revised: 01 and t = 40 for the MNIST results and α = 1 .and

Original: t = 10 for CIFAR-10 throughout.
Revised: t = 10 for CIFAR-10 throughout.

Original: The same parameters are used for the Iterative FGS attack results given in Appendix J.5.
Revised: The same parameters are used for the Iterative FGS attack results given in Appendix J.5.

Original: This translates to 62720 queries for MNIST (40 steps of iteration) and 61440 queries (10 steps of iteration) for CIFAR-10 per sample.
Revised: This translates to 62720 queries for MNIST (40 steps of iteration) and 61440 queries (10 steps of iteration) for CIFAR-10 per sample.

Original: We ﬁnd these choices work well, and keep the running time of the gradient estimation attacks at a manageable level.
Revised: We ﬁnd these choices work well, and keep the running time of the gradient estimation attacks at a manageable level.

Original: However, we ﬁnd that we can achieve similar adversarial success rates with much fewer queries using query reduction methods which we describe in the next section.
Revised: However, we ﬁnd that we can achieve similar adversarial success rates with much fewer queries using query reduction methods which we describe in the next section.

Original: 3. Q UERY REDUCTION
Revised: 3. Q UERY REDUCTION

Original: The major drawback of the approximation based black-box attacks is that the number of queries needed per adversarial sample is large.
Revised: The major drawback of the approximation based black-box attacks is that the number of queries needed per adversarial sample is large.

Original: For an input with dimension d , the number of queries will be exactly 2 d for a two-sided approximation.
Revised: For an input with dimension d , the number of queries will be exactly 2 d for a two-sided approximation.

Original: This may be too large when the input is of high dimension.
Revised: This may be too large when the input is of high dimension.

Original: So we examined two techniques in order to reduce the number of queries the adversary has to make.
Revised: So we examined two techniques in order to reduce the number of queries the adversary has to make.

Original: Both techniques involve estimating the gradient for groups of features, instead of estimating it one feature at a time.
Revised: Both techniques involve estimating the gradient for groups of features, instead of estimating it one feature at a time.

Original: The justiﬁcation for the use of feature grouping comes from the relation between gradients and directional derivatives (Hildebrand (1962)) for differentiable functions.
Revised: The justiﬁcation for the use of feature grouping comes from the relation between gradients and directional derivatives (Hildebrand (1962)) for differentiable functions.

Original: The directional derivative of a function g is deﬁned as ∇ v g ( x ) =
Revised: The directional derivative of a function g is deﬁned as ∇ v g ( x ) =

Original: lim h → 0 g ( x + h v ) − g ( x ) h .
Revised: lim h → 0 g ( x + h v ) − g ( x ) h .

Original: It is a generalization of a partial derivative.
Revised: It is a generalization of a partial derivative.

Original: For differentiable functions, ∇ v g ( x ) = ∇ g ( x ) · v , which implies that the directional derivative is just the projection of the gradient along the direction v .
Revised: For differentiable functions, ∇ v g ( x ) = ∇ g ( x ) · v , which implies that the directional derivative is just the projection of the gradient along the direction v .

Original: Thus, estimating the gradient by grouping features is equivalent to estimating an approximation of the gradient constructed by projecting it along appropriately chosen directions.
Revised: Thus, estimating the gradient by grouping features is equivalent to estimating an approximation of the gradient constructed by projecting it along appropriately chosen directions.

Original: The estimated gradient ˆ ∇ x g ( x ) of any function g can be computed using the techniques below, and then plugged in to Equations 3, 4 and 5 instead of the ﬁnite difference term to create an adversarial sample.
Revised: The estimated gradient ˆ ∇ x g ( x ) of any function g can be computed using the techniques below, and then plugged in to Equations 3, 4 and 5 instead of the ﬁnite difference term to create an adversarial sample.

Original: Next, we will introduce the techniques applied to group the features for estimation.
Revised: Next, we will introduce the techniques applied to group the features for estimation.

Original: Detailed algorithms for these techniques are given in Appendix C.
Revised: Detailed algorithms for these techniques are given in Appendix C.

Original: 3.2. Q UERY REDUCTION BASED ON RANDOM GROUPING
Revised: 3.2. Q UERY REDUCTION BASED ON RANDOM GROUPING

Original: The simplest way to group features is to choose, without replacement, a random set of features.
Revised: The simplest way to group features is to choose, without replacement, a random set of features.

Original: If the size of the set chosen is k , then the number of queries the adversary has to make is (cid:100) dk (cid:101) .
Revised: If the size of the set chosen is k , then the number of queries the adversary has to make is (cid:100) dk (cid:101) .

Original: When k
Revised: When k

Original: = 1 , this reduces to the case where the partial derivative with respect to every feature is found, as in Section 3.1.
Revised: = 1 , this reduces to the case where the partial derivative with respect to every feature is found, as in Section 3.1.

Original: In each iteration of Algorithm 1, there is a set of indices S according to which v is determined, with v i = 

Revised: In each iteration of Algorithm 1, there is a set of indices S according to which v is determined, with v i =

Original: if and only if i ∈ S .
Revised: if and only if i ∈ S .

Original: Thus, the directional derivative being estimated is (cid:80) 
Revised: Thus, the directional derivative being estimated is (cid:80) i ∈

Original: i [Equation]
Revised: S ∂g

Original:  x ) ∂ x i , which is an average of partial derivatives.
Revised: ( x ) ∂ x i , which is an average of partial derivatives.

Original: Thus, the quantity being estimated is not the gradient itself, but an index-wise averaged version of it.
Revised: Thus, the quantity being estimated is not the gradient itself, but an index-wise averaged version of it.

Original: 3.2. Q UERY REDUCTION USING PCA COMPONENTS
Revised: 3.2. Q UERY REDUCTION USING PCA COMPONENTS

Original: A more principled way to reduce the number of queries the adversary has to make to estimate the gradient is to compute directional derivatives along the principal components as determined by principal component analysis (PCA)
Revised: A more principled way to reduce the number of queries the adversary has to make to estimate the gradient is to compute directional derivatives along the principal components as determined by principal component analysis (PCA)

Original: (Shlens (2014)), which requires the adversary to have access to a subset of the training data.
Revised: (Shlens (2014)), which requires the adversary to have access to a subset of the training data.

Original: A more detailed description of PCA and the Gradient Estimation attack using PCA components for query reduction is given in Appendix C.2.
Revised: A more detailed description of PCA and the Gradient Estimation attack using PCA components for query reduction is given in Appendix C.2.

Original: In Algorithm 2, U is the d × d matrix whose columns are the principal components.
Revised: In Algorithm 2, U is the d × d matrix whose columns are the principal components.

Original: The quantity being estimated in Algorithm 2 in the Appendix is an approximation of the gradient in the PCA basis:
Revised: The quantity being estimated in Algorithm 2 in the Appendix is an approximation of the gradient in the PCA basis:

Original: [Equation]
Revised: [Equation]

Original: where the term on the left represents an approximation of the true gradient by the sum of its projection along the top k principal components.
Revised: where the term on the left represents an approximation of the true gradient by the sum of its projection along the top k principal components.

Original: In Algorithm 2, the weights of the representation in the PCA basis are approximated using the approximate directional derivatives along the principal components.
Revised: In Algorithm 2, the weights of the representation in the PCA basis are approximated using the approximate directional derivatives along the principal components.

Original: 3. I TERATIVE ATTACKS WITH QUERY REDUCTION
Revised: 3. I TERATIVE ATTACKS WITH QUERY REDUCTION

Original: Performing an iterative attack with the gradient estimated using the ﬁnite difference method (Equation 1) could be expensive for an adversary, needing 2 td queries to the target model, for t iterations with the two-sided ﬁnite difference estimation of the gradient.
Revised: Performing an iterative attack with the gradient estimated using the ﬁnite difference method (Equation 1) could be expensive for an adversary, needing 2 td queries to the target model, for t iterations with the two-sided ﬁnite difference estimation of the gradient.

Original: To lower the number of queries needed, the adversary can use either of the query reduction techniques described above to reduce the number of queries to 2 tk ( k < d ).
Revised: To lower the number of queries needed, the adversary can use either of the query reduction techniques described above to reduce the number of queries to 2 tk ( k < d ).

Original: These attacks using the cross-entropy loss are denoted as IGE-QR (RG- k , xent) for the random grouping technique and IGE-QR (PCA- k , xent) for the PCA-based technique.
Revised: These attacks using the cross-entropy loss are denoted as IGE-QR (RG- k , xent) for the random grouping technique and IGE-QR (PCA- k , xent) for the PCA-based technique.

Original: Gradient estimation with query reduction maintains high attack success rates :
Revised: Gradient estimation with query reduction maintains high attack success rates :

Original: For both datasets, the Gradient Estimation attack with PCA based query reduction ( GE-QR (PCA- k , logit) ) is effective, with performance close to that of FD-logit with k = 100 for MNIST (Figure 2a) and k
Revised: For both datasets, the Gradient Estimation attack with PCA based query reduction ( GE-QR (PCA- k , logit) ) is effective, with performance close to that of FD-logit with k = 100 for MNIST (Figure 2a) and k

Original: = 400 for
Revised: = 400 for

Original: CIFAR-10 (Figure 2b).
Revised: CIFAR-10 (Figure 2b).

Original: The Iterative Gradient Estimation attacks with both Random Grouping and
Revised: The Iterative Gradient Estimation attacks with both Random Grouping and

Original: PCA based query reduction ( IGE-QR (RG- k , logit) and IGE-QR (PCA- k , logit) ) achieve close to 100% success rates for untargeted attacks and above 80% for targeted attacks on Model A on MNIST and Resnet-32 on CIFAR-10 (Figure 3).
Revised: PCA based query reduction ( IGE-QR (RG- k , logit) and IGE-QR (PCA- k , logit) ) achieve close to 100% success rates for untargeted attacks and above 80% for targeted attacks on Model A on MNIST and Resnet-32 on CIFAR-10 (Figure 3).

Original: Figure 3 clearly shows the effectiveness of the gradient estimation attack across models, datasets and adversarial goals.
Revised: Figure 3 clearly shows the effectiveness of the gradient estimation attack across models, datasets and adversarial goals.

Original: While random grouping is not as effective as the PCA based method for Single-step attacks, it is as effective for iterative attacks.
Revised: While random grouping is not as effective as the PCA based method for Single-step attacks, it is as effective for iterative attacks.

Original: Thus, powerful black-box attacks can be carried out purely using query access.
Revised: Thus, powerful black-box attacks can be carried out purely using query access.

Original: Detailed evaluation results are contained in Appendix J. In particular, discussions of the results on baseline attacks (Appendix J.1), effect of dimension on query reduced Gradient Estimation attacks (Appendix J.2), Single-step attacks on defenses (Appendix J.3) and the efﬁciency of Gradient
Revised: Detailed evaluation results are contained in Appendix J. In particular, discussions of the results on baseline attacks (Appendix J.1), effect of dimension on query reduced Gradient Estimation attacks (Appendix J.2), Single-step attacks on defenses (Appendix J.3) and the efﬁciency of Gradient

Original: Estimation attacks (Appendix J.4) are given.
Revised: Estimation attacks (Appendix J.4) are given.

Original: Sample adversarial examples are shown in Appendix I.
Revised: Sample adversarial examples are shown in Appendix I.

Original:  A TTACKING DEFENSES AND REAL - WORLD SYSTEMS
Revised:  A TTACKING DEFENSES AND REAL - WORLD SYSTEMS

Original: In this section, we evaluate black-box attacks against different defenses based on adversarial training and its variants.
Revised: In this section, we evaluate black-box attacks against different defenses based on adversarial training and its variants.

Original: The details about the adversarially trained models can be found in Appendix D. 
Revised: The details about the adversarially trained models can be found in Appendix D. We focus on adversarial training based defenses as they aim to directly improve the robustness of DNNs, and are among the most effective defenses demonstrated so far in the literature.

Original: We 
Revised: We also conduct real-world attacks on a deployed system, Clarifai.

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: focus on adversarial training based defenses as they aim to directly improve the robustness of DNNs, and are among the most effective defenses demonstrated so far in the literature.
Revised: [Figure]

Original: We also conduct real-world attacks on a deployed system, Clarifai.
Revised: [Figure]

Original: 4. MNIST SETUP AND RESULTS
Revised: 4. MNIST SETUP AND RESULTS

Original: We train variants of Model A with the 3 adversarial training strategies described in Appendix D using adversarial samples based on an L ∞ constraint of 0.3.
Revised: We train variants of Model A with the 3 adversarial training strategies described in Appendix D using adversarial samples based on an L ∞ constraint of 0.3.

Original: Model A adv-0.3 is trained with FGS samples, while Model A adv-iter-0.3 is trained with iterative FGS samples using t = 40 and α = 0 .
Revised: Model A adv-0.3 is trained with FGS samples, while Model A adv-iter-0.3 is trained with iterative FGS samples using t = 40 and α = 0 .

Original: For the model with ensemble training, Model A adv-ens-0.3 is trained with pre-generated FGS samples for
Revised: For the model with ensemble training, Model A adv-ens-0.3 is trained with pre-generated FGS samples for

Original: Models A , C , and D , as well as FGS samples.
Revised: Models A , C , and D , as well as FGS samples.

Original: The source of the samples is chosen randomly for each minibatch during training.
Revised: The source of the samples is chosen randomly for each minibatch during training.

Original:  of iterative attacks on different adversarial training defenses : While single-step black-box attacks are less effective at (cid:15) lower than the one used for training, our experiments show that iterative black-box attacks continue to work well even against adversarially trained networks.
Revised: Evaluation of iterative attacks on different adversarial training defenses : While single-step black-box attacks are less effective at (cid:15) lower than the one used for training, our experiments show that iterative black-box attacks continue to work well even against adversarially trained networks.

Original: For example, the Iterative Gradient Estimation attack using Finite Differences with a logit loss ( IFD-logit ) achieves an adversarial success rate of 96.4% against Model A adv-ens-0.3 , while the best transferability attack has a success rate of 4.9%.
Revised: For example, the Iterative Gradient Estimation attack using Finite Differences with a logit loss ( IFD-logit ) achieves an adversarial success rate of 96.4% against Model A adv-ens-0.3 , while the best transferability attack has a success rate of 4.9%.

Original: It is comparable to the white-box attack success rate of 93% from
Revised: It is comparable to the white-box attack success rate of 93% from

Original: Table 9.
Revised: Table 9.

Original: However, Model A adv-iter-0.3 is quite robust even against iterative attacks, with the highest black-box attack success rate achieved being 14.5%.
Revised: However, Model A adv-iter-0.3 is quite robust even against iterative attacks, with the highest black-box attack success rate achieved being 14.5%.

Original: Further, in Figure 3, we can see that using just 4000 queries per sample, the Iterative Gradient
Revised: Further, in Figure 3, we can see that using just 4000 queries per sample, the Iterative Gradient

Original: Estimation attack using PCA for query reduction ( IGE-QR (PCA-400, logit) ) achieves 100% (untargeted) and 74.5% (targeted) adversarial success rates against Model A adv-0.3 .
Revised: Estimation attack using PCA for query reduction ( IGE-QR (PCA-400, logit) ) achieves 100% (untargeted) and 74.5% (targeted) adversarial success rates against Model A adv-0.3 .

Original: Our methods far outperform the other black-box attacks as shown in Table 9.
Revised: Our methods far outperform the other black-box attacks as shown in Table 9.

Original: Remarks.
Revised: Remarks.

Original: On MNIST, our proposed gradient estimation attacks outperform other black-box attacks, achieving high attack success rates matching those of white-box attacks even on adversarially trained networks, as can be seen in Table 2 and Figure 3.
Revised: On MNIST, our proposed gradient estimation attacks outperform other black-box attacks, achieving high attack success rates matching those of white-box attacks even on adversarially trained networks, as can be seen in Table 2 and Figure 3.

Original: 4. CIFAR-10 SETUP AND RESULTS
Revised: 4. CIFAR-10 SETUP AND RESULTS

Original: We train variants of Resnet-32 using adversarial samples with L ∞ constraint of 8.
Revised: We train variants of Resnet-32 using adversarial samples with L ∞ constraint of 8.

Original: Resnetadv-8 is trained with FGS samples with the same constraint, and Resnet-32 ens-adv-8 is trained with pre-generated FGS samples from Resnet-32 and Std.-CNN as well as FGS samples.
Revised: Resnetadv-8 is trained with FGS samples with the same constraint, and Resnet-32 ens-adv-8 is trained with pre-generated FGS samples from Resnet-32 and Std.-CNN as well as FGS samples.

Original: Resnetadv-iter-8 is trained with iterative FGS samples using t = 10 and α = 1 . 0 .
Revised: Resnetadv-iter-8 is trained with iterative FGS samples using t = 10 and α = 1 . 0 .

Original: Iterative black-box attacks perform well against adversarially trained models for CIFAR-10 as well.
Revised: Iterative black-box attacks perform well against adversarially trained models for CIFAR-10 as well.

Original: IFD-logit achieves attack success rates of 100% against both Resnet-32 adv-8 and Resnetadv-ens-8 (Table 2), which reduces slightly to 97% when IFD-QR (PCA-400, logit) is used.
Revised: IFD-logit achieves attack success rates of 100% against both Resnet-32 adv-8 and Resnetadv-ens-8 (Table 2), which reduces slightly to 97% when IFD-QR (PCA-400, logit) is used.

Original: This matches the performance of white-box attacks as given in Table 9.
Revised: This matches the performance of white-box attacks as given in Table 9.

Original: IFD-QR (PCA-400, logit) also achieves a 72% success rate for targeted attacks at (cid:15)
Revised: IFD-QR (PCA-400, logit) also achieves a 72% success rate for targeted attacks at (cid:15)

Original: = 8 as shown in Figure 3.
Revised: = 8 as shown in Figure 3.

Original: The iteratively trained model has poor performance on both benign as well as adversarial samples.
Revised: The iteratively trained model has poor performance on both benign as well as adversarial samples.

Original: Resnet-32 adv-iter-8 has an accuracy of only 79.1% on benign data, as shown in Table 3.
Revised: Resnet-32 adv-iter-8 has an accuracy of only 79.1% on benign data, as shown in Table 3.

Original: The Iterative
Revised: The Iterative

Original: Gradient Estimation attack using Finite Differences with cross-entropy loss ( IFD-xent ) achieves an untargeted attack success rate of 55% on this model, which is lower than on the other adversarially trained models, but still signiﬁcant.
Revised: Gradient Estimation attack using Finite Differences with cross-entropy loss ( IFD-xent ) achieves an untargeted attack success rate of 55% on this model, which is lower than on the other adversarially trained models, but still signiﬁcant.

Original: This is in line with the observation by M ˛adry et al.
Revised: This is in line with the observation by M ˛adry et al.

Original: M ˛adry et al.
Revised: M ˛adry et al.

Original: (2017) that iterative adversarial training needs models with large capacity for it to be effective.
Revised: (2017) that iterative adversarial training needs models with large capacity for it to be effective.

Original: This highlights a limitation of this defense, since it is not clear what model capacity is needed although the models we use already have a large number of parameters.
Revised: This highlights a limitation of this defense, since it is not clear what model capacity is needed although the models we use already have a large number of parameters.

Original: Remarks.
Revised: Remarks.

Original: Both single-step and iterative variants of the Gradient Estimation attacks outperform other black-box attacks, achieving attack success rates close to those of white-box attacks even on adversarially trained models.
Revised: Both single-step and iterative variants of the Gradient Estimation attacks outperform other black-box attacks, achieving attack success rates close to those of white-box attacks even on adversarially trained models.

Original: 4. A TTACKING C LARIFAI MODELS
Revised: 4. A TTACKING C LARIFAI MODELS

Original: Since the only requirement for carrying out the Gradient Estimation based attacks is query-based access to the target model, there are a number of deployed public systems providing classiﬁcation as a service that can be used to evaluate our methods.
Revised: Since the only requirement for carrying out the Gradient Estimation based attacks is query-based access to the target model, there are a number of deployed public systems providing classiﬁcation as a service that can be used to evaluate our methods.

Original: We choose Clarifai as it has a number of models trained to classify different image datasets.
Revised: We choose Clarifai as it has a number of models trained to classify different image datasets.

Original: Clarifai provides black-box access and returns conﬁdence scores upon querying.
Revised: Clarifai provides black-box access and returns conﬁdence scores upon querying.

Original: In addition, Clarifai has models used for the detection of Not Safe
Revised: In addition, Clarifai has models used for the detection of Not Safe

Original: For Work (NSFW) content, as well as for Content Moderation.
Revised: For Work (NSFW) content, as well as for Content Moderation.

Original: These are important applications where the presence of adversarial samples presents a real danger, as an attacker, using query access to the model, could generate an adversarial sample which will no longer be classiﬁed as inappropriate.
Revised: These are important applications where the presence of adversarial samples presents a real danger, as an attacker, using query access to the model, could generate an adversarial sample which will no longer be classiﬁed as inappropriate.

Original: For example, an adversary could upload violent images, adversarially modiﬁed, such that they are marked incorrectly as“safe" by the Content Moderation model.
Revised: For example, an adversary could upload violent images, adversarially modiﬁed, such that they are marked incorrectly as“safe" by the Content Moderation model.

Original: An example of an attack on the Content Moderation API is given in Figure 1, where the original image on the left (Figure 1) is clearly of some kind of drug on a table, with a spoon and a syringe.
Revised: An example of an attack on the Content Moderation API is given in Figure 1, where the original image on the left (Figure 1) is clearly of some kind of drug on a table, with a spoon and a syringe.

Original: It is classiﬁed as a drug by the Content Moderation model with a conﬁdence score of 0.99.
Revised: It is classiﬁed as a drug by the Content Moderation model with a conﬁdence score of 0.99.

Original: The image on the right (Figure 1) is an adversarial image generated with 192 queries to the Content
Revised: The image on the right (Figure 1) is an adversarial image generated with 192 queries to the Content

Original: Moderation API, with an (cid:96) ∞ constraint on the perturbation of (cid:15) = 32 .
Revised: Moderation API, with an (cid:96) ∞ constraint on the perturbation of (cid:15) = 32 .

Original: While the image can still clearly be classiﬁed by a human as being of drugs on a table, the Content Moderation model now classiﬁes it as ‘safe’ with a conﬁdence score of 0.96.
Revised: While the image can still clearly be classiﬁed by a human as being of drugs on a table, the Content Moderation model now classiﬁes it as ‘safe’ with a conﬁdence score of 0.96.

Original: Remarks.
Revised: Remarks.

Original: The proposed Gradient Estimation attacks can successfully generate adversarial examples that are misclassiﬁed by a real-world system hosted by Clarifai withou prior knowledge of the training set or model.
Revised: The proposed Gradient Estimation attacks can successfully generate adversarial examples that are misclassiﬁed by a real-world system hosted by Clarifai withou prior knowledge of the training set or model.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Overall, in this paper, we conduct a systematic analysis of black-box attacks and provide a taxonomy for them.
Revised: Overall, in this paper, we conduct a systematic analysis of black-box attacks and provide a taxonomy for them.

Original: We propose Gradient Estimation attacks which achieve high attack success rates comparable with even white-box attacks and outperform other state-of-the-art black-box attacks.
Revised: We propose Gradient Estimation attacks which achieve high attack success rates comparable with even white-box attacks and outperform other state-of-the-art black-box attacks.

Original: We apply random grouping and PCA based methods to reduce the number of queries required to a small constant and demonstrate the effectiveness of the Gradient Estimation attack even in this setting.
Revised: We apply random grouping and PCA based methods to reduce the number of queries required to a small constant and demonstrate the effectiveness of the Gradient Estimation attack even in this setting.

Original: We also apply our black-box attack against a real-world classiﬁer and state-of-the-art defenses.
Revised: We also apply our black-box attack against a real-world classiﬁer and state-of-the-art defenses.

Original: All of our results show that Gradient Estimation attacks are extremely effective in a variety of settings, which motivates better defenses against black-box attacks.
Revised: All of our results show that Gradient Estimation attacks are extremely effective in a variety of settings, which motivates better defenses against black-box attacks.

Original: A E XISTING ATTACKS
Revised: A E XISTING ATTACKS

Original: In this section, we describe existing methods for generating adversarial examples.
Revised: In this section, we describe existing methods for generating adversarial examples.

Original: A. B LACK - BOX A DVERSARIAL E XAMPLES
Revised: A. B LACK - BOX A DVERSARIAL E XAMPLES

Original: An adversary can generate adversarial example x adv from a benign sample x by adding small magnitude of perturbation (Szegedy et al., 2014).
Revised: An adversary can generate adversarial example x adv from a benign sample x by adding small magnitude of perturbation (Szegedy et al., 2014).

Original: Such adversarial example x adv will attack the classiﬁer to misclassify it into a targeted class (targeted attack), or any class other than the ground truth class (untargeted attack).
Revised: Such adversarial example x adv will attack the classiﬁer to misclassify it into a targeted class (targeted attack), or any class other than the ground truth class (untargeted attack).

Original: Now we describe 2 baseline black-box attacks which can be carried out without any knowledge of or query access to the target model.
Revised: Now we describe 2 baseline black-box attacks which can be carried out without any knowledge of or query access to the target model.

Original: A.1. B ASELINE ATTACKS
Revised: A.1. B ASELINE ATTACKS

Original: Random perturbations.
Revised: Random perturbations.

Original: With no knowledge of f or the training set, the simplest manner in which an adversary may seek to carry out an attack is by adding a random perturbation to the input (Szegedy et al.
Revised: With no knowledge of f or the training set, the simplest manner in which an adversary may seek to carry out an attack is by adding a random perturbation to the input (Szegedy et al.

Original: ; Goodfellow et al.
Revised: ; Goodfellow et al.

Original: ; Fawzi et al.
Revised: ; Fawzi et al.

Original: These perturbations can be generated by any distribution of the adversary’s choice and constrained according to an appropriate norm.
Revised: These perturbations can be generated by any distribution of the adversary’s choice and constrained according to an appropriate norm.

Original: If we let
Revised: If we let

Original: P be a distribution over D , and p is a random variable drawn according to P , then a noisy sample is just x noise = x + p .
Revised: P be a distribution over D , and p is a random variable drawn according to P , then a noisy sample is just x noise = x + p .

Original: Since random noise is added, it is not possible to generate targeted adversarial samples in a principled manner.
Revised: Since random noise is added, it is not possible to generate targeted adversarial samples in a principled manner.

Original: This attack is denoted as Rand.
Revised: This attack is denoted as Rand.

Original: throughout.
Revised: throughout.

Original: Difference of means.
Revised: Difference of means.

Original: A perturbation aligned with the difference of means of two classes is likely to be effective for an adversary hoping to cause misclassiﬁcation for a broad range of classiﬁers (Tramèr et al.
Revised: A perturbation aligned with the difference of means of two classes is likely to be effective for an adversary hoping to cause misclassiﬁcation for a broad range of classiﬁers (Tramèr et al.

Original: (2017b)).
Revised: (2017b)).

Original: While these perturbations are far from optimal for DNNs, they provide a useful baseline to compare against.
Revised: While these perturbations are far from optimal for DNNs, they provide a useful baseline to compare against.

Original: Adversaries with at least partial access to the training or test sets can carry out this attack.
Revised: Adversaries with at least partial access to the training or test sets can carry out this attack.

Original: An adversarial sample generated using this method, and with L ∞ constraints is x adv = x + (cid:15) · sign ( µ t − µ o ) , where µ t is the mean of the target class and µ o is the mean of the original ground truth class.
Revised: An adversarial sample generated using this method, and with L ∞ constraints is x adv = x + (cid:15) · sign ( µ t − µ o ) , where µ t is the mean of the target class and µ o is the mean of the original ground truth class.

Original: For an untargeted attack, t = argmin i d ( µ i − µ o ) , where d ( · , · ) is an appropriately chosen distance function.
Revised: For an untargeted attack, t = argmin i d ( µ i − µ o ) , where d ( · , · ) is an appropriately chosen distance function.

Original: In other words, the class whose mean is closest to the original class in terms of the Euclidean distance is chosen to be the target.
Revised: In other words, the class whose mean is closest to the original class in terms of the Euclidean distance is chosen to be the target.

Original: This attack is denoted as D. of M. throughout.
Revised: This attack is denoted as D. of M. throughout.

Original: Now, we describe 2 white-box attack methods for which we will construct approximate, gradientfree versions in Section 3.
Revised: Now, we describe 2 white-box attack methods for which we will construct approximate, gradientfree versions in Section 3.

Original: These attacks are based on either iterative or single-step gradient based minimization of appropriately deﬁned loss functions of neural networks.
Revised: These attacks are based on either iterative or single-step gradient based minimization of appropriately deﬁned loss functions of neural networks.

Original: Since these methods all require the knowledge of the model’s gradient, we assume the adversary has access to a local model f s .
Revised: Since these methods all require the knowledge of the model’s gradient, we assume the adversary has access to a local model f s .

Original: Adversarial samples generated for f s can then be transferred to the target model f t to carry out a transferability-based attack (Papernot et al.
Revised: Adversarial samples generated for f s can then be transferred to the target model f t to carry out a transferability-based attack (Papernot et al.

Original: (2016); Moosavi-Dezfooli et al.
Revised: (2016); Moosavi-Dezfooli et al.

Original: An ensemble of local models (Liu et al.
Revised: An ensemble of local models (Liu et al.

Original: (2017)) may also be used.
Revised: (2017)) may also be used.

Original: Transferability-based attacks are described in
Revised: Transferability-based attacks are described in

Original: Appendix G.
Revised: Appendix G.

Original: A.1. S INGLE - STEP AND I TERATIVE F AST G RADIENT M ETHODS
Revised: A.1. S INGLE - STEP AND I TERATIVE F AST G RADIENT M ETHODS

Original: The single-step Fast Gradient method, ﬁrst introduced in Goodfellow et al. (2015), utilizes a ﬁrstorder approximation of the loss function in order to construct adversarial samples for the adversary’s surrogate local model f s .
Revised: The single-step Fast Gradient method, ﬁrst introduced in Goodfellow et al. (2015), utilizes a ﬁrstorder approximation of the loss function in order to construct adversarial samples for the adversary’s surrogate local model f s .

Original: The samples are constructed by performing a single step of gradient ascent for untargeted attacks.
Revised: The samples are constructed by performing a single step of gradient ascent for untargeted attacks.

Original: Formally, the adversary generates samples x adv with L ∞ constraints (known as the Fast Gradient Sign (FGS) method) in the untargeted attack setting as
Revised: Formally, the adversary generates samples x adv with L ∞ constraints (known as the Fast Gradient Sign (FGS) method) in the untargeted attack setting as

Original: [Equation]
Revised: [Equation]

Original: (7) where (cid:96) f s ( x , y ) is the loss function with respect to which the gradient is taken.
Revised: (7) where (cid:96) f s ( x , y ) is the loss function with respect to which the gradient is taken.

Original: The loss function typically used is the cross-entropy loss (Goodfellow et al.
Revised: The loss function typically used is the cross-entropy loss (Goodfellow et al.

Original: Iterative Fast Gradient Methods are simply multi-step variants of the Fast Gradient method described above (Kurakin et al.
Revised: Iterative Fast Gradient Methods are simply multi-step variants of the Fast Gradient method described above (Kurakin et al.

Original: (2016)), where the signed gradient of the loss is added to the sample for t +iterations, starting from the benign sample,
Revised: (2016)), where the signed gradient of the loss is added to the sample for t +iterations, starting from the benign sample,

Original: and the updated sample is projected back to the constraint region H in every step:
Revised: and the updated sample is projected back to the constraint region H in every step:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: with x 0 adv = x .
Revised: with x 0 adv = x .

Original: Iterative fast gradient methods thus essentially carry out projected gradient descent (PGD) with the goal of maximizing the loss as pointed out by M ˛adry et al.
Revised: Iterative fast gradient methods thus essentially carry out projected gradient descent (PGD) with the goal of maximizing the loss as pointed out by M ˛adry et al.

Original: A.1. B EYOND THE CROSS - ENTROPY LOSS
Revised: A.1. B EYOND THE CROSS - ENTROPY LOSS

Original: Prior work by Carlini & Wagner (2017) investigates a variety of loss functions for white-box attacks based on the minimization of an appropriately deﬁned loss function.
Revised: Prior work by Carlini & Wagner (2017) investigates a variety of loss functions for white-box attacks based on the minimization of an appropriately deﬁned loss function.

Original: In our experiments with neural networks, for untargeted attacks, we use a loss function based on logits which was found to work well for white-box attacks in Carlini & Wagner (2017).
Revised: In our experiments with neural networks, for untargeted attacks, we use a loss function based on logits which was found to work well for white-box attacks in Carlini & Wagner (2017).

Original: The loss function is given by
Revised: The loss function is given by

Original: [Equation]
Revised: [Equation]

Original: where y represents the ground truth label for the benign sample x , φ ( · ) are the logits.
Revised: where y represents the ground truth label for the benign sample x , φ ( · ) are the logits.

Original: κ is a conﬁdence parameter that can be adjusted to control the strength of the adversarial sample.
Revised: κ is a conﬁdence parameter that can be adjusted to control the strength of the adversarial sample.

Original: B E VALUATION SETUP DETAILS B. D ATASETS
Revised: B E VALUATION SETUP DETAILS B. D ATASETS

Original: MNIST.
Revised: MNIST.

Original: This is a dataset of images of handwritten digits LeCun & Cortes (1998).
Revised: This is a dataset of images of handwritten digits LeCun & Cortes (1998).

Original: There are 60, training examples and 10,000 test examples.
Revised: There are 60, training examples and 10,000 test examples.

Original: Each image belongs to a single class from 0 to 9.
Revised: Each image belongs to a single class from 0 to 9.

Original: The images have a dimension d of 28 × 28 pixels (total of 784) and are grayscale.
Revised: The images have a dimension d of 28 × 28 pixels (total of 784) and are grayscale.

Original: Each pixel value lies in [0 , 1] .
Revised: Each pixel value lies in [0 , 1] .

Original: The digits are size-normalized and centered.
Revised: The digits are size-normalized and centered.

Original: This dataset is used commonly as a ‘sanity-check’ or ﬁrst-level benchmark for state-of-the-art classiﬁers.
Revised: This dataset is used commonly as a ‘sanity-check’ or ﬁrst-level benchmark for state-of-the-art classiﬁers.

Original: We use this dataset since it has been extensively studied from the attack perspective by previous work.
Revised: We use this dataset since it has been extensively studied from the attack perspective by previous work.

Original: CIFAR-10.
Revised: CIFAR-10.

Original: This is a dataset of color images from 10 classes Krizhevsky & Hinton (2009).
Revised: This is a dataset of color images from 10 classes Krizhevsky & Hinton (2009).

Original: The images belong to 10 mutually exclusive classes (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck).
Revised: The images belong to 10 mutually exclusive classes (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck).

Original: There are 50,000 training examples and 10,000 test examples.
Revised: There are 50,000 training examples and 10,000 test examples.

Original: There are exactly 6000 examples in each class.
Revised: There are exactly 6000 examples in each class.

Original: The images have a dimension of 32 × 32 pixels (total of 1024) and have 3 channels (Red, Green, and
Revised: The images have a dimension of 32 × 32 pixels (total of 1024) and have 3 channels (Red, Green, and

Original: Blue).
Revised: Blue).

Original: Each pixel value lies in [0 , 255] .
Revised: Each pixel value lies in [0 , 255] .

Original: B. M ODEL TRAINING DETAILS
Revised: B. M ODEL TRAINING DETAILS

Original: In this section, we present the architectures and training details for both the normally and adversarially trained variants of the models on both the MNIST and CIFAR-10 datasets.
Revised: In this section, we present the architectures and training details for both the normally and adversarially trained variants of the models on both the MNIST and CIFAR-10 datasets.

Original: The accuracy of each model on benign data is given in Table 3.
Revised: The accuracy of each model on benign data is given in Table 3.

Original: MNIST.
Revised: MNIST.

Original: The model details for the 4 models trained on the MNIST dataset are as follows:
Revised: The model details for the 4 models trained on the MNIST dataset are as follows:

Original: Model A (3,382,346 parameters): Conv(64, 5, 5) + Relu, Conv(64, 5, 5) + Relu, Dropout(0.25), FC(128)
Revised: Model A (3,382,346 parameters): Conv(64, 5, 5) + Relu, Conv(64, 5, 5) + Relu, Dropout(0.25), FC(128)

Original: + Relu, Dropout(0.5), FC + Softmax 2.
Revised: + Relu, Dropout(0.5), FC + Softmax 2.

Original: Model B (710,218 parameters) - Dropout(0.2), Conv(64, 8, 8) + Relu, Conv(128, 6, 6) + Relu, Conv(128, 5, 5) + Relu, Dropout(0.5), FC + Softmax 3.
Revised: Model B (710,218 parameters) - Dropout(0.2), Conv(64, 8, 8) + Relu, Conv(128, 6, 6) + Relu, Conv(128, 5, 5) + Relu, Dropout(0.5), FC + Softmax 3.

Original: Model C (4,795,082 parameters) - Conv(128, 3, 3) + Relu, Conv(64, 3, 3) + Relu, Dropout(0.25), FC(128)
Revised: Model C (4,795,082 parameters) - Conv(128, 3, 3) + Relu, Conv(64, 3, 3) + Relu, Dropout(0.25), FC(128)

Original: + Relu, Dropout(0.5), FC + Softmax 4.
Revised: + Relu, Dropout(0.5), FC + Softmax 4.

Original: Model D (509,410 parameters) -
Revised: Model D (509,410 parameters) -

Original: [FC(300) + Relu, Dropout(0.5)] × 4, FC + Softmax
Revised: [FC(300) + Relu, Dropout(0.5)] × 4, FC + Softmax

Original: Models A and C have both convolutional layers as well as fully connected layers.
Revised: Models A and C have both convolutional layers as well as fully connected layers.

Original: They also have the same order of magnitude of parameters.
Revised: They also have the same order of magnitude of parameters.

Original: Model B, on the other hand, does not have fully connected layers and has an order of magnitude fewer parameters.
Revised: Model B, on the other hand, does not have fully connected layers and has an order of magnitude fewer parameters.

Original: Similarly, Model D has no convolutional layers and has fewer parameters than all the other models.
Revised: Similarly, Model D has no convolutional layers and has fewer parameters than all the other models.

Original: The details of the training for these models are given in Appendix B.2.
Revised: The details of the training for these models are given in Appendix B.2.

Original: Models A, B, and C all achieve greater than 99% classiﬁcation accuracy on the test data.
Revised: Models A, B, and C all achieve greater than 99% classiﬁcation accuracy on the test data.

Original: Model D achieves 97.2% classiﬁcation accuracy, due to the lack of convolutional layers.
Revised: Model D achieves 97.2% classiﬁcation accuracy, due to the lack of convolutional layers.

Original: For all adversarially trained models, each training batch contains 128 samples of which 64 are benign and 64 are adversarial samples (either FGSM or iterative FGSM).
Revised: For all adversarially trained models, each training batch contains 128 samples of which 64 are benign and 64 are adversarial samples (either FGSM or iterative FGSM).

Original: This implies that the loss for each is weighted equally during training; i.e., in Eq.
Revised: This implies that the loss for each is weighted equally during training; i.e., in Eq.

Original: 10, α is set to 0.5.
Revised: 10, α is set to 0.5.

Original: For ensemble adversarial training, the source of the FGSM samples is chosen randomly for each training batch.
Revised: For ensemble adversarial training, the source of the FGSM samples is chosen randomly for each training batch.

Original: Networks using standard and ensemble adversarial training are trained for 12 epochs, while those using iterative adversarial training are trained for 64 epochs.
Revised: Networks using standard and ensemble adversarial training are trained for 12 epochs, while those using iterative adversarial training are trained for 64 epochs.

Original: CIFAR-10.
Revised: CIFAR-10.

Original: As their name indicates, Resnet-32 and Resnet-28-10 are ResNet variants
Revised: As their name indicates, Resnet-32 and Resnet-28-10 are ResNet variants

Original: He et al.
Revised: He et al.

Original: (2016); Zagoruyko & Komodakis (2016), while Std-CNN is a standard CNN std.
Revised: (2016); Zagoruyko & Komodakis (2016), while Std-CNN is a standard CNN std.

Original: In particular,
Revised: In particular,

Original: Resnet-32 is a standard 32 layer ResNet with no width expansion, and Resnet-28-10 is a wide
Revised: Resnet-32 is a standard 32 layer ResNet with no width expansion, and Resnet-28-10 is a wide

Original: Resnet with 28 layers with the width set to 10, based on the best performing ResNet from Zagoruyko and Komodakis Zagoruyko & Komodakis (2016); res.
Revised: Resnet with 28 layers with the width set to 10, based on the best performing ResNet from Zagoruyko and Komodakis Zagoruyko & Komodakis (2016); res.

Original: The width indicates the multiplicative factor by which the number of ﬁlters in each residual layer is increased.
Revised: The width indicates the multiplicative factor by which the number of ﬁlters in each residual layer is increased.

Original: Std-CNN is a CNN with two convolutional layers, each followed by a max-pooling and normalization layer and two fully connected layers, each of which has weight decay.
Revised: Std-CNN is a CNN with two convolutional layers, each followed by a max-pooling and normalization layer and two fully connected layers, each of which has weight decay.

Original: For each model architecture, we train 3 models, one on only the CIFAR-10 training data, one using standard adversarial training and one using ensemble adversarial training.
Revised: For each model architecture, we train 3 models, one on only the CIFAR-10 training data, one using standard adversarial training and one using ensemble adversarial training.

Original: Resnet-32 is trained for 125,000 steps, Resnet-28-10 is trained for 167,000 steps and Std.-CNN is trained for 100, steps on the benign training data.
Revised: Resnet-32 is trained for 125,000 steps, Resnet-28-10 is trained for 167,000 steps and Std.-CNN is trained for 100, steps on the benign training data.

Original: Models Resnet-32 and Resnet-28-10 are much more accurate than Std.-CNN .
Revised: Models Resnet-32 and Resnet-28-10 are much more accurate than Std.-CNN .

Original: The adversarial variants of Resnet-32 is trained for 80,000 steps.
Revised: The adversarial variants of Resnet-32 is trained for 80,000 steps.

Original: All models were trained with a batch size of 128.
Revised: All models were trained with a batch size of 128.

Original: The two ResNets achieve close to state-of-the-art accuracy ima on the CIFAR-10 test set, with
Revised: The two ResNets achieve close to state-of-the-art accuracy ima on the CIFAR-10 test set, with

Original: Resnet-32 at 92.4% and Resnet-28-10 at 94.4%.
Revised: Resnet-32 at 92.4% and Resnet-28-10 at 94.4%.

Original: The Std-CNN , on the other hand, only achieves an accuracy of 81.4%, reﬂecting its simple architecture and the complexity of the task.
Revised: The Std-CNN , on the other hand, only achieves an accuracy of 81.4%, reﬂecting its simple architecture and the complexity of the task.

Original: B. A LTERNATIVE ADVERSARIAL SUCCESS METRIC
Revised: B. A LTERNATIVE ADVERSARIAL SUCCESS METRIC

Original: Note that the adversarial success rate can also be computed by considering only the fraction of inputs that meet the adversary’s objective, given that the original sample was correctly classiﬁed .
Revised: Note that the adversarial success rate can also be computed by considering only the fraction of inputs that meet the adversary’s objective, given that the original sample was correctly classiﬁed .

Original: So, we count the fraction of correctly classiﬁed inputs (i.e. f ( x ) =
Revised: So, we count the fraction of correctly classiﬁed inputs (i.e. f ( x ) =

Original: y ) for which f ( x adv ) (cid:54) =
Revised: y ) for which f ( x adv ) (cid:54) =

Original: y in the untargeted case, and f t ( x adv )
Revised: y in the untargeted case, and f t ( x adv )

Original: =
Revised: =

Original: T in the targeted case.
Revised: T in the targeted case.

Original: In a sense, this fraction represents those samples which are truly adversarial, since they are misclassiﬁed solely due to the adversarial perturbation added, and not due to the classiﬁer’s failure to generalize well.
Revised: In a sense, this fraction represents those samples which are truly adversarial, since they are misclassiﬁed solely due to the adversarial perturbation added, and not due to the classiﬁer’s failure to generalize well.

Original: In practice, both these methods of measuring the adversarial success rate lead to similar results for classiﬁers with high accuracy on the test data.
Revised: In practice, both these methods of measuring the adversarial success rate lead to similar results for classiﬁers with high accuracy on the test data.

Original: C G RADIENT ESTIMATION WITH Q UERY REDUCTION C. R ANDOM GROUPING
Revised: C G RADIENT ESTIMATION WITH Q UERY REDUCTION C. R ANDOM GROUPING

Original: This section contains the detailed algorithm for query reduction using random grouping.
Revised: This section contains the detailed algorithm for query reduction using random grouping.

Original: C. PCA
Revised: C. PCA

Original: Concretely, let the samples the adversary wants to misclassify be column vectors x i ∈ R d for i ∈ { 1 , . .
Revised: Concretely, let the samples the adversary wants to misclassify be column vectors x i ∈ R d for i ∈ { 1 , . .

Original: . , n } and let X be the d × n matrix of centered data samples (i.e. X
Revised: . , n } and let X be the d × n matrix of centered data samples (i.e. X

Original: =
Revised: =

Original: [˜ x 1 ˜ x 2 . . .
Revised: [˜ x 1 ˜ x 2 . . .

Original: ˜ x n ] , where ˜ x i = x − 1 n (cid:80) nj =1 x j ).
Revised: ˜ x n ] , where ˜ x i = x − 1 n (cid:80) nj =1 x j ).

Original: The principal components of X are the normalized eigenvectors of its sample covariance matrix C =
Revised: The principal components of X are the normalized eigenvectors of its sample covariance matrix C =

Original: XX T .
Revised: XX T .

Original: Since C is a positive semideﬁnite matrix, there is a decomposition
Revised: Since C is a positive semideﬁnite matrix, there is a decomposition

Original: C = U Λ U T where U is an orthogonal matrix, Λ = diag( λ 1 , . . .
Revised: C = U Λ U T where U is an orthogonal matrix, Λ = diag( λ 1 , . . .

Original: , λ d ) , and λ 1 ≥ . .
Revised: , λ d ) , and λ 1 ≥ . .

Original: ≥ λ d ≥ 0 .
Revised: ≥ λ d ≥ 0 .

Original: Thus, U in Algorithm 2 is the d × d matrix whose columns are unit eigenvectors of C .
Revised: Thus, U in Algorithm 2 is the d × d matrix whose columns are unit eigenvectors of C .

Original: The eigenvalue λ i is the variance of X along the i th component.
Revised: The eigenvalue λ i is the variance of X along the i th component.

Original: Further, PCA minimizes reconstruction error in terms of the (cid:96) 2 norm, i.e. it provides a basis in which the Euclidean distance to the original sample from a sample reconstructed using a subset of the basis vectors is the smallest.
Revised: Further, PCA minimizes reconstruction error in terms of the (cid:96) 2 norm, i.e. it provides a basis in which the Euclidean distance to the original sample from a sample reconstructed using a subset of the basis vectors is the smallest.

Original: Input
Revised: Input

Original: Output: Estimated gradient
Revised: Output: Estimated gradient

Original: ∇
Revised: ∇

Original: 1: for i
Revised: 1: for i

Original: 1 to k do 2:
Revised: 1 to k do 2:

Original: Initialize v such that v =
Revised: Initialize v such that v =

Original: u
Revised: u

Original: i (cid:107) u i (cid:107) , where u i is the i th column of U
Revised: i (cid:107) u i (cid:107) , where u i is the i th column of U

Original: Compute
Revised: Compute

Original: [Equation]
Revised: [Equation]

Original: which is the two-sided approximation of the directional derivative along v
Revised: which is the two-sided approximation of the directional derivative along v

Original: [Equation]
Revised: [Equation]

Original: : end for
Revised: : end for

Original: D B ACKGROUND ON A DVERSARIAL T RAINING .
Revised: D B ACKGROUND ON A DVERSARIAL T RAINING .

Original: Szegedy et al.
Revised: Szegedy et al.

Original: (2014) and Goodfellow et al.
Revised: (2014) and Goodfellow et al.

Original: (2015) introduced the concept of adversarial training, where the standard loss function for a neural network f is modiﬁed as follows:
Revised: (2015) introduced the concept of adversarial training, where the standard loss function for a neural network f is modiﬁed as follows:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where y is the true label of the sample x .
Revised: where y is the true label of the sample x .

Original: The underlying objective of this modiﬁcation is to make the neural networks more robust by penalizing it during training to count for adversarial samples.
Revised: The underlying objective of this modiﬁcation is to make the neural networks more robust by penalizing it during training to count for adversarial samples.

Original: Ensemble adversarial training.
Revised: Ensemble adversarial training.

Original: Tramèr et al.
Revised: Tramèr et al.

Original: (2017a) proposed an extension of the adversarial training paradigm which is called ensemble adversarial training .
Revised: (2017a) proposed an extension of the adversarial training paradigm which is called ensemble adversarial training .

Original: As the name suggests, in ensemble adversarial training, the network is trained with adversarial samples from multiple networks.
Revised: As the name suggests, in ensemble adversarial training, the network is trained with adversarial samples from multiple networks.

Original: Iterative adversarial training.
Revised: Iterative adversarial training.

Original: A further modiﬁcation of the adversarial training paradigm proposes training with adversarial samples generated using iterative methods such as the iterative FGSM attack described earlier (M ˛adry et al.
Revised: A further modiﬁcation of the adversarial training paradigm proposes training with adversarial samples generated using iterative methods such as the iterative FGSM attack described earlier (M ˛adry et al.

Original: In the discussion of our results
Revised: In the discussion of our results

Original: [Table]
Revised: [Table]

Original: we focus on the attack success rate obtained by Iterative Gradient
Revised: we focus on the attack success rate obtained by Iterative Gradient

Original: Estimation attacks, since they perform much better than any single-step black-box attack.
Revised: Estimation attacks, since they perform much better than any single-step black-box attack.

Original: Nevertheless, in Figure 6 and Appendix J.
Revised: Nevertheless, in Figure 6 and Appendix J.

Original: [Table]
Revised: [Table]

Original: we show that with the addition of an initial random perturbation to overcome “gradient masking” (Tramèr et al.
Revised: we show that with the addition of an initial random perturbation to overcome “gradient masking” (Tramèr et al.

Original: (2017a)), the Gradient Estimation attack with Finite
Revised: (2017a)), the Gradient Estimation attack with Finite

Original: Differences is the most effective single-step black-box attack on adversarially trained models on
Revised: Differences is the most effective single-step black-box attack on adversarially trained models on

Original: MNIST.
Revised: MNIST.

Original: E T ARGETED ATTACKS BASED ON FINITE DIFFERENCES
Revised: E T ARGETED ATTACKS BASED ON FINITE DIFFERENCES

Original: [Table]
Revised: [Table]

Original: The expressions for targeted white-box and Gradient Estimation attacks are given in this section.
Revised: The expressions for targeted white-box and Gradient Estimation attacks are given in this section.

Original: Targeted transferability attacks are carried out using locally generated targeted white-box adversarial samples.
Revised: Targeted transferability attacks are carried out using locally generated targeted white-box adversarial samples.

Original: Adversarial samples generated using the targeted FGS attack are
Revised: Adversarial samples generated using the targeted FGS attack are

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where T is the target class.
Revised: where T is the target class.

Original: Similarly, the adversarial samples generated using iterative FGS are
Revised: Similarly, the adversarial samples generated using iterative FGS are

Original: [Equation]
Revised: [Equation]

Original: For the logit based loss, targeted adversarial samples are generated using the following loss term:
Revised: For the logit based loss, targeted adversarial samples are generated using the following loss term:

Original: [Equation]
Revised: [Equation]

Original: Targeted black-box adversarial samples generated using the Gradient Estimation method are then
Revised: Targeted black-box adversarial samples generated using the Gradient Estimation method are then

Original: [Equation]
Revised: [Equation]

Original: Similarly, in the case of a black-box adversary with query-access to the logits, the adversarial sample is
Revised: Similarly, in the case of a black-box adversary with query-access to the logits, the adversarial sample is

Original: [Equation]
Revised: [Equation]

Original: F F ORMAL DEFINITIONS FOR QUERY - BASED ATTACKS
Revised: F F ORMAL DEFINITIONS FOR QUERY - BASED ATTACKS

Original: Here, we provide a uniﬁed framework assuming an adversary can make active queries to the model.
Revised: Here, we provide a uniﬁed framework assuming an adversary can make active queries to the model.

Original: Existing attacks making zero queries are a special case in this framework.
Revised: Existing attacks making zero queries are a special case in this framework.

Original: Given an input instance x , the adversary makes a sequence of queries based on the adversarial constraint set H , and iteratively adds perturbations until the desired query results are obtained, using which the corresponding adversarial example x adv is generated.
Revised: Given an input instance x , the adversary makes a sequence of queries based on the adversarial constraint set H , and iteratively adds perturbations until the desired query results are obtained, using which the corresponding adversarial example x adv is generated.

Original: We formally deﬁne the targeted and untargeted black-box attacks based on the framework as below.
Revised: We formally deﬁne the targeted and untargeted black-box attacks based on the framework as below.

Original: Deﬁnition 1 (Untargeted black-box attack) .
Revised: Deﬁnition 1 (Untargeted black-box attack) .

Original: Given an input instance x and an iterative active query attack strategy A , a query sequence can generated as x 2 =
Revised: Given an input instance x and an iterative active query attack strategy A , a query sequence can generated as x 2 =

Original: A ( { ( x 1 , q 1 f ) } , H ) , ..., x i =
Revised: A ( { ( x 1 , q 1 f ) } , H ) , ..., x i =

Original: A ( { ( x 1 , q 1 f ) , . . .
Revised: A ( { ( x 1 , q 1 f ) , . . .

Original: , ( x i − 1 , q i − 1 f ) } , H ) , where q if denotes the i th corresponding query result on x i , and we set x 1 = x .
Revised: , ( x i − 1 , q i − 1 f ) } , H ) , where q if denotes the i th corresponding query result on x i , and we set x 1 = x .

Original: A black-box attack on f ( · ; θ ) is untargeted if the adversarial example x adv = x k satisﬁes f ( x adv ; θ ) (cid:54) =
Revised: A black-box attack on f ( · ; θ ) is untargeted if the adversarial example x adv = x k satisﬁes f ( x adv ; θ ) (cid:54) =

Original: f ( x ; θ ) , where k is the number of queries made.
Revised: f ( x ; θ ) , where k is the number of queries made.

Original: Deﬁnition 2 (Targeted black-box attack) .
Revised: Deﬁnition 2 (Targeted black-box attack) .

Original: Given an input instance x and an iterative active query attack strategy A , a query sequence can generated as x 2 =
Revised: Given an input instance x and an iterative active query attack strategy A , a query sequence can generated as x 2 =

Original: A ( { ( x 1 , q 1 f ) } , H ) , ..., x i =
Revised: A ( { ( x 1 , q 1 f ) } , H ) , ..., x i =

Original: A ( { ( x 1 , q 1 f ) , . . .
Revised: A ( { ( x 1 , q 1 f ) , . . .

Original: , ( x i − 1 , q i − 1 f ) } , H ) , where q if denotes the i th corresponding query result on x i , and we set x 1 = x .
Revised: , ( x i − 1 , q i − 1 f ) } , H ) , where q if denotes the i th corresponding query result on x i , and we set x 1 = x .

Original: A black-box attack on f ( · ; θ ) is targeted if the adversarial example x adv = x
Revised: A black-box attack on f ( · ; θ ) is targeted if the adversarial example x adv = x

Original: k satisﬁes f ( x adv ; θ ) =
Revised: k satisﬁes f ( x adv ; θ ) =

Original: T , where T and k are the target class and the number of queries made, respectively.
Revised: T , where T and k are the target class and the number of queries made, respectively.

Original: The case where the adversary makes no queries to the target classiﬁer is a special case we refer to as a zero-query attack.
Revised: The case where the adversary makes no queries to the target classiﬁer is a special case we refer to as a zero-query attack.

Original: In the literature, a number of these zero-query attacks have been carried out with varying degrees of success (Papernot et al.
Revised: In the literature, a number of these zero-query attacks have been carried out with varying degrees of success (Papernot et al.

Original: ; Liu et al.
Revised: ; Liu et al.

Original: (2017); Moosavi-Dezfooli et al.
Revised: (2017); Moosavi-Dezfooli et al.

Original: Mopuri et al.
Revised: Mopuri et al.

Original: Taxonomy of black-box attacks : To deepen our understanding of the effectiveness of black-box attacks, in this work, we propose a taxonomy of black-box attacks, intuitively based on the number of queries on the target model used in the attack.
Revised: Taxonomy of black-box attacks : To deepen our understanding of the effectiveness of black-box attacks, in this work, we propose a taxonomy of black-box attacks, intuitively based on the number of queries on the target model used in the attack.

Original: The details are provided in Table 6.
Revised: The details are provided in Table 6.

Original: G T RANSFERABILITY BASED ATTACK
Revised: G T RANSFERABILITY BASED ATTACK

Original: Here we describe black-box attacks that assume the adversary has access to a representative set of training data in order to train a local model.
Revised: Here we describe black-box attacks that assume the adversary has access to a representative set of training data in order to train a local model.

Original: One of the earliest observations with regards to adversarial samples for neural networks was that they transfer , i.e. adversarial attack samples generated for one network are also adversarial for another network.
Revised: One of the earliest observations with regards to adversarial samples for neural networks was that they transfer , i.e. adversarial attack samples generated for one network are also adversarial for another network.

Original: This observation directly led to the proposal of black-box attacks where an adversary would generate samples for a local network and transfer these to the target model, which is referred to as Transferability based attack.
Revised: This observation directly led to the proposal of black-box attacks where an adversary would generate samples for a local network and transfer these to the target model, which is referred to as Transferability based attack.

Original: Transferability attack (single local model).
Revised: Transferability attack (single local model).

Original: These attacks use a surrogate local model
Revised: These attacks use a surrogate local model

Original: f s to craft adversarial samples, which are then submitted to f in order to cause misclassiﬁcation.
Revised: f s to craft adversarial samples, which are then submitted to f in order to cause misclassiﬁcation.

Original: Most existing black-box attacks are based on transferability from a single local model Papernot et al.
Revised: Most existing black-box attacks are based on transferability from a single local model Papernot et al.

Original: Moosavi-Dezfooli et al.
Revised: Moosavi-Dezfooli et al.

Original: The different attack strategies to generate adversarial instances introduced in Section A.1 can be used here to generate adversarial instances against f s , so as to attack f .
Revised: The different attack strategies to generate adversarial instances introduced in Section A.1 can be used here to generate adversarial instances against f s , so as to attack f .

Original: Transferability attack (local model ensemble).
Revised: Transferability attack (local model ensemble).

Original: Since it is not clear which local model f s is best suited for generating adversarial sample that transfer well to the target model f , Liu et al.
Revised: Since it is not clear which local model f s is best suited for generating adversarial sample that transfer well to the target model f , Liu et al.

Original: Liu et al.
Revised: Liu et al.

Original: (2017) propose the generation of adversarial examples for an ensemble of local models.
Revised: (2017) propose the generation of adversarial examples for an ensemble of local models.

Original: This method modiﬁes each of the existing Transferability attack (single local model) by substituting a sum over the loss functions in place of the loss from a single local model.
Revised: This method modiﬁes each of the existing Transferability attack (single local model) by substituting a sum over the loss functions in place of the loss from a single local model.

Original: Concretely, let the ensemble of m local models to be used to generate the local loss be { f s 1 , . .
Revised: Concretely, let the ensemble of m local models to be used to generate the local loss be { f s 1 , . .

Original: , f s m } .
Revised: , f s m } .

Original: The ensemble loss is then computed as (cid:96) ens ( x , y )
Revised: The ensemble loss is then computed as (cid:96) ens ( x , y )

Original: = (cid:80) mi =

Revised: = (cid:80) mi =

Original: α i (cid:96) f si ( x , y ) , where α i is the weight given to each model in the ensemble.
Revised: α i (cid:96) f si ( x , y ) , where α i is the weight given to each model in the ensemble.

Original: The FGS attack in the ensemble setting then becomes x adv = x + (cid:15) · sign ( ∇ x (cid:96) ens ( x , y )) .
Revised: The FGS attack in the ensemble setting then becomes x adv = x + (cid:15) · sign ( ∇ x (cid:96) ens ( x , y )) .

Original: The Iterative FGS attack is modiﬁed similarly.
Revised: The Iterative FGS attack is modiﬁed similarly.

Original: Liu et al.
Revised: Liu et al.

Original: show that the Transferability attack (local model ensemble) performs well even in the targeted attack case, while Transferability attack (single local model) is usually only effective for untargeted attacks.
Revised: show that the Transferability attack (local model ensemble) performs well even in the targeted attack case, while Transferability attack (single local model) is usually only effective for untargeted attacks.

Original: The intuition is that while one model’s gradient may not be adversarial for a target model, it is likely that at least one of the gradient directions from the ensemble represents a direction that is somewhat adversarial for the target model.
Revised: The intuition is that while one model’s gradient may not be adversarial for a target model, it is likely that at least one of the gradient directions from the ensemble represents a direction that is somewhat adversarial for the target model.

Original: G. T RANSFERABILITY OF ADVERSARIAL EXAMPLES
Revised: G. T RANSFERABILITY OF ADVERSARIAL EXAMPLES

Original: For the transferability experiments, we choose to transfer from Model B for MNIST dataset and from Resnet-28-10 for CIFAR-10 dataset as these models are each similar to at least one of the other models for their respective dataset, and different from one of the others.
Revised: For the transferability experiments, we choose to transfer from Model B for MNIST dataset and from Resnet-28-10 for CIFAR-10 dataset as these models are each similar to at least one of the other models for their respective dataset, and different from one of the others.

Original: They are also fairly representative instances of DNNs used in practice.
Revised: They are also fairly representative instances of DNNs used in practice.

Original: Adversarial samples generated using single-step methods and transferred from Model B to the other models have higher success rates for untargeted attacks when they are generated using the logit loss as compared to the cross entropy loss as can be seen in Table 1.
Revised: Adversarial samples generated using single-step methods and transferred from Model B to the other models have higher success rates for untargeted attacks when they are generated using the logit loss as compared to the cross entropy loss as can be seen in Table 1.

Original: For iterative adversarial samples, however, the untargeted attack success rates are roughly the same for both loss functions.
Revised: For iterative adversarial samples, however, the untargeted attack success rates are roughly the same for both loss functions.

Original: As has been observed before, the adversarial success rate for targeted attacks with transferability is much lower than the untargeted case, even when iteratively generated samples are used.
Revised: As has been observed before, the adversarial success rate for targeted attacks with transferability is much lower than the untargeted case, even when iteratively generated samples are used.

Original: For example, the highest
Revised: For example, the highest

Original: Untargeted Transferability to Model A
Revised: Untargeted Transferability to Model A

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: targeted transferability rate in Table 4 is 54.5%, compared to 100.0% achieved by IFD-xent-T across models.
Revised: targeted transferability rate in Table 4 is 54.5%, compared to 100.0% achieved by IFD-xent-T across models.

Original: One attempt to improve the transferability rate is to use an ensemble of local models, instead of a single one.
Revised: One attempt to improve the transferability rate is to use an ensemble of local models, instead of a single one.

Original: The results for this on the MNIST data are presented in Table 5.
Revised: The results for this on the MNIST data are presented in Table 5.

Original: In general, both untargeted and targeted transferability increase when an ensemble is used.
Revised: In general, both untargeted and targeted transferability increase when an ensemble is used.

Original: However, the increase is not monotonic in the number of models used in the ensemble, and we can see that the transferability rate for IFGS-xent samples falls sharply when Model D is added to the ensemble.
Revised: However, the increase is not monotonic in the number of models used in the ensemble, and we can see that the transferability rate for IFGS-xent samples falls sharply when Model D is added to the ensemble.

Original: This may be due to it having a very different architecture as compared to the models, and thus also having very different gradient directions.
Revised: This may be due to it having a very different architecture as compared to the models, and thus also having very different gradient directions.

Original: This highlights one of the pitfalls of transferability, where it is important to use a local surrogate model similar to the target model for achieving high attack success rates.
Revised: This highlights one of the pitfalls of transferability, where it is important to use a local surrogate model similar to the target model for achieving high attack success rates.

Original: H S UMMARY OF ATTACKS EVALUATED
Revised: H S UMMARY OF ATTACKS EVALUATED

Original: We evaluate the following attacks:
Revised: We evaluate the following attacks:

Original: Zero-query attacks (a) Baseline attacks: Random-Gaussian perturbations ( Rand. ) and Difference-of-Means aligned perturbations ( D. of M. )
Revised: Zero-query attacks (a) Baseline attacks: Random-Gaussian perturbations ( Rand. ) and Difference-of-Means aligned perturbations ( D. of M. )

Original: [Figure]
Revised: [Figure]

Original: (b) Transferability attack (single local model) using Fast Gradient Sign (FGS) and Iterative FGS (IFGS) samples generated on a single source model for both loss functions (Transfer model FGS/IFGS- loss ); e
Revised: (b) Transferability attack (single local model) using Fast Gradient Sign (FGS) and Iterative FGS (IFGS) samples generated on a single source model for both loss functions (Transfer model FGS/IFGS- loss ); e

Original: g., Transfer Model A FGS-logit (c) Transferability attack (local model ensemble) using FGS and IFGS samples generated on a source model for both loss functions (Transfer models FGS/IFGS- loss ); e.g., Transfer Model B , Model C IFGS-logit
Revised: g., Transfer Model A FGS-logit (c) Transferability attack (local model ensemble) using FGS and IFGS samples generated on a source model for both loss functions (Transfer models FGS/IFGS- loss ); e.g., Transfer Model B , Model C IFGS-logit

Original: 2. Query based attacks
Revised: 2. Query based attacks

Original: (a) Finite-difference and Iterative Finite-difference attacks for the gradient estimation attack for both loss functions ( FD/IFD- loss ); e.g., FD-logit (b) Gradient Estimation and Iterative Gradient Estimation with Query reduction attacks ( IGE/GE-QR ( Technique - k , loss ) ) using two query reduction techniques, random grouping ( RG ) and principal component analysis components ( PCA ); e.g., GE-QR (PCA- k , logit)
Revised: (a) Finite-difference and Iterative Finite-difference attacks for the gradient estimation attack for both loss functions ( FD/IFD- loss ); e.g., FD-logit (b) Gradient Estimation and Iterative Gradient Estimation with Query reduction attacks ( IGE/GE-QR ( Technique - k , loss ) ) using two query reduction techniques, random grouping ( RG ) and principal component analysis components ( PCA ); e.g., GE-QR (PCA- k , logit)

Original: White-box FGS and IFGS attacks for both loss functions ( WB FGS/IFGS ( loss ) )
Revised: White-box FGS and IFGS attacks for both loss functions ( WB FGS/IFGS ( loss ) )

Original: I A DVERSARIAL SAMPLES
Revised: I A DVERSARIAL SAMPLES

Original: In Figure we show some examples of successful untargeted adversarial samples against Model A on MNIST and Resnet-32 on CIFAR-10.
Revised: In Figure we show some examples of successful untargeted adversarial samples against Model A on MNIST and Resnet-32 on CIFAR-10.

Original: These images were generated with an L ∞ constraint of (cid:15) = 0 .
Revised: These images were generated with an L ∞ constraint of (cid:15) = 0 .

Original: 3 for MNIST and (cid:15) = 8 for CIFAR-10.
Revised: 3 for MNIST and (cid:15) = 8 for CIFAR-10.

Original: Clearly, the amount of perturbation added by iterative attacks is much smaller, barely being visible in the images.
Revised: Clearly, the amount of perturbation added by iterative attacks is much smaller, barely being visible in the images.

Original: J D ETAILED EVALUATION RESULTS J. E FFECTIVENESS OF BASELINE ATTACKS
Revised: J D ETAILED EVALUATION RESULTS J. E FFECTIVENESS OF BASELINE ATTACKS

Original: In the baseline attacks described in Section A.1.1, the choice of distribution for the random perturbation attack and the choice of distance function for the difference of means attack are not ﬁxed.
Revised: In the baseline attacks described in Section A.1.1, the choice of distribution for the random perturbation attack and the choice of distance function for the difference of means attack are not ﬁxed.

Original: Here, we describe the choices we make for both attacks.
Revised: Here, we describe the choices we make for both attacks.

Original: The random perturbation p for each sample (for both MNIST and CIFAR-10) is chosen independently according to a multivariate normal distribution with mean 0 , i.e. p ∼ N ( 0 , I d ) .
Revised: The random perturbation p for each sample (for both MNIST and CIFAR-10) is chosen independently according to a multivariate normal distribution with mean 0 , i.e. p ∼ N ( 0 , I d ) .

Original: Then, depending on the norm constraint, either a signed and scaled version of the random perturbation ( L ∞ ) or a scaled unit vector in the direction of the perturbation ( L 2 ) is added.
Revised: Then, depending on the norm constraint, either a signed and scaled version of the random perturbation ( L ∞ ) or a scaled unit vector in the direction of the perturbation ( L 2 ) is added.

Original: For an untargeted attack utilizing perturbations aligned with the difference of means, for each sample, the mean of the class closest to the original class in the L 2 distance is determined.
Revised: For an untargeted attack utilizing perturbations aligned with the difference of means, for each sample, the mean of the class closest to the original class in the L 2 distance is determined.

Original: [Figure]
Revised: [Figure]

Original: A
Revised: A

Original: [Figure]
Revised: [Figure]

Original: e r s a r i a l s u cce
Revised: e r s a r i a l s u cce

Original: [Figure]
Revised: [Figure]

Original: ( % ) (cid:15)
Revised: ( % ) (cid:15)

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: QR ( Technique , logit) ) on Model A (MNIST) and Resnet-32 (CIFAR-10) , where Technique is either PCA or RG .
Revised: QR ( Technique , logit) ) on Model A (MNIST) and Resnet-32 (CIFAR-10) , where Technique is either PCA or RG .

Original: ‘None’ refers to FD-logit , the case where the number of queries is 2 d , where d is the dimension of the input.
Revised: ‘None’ refers to FD-logit , the case where the number of queries is 2 d , where d is the dimension of the input.

Original: As expected, adversarial samples generated using Rand.
Revised: As expected, adversarial samples generated using Rand.

Original: do not achieve high adversarial success rates in spite of having similar or larger average distortion than the other black-box attacks for both the
Revised: do not achieve high adversarial success rates in spite of having similar or larger average distortion than the other black-box attacks for both the

Original: MNIST and CIFAR-10 models.
Revised: MNIST and CIFAR-10 models.

Original: However, the D. of M. method is quite effective at higher perturbation values for the MNIST dataset as can be seen in Figure 2a.
Revised: However, the D. of M. method is quite effective at higher perturbation values for the MNIST dataset as can be seen in Figure 2a.

Original: Also, for Models B and D , the D. of M. attack is more effective than FD-xent .
Revised: Also, for Models B and D , the D. of M. attack is more effective than FD-xent .

Original: The D. of M. method is less effective in the targeted attack case, but for Model D , it outperforms the transferability based attack considerably.
Revised: The D. of M. method is less effective in the targeted attack case, but for Model D , it outperforms the transferability based attack considerably.

Original: Its success rate is comparable to the targeted transferability based attack for Model A as well.
Revised: Its success rate is comparable to the targeted transferability based attack for Model A as well.

Original: The relative effectiveness of the two baseline methods is reversed for the CIFAR-10 dataset, however, where Rand.
Revised: The relative effectiveness of the two baseline methods is reversed for the CIFAR-10 dataset, however, where Rand.

Original: outperforms D. of M. considerably as (cid:15) is increased.
Revised: outperforms D. of M. considerably as (cid:15) is increased.

Original: This indicates that the models trained on MNIST have normal vectors to decision boundaries which are more aligned with the vectors along the difference of means as compared to the models on CIFAR-10.
Revised: This indicates that the models trained on MNIST have normal vectors to decision boundaries which are more aligned with the vectors along the difference of means as compared to the models on CIFAR-10.

Original: J. E FFECT OF DIMENSION ON G RADIENT E STIMATION ATTACKS WITH QUERY REDUCTION
Revised: J. E FFECT OF DIMENSION ON G RADIENT E STIMATION ATTACKS WITH QUERY REDUCTION

Original: We consider the effectiveness of Gradient Estimation with random grouping based query reduction and the logit loss ( GE-QR (RG- k , logit) ) on Model A on MNIST data in Figure 5a, where k is the number of indices chosen in each iteration of Algorithm 1.
Revised: We consider the effectiveness of Gradient Estimation with random grouping based query reduction and the logit loss ( GE-QR (RG- k , logit) ) on Model A on MNIST data in Figure 5a, where k is the number of indices chosen in each iteration of Algorithm 1.

Original: Thus, as k increases and the number of groups decreases, we expect adversarial success to decrease as gradients over larger groups of features are averaged.
Revised: Thus, as k increases and the number of groups decreases, we expect adversarial success to decrease as gradients over larger groups of features are averaged.

Original: This is the effect we see in Figure 5a, where the adversarial success rate drops from 93% to 63% at (cid:15) = 0 .
Revised: This is the effect we see in Figure 5a, where the adversarial success rate drops from 93% to 63% at (cid:15) = 0 .

Original: 3 as k increases from 1 to 7.
Revised: 3 as k increases from 1 to 7.

Original: Grouping with k = 7 translates toqueries per MNIST image, down from 784.
Revised: Grouping with k = 7 translates toqueries per MNIST image, down from 784.

Original: Thus, in order to achieve high adversarial success rates with the random grouping method, larger perturbation magnitudes are needed.
Revised: Thus, in order to achieve high adversarial success rates with the random grouping method, larger perturbation magnitudes are needed.

Original: On the other hand, the PCA-based approach GE-QR (PCA- k , logit) is much more effective, as can be seen in Figure 5b.
Revised: On the other hand, the PCA-based approach GE-QR (PCA- k , logit) is much more effective, as can be seen in Figure 5b.

Original: Using 100 principal components to estimate the gradient for Model A on
Revised: Using 100 principal components to estimate the gradient for Model A on

Original: MNIST as in Algorithm 2, the adversarial success rate at (cid:15) = 0 .
Revised: MNIST as in Algorithm 2, the adversarial success rate at (cid:15) = 0 .

Original: 3 is 88.09%, as compared to 92.9% without any query reduction.
Revised: 3 is 88.09%, as compared to 92.9% without any query reduction.

Original: Similarly, using 400 principal components for Resnet-32 on CIFAR-
Revised: Similarly, using 400 principal components for Resnet-32 on CIFAR-

Original: (Figure 5c), an adversarial success rate of 66.9% can be achieved at (cid:15)
Revised: (Figure 5c), an adversarial success rate of 66.9% can be achieved at (cid:15)

Original: = 8 .
Revised: = 8 .

Original: At (cid:15) = 16 , the adversarial success rate rises to 80.1%.
Revised: At (cid:15) = 16 , the adversarial success rate rises to 80.1%.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: J. S INGLE - STEP ATTACKS ON DEFENSES
Revised: J. S INGLE - STEP ATTACKS ON DEFENSES

Original: In this section, we analyse the effectiveness of single-step black-box attacks on adversarially trained models and show that the Gradient Estimation attacks using Finite Differences with the addition of random perturbations outperform other black-box attacks.
Revised: In this section, we analyse the effectiveness of single-step black-box attacks on adversarially trained models and show that the Gradient Estimation attacks using Finite Differences with the addition of random perturbations outperform other black-box attacks.

Original: Evaluation of single-step attacks on model with basic adversarial training :
Revised: Evaluation of single-step attacks on model with basic adversarial training :

Original: In Figure 6a, we can see that both single-step black-box and white-box attacks have much lower adversarial success rates on Model A adv-0.3 as compared to Model A .
Revised: In Figure 6a, we can see that both single-step black-box and white-box attacks have much lower adversarial success rates on Model A adv-0.3 as compared to Model A .

Original: The success rate of the Gradient Estimation attacks matches that of white-box attacks on these adversarially trained networks as well.
Revised: The success rate of the Gradient Estimation attacks matches that of white-box attacks on these adversarially trained networks as well.

Original: To overcome this, we add an initial random perturbation to samples before using the Gradient Estimation attack with
Revised: To overcome this, we add an initial random perturbation to samples before using the Gradient Estimation attack with

Original: Finite Differences and the logit loss ( FD-logit ) as described in the next paragraph.
Revised: Finite Differences and the logit loss ( FD-logit ) as described in the next paragraph.

Original: This is then the most effective single step black-box attacks on Model A adv-0.3 at (cid:15) = 0 .
Revised: This is then the most effective single step black-box attacks on Model A adv-0.3 at (cid:15) = 0 .

Original: 3 with an adversarial success rate of 32.2%, surpassing the Transferability attack (single local model) from B.
Revised: 3 with an adversarial success rate of 32.2%, surpassing the Transferability attack (single local model) from B.

Original: In Figure 6b, we again see that the Gradient Estimation attacks using Finite Differences ( FD-xent and FD-logit ) and white-box FGS attacks ( FGS-xent and FGS-logit ) against Resnet-32 .
Revised: In Figure 6b, we again see that the Gradient Estimation attacks using Finite Differences ( FD-xent and FD-logit ) and white-box FGS attacks ( FGS-xent and FGS-logit ) against Resnet-32 .

Original: As (cid:15) is increased, the attacks that perform the best are Random Perturbations ( Rand. )
Revised: As (cid:15) is increased, the attacks that perform the best are Random Perturbations ( Rand. )

Original: , Difference-ofmeans ( D. of M. ) and Transferability attack (single local model) from Resnet-28-10 with the latter performing slightly better than the baseline attacks.
Revised: , Difference-ofmeans ( D. of M. ) and Transferability attack (single local model) from Resnet-28-10 with the latter performing slightly better than the baseline attacks.

Original: This is due to the ‘gradient masking’ phenomenon, and can be overcome by adding random perturbations as for MNIST.
Revised: This is due to the ‘gradient masking’ phenomenon, and can be overcome by adding random perturbations as for MNIST.

Original: An interesting effect is observed at (cid:15)
Revised: An interesting effect is observed at (cid:15)

Original: = 4 , where the adversarial success rate is higher than at (cid:15) = 8 .
Revised: = 4 , where the adversarial success rate is higher than at (cid:15) = 8 .

Original: The likely explanation for this effect is that the model has overﬁtted to adversarial samples at (cid:15)
Revised: The likely explanation for this effect is that the model has overﬁtted to adversarial samples at (cid:15)

Original: = 8 .
Revised: = 8 .

Original: Our Gradient Estimation attack closely tracks the adversarial success rate of white-box attacks in this setting as well.
Revised: Our Gradient Estimation attack closely tracks the adversarial success rate of white-box attacks in this setting as well.

Original: Increasing effectiveness of single-step attacks using initial random perturbation : Since the
Revised: Increasing effectiveness of single-step attacks using initial random perturbation : Since the

Original: Gradient Estimation attack with Finite Differences ( FD-xent and FD-logit ) were not performing well due the masking of gradients at the benign sample x , we added an initial random perturbation to escape this low-gradient region as in the RAND-FGSM attack Tramèr et al. (2017a).
Revised: Gradient Estimation attack with Finite Differences ( FD-xent and FD-logit ) were not performing well due the masking of gradients at the benign sample x , we added an initial random perturbation to escape this low-gradient region as in the RAND-FGSM attack Tramèr et al. (2017a).

Original: Figure 7 shows the effect of adding an initial L ∞ constrained perturbation of magnitude 0.05.
Revised: Figure 7 shows the effect of adding an initial L ∞ constrained perturbation of magnitude 0.05.

Original: With the addition of
Revised: With the addition of

Original: [Table]
Revised: [Table]

Original: a random perturbation
Revised: a random perturbation

Original: , FD-logit has a much improved adversarial success rate on Model A adv-0.3 , going up to 32.2% from 2.8% without the perturbation at a total perturbation value of 0 . 3 .
Revised: , FD-logit has a much improved adversarial success rate on Model A adv-0.3 , going up to 32.2% from 2.8% without the perturbation at a total perturbation value of 0 . 3 .

Original: It even outperforms the white-box FGS ( FGS-logit ) with the same random perturbation added.
Revised: It even outperforms the white-box FGS ( FGS-logit ) with the same random perturbation added.

Original: This effect is also observed for Model A adv-ens-0.3 but Model A adv-iter-0.3 appears to be resistant to single-step gradient based attacks.
Revised: This effect is also observed for Model A adv-ens-0.3 but Model A adv-iter-0.3 appears to be resistant to single-step gradient based attacks.

Original: Thus, our attacks work well for single-step attacks on DNNs with standard and ensemble adversarial training, and achieve performance levels close to that of white-box attacks.
Revised: Thus, our attacks work well for single-step attacks on DNNs with standard and ensemble adversarial training, and achieve performance levels close to that of white-box attacks.

Original: J. E FFICIENCY OF GRADIENT ESTIMATION ATTACKS
Revised: J. E FFICIENCY OF GRADIENT ESTIMATION ATTACKS

Original: In our evaluations, all models were run on a GPU with a batch size of 100.
Revised: In our evaluations, all models were run on a GPU with a batch size of 100.

Original: On Model A on MNIST data, single-step attacks FD-xent and FD-logit take 6 .
Revised: On Model A on MNIST data, single-step attacks FD-xent and FD-logit take 6 .

Original: 2 × 10 − 2 and 8 . 8 × 10 − 2 seconds per sample respectively.
Revised: 2 × 10 − 2 and 8 . 8 × 10 − 2 seconds per sample respectively.

Original: Thus, these attacks can be carried out on the entire MNIST test set of 10,000 images in about 10 minutes.
Revised: Thus, these attacks can be carried out on the entire MNIST test set of 10,000 images in about 10 minutes.

Original: For iterative attacks with no query reduction, with 40 iterations per sample ( α set to 0.01), both IFD-xent and IFD-xent-T taking about 2 . 4 seconds per sample.
Revised: For iterative attacks with no query reduction, with 40 iterations per sample ( α set to 0.01), both IFD-xent and IFD-xent-T taking about 2 . 4 seconds per sample.

Original: Similarly, IFD-logit and IFD-logit-T take about 3 .
Revised: Similarly, IFD-logit and IFD-logit-T take about 3 .

Original: 5 seconds per sample.
Revised: 5 seconds per sample.

Original: With query reduction, using IGE-QR (PCA- k , logit) with k = 100 the time taken is just 0.5 seconds per sample.
Revised: With query reduction, using IGE-QR (PCA- k , logit) with k = 100 the time taken is just 0.5 seconds per sample.

Original: For Resnet-32 on the CIFAR-10 dataset, FD-xent , FD-xent-T , FD-logit and FD-logit-T all take roughly 3s per sample.
Revised: For Resnet-32 on the CIFAR-10 dataset, FD-xent , FD-xent-T , FD-logit and FD-logit-T all take roughly 3s per sample.

Original: The iterative variants of these attacks with 10 iterations ( α set to 1.0) take roughly 30s per sample.
Revised: The iterative variants of these attacks with 10 iterations ( α set to 1.0) take roughly 30s per sample.

Original: Using query reduction, IGE-QR (PCA- k , logit) with k = 100 withiterations takes just 5s per sample.
Revised: Using query reduction, IGE-QR (PCA- k , logit) with k = 100 withiterations takes just 5s per sample.

Original: The time required per sample increases with the complexity of the network, which is observed even for white-box attacks.
Revised: The time required per sample increases with the complexity of the network, which is observed even for white-box attacks.

Original: All the above numbers are for the case when queries are not made in parallel.
Revised: All the above numbers are for the case when queries are not made in parallel.

Original: Our attack algorithm allows for queries to be made in parallel as well.
Revised: Our attack algorithm allows for queries to be made in parallel as well.

Original: We ﬁnd that a simple parallelization of the queries gives us a 2 − 4 × speedup.
Revised: We ﬁnd that a simple parallelization of the queries gives us a 2 − 4 × speedup.

Original: The limiting factor is the fact that the model is loaded on a single GPU, which implies that the current setup is not fully optimized to take advantage of the inherently parallel nature of our attack.
Revised: The limiting factor is the fact that the model is loaded on a single GPU, which implies that the current setup is not fully optimized to take advantage of the inherently parallel nature of our attack.

Original: With further optimization, greater speedups can be achieved.
Revised: With further optimization, greater speedups can be achieved.

Original: Remarks : Overall, our attacks are very efﬁcient and allow an adversary to generate a large number of adversarial samples in a short period of time.
Revised: Remarks : Overall, our attacks are very efﬁcient and allow an adversary to generate a large number of adversarial samples in a short period of time.

Original: J. W HITE - BOX ATTACK RESULTS
Revised: J. W HITE - BOX ATTACK RESULTS

Original: In this section, we present the white-box attack results for various cases.
Revised: In this section, we present the white-box attack results for various cases.

Original: Where relevant, our results match previous work Goodfellow et al.
Revised: Where relevant, our results match previous work Goodfellow et al.

Original: ; Kurakin et al.
Revised: ; Kurakin et al.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: K A TTACKS THAT WERE TOO EXPENSIVE
Revised: K A TTACKS THAT WERE TOO EXPENSIVE

Original: We experimented with Particle Swarm Optimization to construct adversarial samples as was done in Sharif et al.
Revised: We experimented with Particle Swarm Optimization to construct adversarial samples as was done in Sharif et al.

Original: Sharif et al.
Revised: Sharif et al.

Original: , but found it to be prohibitively expensive for a large dataset and unable to achieve high adversarial success rates.
Revised: , but found it to be prohibitively expensive for a large dataset and unable to achieve high adversarial success rates.

Original: We also tried to use a class of gradient free optimization techniques from Nesterov and Spokoiny Nesterov & Spokoiny (2017) that use query access to the target model to directly optimize loss functions constructed for white-box attacks Carlini & Wagner (2017), but found it was hard to reach convergence in a reasonable number of iterations.
Revised: We also tried to use a class of gradient free optimization techniques from Nesterov and Spokoiny Nesterov & Spokoiny (2017) that use query access to the target model to directly optimize loss functions constructed for white-box attacks Carlini & Wagner (2017), but found it was hard to reach convergence in a reasonable number of iterations.