Original: D EFENDING AGAINST BLACK - BOX ADVERSARIAL ATTACKS WITH GRADIENT - FREE TRAINED SIGN ACTIVATION NEURAL NETWORKS
Revised: D EFENDING AGAINST BLACK - BOX ADVERSARIAL ATTACKS WITH GRADIENT - FREE TRAINED SIGN ACTIVATION NEURAL NETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Revised: While machine learning models today can achieve high accuracies on classiﬁcation tasks, they can be deceived by minor imperceptible distortions to the data.

Revised: These are known as adversarial attacks and can be lethal in the black-box setting which does not require knowledge of the target model type or its parameters.

Original: Binary neural networks  have been shown to be more adversarially robust than fullprecision networks but their improvements are marginal.
Revised: Binary neural networks that have sign activation and are trained with gradient descent have been shown to be harder to attack than conventional sigmoid activation networks but their improvements are marginal.

Original: We propose an ensemble of sign activationneural networkstrained with a novel gradient free stochastic coordinate descent algorithm.
Revised: We instead train sign activation networks with a novel gradient-free stochastic coordinate descent algorithm and propose an ensemble of such networks as a defense model.

Original: On the CIFAR10 image benchmark we show that our model has a much higher minimum distortion, as measured by the black box boundary attack method HopSkipJump, than ensembles of binary, full precision, and convolutional neural networks, and than random forests while attaining comparable clean test accuracy.We also show that our ensemble model attains higher accuracy on adversarial examples in text black box attacks across several datasets and is also hard to attack on medical ECG data.
Revised: We evaluate the robustness of our model (a hard problem in itself) on image, text, and medical image and ECG data and ﬁnd it to be more robust than ensembles of binary, full precision, and convolutional neural networks, and than random forests while attaining comparable clean test accuracy.

Original: In order to explain our model’s robustness we turn to non-transferability. We empirically measure the probability that a black box adversary targeting a single network in our ensemble will transfer to other networks and ﬁnd this to be lowest compared to ensembles of other networks and models.
Revised: In order to explain our model’s robustness we show that an adversary targeting a single network in our ensemble fails to attack (and thus non-transferable to) other networks in the ensemble.

Original: Thus an image requires greater distortion to fool the majority of networks in our ensemble.
Revised: Thus a datapoint requires a large distortion to fool the majority of networks in our ensemble and is likely to be detected in advance.

Original: The non-transferability in our ensemble also makes it a powerful defense to substitute model black box attacks that we show require a much greater distortion than binary and full precision networks to bring our model to zero adversarial accuracy.

Original: This property of non-transferabilityin our ensemble arises naturally from the non-convexity of sign  networks and randomization in our  training algorithm without any adversarial defense effort.
Revised: This property of non-transferability arises naturally from the non-convexity of sign activation networks and randomization in our gradient-free training algorithm without any adversarial defense effort.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: State of the art machine learning algorithms can achieve high accuracies in classiﬁcation tasks but misclassify minor perturbations in the data known as as adversarial attacks Goodfellow et al.
Revised: State of the art machine learning algorithms can achieve high accuracies in classiﬁcation tasks but misclassify minor perturbations in the data known as as adversarial attacks Goodfellow et al.

Original: Papernot et al.
Revised: Papernot et al.

Original: (2016b); Kurakin et al.
Revised: (2016b); Kurakin et al.

Original: ; Carlini & Wagner (2017); Brendel et al.
Revised: ; Carlini & Wagner (2017); Brendel et al.

Original: Adversarial examples have been shown to transfer across models which makes it possible to perform transfer-based (substitute model) black box attacks Papernot et al.
Revised: Adversarial examples have been shown to transfer across models which makes it possible to perform transfer-based (substitute model) black box attacks Papernot et al.

Original: (2016a).
Revised: (2016a).

Original: To counter adversarial attacks many defense methods been proposed with adversarial training being the most popular
Revised: To counter adversarial attacks many defense methods been proposed with adversarial training being the most popular

Original: Szegedy et al.
Revised: Szegedy et al.

Original: ; Tram`er et al.
Revised: (2014); Tram`er et al.

Original: However this tends to lower accuracy on clean test data that has no perturbations Raghunathan et al.
Revised: However this tends to lower accuracy on clean test data that has no perturbations Raghunathan et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: and can still be attacked with better transfer based methods Wu et al.
Revised: and can still be attacked with better transfer based methods Wu et al.

Original: Xie et al. (2019a); Dong et al.
Revised: Xie et al. (2019a); Dong et al.

Original: Many previously proposed defenses have also been shown to be vulnerable Carlini & Wagner (2017);
Revised: Many previously proposed defenses have also been shown to be vulnerable Carlini & Wagner (2017);

Original: Athalye et al.
Revised: Athalye et al.

Original: ; Ghiasi et al.
Revised: ; Ghiasi et al.

Original: (2020) thus leaving adversarial robustness an open problem in machine learning.
Revised: (2020) thus leaving adversarial robustness an open problem in machine learning.

Original: A more lethal and practical attack than substitute model training is a boundary based one that requires only the prediction of the model Brendel et al.
Revised: A more lethal and practical attack than substitute model training is a boundary based one that requires only the prediction of the model Brendel et al.

Original: These attacks are aimed at ﬁnding the minimum distortion to an image such that it will fool a classiﬁer.
Revised: These attacks are aimed at ﬁnding the minimum distortion to an image such that it will fool a classiﬁer.

Original: This is in fact an NP-hard problem for ReLu activated neural networks Katz et al.
Revised: This is in fact an NP-hard problem for ReLu activated neural networks Katz et al.

Original: ; Sinha et al.
Revised: ; Sinha et al.

Original: and tree ensemble classiﬁers Kantchelian et al.
Revised: and tree ensemble classiﬁers Kantchelian et al.

Original: Even approximating the minimum distortion for ReLu activated neural networks is NP-hard Weng et al.
Revised: Even approximating the minimum distortion for ReLu activated neural networks is NP-hard Weng et al.

Original: The HopSkipJumpboundary attack Chen et al.
Revised: Boundary based black box attacks such as HopSkipJump Chen et al.

Revised: , Boundary Attack Brendel et al.

Original: (2019) has been shown to give  the lowest distortion among such methods.
Revised: and RayS Chen & Gu (2020) give an upper bound on the minimum adversarial distortion.

Original: Binary neural networks that have sign activation and binary weights were originally proposed as lightweight models.
Revised: Binary neural networks that have sign activation and binary weights were originally proposed as lightweight models.

Original: These are trained by with gradient descent by approximating the sign activation.
Revised: These are trained by with gradient descent by approximating the sign activation.

Original: Recent work has shown that they tend to be more adversarially robust than full precision networks but the improvements are marginal (see Tables 4 and 5 in Galloway et al.
Revised: Recent work has shown that they tend to be more adversarially robust than full precision networks but the improvements are marginal (see Tables 4 and 5 in Galloway et al.

Original: (2017) and Table 8 in
Revised: (2018) and Table 8 in

Original: Panda et al.
Revised: Panda et al.

Original: In this paper we propose a gradient free stochastic coordinate descent algorithm for training sign activation networks with and without binary weights similar to recent work Xue et al.
Revised: In this paper we propose a gradient free stochastic coordinate descent algorithm for training sign activation networks with and without binary weights similar to recent work Xue et al.

Original: (2020a;b);
Revised: (2020a;b);

Original: Xie et al.
Revised: Xie et al.

Original: (2019b).
Revised: (2019b).

Original: While our original intention was to study the accuracy of a sign activation network trained directly without any approximation we make an interesting ﬁnding on the adversarial robustness of our model.
Revised: While our original intention was to study the accuracy of a sign activation network trained directly without any approximation we make an interesting ﬁnding on the adversarial robustness of our model.

Original: We ﬁnd that ensembling our model gives a high minimum distortion (as measured by HopSkipJump) compared to full precision, binary, and convolutional neural networks.
Revised: We ﬁnd that ensembling our model gives a high minimum distortion (as measured by HopSkipJump) compared to full precision, binary, and convolutional neural networks.

Original: We explain this phenomena by measuring the transferability between networks in an ensemble.
Revised: We explain this phenomena by measuring the transferability between networks in an ensemble.

Original: In summary we make the following observations in our paper:
Revised: In summary we make the following observations in our paper:

Original: • Our model has higher minimum distortion than ensembles of full precision and binary neural networks, than random forests that have the advantage of bootstrapping and random feature selection (although only in L 2 and not L ∞ on the data we show in this paper), and than ensembles of convolutional networks that have the advantage of convolutions and several layers.
Revised: • Our single hidden layer sign activation network has higher minimum distortion than ensembles of full precision and binary neural networks, than random forests that have the advantage of bootstrapping and random feature selection, and than ensembles of convolutional networks that have the advantage of convolutions and several layers.

Original: • Our model’s robustness stems from non-transferability of adversarial examples between networks in our ensemble and its robustness increases as we add more networks to the ensemble.
Revised: • Our model’s robustness stems from non-transferability of adversarial examples between networks in our ensemble and its robustness increases as we add more networks to the ensemble.

Original: • Substitute model black box attacks require a much greater distortion to bring our model to zero adversarial accuracy compared to ensembles of full precision and binary networks.
Revised: • Substitute model black box attacks require a much greater distortion to bring our model to zero adversarial accuracy compared to ensembles of full precision and binary networks.

Original: • Text classiﬁcation black box attacks are less effective on our model than on convolutional networks, random forests, and ensembles of full precision and binary networks.
Revised: • Text classiﬁcation black box attacks are less effective on our model than on convolutional networks, random forests, and ensembles of full precision and binary networks.

Original: • In a medical diagnosis setting, attacks on ECG data on our model have a higher distortions and are visually distinguishable compared to attacks on ensembles of full precision and convolutional networks, and on random forests.
Revised: • In a medical diagnosis setting, attacks on ECG data on our model have a higher distortions and are visually distinguishable compared to attacks on ensembles of full precision and convolutional networks, and on random forests.

Original:  M ETHODS2. G RADIENT - FREE STOCHASTIC COORDINATE DECENT
Revised:  M ETHODS

Revised: G RADIENT - FREE STOCHASTIC COORDINATE DECENT

Original: Our stochastic coordinate descent is inspired by and similar to recent work Xue et al.
Revised: Our stochastic coordinate descent is inspired by and similar to recent work Xue et al.

Original: (2020a;b); Xie et al. (2019b).
Revised: (2020a;b); Xie et al. (2019b).

Original: In order to understand our training algorithm for a single hidden layer network we ﬁrst show it for a simple linear classiﬁer.
Revised: In order to understand our training algorithm for a single hidden layer network we ﬁrst show it for a simple linear classiﬁer.

Original: Suppose we are given binary class data x i ∈ R d and y i ∈ {− 1 , +1 } for i = 0 ...n − 1 .
Revised: Suppose we are given binary class data x i ∈ R d and y i ∈ {− 1 , +1 } for i = 0 ...n − 1 .

Original: We wish to determine a linear classiﬁer w ∈ R d , w 0 ∈ R that minimizes the empirical risk for a given loss function (cid:80) i L ( w, w 0 , x i , y i ) .
Revised: We wish to determine a linear classiﬁer w ∈ R d , w 0 ∈ R that minimizes the empirical risk for a given loss function (cid:80) i L ( w, w 0 , x i , y i ) .

Original: At the high level our approach is simple: start with a random solution w i ∈ N (0 , 1) , w 0 ∈ N (0 , 1) for i = 0 ...
Revised: At the high level our approach is simple: start with a random solution w i ∈ N (0 , 1) , w 0 ∈ N (0 , 1) for i = 0 ...

Original: d − 1 and iteratively make incremental changes that improve the risk.
Revised: d − 1 and iteratively make incremental changes that improve the risk.

Original: In each iteration we select a random set of features (coordinates) from w called F .
Revised: In each iteration we select a random set of features (coordinates) from w called F .

Original: For each feature w i ∈ F we add/subtract a learning rate η
Revised: For each feature w i ∈ F we add/subtract a learning rate η

Original: and then determine the w 0 that optimizes the risk.
Revised: and then determine the w 0 that optimizes the risk.

Original: We consider all possible values of w 0 = w Ti x i + w [-Ti +
-] x
Revised: We consider all possible values of w 0 = w T x i + w [-Ti +
-]T x

Original: i
Revised: i

Original: +1for i
Revised: +1for i

Original: = 0 ...
Revised: = 0 ...

Original: n − 2 and select the one that minimizes the loss.
Revised: n − 2 and select the one that minimizes the loss.

Original: To avoid local minima in our search we consider a random sample of the training data in each iteration.
Revised: To avoid local minima in our search we consider a random sample of the training data in each iteration.

Original: We set this to 75% of the training data in image and text data experiments and 25% in the ECG data.
Revised: We set this to 75% of the training data in image and text data experiments and 25% in the ECG data.

Original: We call the above search stochastic coordinate descent abbreviated by SCD.
Revised: We call the above search stochastic coordinate descent abbreviated by SCD.

Original: In order to train a single hidden layer network we apply SCD to the ﬁnal node and then a randomly selected hidden node in each iteration of the algorithm.
Revised: In order to train a single hidden layer network we apply SCD to the ﬁnal node and then a randomly selected hidden node in each iteration of the algorithm.

Original: We can train sign activation networks with and without binary weights using our SCD training procedure above.
Revised: We can train sign activation networks with and without binary weights using our SCD training procedure above.

Original: In the case of binary weights we don’t need a learning rate.
Revised: In the case of binary weights we don’t need a learning rate.

Original: We apply parallelism and several heuristics in practice to speed up the real runtimes.
Revised: We apply parallelism and several heuristics in practice to speed up the real runtimes.

Original: We provide full algorithmic details of our training procedure in the Supplementary Material.
Revised: We provide full algorithmic details of our training procedure in the Supplementary Material.

Original: 2. I MPLEMENTATION , TEST ACCURACY , AND RUNTIME
Revised: 2. I MPLEMENTATION , TEST ACCURACY , AND RUNTIME

Original: We implement our training procedure in Python, numpy, and Pytorch Paszke et al.
Revised: We implement our training procedure in Python, numpy, and Pytorch Paszke et al.

Original: and make our code freely available from https://github.com/zero-one-loss/scd_github .
Revised: and make our code freely available from https://github.com/zero-one-loss/scd_github .

Original: We train three types of sign activation networks with our algorithm:
Revised: We train three types of sign activation networks with our algorithm:

Original: (1) SCD01: 01-loss in the ﬁnal node, (2) SCDCE: cross-entropy loss in the ﬁnal node, and (3) SCDCEBNN: cross-entropy in the ﬁnal node with binary weights throughout the model.
Revised: (1) SCD01: 01-loss in the ﬁnal node, (2) SCDCE: cross-entropy loss in the ﬁnal node, and (3) SCDCEBNN: cross-entropy in the ﬁnal node with binary weights throughout the model.

Original: Since sign activation is non-convex and our training starts from a different random initialization we run it a 100 times and output the majority vote.
Revised: Since sign activation is non-convex and our training starts from a different random initialization we run it a 100 times and output the majority vote.

Original: To illustrate our real runtimes and clean test accuracies we compare our models with a single hidden layer of 20 nodes to the equivalent network with sigmoid activation and logistic loss (denoted as
Revised: To illustrate our real runtimes and clean test accuracies we compare our models with a single hidden layer of 20 nodes to the equivalent network with sigmoid activation and logistic loss (denoted as

Original: MLP) and the binary neural network (denoted as BNN)
Revised: MLP) and the binary neural network (denoted as BNN)

Original: Courbariaux et al.
Revised: Hubara et al.

Original: We used the MLPClassiﬁer in scikit-learn Pedregosa et al.
Revised: We used the MLPClassiﬁer in scikit-learn Pedregosa et al.

Original: (2011) to implement MLP and the Larq library Geiger & Team (2020) with the approx approximation to the sign activation.
Revised: (2011) to implement MLP and the Larq library Geiger & Team (2020) with the approx approximation to the sign activation.

Original: This has shown to achieve a higher test accuracy than the original straight through estimator (STE) of the sign activation Liu et al.
Revised: This has shown to achieve a higher test accuracy than the original straight through estimator (STE) of the sign activation Liu et al.

Original: (2018b).
Revised: (2018b).

Original: We perform a 1000 iterations of SCD01 and SCDCE and 10000 of SCDCEBNN.
Revised: We perform a 1000 iterations of SCD01 and SCDCE and 10000 of SCDCEBNN.

Original: In Table 1 we show the runtimes of a single run of all models on CIFAR10 Krizhevsky (2009) ( 32 × 32 × 3 , 10K train, 2K test), CelebA facial attributes black hair vs brown hair Liu et al.
Revised: In Table 1 we show the runtimes of a single run of all models on CIFAR10 Krizhevsky (2009) ( 32 × 32 × 3 , 10K train, 2K test), CelebA facial attributes black hair vs brown hair Liu et al.

Original: ( 96 × 96 × 3 , 1K train, 1K test), GTSRB street sign recognition 60 vs 120 speed limit signs Stallkamp et al.
Revised: ( 96 × 96 × 3 , 1K train, 1K test), GTSRB street sign recognition 60 vs 120 speed limit signs Stallkamp et al.

Original: (2011) ( 48 × 48 × 3 , 2816 train, 900 test), and ImageNet class 0 vs. 1 Russakovsky et al. (2015) ( 256 × 256 × 3 ,train, 100 test).
Revised: (2011) ( 48 × 48 × 3 , 2816 train, 900 test), and ImageNet class 0 vs. 1 Russakovsky et al. (2015) ( 256 × 256 × 3 ,train, 100 test).

Original: Our training runtimes are comparable to gradient descent in MLP and BNN and thus practically usable.
Revised: Our training runtimes are comparable to gradient descent in MLP and BNN and thus practically usable.

Original: We can trivially parallelize training an ensemble by doing multiple runs on CPU and GPU cores at the same time.
Revised: We can trivially parallelize training an ensemble by doing multiple runs on CPU and GPU cores at the same time.

Original: We also show test accuracies of 100 vote ensembles of all models and ﬁnd our model accuracies to be comparable to MLP and BNN.
Revised: We also show test accuracies of 100 vote ensembles of all models and ﬁnd our model accuracies to be comparable to MLP and BNN.

Original: [Table]
Revised: [Table]

Original:  R ESULTS
Revised:  R ESULTS

Original: Going forward we compare the adversarial robustness of ensembles of our three models SCD01,
Revised: Going forward we compare the adversarial robustness of ensembles of our three models SCD01,

Original: SCDCE [Table] and SCDCEBNN, their full precision and binary gradient descent trained equivalent counterparts MLP and BNN, two convolutional neural networks: LeNet LeCun et al.
Revised: SCDCE, and SCDCEBNN, their full precision and binary gradient descent trained equivalent counterparts MLP and BNN, two convolutional neural networks: LeNet LeCun et al.

Original: (1998) and
Revised: (1998) and

Original: ResNet50 He et al.
Revised: ResNet50 He et al.

Original: (2016), and random forests Breiman (2001) (denoted as RF).
Revised: (2016), and random forests Breiman (2001) (denoted as RF).

Original: For each model we use the majority vote output of 100 votes each with different initial parameters except for ResNet where we use 10 votes.
Revised: For each model we use the majority vote output of 100 votes each with different initial parameters except for ResNet where we use 10 votes.

Original: In random forest we use an ensemble of 100 trees.
Revised: In random forest we use an ensemble of 100 trees.

Original: We use a single hidden layer of 20 nodes in our three models and in MLP and BNN throughout the paper.
Revised: We use a single hidden layer of 20 nodes in our three models and in MLP and BNN throughout the paper.

Original: The convolutional networks and random forest are not a fair comparison to our model since it has fewer parameters and does not perform bootstrapping or random feature selection as random forest.
Revised: The convolutional networks and random forest are not a fair comparison to our model since it has fewer parameters and does not perform bootstrapping or random feature selection as random forest.

Original: We include them nevertheless since convolutional neural networks serve as state of the art references and random forest serves as an alternative ensemble method.
Revised: We include them nevertheless since convolutional neural networks serve as state of the art references and random forest serves as an alternative ensemble method.

Original: 3. A DVERSARIAL DISTORTION ON IMAGE DATA
Revised: 3. A DVERSARIAL DISTORTION ON IMAGE DATA

Original: The minimum distortion required to make a datapoint adversarial is an indicator of a model’s adversarial and even corruption/general robustness Gilmer et al.
Revised: The minimum distortion required to make a datapoint adversarial is an indicator of a model’s adversarial and even corruption/general robustness Gilmer et al.

Original: We consider 10 randomly selected datapoints from the CIFAR10 benchmark Krizhevsky (2009) and report their minimal adversarial distortion as given by HopSkipJump Chen et al.
Revised: We consider 10 randomly selected datapoints from the CIFAR10 benchmark Krizhevsky (2009) and report their minimal ad- versarial distortion as given by HopSkipJump Chen et al.

Original: Compared to other boundary attack methods Brendel et al.
Revised: , Boundary Attack Brendel et al.

Original: (2017) it is known to give the best estimate of a datapoint’s minimum adversarial distortion.
Revised: and RayS Chen & Gu (2020).

Original: We use the HopSkipJump  implementation in the IBM Adversarial RobustnessToolkit (ART)
Revised: We use the HopSkipJump and Boundary Attack implementation in the IBM Adversarial Robustness

Original:  Nicolae et al.
Revised: Toolkit (ART) Nicolae et al.

Original: In order to obtain as accurate an estimate as possible we run HopSkipJump 10 times each with an initial pool size of 1000 random datapoints and maximum iterations of 100 and report the minimum value.
Revised: In order to obtain as accurate an estimate as possible we run both mthods 10 times each with an initial pool size of 1000 random datapoints and maximum iterations of 100 and report the minimum value.

Original: For a single datapoint this typically takes several hours to ﬁnish and thus we are able to report the distortion of only 10 random points in this study.
Revised: For a single datapoint this typically takes several hours to ﬁnish and thus we are able to report the distortion of only 10 random points in this study.

Revised: We use the RayS implementation from their GitHub site https://github.com/uclaml/RayS and run it with default parameters of 40,000 queries to obtain a distortion estimate.

Original: In Table 2 ﬁrst row we show the clean test accuracy of all models on CIFAR10 class 0 vs. 1.
Revised: In Table 2 ﬁrst row we show the clean test accuracy of all models on CIFAR10 class 0 vs. 1.

Original: The convolutional networks LeNet and ResNet50 have higher accuracies since they have the advantage of convolutions.
Revised: The convolutional networks LeNet and ResNet50 have higher accuracies since they have the advantage of convolutions.

Revised: In the following three rows of Table 2 we see the minimum adversarial distortion of models as estimate by three boundary attack methods.

Revised: We were unable to attack some models with

Revised: Boundary Attack and RayS due to time constraints and mark them as NA.

Revised: We see that HopSkipJump gives the lowest distortion for each model except for SCD01 and SCDCE where it is comparable to

Revised: RayS.

Original: However, our sign activation trained models have the highest adversarial distortion with the binary weights cross-entropy variant as the winner.
Revised: Amongst HopSkipJump distortions our sign activation trained models have the highest adversarial distortion with the binary weights cross-entropy variant as the winner.

Original: All other neural networks lag far behind and have distortion even lower than random forest.
Revised: All other neural networks lag far behind and have distortion even lower than random forest.

Original: Even though BNN also has sign activations its distortions are similar to MLP possibly due to its approximation of the sign activation and gradient descent search.
Revised: Even though BNN also has sign activations its distortions are similar to MLP possibly due to its approximation of the sign activation and gradient descent search.

Original: If we use the the straight through estimator and swish approximations Darabi et al.
Revised: If we use the the straight through estimator and swish approximations Darabi et al.

Original: (2018) the distortions remain similar to what we report here.
Revised: the distortions remain similar to what we report here.

Original: [Table]
Revised: [Table]

Original: In Table 3 we compare theL ∞ distortionsand make similar observations as above except that random forest has the highest L ∞ distortion.
Revised: To further validate the distortions above we run HopSkipJump on SCD01, MLP, LeNet, and RF with 10 maximum iterations on the ﬁrst 100 CIFAR10 test datapoints.

Original: This suggests that in order for an image to fool our model changes have to made in many pixels whereas in random forest a larger change in fewer pixels is sufﬁcient.
Revised: We used a ﬁxed image as the initial one in these experiments. In Table 3 we see that SCD01 distortions are the highest and the relative ranking is the same as we saw for the 10 images above with 100 maximum iterations of HopSkipJump. [Table]

Original: 3. T RANSFERABILITY WITHIN ENSEMBLES AND EFFECT OF ENSEMBLE SIZE
Revised: 3. T RANSFERABILITY WITHIN ENSEMBLES AND EFFECT OF ENSEMBLE SIZE

Original: To understand the above phenomena we estimate the probability that an adversarial example targeting a single model in the ensemble will also be adversarial to other models in the ensemble.
Revised: To understand the above phenomena we estimate the probability that an adversarial example targeting a single model in the ensemble will also be adversarial to other models in the ensemble.

Original: We can estimate this by ﬁrst performing a HopSkipJump attack on each model in the ensemble separately.
Revised: We can estimate this by ﬁrst performing a HopSkipJump attack on each model in the ensemble separately.

Original: Let x (cid:48)
Revised: Let x (cid:48)

Original: i be the adversary obtained by targeting model m i in the ensemble.
Revised: i be the adversary obtained by targeting model m i in the ensemble.

Original: Let k i be the number of models in the ensemble that are also misclassiﬁed by the adversary x (cid:48) i (thus transferable).
Revised: Let k i be the number of models in the ensemble that are also misclassiﬁed by the adversary x (cid:48) i (thus transferable).

Original: We sum k
Revised: We sum k

Original: i for i = 0 ...n − 1 and divide by 9900 which is the maximum value of this sum (obtained when the adversary attacks all models in the ensemble excluding the target of course).
Revised: i for i = 0 ...n − 1 and divide by 9900 which is the maximum value of this sum (obtained when the adversary attacks all models in the ensemble excluding the target of course).

Original: We average this probability for Images 0 through 7 for each method.
Revised: We average this probability for Images 0 through 7 for each method.

Original: In Table 4 we see that this probability is lowest for our models and highest for MLP and BNN.
Revised: In Table 4 we see that this probability is lowest for our models and highest for MLP and BNN.

Original: The fact that this probability is very low for our models indicates that for several of the networks in our ensemble the adversary targeting a ﬁxed network does not transfer to other ones.
Revised: The fact that this probability is very low for our models indicates that for several of the networks in our ensemble the adversary targeting a ﬁxed network does not transfer to other ones.

Original: The low transferability of our models indicates that a greater distortion is required for an image to be adversarial.
Revised: The low transferability of our models indicates that a greater distortion is required for an image to be adversarial.

Original: [Table]
Revised: [Table]

Original: [Table] In fact as we see in Figure 1 the robustness of our models increases as we increase the ensemble size to a much larger degree than ensembles of MLP and BNN, and than RF.
Revised: In fact as we see in Figure 1 the robustness of our models increases as we increase the ensemble size to a much larger degree than ensembles of MLP and BNN, and than RF.

Original: We use ensemble sizes of 100 in this study but the ﬁgure suggests that increasing our ensemble size is likely to further increase robustness.
Revised: We use ensemble sizes of 100 in this study but the ﬁgure suggests that increasing our ensemble size is likely to further increase robustness.

Original: 3. S UBSTITUTE MODEL BLACK - BOX ATTACKS
Revised: 3. S UBSTITUTE MODEL BLACK - BOX ATTACKS

Original: Our model’s high distortion in CIFAR10 is reﬂected in substitute model black box attacks on this dataset Papernot et al.
Revised: Our model’s high distortion in CIFAR10 is reﬂected in substitute model black box attacks on this dataset Papernot et al.

Original: (2016a).
Revised: (2016a).

Original: We train a two hidden layer neural network each with 200 nodes as the substitute using the standard adversarial augmented algorithm Papernot et al.
Revised: We train a two hidden layer neural network each with 200 nodes as the substitute using the standard adversarial augmented algorithm Papernot et al.

Original: (2017) (described fully in the Supplementary Material).
Revised: (2017) (described fully in the Supplementary Material).

Original: In Figure 2 we that our models require a much higher distortion than their gradient descent trained equivalents MLP and BNN in order to reach zero percent adversarial accuracy.
Revised: In Figure 2 we that our models require a much higher distortion than their gradient descent trained equivalents MLP and BNN in order to reach zero percent adversarial accuracy.

Original: We also see that all models attacked with random Gaussian noise of the same distortion added to the test examples are barely affected thus showing the effectiveness of the black box adversarial examples.
Revised: We also see that all models attacked with random Gaussian noise of the same distortion added to the test examples are barely affected thus showing the effectiveness of the black box adversarial examples.

Original: 3. T EXT BLACK - BOX ATTACKS
Revised: 3. T EXT BLACK - BOX ATTACKS

Original: The TextFooler Jin et al.
Revised: The TextFooler Jin et al.

Original: (2019) method is designed to ﬁnd syntactically and semantically similar adversarial documents by replacing important words with similar ones until the document is misclassiﬁed.
Revised: (2020) method is designed to ﬁnd syntactically and semantically similar adversarial documents by replacing important words with similar ones until the document is misclassiﬁed.

Original: We apply this to all ensemble models on four document classiﬁcation datasets: Internet
Revised: We apply this to all ensemble models on four document classiﬁcation datasets: Internet

Original: Movie Database (25K train, 25K test, mean words per document: 215) and Yelp (560K train, 38K test, mean words per document: 152) positive and negative reviews (IMDB and Yelp), sentence classiﬁcation of positive and negative sentiments (9K train, 1K test, mean words per document: 20, denoted as MR), and sentence-level classiﬁcation of news items in World and Sports categories (120K train, 7.6K test, mean words per document: 43, denoted as AG)

Original: Jin et al.

Original: WordCNN stacks word vectors Pennington et al.
Revised: WordCNN stacks word vectors Pennington et al.

Original: (2014) of each word in a document into a matrix to treat it as 2D image Kim (2014).
Revised: (2014) of each word in a document into a matrix to treat it as 2D image Kim (2014).

Original: In the other models that take feature vectors as inputs we consider the averaged word vector of all words in a document Lilleberg et al. (2015).
Revised: In the other models that take feature vectors as inputs we consider the averaged word vector of all words in a document Lilleberg et al. (2015).

Original: For all models we use 200 dimensional Glove word embeddings pre-trained on 6 billion tokens from Wikipedia and
Revised: For all models we use 200 dimensional Glove word embeddings pre-trained on 6 billion tokens from Wikipedia and

Original: Gigawords Pennington et al.
Revised: Gigawords Pennington et al.

Original: This gives a lower clean test accuracy than WordCNN but still above an acceptable level in practice.
Revised: This gives a lower clean test accuracy than WordCNN but still above an acceptable level in practice.

Original: In Table 5 we see that ensembles of our models give the highest adversarial accuracy on all four datasets and require the greatest number of queries.
Revised: In Table 5 we see that ensembles of our models give the highest adversarial accuracy on all four datasets and require the greatest number of queries.

Original: If a smaller limit was placed on the allowed queries (for example imposed by the system being attacked) we can expect a higher adversarial accuracy for our models.
Revised: If a smaller limit was placed on the allowed queries (for example imposed by the system being attacked) we can expect a higher adversarial accuracy for our models.

Original: Here we show ensembles of 8 votes for each model.
Revised: Here we show ensembles of 8 votes for each model.

Original: If we increase the ensemble size to 100 we ﬁnd the adversarial accuracy of our models, BNN, and RF slightly drop but their relative difference remains the same.
Revised: If we increase the ensemble size to 100 we ﬁnd the adversarial accuracy of our models, BNN, and RF slightly drop but their relative difference remains the same.

Original: 3. ECG BLACK - BOX ATTACKS
Revised: 3. ECG BLACK - BOX ATTACKS

Original: ECG time-series data is increasingly being used in automatic diagnosis by machine learning systems
Revised: ECG time-series data is increasingly being used in automatic diagnosis by machine learning systems

Original: Ribeiro et al.
Revised: Ribeiro et al.

Original: Tailored adversarial attacks have recently been proposed Han et al.
Revised: Tailored adversarial attacks have recently been proposed Han et al.

Original: Chen et al.
Revised: Chen et al.

Original: but HopSkipJump can also be used to produce adversarial ECG examples.
Revised: but HopSkipJump can also be used to produce adversarial ECG examples.

Original: To[Table] illustrate this and evaluate our model’s robustness on this data we consider the PTB Diagnostic
Revised: To illustrate this and evaluate our model’s robustness on this data we consider the PTB Diagnostic

Original: ECG dataset Bousseljot et al.
Revised: ECG dataset Bousseljot et al.

Original: Goldberger et al.
Revised: Goldberger et al.

Original: (2000) available from this URL https: //www.kaggle.com
Revised: (2000) available from this URL https: //www.kaggle.com

Original: /shayanfazeli/heartbeat .
Revised: /shayanfazeli/heartbeat .

Original: We randomly split this dataset into an 80:20 train test split.
Revised: We randomly split this dataset into an 80:20 train test split.

Original: We train 100-model ensembles of SCD01, SCDCE, and MLP.
Revised: We train 100-model ensembles of SCD01, SCDCE, and MLP.

Original: We also train an ensemble ofconvolutional neural networks (CNN) with 1D convolutional kernels and random forest (RF) with 100 trees.
Revised: We also train an ensemble ofconvolutional neural networks (CNN) with 1D convolutional kernels and random forest (RF) with 100 trees.

Original: Each of our CNNs has the following structure: 64 1x16 Conv1D kernels → MaxPool 1x → 128 1x16 Conv1D kernels → MaxPool 1x4 → 256 1x16 Conv1D kernels → MaxPool 1x2 →
Revised: Each of our CNNs has the following structure: 64 1x16 Conv1D kernels → MaxPool 1x → 128 1x16 Conv1D kernels → MaxPool 1x4 → 256 1x16 Conv1D kernels → MaxPool 1x2 →

Original: FullyConnected → Output.
Revised: FullyConnected → Output.

Original: In Table 6 we see the clean test accuracy of our model is slightly lower than gradient-descent trained models and random forest.
Revised: In Table 6 we see the clean test accuracy of our model is slightly lower than gradient-descent trained models and random forest.

Original: We picked 37 random datapoints from the  test and attacked all models on these points.
Revised: We picked 37 random datapoints from the [Table] test and attacked all models on these points.

Original: We attack each point 10 times and report the minimum with the same parameters as we did in our CIFAR10 attacks described above.
Revised: We attack each point 10 times and report the minimum with the same parameters as we did in our CIFAR10 attacks described above.

Original: In Table 6 second row we show the average min L 2 distortion and ﬁnd SCD01 to have the highest one.
Revised: In Table 6 second row we show the average min L 2 distortion and ﬁnd SCD01 to have the highest one.

Original: The L 2 difference between SCD01 and the next best RF (after SCDCE) turns out to be statistically signiﬁcant with a p-value of .008.
Revised: The L 2 difference between SCD01 and the next best RF (after SCDCE) turns out to be statistically signiﬁcant with a p-value of .008.

Original: As we saw earlier in CIFAR10 the L ∞ distortion of random forest is higher but not statistically signiﬁcant with a p-value of 0.07. 
Revised: As we saw earlier in CIFAR10 the L ∞ distortion of random forest is higher but not statistically signiﬁcant with a p-value of 0.07. [Table] Clean acc [Table]

Revised: Average

Original: [Table]
Revised: [Table]

Original:  In Figure 3 we visualize an original ECG sample and its adversarial versions targeting SCD01,
Revised: Average [Table] [Table] In Figure 3 we visualize an original ECG sample and its adversarial versions targeting SCD01,

Original: CNN, and RF.
Revised: CNN, and RF.

Original: The SCD01 adversary is rigid and has many more bumps compared to the CNN and RF adversaries and is thus likely to be detected by an observer or a system that checks for smoothness (that we expect to see as in the original sample). 
Revised: The SCD01 adversary is rigid and has many more bumps compared to the CNN and RF adversaries and is thus likely to be detected by an observer or a system that checks for smoothness (that we expect to see as in the original sample). [Figure]

Original: 3. D ISCUSSION
Revised: 3. D ISCUSSION

Original: Using ensembles of neural networks and promoting diverse ensembles has been previously proposed as a defense against adversarial attacks.
Revised: Using ensembles of neural networks and promoting diverse ensembles has been previously proposed as a defense against adversarial attacks.

Original: Studies using ensembles with different initializations (like we do), bootstrapping, and Gaussian noise have shown robustness but only in the white box setting
Revised: Studies using ensembles with different initializations (like we do), bootstrapping, and Gaussian noise have shown robustness but only in the white box setting

Original: Strauss et al.
Revised: Strauss et al.

Original: (2017) (which is somewhat unrealistic since it assumes the attacker has full knowledge of the model and its parameters).
Revised: (2017) (which is somewhat unrealistic since it assumes the attacker has full knowledge of the model and its parameters).

Original: Other studies combine the loss of all models in the classiﬁer and add a regularizer that promotes diversity.
Revised: Other studies combine the loss of all models in the classiﬁer and add a regularizer that promotes diversity.

Original: [Figure]

Original: For example we could try to maximize the angle between gradients of models in the ensemble
Revised: For example we could try to maximize the angle between gradients of models in the ensemble

Original: Kariyappa & Qureshi (2019) to make them misaligned .
Revised: Kariyappa & Qureshi (2019) to make them misaligned .

Original: In their diversity training they use a Gaussian noise augmented dataset which raises concerns about the effectiveness of their method since augmentation alone has been shown to be effective in ensemble training Strauss et al.
Revised: In their diversity training they use a Gaussian noise augmented dataset which raises concerns about the effectiveness of their method since augmentation alone has been shown to be effective in ensemble training Strauss et al.

Original: Another study maximizes diversity between classes Pang et al.
Revised: Another study maximizes diversity between classes Pang et al.

Original: and thus does not apply to our work here that focuses on binary classes only.
Revised: and thus does not apply to our work here that focuses on binary classes only.

Original: Even for multiple classes their method is computationally expensive as it uses a joint loss function.
Revised: Even for multiple classes their method is computationally expensive as it uses a joint loss function.

Original: Other methods inject noise to models in the ensemble
Revised: Other methods inject noise to models in the ensemble

Original: Liu et al.
Revised: Liu et al.

Original: (2018a)
Revised: (2018a)

Original: but their evaluation is only in the white box setting.
Revised: but their evaluation is only in the white box setting.

Original: Various measures for ensemble diversity have been previously proposed for deep networks Liu et al.
Revised: Various measures for ensemble diversity have been previously proposed for deep networks Liu et al.

Original: and evaluated in the white-box setting.
Revised: and evaluated in the white-box setting.

Original: We can apply all of the above diversity training methods to our ensemble of sign networks.
Revised: We can apply all of the above diversity training methods to our ensemble of sign networks.

Original: Our work, however, is not explicitly aimed at enhancing diversity.
Revised: Our work, however, is not explicitly aimed at enhancing diversity.

Original: As we show it is naturally diverse and we conjecture this is due to the non-convexity of sign activation and our randomized training method.
Revised: As we show it is naturally diverse and we conjecture this is due to the non-convexity of sign activation and our randomized training method.

Original: Even sigmoid activation networks have a non-convex search space but we can imagine that sign activation gives a greater degree of freedom.
Revised: Even sigmoid activation networks have a non-convex search space but we can imagine that sign activation gives a greater degree of freedom.

Original: This can easily be seen in the case of a linear classiﬁer with logistic or hinge loss vs. 01 loss Xue et al.
Revised: This can easily be seen in the case of a linear classiﬁer with logistic or hinge loss vs. 01 loss Xue et al.

Original: (2020b).
Revised: (2020b).

Original: Our model accuracy is not the same as convolutional networks understandably due to lack of convolutions in our networks.
Revised: Our model accuracy is not the same as convolutional networks understandably due to lack of convolutions in our networks.

Original: But they are close to sigmoid activated networks and random forests, and better than binary neural networks in most cases.
Revised: But they are close to sigmoid activated networks and random forests, and better than binary neural networks in most cases.

Original: It is possible to extend our training procedure to allow for convolutions and this may increase accuracy making our model comparable to convolutional networks and much more robust.
Revised: It is possible to extend our training procedure to allow for convolutions and this may increase accuracy making our model comparable to convolutional networks and much more robust.

Original: It is hard to make a general claim of robustness with only 10 images from CIFAR10.
Revised: It is hard to make a general claim of robustness with only 100 images from CIFAR10.

Original: We would need to show more images from CIFAR10 and other image benchmarks as well but our preliminary experiments suggest higher distortion on other image data as well.
Revised: We would need to show more images from CIFAR10 and other image benchmarks as well but our preliminary experiments suggest higher distortion on other image data as well.

Original: For example on a single random test datapoint in CelebA our SCDCE ensemble has minimum L 2 distortion of 8.6 while the next best
Revised: For example on a single random test datapoint in CelebA our SCDCE ensemble has minimum L 2 distortion of 8.6 while the next best

Original: LeNet ensemble is 3.3.
Revised: LeNet ensemble is 3.3.

Original: Due to computational limitations we are unable to show more image data here but instead we take another route to show generality of our results.
Revised: Due to computational limitations we are unable to show more image data here but instead we take another route to show generality of our results.

Original: We show that our model is robust even to text classiﬁcation black box attacks and on ECG data attacks.
Revised: We show that our model is robust even to text classiﬁcation black box attacks and on ECG data attacks.

Original: Both of these are outside the domain of images and our model’s robustness there suggests a greater generalization.
Revised: Both of these are outside the domain of images and our model’s robustness there suggests a greater generalization.

Original: Future work entails extending our training to sign activated convolutions and multi-class networks.
Revised: Future work entails extending our training to sign activated convolutions and multi-class networks.

Original: 3. C ONCLUSION
Revised: 3. C ONCLUSION

Original: We show that our ensemble of gradient-free sign activation networks have a higher image adversarial distortion than ensembles of several other networks and random forests on10 random images in CIFAR10.
Revised: We show that our ensemble of gradient-free sign activation networks have a higher image adversarial distortion than ensembles of several other networks and random forests on images in CIFAR10 class 0 vs. 1.

Original: We then give a transferability based argument to support this result and show robustness on text document and ECG data classiﬁcation.
Revised: We then give a transferability based argument to support this result and show robustness on text document and ECG data classiﬁcation.