Original: D EFENDING AGAINST BLACK - BOX ADVERSARIAL ATTACKS WITH GRADIENT - FREE TRAINED SIGN ACTIVATION NEURAL NETWORKS
Revised: D EFENDING AGAINST BLACK - BOX ADVERSARIAL ATTACKS WITH GRADIENT - FREE TRAINED SIGN ACTIVATION NEURAL NETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: While machine learning models today can achieve high accuracies on classiﬁcation tasks, they can be deceived by minor imperceptible distortions to the data.
Revised: While machine learning models today can achieve high accuracies on classiﬁcation tasks, they can be deceived by minor imperceptible distortions to the data.

Original: These are known as adversarial attacks and can be lethal in the black-box setting which does not require knowledge of the target model type or its parameters.
Revised: These are known as adversarial attacks and can be lethal in the black-box setting which does not require knowledge of the target model type or its parameters.

Original: Binary neural networks that have sign activation and are trained with gradient descent have been shown to be harder to attack than conventional sigmoid activation networks but their improvements are marginal.
Revised: Binary neural networks that have sign activation and are trained with gradient descent have been shown to be harder to attack than conventional sigmoid activation networks but their improvements are marginal.

Original: We instead train sign activation networks with a novel gradient-free stochastic coordinate descent algorithm and propose an ensemble of such networks as a defense model.
Revised: We instead train sign activation networks with a novel gradient-free stochastic coordinate descent algorithm and propose an ensemble of such networks as a defense model.

Original: We evaluate the robustness of our model (a hard problem in itself) on image, text, and medicalimage and ECG data and ﬁnd it to be more robust than ensembles of binary, full precision, and convolutional neural networks, and than random forests while attaining comparable clean test accuracy.
Revised: We evaluate the robustness of our model (a hard problem in itself) on image, text, and medical ECG data and ﬁnd it to be more robust than ensembles of binary, full precision, and convolutional neural networks, and than random forests while attaining comparable clean test accuracy.

Original: In order to explain our model’s robustness we show that an adversary targeting a single network in our ensemble fails to attack (and thus non-transferable to) other networks in the ensemble.
Revised: In order to explain our model’s robustness we show that an adversary targeting a single network in our ensemble fails to attack (and thus non-transferable to) other networks in the ensemble.

Original: Thus a datapoint requires a large distortion to fool the majority of networks in our ensemble and is likely to be detected in advance.
Revised: Thus a datapoint requires a large distortion to fool the majority of networks in our ensemble and is likely to be detected in advance.

Original: This property of non-transferability arises naturally from the non-convexity of sign activation networks and randomization in our gradient-free training algorithm without any adversarial defense effort.
Revised: This property of non-transferability arises naturally from the non-convexity of sign activation networks and randomization in our gradient-free training algorithm without any adversarial defense effort.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: State of the art machine learning algorithms can achieve high accuracies in classiﬁcation tasks but misclassify minor perturbations in the data known as as adversarial attacks Goodfellow et al.
Revised: State of the art machine learning algorithms can achieve high accuracies in classiﬁcation tasks but misclassify minor perturbations in the data known as as adversarial attacks Goodfellow et al.

Original: Papernot et al.
Revised: Papernot et al.

Original: (2016b); Kurakin et al.
Revised: (2016b); Kurakin et al.

Original: ; Carlini & Wagner (2017); Brendel et al.
Revised: ; Carlini & Wagner (2017); Brendel et al.

Original: Adversarial examples have been shown to transfer across models which makes it possible to perform transfer-based (substitute model) black box attacks Papernot et al.
Revised: Adversarial examples have been shown to transfer across models which makes it possible to perform transfer-based (substitute model) black box attacks Papernot et al.

Original: (2016a).
Revised: (2016a).

Original: To counter adversarial attacks many defense methods been proposed with adversarial training being the most popular
Revised: To counter adversarial attacks many defense methods been proposed with adversarial training being the most popular

Original: Szegedy et al.
Revised: Szegedy et al.

Original: (2014); Tram`er et al.
Revised: (2014); Tram`er et al.

Original: However this tends to lower accuracy on clean test data that has no perturbations Raghunathan et al.
Revised: However this tends to lower accuracy on clean test data that has no perturbations Raghunathan et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: and can still be attacked with better transfer based methods Wu et al.
Revised: and can still be attacked with better transfer based methods Wu et al.

Original: Xie et al. (2019a); Dong et al.
Revised: Xie et al. (2019a); Dong et al.

Original: Many previously proposed defenses have also been shown to be vulnerable Carlini & Wagner (2017);
Revised: Many previously proposed defenses have also been shown to be vulnerable Carlini & Wagner (2017);

Original: Athalye et al.
Revised: Athalye et al.

Original: ; Ghiasi et al.
Revised: ; Ghiasi et al.

Original: (2020) thus leaving adversarial robustness an open problem in machine learning.
Revised: (2020) thus leaving adversarial robustness an open problem in machine learning.

Original: A more lethal and practical attack than substitute model training is a boundary based one that requires only the prediction of the model Brendel et al.
Revised: A more lethal and practical attack than substitute model training is a boundary based one that requires only the prediction of the model Brendel et al.

Original: These attacks are aimed at ﬁnding the minimum distortion to an image such that it will fool a classiﬁer.
Revised: These attacks are aimed at ﬁnding the minimum distortion to an image such that it will fool a classiﬁer.

Original: This is in fact an NP-hard problem for ReLu activated neural networks Katz et al.
Revised: This is in fact an NP-hard problem for ReLu activated neural networks Katz et al.

Original: ; Sinha et al.
Revised: ; Sinha et al.

Original: and tree ensemble classiﬁers Kantchelian et al.
Revised: and tree ensemble classiﬁers Kantchelian et al.

Original: Even approximating the minimum distortion for ReLu activated neural networks is NP-hard Weng et al.
Revised: Even approximating the minimum distortion for ReLu activated neural networks is NP-hard Weng et al.

Original: Boundary based black box attacks such as HopSkipJump Chen et al.
Revised: Boundary based black box attacks such as HopSkipJump Chen et al.

Original: , Boundary Attack Brendel et al.
Revised: , Boundary Attack Brendel et al.

Original: and RayS Chen & Gu (2020) give an upper bound on the minimum adversarial distortion.
Revised: and RayS Chen & Gu (2020) give an upper bound on the minimum adversarial distortion.

Original: Binary neural networks that have sign activation and binary weights were originally proposed as lightweight models.
Revised: Binary neural networks that have sign activation and binary weights were originally proposed as lightweight models.

Original: These are trained with gradient descent by approximating the sign activation.
Revised: These are trained with gradient descent by approximating the sign activation.

Original: Recent work has shown that they tend to be more adversarially robust than full precision networks but the improvements are marginal (see Tables 4 and 5 in Galloway et al.
Revised: Recent work has shown that they tend to be more adversarially robust than full precision networks but the improvements are marginal (see Tables 4 and 5 in Galloway et al.

Original: (2018) and Table 8 in
Revised: (2018) and Table 8 in

Original: Panda et al.
Revised: Panda et al.

Original: In this paper we propose a gradient free stochastic coordinate descent algorithm for training sign activation networks with and without binary weights similar to recent work Xue et al.
Revised: In this paper we propose a gradient free stochastic coordinate descent algorithm for training sign activation networks with and without binary weights similar to recent work Xue et al.

Original: (2020a;b);
Revised: (2020a;b);

Original: Xie et al.
Revised: Xie et al.

Original: (2019b).
Revised: (2019b).

Original: While our original intention was to study the accuracy of a sign activation network trained directly without any approximation we make an interesting ﬁnding on the adversarial robustness of our model.
Revised: While our original intention was to study the accuracy of a sign activation network trained directly without any approximation we make an interesting ﬁnding on the adversarial robustness of our model.

Original: We ﬁnd that ensembling our model gives a high minimum distortion (as measured by HopSkipJump) compared to full precision, binary, and convolutional neural networks.
Revised: We ﬁnd that ensembling our model gives a high minimum distortion (as measured by HopSkipJump) compared to full precision, binary, and convolutional neural networks.

Original: We explain this phenomena by measuring the transferability between networks in an ensemble.
Revised: We explain this phenomena by measuring the transferability between networks in an ensemble.

Original: In summary we make the following observations in our paper:
Revised: In summary we make the following observations in our paper:

Original: • Our single hidden layer sign activation network has higher minimum distortion than ensembles of full precision and binary neural networks, than random forests that have the advantage of bootstrapping and random feature selection, and than ensembles of convolutional networks that have the advantage of convolutions and several layers.
Revised: • Our single hidden layer sign activation network has higher minimum distortion than ensembles of full precision and binary neural networks, than random forests that have the advantage of bootstrapping and random feature selection, and than ensembles of convolutional networks that have the advantage of convolutions and several layers.

Original: • Our model’s robustness stems from non-transferability of adversarial examples between networks in our ensemble and its robustness increases as we add more networks to the ensemble.
Revised: • Our model’s robustness stems from non-transferability of adversarial examples between networks in our ensemble and its robustness increases as we add more networks to the ensemble.

Original: • Substitute model black box attacks require a much greater distortion to bring our model to zero adversarial accuracy compared to ensembles of full precision and binary networks.
Revised: • Substitute model black box attacks require a much greater distortion to bring our model to zero adversarial accuracy compared to ensembles of full precision and binary networks.

Original: • Text classiﬁcation black box attacks are less effective on our model than on convolutional networks, random forests, and ensembles of full precision and binary networks.
Revised: • Text classiﬁcation black box attacks are less effective on our model than on convolutional networks, random forests, and ensembles of full precision and binary networks.

Original: • In a medical diagnosis setting, attacks on ECG data on our model have a higher distortions and are visually distinguishable compared to attacks on ensembles of full precision and convolutional networks, and on random forests.
Revised: • In a medical diagnosis setting, attacks on ECG data on our model have a higher distortions and are visually distinguishable compared to attacks on ensembles of full precision and convolutional networks, and on random forests.

Original:  M ETHODS 2. G RADIENT
Revised:  M ETHODS 2. G RADIENT

Original: FREE STOCHASTIC
Revised: FREE STOCHASTIC

Original: Suppose we are given binary class data x i ∈ R d and y i ∈ {− 1 , +1 } for i = 0 ...n − 1 .
Revised: Suppose we are given binary class data x i ∈ R d and y i ∈ {− 1 , +1 } for i = 0 ...n − 1 .

Original: Consider the objective function of a single hidden layer neural network with sign activation and 01 loss given below.
Revised: Consider the objective function of a single hidden layer neural network with sign activation and 01 loss given below.

Original: We employ a stochastic coordinate descent algorithm shown in Algorithm 1 (similar to recent work Xue et al.
Revised: We employ a stochastic coordinate descent algorithm shown in Algorithm 1 (similar to recent work Xue et al.

Original: (2020a;b); Xie et al. (2019b)) to minimize this objective.
Revised: (2020a;b); Xie et al. (2019b)) to minimize this objective.

Original: [Equation]
Revised: [Equation]

Original: We can train sign activation networks with and without binary weights using our SCD training procedure above.
Revised: We can train sign activation networks with and without binary weights using our SCD training procedure above.

Original: In the case of binary weights we don’t need a learning rate.
Revised: In the case of binary weights we don’t need a learning rate.

Original: We apply GPU parallelism to simultaneously update features and other heuristics to speed up runtimes (with additional details given in the Supplementary Material).
Revised: We apply GPU parallelism to simultaneously update features and other heuristics to speed up runtimes (with additional details given in the Supplementary Material).

Original: 2. I MPLEMENTATION , TEST ACCURACY , AND RUNTIME
Revised: 2. I MPLEMENTATION , TEST ACCURACY , AND RUNTIME

Original: We implement our training procedure in Python, numpy, and Pytorch Paszke et al.
Revised: We implement our training procedure in Python, numpy, and Pytorch Paszke et al.

Original: and make our code freely available from https://github.com/zero-one-loss/scd_github .
Revised: and make our code freely available from https://github.com/zero-one-loss/scd_github .

Original: We train three types of sign activation networks with our algorithm:
Revised: We train three types of sign activation networks with our algorithm:

Original: (1) SCD01: 01-loss in the ﬁnal node, (2) SCDCE: cross-entropy loss in the ﬁnal node, and (3) SCDCEBNN: cross-entropy in the ﬁnal node with binary weights throughout the model.
Revised: (1) SCD01: 01-loss in the ﬁnal node, (2) SCDCE: cross-entropy loss in the ﬁnal node, and (3) SCDCEBNN: cross-entropy in the ﬁnal node with binary weights throughout the model.

Original: Since sign activation is non-convex and our training starts from a different random initialization we run it a 100 times and output the majority vote.
Revised: Since sign activation is non-convex and our training starts from a different random initialization we run it a 100 times and output the majority vote.

Original: Procedure:
Revised: Procedure:

Original: Initialize all network weights W, w to random values from the Normal distribution N (0 , 1) .
Revised: Initialize all network weights W, w to random values from the Normal distribution N (0 , 1) .

Original: Set network thresholds W 0 to the median projection value on their corresponding weight vectors and w 0 to the projection value that minimizes our network objective.
Revised: Set network thresholds W 0 to the median projection value on their corresponding weight vectors and w 0 to the projection value that minimizes our network objective.

Original: while i < epochs do
Revised: while i < epochs do

Original: Randomly sample a batch of data equally from each class.
Revised: Randomly sample a batch of data equally from each class.

Original: (We set this to 75% of the training data in image and text data experiments and 25% in the ECG data.)
Revised: (We set this to 75% of the training data in image and text data experiments and 25% in the ECG data.)

Original: Perform coordinate descent separately ﬁrst on the ﬁnal node w and then a randomly selected hidden node u (a random column from the hidden layer weight matrix W ) 3.
Revised: Perform coordinate descent separately ﬁrst on the ﬁnal node w and then a randomly selected hidden node u (a random column from the hidden layer weight matrix W ) 3.

Original: Suppose we are performing coordinate descent on node w .
Revised: Suppose we are performing coordinate descent on node w .

Original: We select a random set of features (coordinates) from w called F .
Revised: We select a random set of features (coordinates) from w called F .

Original: For each feature w i ∈ F we add/subtract a learning rate η
Revised: For each feature w i ∈ F we add/subtract a learning rate η

Original: and then determine the w 0 that optimizes the loss (done in parallel on a GPU).
Revised: and then determine the w 0 that optimizes the loss (done in parallel on a GPU).

Original: We consider all possible values of w 0 = w T x
Revised: We consider all possible values of w 0 = w T x

Original: i + w T x
Revised: i + w T x

Original: i
Revised: i

Original: +1for i
Revised: +1for i

Original: = 0 ...
Revised: = 0 ...

Original: n − 2 and select the one that minimizes the loss (also performed in parallel on a GPU).
Revised: n − 2 and select the one that minimizes the loss (also performed in parallel on a GPU).

Original: After making the update above we evaluate the loss on the full dataset (performed on a GPU for parallel speedups) and accept the change if it improves the loss.
Revised: After making the update above we evaluate the loss on the full dataset (performed on a GPU for parallel speedups) and accept the change if it improves the loss.

Original: end while
Revised: end while

Original: To illustrate our real runtimes and clean test accuracies we compare our models with a single hidden layer of 20 nodes to the equivalent network with sigmoid activation and logistic loss (denoted as
Revised: To illustrate our real runtimes and clean test accuracies we compare our models with a single hidden layer of 20 nodes to the equivalent network with sigmoid activation and logistic loss (denoted as

Original: MLP) and the binary neural network (denoted as BNN)
Revised: MLP) and the binary neural network (denoted as BNN)

Original: Hubara et al.
Revised: Hubara et al.

Original: We used the MLPClassiﬁer in scikit-learn Pedregosa et al.
Revised: We used the MLPClassiﬁer in scikit-learn Pedregosa et al.

Original: (2011) to implement MLP and the Larq library Geiger & Team (2020) with the approx approximation to the sign activation.
Revised: (2011) to implement MLP and the Larq library Geiger & Team (2020) with the approx approximation to the sign activation.

Original: This has shown to achieve a higher test accuracy than the original straight through estimator (STE) of the sign activation Liu et al.
Revised: This has shown to achieve a higher test accuracy than the original straight through estimator (STE) of the sign activation Liu et al.

Original: (2018b).
Revised: (2018b).

Original: We perform a 1000 iterations of SCD01 and SCDCE and 10000 of SCDCEBNN.
Revised: We perform a 1000 iterations of SCD01 and SCDCE and 10000 of SCDCEBNN.

Original: In Table 1 we show the runtimes of a single run of all models on CIFAR10 Krizhevsky (2009) ( 32 × 32 × 3 , 10K train, 2K test), CelebA facial attributes black hair vs brown hair Liu et al.
Revised: In Table 1 we show the runtimes of a single run of all models on CIFAR10 Krizhevsky (2009) ( 32 × 32 × 3 , 10K train, 2K test), CelebA facial attributes black hair vs brown hair Liu et al.

Original: ( 96 × 96 × 3 , 1K train, 1K test), GTSRB street sign recognition 60 vs 120 speed limit signs Stallkamp et al.
Revised: ( 96 × 96 × 3 , 1K train, 1K test), GTSRB street sign recognition 60 vs 120 speed limit signs Stallkamp et al.

Original: (2011) ( 48 × 48 × 3 , 2816 train, 900 test), and ImageNet class 0 vs. 1 Russakovsky et al. (2015) ( 256 × 256 × 3 ,train, 100 test).
Revised: (2011) ( 48 × 48 × 3 , 2816 train, 900 test), and ImageNet class 0 vs. 1 Russakovsky et al. (2015) ( 256 × 256 × 3 ,train, 100 test).

Original: Our training runtimes are comparable to gradient descent in MLP and BNN and thus practically usable.
Revised: Our training runtimes are comparable to gradient descent in MLP and BNN and thus practically usable.

Original: We can trivially parallelize training an ensemble by doing multiple runs on CPU and GPU cores at the same time.
Revised: We can trivially parallelize training an ensemble by doing multiple runs on CPU and GPU cores at the same time.

Original: We also show test accuracies of 100 vote ensembles of all models and ﬁnd our model accuracies to be comparable to MLP and BNN.
Revised: We also show test accuracies of 100 vote ensembles of all models and ﬁnd our model accuracies to be comparable to MLP and BNN.

Original: [Table]
Revised: [Table]

Original:  R ESULTS
Revised:  R ESULTS

Original: Going forward we compare the adversarial robustness of ensembles of our three models SCD01,
Revised: Going forward we compare the adversarial robustness of ensembles of our three models SCD01,

Original: SCDCE, and SCDCEBNN, their full precision and binary gradient descent trained equivalent counterparts MLP and BNN, two convolutional neural networks: LeNet LeCun et al.
Revised: SCDCE, and SCDCEBNN, their full precision and binary gradient descent trained equivalent counterparts MLP and BNN, two convolutional neural networks: LeNet LeCun et al.

Original: (1998) and
Revised: (1998) and

Original: ResNet50 He et al.
Revised: ResNet50 He et al.

Original: (2016), and random forests Breiman (2001) (denoted as RF).
Revised: (2016), and random forests Breiman (2001) (denoted as RF).

Original: For each model we use the majority vote output of 100 votes each with different initial parameters except for ResNet where we use 10 votes.
Revised: For each model we use the majority vote output of 100 votes each with different initial parameters except for ResNet where we use 10 votes.

Original: In random forest we use an ensemble of 100 trees.
Revised: In random forest we use an ensemble of 100 trees.

Original: We use a single hidden layer of 20 nodes in our three models and in MLP and BNN throughout the paper.
Revised: We use a single hidden layer of 20 nodes in our three models and in MLP and BNN throughout the paper.

Original: The convolutional networks and random forest are not a fair comparison to our model since it has fewer parameters and does not perform bootstrapping or random feature selection as random forest.
Revised: The convolutional networks and random forest are not a fair comparison to our model since it has fewer parameters and does not perform bootstrapping or random feature selection as random forest.

Original: We include them nevertheless since convolutional neural networks serve as state of the art references and random forest serves as an alternative ensemble method.
Revised: We include them nevertheless since convolutional neural networks serve as state of the art references and random forest serves as an alternative ensemble method.

Original: 3. A DVERSARIAL DISTORTION ON IMAGE DATA
Revised: 3. A DVERSARIAL DISTORTION ON IMAGE DATA

Original: The minimum distortion required to make a datapoint adversarial is an indicator of a model’s adversarial and even corruption/general robustness Gilmer et al.
Revised: The minimum distortion required to make a datapoint adversarial is an indicator of a model’s adversarial and even corruption/general robustness Gilmer et al.

Original: We consider 10 randomly selected datapoints from the CIFAR10 benchmark Krizhevsky (2009) and report their minimal adversarial distortion as given by HopSkipJump Chen et al.
Revised: We consider 10 randomly selected datapoints from the CIFAR10 benchmark Krizhevsky (2009) and report their minimal adversarial distortion as given by HopSkipJump Chen et al.

Original: , Boundary Attack Brendel et al.
Revised: , Boundary Attack Brendel et al.

Original: and RayS Chen & Gu (2020).
Revised: and RayS Chen & Gu (2020).

Original: We use the HopSkipJump and Boundary Attack implementation in the IBM Adversarial Robustness
Revised: We use the HopSkipJump and Boundary Attack implementation in the IBM Adversarial Robustness

Original: Toolkit (ART) Nicolae et al.
Revised: Toolkit (ART) Nicolae et al.

Original: In order to obtain as accurate an estimate as possible we run both mthods 10 times each with an initial pool size of 1000 random datapoints and maximum iterations of 100 and report the minimum value.
Revised: In order to obtain as accurate an estimate as possible we run both mthods 10 times each with an initial pool size of 1000 random datapoints and maximum iterations of 100 and report the minimum value.

Original: For a single datapoint this typically takes several hours to ﬁnish and thus we are able to report the distortion of only 10 random points in this study.
Revised: For a single datapoint this typically takes several hours to ﬁnish and thus we are able to report the distortion of only 10 random points in this study.

Original: We use the RayS implementation from their GitHub site https://github.com/uclaml/RayS and run it with default parameters of 40,000 queries to obtain a distortion estimate.
Revised: We use the RayS implementation from their GitHub site https://github.com/uclaml/RayS and run it with default parameters of 40,000 queries to obtain a distortion estimate.

Original: In Table 2 ﬁrst row we show the clean test accuracy of all models on CIFAR10 class 0 vs. 1.
Revised: In Table 2 ﬁrst row we show the clean test accuracy of all models on CIFAR10 class 0 vs. 1.

Original: The convolutional networks LeNet and ResNet50 have higher accuracies since they have the advantage of convolutions.
Revised: The convolutional networks LeNet and ResNet50 have higher accuracies since they have the advantage of convolutions.

Original: In the following three rows of Table 2 we see the minimum adversarial distortion of models as estimate by three boundary attack methods.
Revised: In the following three rows of Table 2 we see the minimum adversarial distortion of models as estimate by three boundary attack methods.

Original: We were unable to attack some models with
Revised: We were unable to attack some models with

Original: Boundary Attack and RayS due to time constraints and mark them as NA.
Revised: Boundary Attack and RayS due to time constraints and mark them as NA.

Original: We see that HopSkipJump gives the lowest distortion for each model except for SCD01 and SCDCE where it is comparable to
Revised: We see that HopSkipJump gives the lowest distortion for each model except for SCD01 and SCDCE where it is comparable to

Original: RayS.
Revised: RayS.

Original: Amongst HopSkipJump distortions our sign activation trained models have the highest adversarial distortion with the binary weights cross-entropy variant as the winner.
Revised: Amongst HopSkipJump distortions our sign activation trained models have the highest adversarial distortion with the binary weights cross-entropy variant as the winner.

Original: All other neural networks lag far behind and have distortion even lower than random forest.
Revised: All other neural networks lag far behind and have distortion even lower than random forest.

Original: Even though BNN also has sign activations its distortions are similar to MLP possibly due to its approximation of the sign activation and gradient descent search.
Revised: Even though BNN also has sign activations its distortions are similar to MLP possibly due to its approximation of the sign activation and gradient descent search.

Original: If we use the the straight through estimator and swish approximations
Revised: If we use the the straight through estimator and swish approximations

Original: Darabi et al.
Revised: Darabi et al.

Original: the distortions remain similar to what we report here.
Revised: the distortions remain similar to what we report here.

Original: [Table]
Revised: [Table]

Original: To further validate the distortions above we run HopSkipJump on SCD01, MLP, LeNet, and RF with 10 maximum iterations on the ﬁrst 100 CIFAR10 test datapoints.
Revised: To further validate the distortions above we run HopSkipJump on SCD01, MLP, LeNet, and RF with 10 maximum iterations on the ﬁrst 100 CIFAR10 test datapoints.

Original: We used a ﬁxed image as the initial one in these experiments.
Revised: We used a ﬁxed image as the initial one in these experiments.

Original: In Table 3 we see that SCD01 distortions are the highest and the relative ranking is the same as we saw for the 10 images above with 100 maximum iterations of
Revised: In Table 3 we see that SCD01 distortions are the highest and the relative ranking is the same as we saw for the 10 images above with 100 maximum iterations of

Original: HopSkipJump.
Revised: HopSkipJump.

Revised: In Table 4 below we show HopSkipJump distortions (min of 10 runs 100 max iterations each) on a single random image from CelebA, GTSRB, and ImageNet datasets.

Revised: We ﬁnd our SCD models to have a higher distortion on both CelebA and GTSRB but comparable to MLP on ImageNet.

Revised: To illustrate our

Original: 3. T RANSFERABILITY WITHIN ENSEMBLES AND EFFECT OF ENSEMBLE SIZE
Revised: [Table] [Table] [Table] T RANSFERABILITY WITHIN ENSEMBLES AND EFFECT OF ENSEMBLE SIZE

Original: To understand the above phenomena we estimate the probability that an adversarial example targeting a single model in the ensemble will also be adversarial to other models in the ensemble.We can
Revised: To understand the above phenomena we estimate the probability that an adversarial example targeting a single model in the ensemble will also be adversarial to other models in the ensemble.

Original: [Table]

Original:  estimate this by ﬁrst performing a HopSkipJump attack on each model in the ensemble separately.
Revised: We can estimate this by ﬁrst performing a HopSkipJump attack on each model in the ensemble separately.

Original: Let x (cid:48)
Revised: Let x (cid:48)

Original: i be the adversary obtained by targeting model m i in the ensemble.
Revised: i be the adversary obtained by targeting model m i in the ensemble.

Original: Let k i be the number of models in the ensemble that are also misclassiﬁed by the adversary x (cid:48) i (thus transferable).
Revised: Let k i be the number of models in the ensemble that are also misclassiﬁed by the adversary x (cid:48) i (thus transferable).

Original: We sum k
Revised: We sum k

Original: i for i = 0 ...n − 1 and divide by 9900 which is the maximum value of this sum (obtained when the adversary attacks all models in the ensemble excluding the target of course
Revised: i for i = 0 ...n − 1 and divide by 9900 which is the maximum value of this sum (obtained when the adversary attacks all models in the ensemble excluding the target of course).

Original: We average this probability for Images 0 through 7 for each method.
Revised: We average this probability for Images 0 through 7 for each method.

Original: In Table 4 we see that this probability is lowest for our models and highest for MLP and BNN.
Revised: In Table 6 we see that this probability is lowest for our models and highest for MLP and BNN.

Original: The fact that this probability is very low for our models indicates that for several of the networks in our ensemble the adversary targeting a ﬁxed network does not transfer to other ones.
Revised: The fact that this probability is very low for our models indicates that for several of the networks in our ensemble the adversary targeting a ﬁxed network does not transfer to other ones.

Original: The low transferability of our models indicates that a greater distortion is required for an image to be adversarial.
Revised: The low transferability of our models indicates that a greater distortion is required for an image to be adversarial.

Original: [Table]
Revised: [Table]

Original: In fact as we see in Figure 1 the robustness of our models increases as we increase the ensemble size to a much larger degree than ensembles of MLP and BNN, and than RF.
Revised: In fact as we see in Figure 1 the robustness of our models increases as we increase the ensemble size to a much larger degree than ensembles of MLP and BNN, and than RF.

Original: We use ensemble sizes of 100 in this study but the ﬁgure suggests that increasing our ensemble size is likely to further increase robustness.
Revised: We use ensemble sizes of 100 in this study but the ﬁgure suggests that increasing our ensemble size is likely to further increase robustness.

Original: 3. S UBSTITUTE MODEL BLACK - BOX ATTACKS
Revised: 3. S UBSTITUTE MODEL BLACK - BOX ATTACKS

Original: Our model’s high distortion in CIFAR10 is reﬂected in substitute model black box attacks on this dataset Papernot et al.
Revised: Our model’s high distortion in CIFAR10 is reﬂected in substitute model black box attacks on this dataset Papernot et al.

Original: (2016a).
Revised: (2016a).

Original: We train a two hidden layer neural network each with 200 nodes as the substitute using the standard adversarial augmented algorithm Papernot et al.
Revised: We train a two hidden layer neural network each with 200 nodes as the substitute using the standard adversarial augmented algorithm Papernot et al.

Original: (2017) (described fully in the Supplementary Material).
Revised: (2017) (described fully in the Supplementary Material).

Original: In Figure 2 we that our models require a much higher distortion than their gradient descent trained equivalents MLP and BNN in order to reach zero percent adversarial accuracy.
Revised: In Figure 2 we that our models require a much higher distortion than their gradient descent trained equivalents MLP and BNN in order to reach zero percent adversarial accuracy.

Original: We also see that all models attacked with random Gaussian noise of the same distortion added to the test examples are barely affected thus showing the effectiveness of the black box adversarial examples.
Revised: We also see that all models attacked with random Gaussian noise of the same distortion added to the test examples are barely affected thus showing the effectiveness of the black box adversarial examples.

Original: 3. T EXT BLACK - BOX ATTACKS
Revised: 3. T EXT BLACK - BOX ATTACKS

Original: The TextFooler Jin et al.
Revised: The TextFooler Jin et al.

Original: (2020) method is designed to ﬁnd syntactically and semantically similar adversarial documents by replacing important words with similar ones until the document is misclassiﬁed.
Revised: (2020) method is designed to ﬁnd syntactically and semantically similar adversarial documents by replacing important words with similar ones until the document is misclassiﬁed.

Original: We apply this to all ensemble models on four document classiﬁcation datasets: Internet
Revised: We apply this to all ensemble models on four document classiﬁcation datasets: Internet

Original: Movie Database (25K train, 25K test, mean words per document: 215) and Yelp (560K train, 38K test, mean words per document: 152) positive and negative reviews (IMDB and Yelp), sentence classiﬁcation of positive and negative sentiments (9K train, 1K test, mean words per document: 20, denoted as MR), and sentence-level classiﬁcation of news items in World and Sports categories (120K train, 7.6K test, mean words per document: 43, denoted as AG)
Revised: Movie Database (25K train, 25K test, mean words per document: 215) and Yelp (560K train, 38K test, mean words per document: 152) positive and negative reviews (IMDB and Yelp), sentence classiﬁcation of positive and negative sentiments (9K train, 1K test, mean words per document: 20, denoted as MR), and sentence-level classiﬁcation of news items in World and Sports categories (120K train, 7.6K test, mean words per document: 43, denoted as AG)

Original: Jin et al.
Revised: Jin et al.

Original: WordCNN stacks word vectors Pennington et al.
Revised: WordCNN stacks word vectors Pennington et al.

Original: (2014) of each word in a document into a matrix to treat it as 2D image Kim (2014).
Revised: (2014) of each word in a document into a matrix to treat it as 2D image Kim (2014).

Original: In the other models that take feature vectors as inputs we consider the averaged word vector of all words in a document Lilleberg et al. (2015).
Revised: In the other models that take feature vectors as inputs we consider the averaged word vector of all words in a document Lilleberg et al. (2015).

Original: For all models we use 200 dimensional Glove word embeddings pre-trained on 6 billion tokens from Wikipedia and
Revised: For all models we use 200 dimensional Glove word embeddings pre-trained on 6 billion tokens from Wikipedia and

Original: Gigawords Pennington et al.
Revised: Gigawords Pennington et al.

Original: This gives a lower clean test accuracy than WordCNN but still above an acceptable level in practice.
Revised: This gives a lower clean test accuracy than WordCNN but still above an acceptable level in practice.

Original: In Table 5 we see that ensembles of our models give the highest adversarial accuracy on all four datasets and require the greatest number of queries.
Revised: In Table 7 we see that ensembles of our models give the highest adversarial accuracy on all four datasets and require the greatest number of queries.

Original: If a smaller limit was placed on the allowed queries (for example imposed by the system being attacked) we can expect a higher adversarial accuracy for our models.
Revised: If a smaller limit was placed on the allowed queries (for example imposed by the system being attacked) we can expect a higher adversarial accuracy for our models.

Original: Here we show ensembles of 8 votes for each model.
Revised: Here we show ensembles of 8 votes for each model.

Original: If we increase the ensemble size to 100 we ﬁnd the adversarial accuracy of our models, BNN, and RF slightly drop but their relative difference remains the same.[Table]
Revised: If we increase the ensemble size to 100 we ﬁnd the adversarial accuracy of our models, BNN, and RF slightly drop but their relative difference remains the same.

Original: [Table]
Revised: [Table]

Original:  ECG BLACK - BOX ATTACKS
Revised: 3. ECG BLACK - BOX ATTACKS

Original: ECG time-series data is increasingly being used in automatic diagnosis by machine learning systems
Revised: ECG time-series data is increasingly being used in automatic diagnosis by machine learning systems

Original: Ribeiro et al.
Revised: Ribeiro et al.

Original: Tailored adversarial attacks have recently been proposed Han et al.
Revised: Tailored adversarial attacks have recently been proposed Han et al.

Original: Chen et al.
Revised: Chen et al.

Original: but HopSkipJump can also be used to produce adversarial ECG examples.
Revised: but HopSkipJump can also be used to produce adversarial ECG examples.

Original: To illustrate this and evaluate our model’s robustness on this data we consider the PTB Diagnostic
Revised: To illustrate this and evaluate our model’s robustness on this data we consider the PTB Diagnostic

Original: ECG dataset Bousseljot et al.
Revised: ECG dataset Bousseljot et al.

Original: Goldberger et al.
Revised: Goldberger et al.

Original: (2000) available from this URL https: //www.kaggle.com
Revised: (2000) available from this URL https: //www.kaggle.com

Original: /shayanfazeli/heartbeat .
Revised: /shayanfazeli/heartbeat .

Original: We randomly split this dataset into an 80:20 train test split.
Revised: We randomly split this dataset into an 80:20 train test split (yielding 13096 train and 1456 test points).

Original: We train 100-model ensembles of SCD01, SCDCE, and MLP.
Revised: We train 100-model ensembles of SCD01, SCDCE, and MLP.

Original: We also train an ensemble ofconvolutional neural networks (CNN) with 1D convolutional kernels, random forest (RF) withtrees, and a 10-model ensemble of BNN (as opposed to a 100-model ensemble which is slow to attack and did not show a better distortion on selected datapoints).
Revised: We also train an ensemble ofconvolutional neural networks (CNN) with 1D convolutional kernels, random forest (RF) withtrees, and a 10-model ensemble of BNN (as opposed to a 100-model ensemble which is slow to attack and did not show a better distortion on selected datapoints).

Original: Each of our CNNs has the following structure: 64 1x16 Conv1D kernels → MaxPool 1x4 → 128 1x16 Conv1D kernels →
Revised: Each of our CNNs has the following structure: 64 1x16 Conv1D kernels → MaxPool 1x4 → 128 1x16 Conv1D kernels →

Original: MaxPool 1x4 → 256 1x16 Conv1D kernels → MaxPool 1x2 → FullyConnected → Output.
Revised: MaxPool 1x4 → 256 1x16 Conv1D kernels → MaxPool 1x2 → FullyConnected → Output.

Original: In
Revised: In

Original: Table 6 we see the clean test accuracy of our model is slightly lower than gradient-descent trained models and random forest.
Revised: Table 8 we see the clean test accuracy of our model is slightly lower than gradient-descent trained models and random forest.

Original: We picked 37 random datapoints from the test and attacked all models on these points.
Revised: We picked 37 random datapoints from the test and attacked all models on these points.

Original: We attack each point 10 times and report the minimum with the same parameters as we did in our CIFAR10 attacks described above.
Revised: We attack each point 10 times and report the minimum with the same parameters as we did in our CIFAR10 attacks described above.

Original: In Table 6 second row we show the average min L 2 distortion and ﬁnd SCD01 to have the highest one.
Revised: In Table 8 second row we show the average min L 2 distortion and ﬁnd SCD01 to have the highest one.

Original: The L 2 difference between SCD01 and the next best RF (after SCDCE) turns out to be statistically signiﬁcant with a p-value of .008.
Revised: The L 2 difference between SCD01 and the next best RF (after SCDCE) turns out to be statistically signiﬁcant with a p-value of .008.

Original: [Table]
Revised: [Table]

Original: In Figure 3 we visualize an original ECG sample and its adversarial versions targeting SCD01,
Revised: In Figure 3 we visualize an original ECG sample and its adversarial versions targeting SCD01,

Original: CNN, and RF.
Revised: CNN, and RF.

Original: The SCD01 adversary is rigid and has many more bumps compared to the CNN and RF adversaries and is thus likely to be detected by an observer or a system that checks 
Revised: The SCD01 adversary is rigid and has many more bumps compared to the CNN and RF adversaries and is thus likely to be detected by an observer or a system that checks for smoothness (that we expect to see as in the original sample). Original ECG [Table]

Original: [Figure]
Revised: [Figure]

Original: 3. D ISCUSSION
Revised: 3. D ISCUSSION

Original: Using ensembles of neural networks and promoting diverse ensembles has been previously proposed as a defense against adversarial attacks.
Revised: Using ensembles of neural networks and promoting diverse ensembles has been previously proposed as a defense against adversarial attacks.

Original: Studies using ensembles with different initializations (like we do), bootstrapping, and Gaussian noise have shown robustness but only in the white box setting
Revised: Studies using ensembles with different initializations (like we do), bootstrapping, and Gaussian noise have shown robustness but only in the white box setting

Original: Strauss et al.
Revised: Strauss et al.

Original: (2017) (which is somewhat unrealistic since it assumes the attacker has full knowledge of the model and its parameters).
Revised: (2017) (which is somewhat unrealistic since it assumes the attacker has full knowledge of the model and its parameters).

Original: Other studies combine the loss of all models in the classiﬁer and add a regularizer that promotes diversity.
Revised: Other studies combine the loss of all models in the classiﬁer and add a regularizer that promotes diversity.

Original: For example we could try to maximize the angle between gradients of models in the ensemble
Revised: For example we could try to maximize the angle between gradients of models in the ensemble

Original: Kariyappa & Qureshi (2019) to make them misaligned .
Revised: Kariyappa & Qureshi (2019) to make them misaligned .

Original: In their diversity training they use a Gaussian noise augmented dataset which raises concerns about the effectiveness of their method since augmentation alone has been shown to be effective in ensemble training Strauss et al.
Revised: In their diversity training they use a Gaussian noise augmented dataset which raises concerns about the effectiveness of their method since augmentation alone has been shown to be effective in ensemble training Strauss et al.

Original: Another study maximizes diversity between classes Pang et al.
Revised: An- other study maximizes diversity between classes Pang et al.

Original: and thus does not apply to our work here that focuses on binary classes only.
Revised: and thus does not apply to our work here that focuses on binary classes only.

Original: Even for multiple classes their method is computationally expensive as it uses a joint loss function.
Revised: Even for multiple classes their method is computationally expensive as it uses a joint loss function.

Original: Other methods inject noise to models in the ensemble
Revised: Other methods inject noise to models in the ensemble

Original: Liu et al.
Revised: Liu et al.

Original: (2018a)
Revised: (2018a)

Original: but their evaluation is only in the white box setting.
Revised: but their evaluation is only in the white box setting.

Original: Various measures for ensemble diversity have been previously proposed for deep networks Liu et al.
Revised: Various measures for ensemble diversity have been previously proposed for deep networks Liu et al.

Original: and evaluated in the white-box setting.
Revised: and evaluated in the white-box setting.

Original: We can apply all of the above diversity training methods to our ensemble of sign networks.
Revised: We can apply all of the above diversity training methods to our ensemble of sign networks.

Original: Our work, however, is not explicitly aimed at enhancing diversity.
Revised: Our work, however, is not explicitly aimed at enhancing diversity.

Original: As we show it is naturally diverse and we conjecture this is due to the non-convexity of sign activation and our randomized training method.
Revised: As we show it is naturally diverse and we conjecture this is due to the non-convexity of sign activation and our randomized training method.

Original: Even sigmoid activation networks have a non-convex search space but we can imagine that sign activation gives a greater degree of freedom.
Revised: Even sigmoid activation networks have a non-convex search space but we can imagine that sign activation gives a greater degree of freedom.

Original: This can easily be seen in the case of a linear classiﬁer with logistic or hinge loss vs. 01 loss Xue et al.
Revised: This can easily be seen in the case of a linear classiﬁer with logistic or hinge loss vs. 01 loss Xue et al.

Original: (2020b).
Revised: (2020b).

Original: Our model accuracy is not the same as convolutional networks understandably due to lack of convolutions in our networks.
Revised: Our model accuracy is not the same as convolutional networks understandably due to lack of convolutions in our networks.

Original: But they are close to sigmoid activated networks and random forests, and better than binary neural networks in most cases.
Revised: But they are close to sigmoid activated networks and random forests, and better than binary neural networks in most cases.

Original: It is possible to extend our training procedure to allow for convolutions and this may increase accuracy making our model comparable to convolutional networks and much more robust.
Revised: It is possible to extend our training procedure to allow for convolutions and this may increase accuracy making our model comparable to convolutional networks and much more robust.

Original: It is hard to make a general claim of robustness with only 100 images from CIFAR10.
Revised: It is hard to make a general claim of robustness with only 100 images from CIFAR10.

Original: We would need to show more images from CIFAR10 and other image benchmarks as well but our preliminary experiments  suggest higher distortion on other image data as well.For example on a single random test datapoint in CelebA our SCDCE ensemble has minimum L 2 distortion of 8.6 while the next best LeNet ensemble is 3.3.
Revised: We would need to show more images from CIFAR10 and other image benchmarks as well but our preliminary experiments on CelebA, GTSRB and ImageNet (shown in Table 4) suggest higher distortion on other image data as well.

Original: Due to computational limitations we are unable to show more image data here but instead we take another route to show generality of our results.
Revised: Due to computational limitations we are unable to show more image data here but instead we take another route to show generality of our results.

Original: We show that our model is robust even to text classiﬁcation black box attacks and on ECG data attacks.
Revised: We show that our model is robust even to text classiﬁcation black box attacks and on ECG data attacks.

Original: Both of these are outside the domain of images and our model’s robustness there suggests a greater generalization.
Revised: Both of these are outside the domain of images and our model’s robustness there suggests a greater generalization.

Original: Future work entails extending our training to sign activated convolutions and multi-class networks.
Revised: Future work entails extending our training to sign activated convolutions and multi-class networks.

Revised: 3. C ONCLUSION

Revised: We show that our ensemble of gradient-free sign activation networks are harder to attack than ensembles of several other networks and random forests on images, text, and medical data.