Original: D EFECTIVE C ONVOLUTIONAL N ETWORKS
Revised: D EFECTIVE C ONVOLUTIONAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Robustness of convolutional neural networks (CNNs) has gained in importance on account of adversarial examples, i.e., inputs added as well-designed perturbations that are imperceptible to humans but can cause the model to predict incorrectly.
Revised: Robustness of convolutional neural networks (CNNs) has gained in importance on account of adversarial examples, i.e., inputs added as well-designed perturbations that are imperceptible to humans but can cause the model to predict incorrectly.

Original: Recent research suggests that the noise in adversarial examples breaks the textural structure, which eventually leads to wrong predictions.
Revised: Recent research suggests that the noise in adversarial examples breaks the textural structure, which eventually leads to wrong predictions.

Original: To mitigate the threat of such adversarial attacks, we propose defective convolutional networks that make predictions rely less on textural information but more on shape information by properly integrating defective convolutional layers into standard CNNs.
Revised: To mitigate the threat of such adversarial attacks, we propose defective convolutional networks that make predictions rely less on textural information but more on shape information by properly integrating defective convolutional layers into standard CNNs.

Original: The defective convolutional layers contain defective neurons whose activations are set to be a constant function.
Revised: The defective convolutional layers contain defective neurons whose activations are set to be a constant function.

Original: As defective neurons contain no information and are far different from standard neurons in its spatial neighborhood, the textural features cannot be accurately extracted, and so the model has to seek other features for classiﬁcation, such as the shape.
Revised: As defective neurons contain no information and are far different from standard neurons in its spatial neighborhood, the textural features cannot be accurately extracted, and so the model has to seek other features for classiﬁcation, such as the shape.

Original: We show extensive evidence to justify our proposal and demonstrate that defective CNNs can defend against black-box attacks better than standard CNNs.
Revised: We show extensive evidence to justify our proposal and demonstrate that defective CNNs can defend against black-box attacks better than standard CNNs.

Original: In particular, they achieve state-of-the-art performance against transfer-based attacks without any adversarial training being applied.
Revised: In particular, they achieve state-of-the-art performance against transfer-based attacks without any adversarial training being applied.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep learning (LeCun et al., 1998; 2015), especially deep Convolutional Neural Network (CNN) (Krizhevsky et al., 2012), has led to state-of-the-art results spanning many machine learning ﬁelds (Girshick, 2015;
Revised: Deep learning (LeCun et al., 1998; 2015), especially deep Convolutional Neural Network (CNN) (Krizhevsky et al., 2012), has led to state-of-the-art results spanning many machine learning ﬁelds (Girshick, 2015;

Original: Chen et al., 2018; Luo et al., 2020).
Revised: Chen et al., 2018; Luo et al., 2020).

Original: Despite the great success in numerous applications, recent studies show that deep CNNs are vulnerable to some well-designed input samples named as Adversarial Examples (Szegedy et al., 2013; Biggio et al., 2013).
Revised: Despite the great success in numerous applications, recent studies show that deep CNNs are vulnerable to some well-designed input samples named as Adversarial Examples (Szegedy et al., 2013; Biggio et al., 2013).

Original: Take the task of image classiﬁcation as an example, for almost every commonly used well-performed CNN, attackers are able to construct a small perturbation on an input image, which is almost imperceptible to humans but can make the model give a wrong prediction.
Revised: Take the task of image classiﬁcation as an example, for almost every commonly used well-performed CNN, attackers are able to construct a small perturbation on an input image, which is almost imperceptible to humans but can make the model give a wrong prediction.

Original: The problem is serious as some well-designed adversarial examples can be transferred among different kinds of CNN architectures (Papernot et al., 2016b).
Revised: The problem is serious as some well-designed adversarial examples can be transferred among different kinds of CNN architectures (Papernot et al., 2016b).

Original: As a result, a machine learning system can be easily attacked even if the attacker does not have access to the model parameters, which seriously affect its use in practical applications.
Revised: As a result, a machine learning system can be easily attacked even if the attacker does not have access to the model parameters, which seriously affect its use in practical applications.

Original: There is a rapidly growing body of work on how to obtain a robust CNN, mainly based on adversarial training (Szegedy et al., 2013; Goodfellow et al., 2015;
Revised: There is a rapidly growing body of work on how to obtain a robust CNN, mainly based on adversarial training (Szegedy et al., 2013; Goodfellow et al., 2015;

Original: Madry et al., 2017; Buckman et al., 2018;
Revised: Madry et al., 2017; Buckman et al., 2018;

Original: Mao et al., 2019).
Revised: Mao et al., 2019).

Original: However, those methods need lots of extra computation to obtain adversarial examples at each time step and tend to overﬁt the attacking method used in training (Buckman et al., 2018).
Revised: However, those methods need lots of extra computation to obtain adversarial examples at each time step and tend to overﬁt the attacking method used in training (Buckman et al., 2018).

Original: In this paper, we tackle the problem in a perspective different from most existing methods.
Revised: In this paper, we tackle the problem in a perspective different from most existing methods.

Original: In particular, we explore the possibility of designing new CNN architectures which can be trained using standard optimization methods on standard benchmark datasets and can enjoy robustness by themselves, without appealing to other techniques.
Revised: In particular, we explore the possibility of designing new CNN architectures which can be trained using standard optimization methods on standard benchmark datasets and can enjoy robustness by themselves, without appealing to other techniques.

Original: Recent studies (Geirhos et al., 2017; 2018; Baker et al., 2018; Brendel & Bethge, 2019) show that the predictions of standard CNNs mainly depend on the texture of objects.
Revised: Recent studies (Geirhos et al., 2017; 2018; Baker et al., 2018; Brendel & Bethge, 2019) show that the predictions of standard CNNs mainly depend on the texture of objects.

Original: However, the textural information has a high degree of redundancy and may be easily injected with adversarial noise (Yang et al., 2019; Hosseini et al., 2019).
Revised: However, the textural information has a high degree of redundancy and may be easily injected with adversarial noise (Yang et al., 2019; Hosseini et al., 2019).

Original: Also,
Revised: Also,

Original: Cao et al.
Revised: Cao et al.

Original: Das et al.
Revised: Das et al.

Original: (2020) ﬁnds adversarial attack methods may perturb local patches to contain textural features of incorrect classes.
Revised: (2020) ﬁnds adversarial attack methods may perturb local patches to contain textural features of incorrect classes.

Original: All the literature suggests that the wrong prediction by CNNs for adversarial examples mainly comes from the change in the textural information.
Revised: All the literature suggests that the wrong prediction by CNNs for adversarial examples mainly comes from the change in the textural information.

Original: The small perturbation of adversarial examples will change the textures and eventually affect the features extracted by the CNNs.
Revised: The small perturbation of adversarial examples will change the textures and eventually affect the features extracted by the CNNs.

Original: Therefore, a natural way to avoid adversarial examples is to let the CNN make predictions relying less on textures but more on other information, such as the shape, which cannot be severely distorted by small perturbations.
Revised: Therefore, a natural way to avoid adversarial examples is to let the CNN make predictions relying less on textures but more on other information, such as the shape, which cannot be severely distorted by small perturbations.

Original: In practice, sometimes a camera might have mechanical failures which cause the output image to have many defective pixels (such pixels are always black in all images).
Revised: In practice, sometimes a camera might have mechanical failures which cause the output image to have many defective pixels (such pixels are always black in all images).

Original: Nonetheless, humans can still recognize objects in the image with defective pixels since we are able to classify the objects even in the absence of local textural information.
Revised: Nonetheless, humans can still recognize objects in the image with defective pixels since we are able to classify the objects even in the absence of local textural information.

Original: Motivated by this, we introduce the concept of defectiveness into the convolutional neural networks: we call a neuron a defective neuron if its output value is ﬁxed to zero no matter what input signal is received; similary, a convolutional layer is a defective convolutional layer if it contains defective neurons.
Revised: Motivated by this, we introduce the concept of defectiveness into the convolutional neural networks: we call a neuron a defective neuron if its output value is ﬁxed to zero no matter what input signal is received; similary, a convolutional layer is a defective convolutional layer if it contains defective neurons.

Original: Before training, we replace the standard convolutional layers with the defective version on a standard CNN and train the network in the standard way.
Revised: Before training, we replace the standard convolutional layers with the defective version on a standard CNN and train the network in the standard way.

Original: As defective neurons of the defective convolutional layer contain no information and are very different from their spatial neighbors, the textural information cannot be accurately extracted from the bottom defective layers to top layers.
Revised: As defective neurons of the defective convolutional layer contain no information and are very different from their spatial neighbors, the textural information cannot be accurately extracted from the bottom defective layers to top layers.

Original: Therefore, we destroy local textural information to a certain extent and prompt the neural network to rely more on other information for classiﬁcation.
Revised: Therefore, we destroy local textural information to a certain extent and prompt the neural network to rely more on other information for classiﬁcation.

Original: We call the architecture deployed with defective convolutional layers as defective convolutional network .
Revised: We call the architecture deployed with defective convolutional layers as defective convolutional network .

Original: We ﬁnd that applying the defective convolutional layers to the bottom 1 layers of the network and introducing various patterns for defective neurons arrangement across channels are critical.
Revised: We ﬁnd that applying the defective convolutional layers to the bottom 1 layers of the network and introducing various patterns for defective neurons arrangement across channels are critical.

Original: In summary, our main contributions are:
Revised: In summary, our main contributions are:

Original: • We propose Defective CNNs and four empirical evidences to justify that, compared to standard CNNs, the defective ones rely less on textures
Revised: • We propose Defective CNNs and four empirical evidences to justify that, compared to standard CNNs, the defective ones rely less on textures

Original: and more
Revised: and more

Original: on shapes of the inputs for making predictions.
Revised: on shapes of the inputs for making predictions.

Original: • Experiments show that Defective CNNs has superior defense performance than standard CNNs against transfer-based attacks, decision-based attacks, and additive Gaussian noise.
Revised: • Experiments show that Defective CNNs has superior defense performance than standard CNNs against transfer-based attacks, decision-based attacks, and additive Gaussian noise.

Original: • Using the standard training method, Defective CNN achieves state-of-the-art results against two transfer-based black-box attacks while maintaining high accuracy on clean test data.
Revised: • Using the standard training method, Defective CNN achieves state-of-the-art results against two transfer-based black-box attacks while maintaining high accuracy on clean test data.

Original: • Through proper implementation, Defective CNNs can save a lot of computation and storage costs; thus may lead to a practical solution in the real world.
Revised: • Through proper implementation, Defective CNNs can save a lot of computation and storage costs; thus may lead to a practical solution in the real world.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Various methods have been proposed to defend against adversarial examples.
Revised: Various methods have been proposed to defend against adversarial examples.

Original: One line of research is to derive a meaningful optimization objective and optimize the model by adversarial training (Szegedy et al., 2013; Goodfellow et al., 2015; Huang et al., 2015; Madry et al., 2017; Buckman et al., 2018; Mao et al., 2019).
Revised: One line of research is to derive a meaningful optimization objective and optimize the model by adversarial training (Szegedy et al., 2013; Goodfellow et al., 2015; Huang et al., 2015; Madry et al., 2017; Buckman et al., 2018; Mao et al., 2019).

Original: The high-level idea of these works is that if we can predict the potential attack to the model during optimization, then we can give the attacked sample a correct signal and use it during training.
Revised: The high-level idea of these works is that if we can predict the potential attack to the model during optimization, then we can give the attacked sample a correct signal and use it during training.

Original: Another line of research is to take an adjustment to the input image before letting it go through the deep neural network (Liao et al., 2017; Song et al., 2017; Samangouei et al., 2018; Sun et al., 2018;
Revised: Another line of research is to take an adjustment to the input image before letting it go through the deep neural network (Liao et al., 2017; Song et al., 2017; Samangouei et al., 2018; Sun et al., 2018;

Original: Xie et al., 2019; Yuan & He, 2020).
Revised: Xie et al., 2019; Yuan & He, 2020).

Original: The basic intuition behind this is that if we can clean the adversarial attack to a certain extent, then such attacks can be defended.
Revised: The basic intuition behind this is that if we can clean the adversarial attack to a certain extent, then such attacks can be defended.

Original: Although these methods achieve some success, a major difﬁculty is that it needs a large extra cost to collect adversarial examples and hard to apply on large-scale datasets.
Revised: Although these methods achieve some success, a major difﬁculty is that it needs a large extra cost to collect adversarial examples and hard to apply on large-scale datasets.

Original: Several studies (Geirhos et al., 2017; 2018;
Revised: Several studies (Geirhos et al., 2017; 2018;

Original: Baker et al., 2018; Brendel & Bethge, 2019) show that the prediction of CNNs is mainly from the texture of objects but not the shape.
Revised: Baker et al., 2018; Brendel & Bethge, 2019) show that the prediction of CNNs is mainly from the texture of objects but not the shape.

Original: Also, Cao et al.
Revised: Also, Cao et al.

Original: (2020); Das et al.
Revised: (2020); Das et al.

Original: (2020) found that adversarial examples usually perturb a patch of the original image to contain the textural feature of incorrect classes.
Revised: (2020) found that adversarial examples usually perturb a patch of the original image to contain the textural feature of incorrect classes.

Original: For example, the adversarial example of the panda image is misclassiﬁed as a monkey because a patch of the panda skin is perturbed adversarially so that it alone looks like the face of a monkey (see Figure 11 in (Cao et al., 2020)).
Revised: For example, the adversarial example of the panda image is misclassiﬁed as a monkey because a patch of the panda skin is perturbed adversarially so that it alone looks like the face of a monkey (see Figure 11 in (Cao et al., 2020)).

Original: All previous works above suggest that the CNN learns textural information more than shape and the adversarial attack might come from textural-level perturbations.
Revised: All previous works above suggest that the CNN learns textural information more than shape and the adversarial attack might come from textural-level perturbations.

Original: This is also correlated with robust features (Tsipras et al., 2018;
Revised: This is also correlated with robust features (Tsipras et al., 2018;

Original: Ilyas et al., 2019; Hosseini et al., 2019;
Revised: Ilyas et al., 2019; Hosseini et al., 2019;

Original: Yang et al., 2019) which has attracted more interest recently.
Revised: Yang et al., 2019) which has attracted more interest recently.

Original: Pixels which encode textural information contain high redundancy and may be easily deteriorated to the distribution of incorrect classes.
Revised: Pixels which encode textural information contain high redundancy and may be easily deteriorated to the distribution of incorrect classes.

Original: However, shape information is more compact and thus may serve as a more robust feature for predicting.
Revised: However, shape information is more compact and thus may serve as a more robust feature for predicting.

Original:  D EFECTIVE C ONVOLUTIONAL N EURAL N ETWORK 3. D ESIGN OF D EFECTIVE C ONVOLUTIONAL L AYERS
Revised:  D EFECTIVE C ONVOLUTIONAL N EURAL N ETWORK 3. D ESIGN OF D EFECTIVE C ONVOLUTIONAL L AYERS

Original: In this subsection, we introduce our proposed defective convolutional neural networks and discuss the differences between the proposed method and related topics.
Revised: In this subsection, we introduce our proposed defective convolutional neural networks and discuss the differences between the proposed method and related topics.

Original: First, we brieﬂy introduce the notations.
Revised: First, we brieﬂy introduce the notations.

Original: For one convolutional layer, denote x as the input and z as the output of neurons in the layer.
Revised: For one convolutional layer, denote x as the input and z as the output of neurons in the layer.

Original: Note that x may be the input image or the output of the last convolutional layer.
Revised: Note that x may be the input image or the output of the last convolutional layer.

Original: The input x is usually a M × N × K tensor in which M/N are the height/width of a feature map, and K is the number of feature maps, or equivalently, channels.
Revised: The input x is usually a M × N × K tensor in which M/N are the height/width of a feature map, and K is the number of feature maps, or equivalently, channels.

Original: Denote w and b as the parameters (e.g., the weights and biases) of the convolutional kernel.
Revised: Denote w and b as the parameters (e.g., the weights and biases) of the convolutional kernel.

Original: Then a standard convolutional layer can be mathematically deﬁned as below.
Revised: Then a standard convolutional layer can be mathematically deﬁned as below.

Original: Standard convolutional layer:
Revised: Standard convolutional layer:

Original: [Equation]
Revised: [Equation]

Original: where f ( · ) is a non-linear activation function such as ReLU 2 and ⊗ conv is the convolutional operation.
Revised: where f ( · ) is a non-linear activation function such as ReLU 2 and ∗ is the convolutional operation.

Original: The convolutional ﬁlter receives signals in a patch and extracts local textural information from the patch.
Revised: The convolutional ﬁlter receives signals in a patch and extracts local textural information from the patch.

Original: As mentioned in the introduction, recent works suggest that the prediction of standard CNNs strongly depends on such textural information, and noises imposed on the texture may lead to wrong predictions.
Revised: As mentioned in the introduction, recent works suggest that the prediction of standard CNNs strongly depends on such textural information, and noises imposed on the texture may lead to wrong predictions.

Original: Therefore, we hope to learn a feature extractor which does not solely rely on textural features and also considers other information.
Revised: Therefore, we hope to learn a feature extractor which does not solely rely on textural features and also considers other information.

Original: To achieve this goal, we introduce the defective convolutional layer in which some neurons are purposely designed to be corrupted.
Revised: To achieve this goal, we introduce the defective convolutional layer in which some neurons are purposely designed to be corrupted.

Original: Deﬁne M defect to be a binary matrix of size M × N × K .
Revised: Deﬁne M defect to be a binary matrix of size M × N × K .

Original: Our defective convolutional layer is deﬁned as follows.
Revised: Our defective convolutional layer is deﬁned as follows.

Original: Defective convolutional layer:
Revised: Defective convolutional layer:

Original: [Equation]
Revised: [Equation]

Original: where ∗ denotes element-wise product.
Revised: where ◦ denotes element-wise product.

Original: M defect is a ﬁxed matrix and is not learnable during training and testing.
Revised: M defect is a ﬁxed matrix and is not learnable during training and testing.

Original: We can see that M defect plays a role of “masking” out values of some neurons in the layer.
Revised: We can see that M defect plays a role of “masking” out values of some neurons in the layer.

Original: This disturbs the distribution of local textural information and decouples the correlation among neurons.
Revised: This disturbs the distribution of local textural information and decouples the correlation among neurons.

Original: With the masked output z as input, the feature extractor of the next convolutional layer cannot accurately capture the local textural feature from x .
Revised: With the masked output z as input, the feature extractor of the next convolutional layer cannot accurately capture the local textural feature from x .

Original: As a consequence, the textural information is hard to pass through the defective CNN from bottom to top.
Revised: As a consequence, the textural information is hard to pass through the defective CNN from bottom to top.

Original: To produce accurate predictions, the deep neural network has to ﬁnd relevant signals other than the texture, e.g., the shape.
Revised: To produce accurate predictions, the deep neural network has to ﬁnd relevant signals other than the texture, e.g., the shape.

Original: Those corrupted neurons have no severe impact on the extraction of shape information since neighbors of those neurons in the same ﬁlter are still capable of passing the shape information to the next layer.
Revised: Those corrupted neurons have no severe impact on the extraction of shape information since neighbors of those neurons in the same ﬁlter are still capable of passing the shape information to the next layer.

Original: In this paper, we ﬁnd that simply setting M defect by random initialization is already helpful for learning a robust CNN.
Revised: In this paper, we ﬁnd that simply setting M defect by random initialization is already helpful for learning a robust CNN.

Original: Before training, we sample each entry in M defect using Bernoulli distribution with keep probability p and then ﬁx M defect during training and testing.
Revised: Before training, we sample each entry in M defect using Bernoulli distribution with keep probability p and then ﬁx M defect during training and testing.

Original: More discussions and ablation studies are provided in Section 4.
Revised: More discussions and ablation studies are provided in Section 4.

Original: As can be seen from Equation (3)-(5), the implementation of our defective convolutional layer is similar to the dropout operation (Srivastava et al., 2014).
Revised: As can be seen from Equation (3)-(5), the implementation of our defective convolutional layer is similar to the dropout operation (Srivastava et al., 2014).

Original: To demonstrate the relationship and differences, we mathematically deﬁne the dropout as below.
Revised: To demonstrate the relationship and differences, we mathematically deﬁne the dropout as below.

Original: Standard convolutional layer + dropout:
Revised: Standard convolutional layer + dropout:

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: The shape of M dp is the same as M defect , and the value of each entry in M dp is sampled in each batch using some sampling strategies at each step during training.
Revised: The shape of M dropout is the same as M defect , and the value of each entry in M dropout is sampled in each batch using some sampling strategies at each step during training.

Original: Generally, entries in M dp are independent and identically sampled in an online fashion using Bernoulli distribution with keep probability p .
Revised: Generally, entries in M dropout are independent and identically sampled in an online fashion using Bernoulli distribution with keep probability p .

Original: There are several signiﬁcant differences between dropout and defective convolutional layer.
Revised: There are several signiﬁcant differences between dropout and defective convolutional layer.

Original: First, the binary matrix M dp is sampled online during training and is removed during testing, while the binary matrix M defect in defective convolutional layers is predeﬁned and keeps ﬁxed in both training and testing.
Revised: First, the binary matrix M dropout is sampled online during training and is removed during testing, while the binary matrix M defect in defective convolutional layers is predeﬁned and keeps ﬁxed in both training and testing.

Original: The predeﬁned way can help Defective CNNs save a lot of computation and storage costs.
Revised: The predeﬁned way can help Defective CNNs save a lot of computation and storage costs.

Original: Second, the motivations behind the two methods are quite different and may lead to differences in the places to applying methods, the values of the keep probability p , and the shape of the masked unit.
Revised: Second, the motivations behind the two methods are quite different and may lead to differences in the places to applying methods, the values of the keep probability p , and the shape of the masked unit.

Original: Dropout tries to reduce overﬁtting by preventing co-adaptations on training data.
Revised: Dropout tries to reduce overﬁtting by preventing co-adaptations on training data.

Original: When comes to CNNs, those methods are applied to top layers, p is set to be large (e.g., 0 . 9 ), and the masked units are chosen to be a whole channel in Tompson et al.
Revised: When comes to CNNs, those methods are applied to top layers, p is set to be large (e.g., 0 . 9 ), and the masked units are chosen to be a whole channel in Tompson et al.

Original: (2015) and a connected block in Ghiasi et al.
Revised: (2015) and a connected block in Ghiasi et al.

Original: However, our method tries to prevent the model extract textural information of inputs for making predictions.
Revised: However, our method tries to prevent the model extract textural information of inputs for making predictions.

Original: We would apply the method to bottom layers, use a small p (e.g. 0 . 1 ), and the masked unit is a single neuron.
Revised: We would apply the method to bottom layers, use a small p (e.g. 0 . 1 ), and the masked unit is a single neuron.

Original: Also, in our experiments, we will show that the proposed method can improve the robustness of CNNs against transfer-based attacks and decision-based attacks, while the dropout methods cannot.
Revised: Also, in our experiments, we will show that the proposed method can improve the robustness of CNNs against transfer-based attacks and decisionbased attacks, while the dropout methods cannot.

Original: 3. D EFECTIVE CNN S R ELY L ESS ON T EXTURE BUT M ORE ON S HAPE FOR P REDICTING
Revised: 3. D EFECTIVE CNN S R ELY L ESS ON T EXTURE BUT M ORE ON S HAPE FOR P REDICTING

Original: In this subsection, we provide extensive analysis to show Defective CNNs that, compared to the standard CNNs, rely less on textures and more on shapes of the inputs for making predictions.
Revised: In this subsection, we provide extensive analysis to show Defective CNNs that, compared to the standard CNNs, rely less on textures and more on shapes of the inputs for making predictions.

Original: [Figure]
Revised: [Figure]

Original: being divided into 2 × 2 , 4 × 4 and 8 ×patches respectively.
Revised: × 4 and 8 ×patches respectively.

Original: First, we design a particular image manipulation in which the local texture of the object in an image is preserved while the shape is destroyed.
Revised: First, we design a particular image manipulation in which the local texture of the object in an image is preserved while the shape is destroyed.

Original: Particularly, we divide an image into k × k patches and randomly relocate those patches to form a new image.
Revised: Particularly, we divide an image into k × k patches and randomly relocate those patches to form a new image.

Original: An example is shown in Figure 1.
Revised: An example is shown in Figure 1.

Original: A model that more focuses on the shape cues should achieve lower performance on such images while it is trained on the normal dataset.
Revised: A model that more focuses on the shape cues should achieve lower performance on such images while it is trained on the normal dataset.

Original: We manipulate a set of images and test whether a defective CNN and a standard CNN can make correct predictions.
Revised: We manipulate a set of images and test whether a defective CNN and a standard CNN can make correct predictions.

Original: The experimental details are described as follows.
Revised: The experimental details are described as follows.

Original: We ﬁrst construct a defective CNN by applying defective convolutional layers to the bottom layers of a standard ResNet-18, and train the defective CNN along with a standard ResNet-18 on the ImageNet dataset.
Revised: We ﬁrst construct a defective CNN by applying defective convolutional layers to the bottom layers of a standard ResNet-18, and train the defective CNN along with a standard ResNet-18 on the ImageNet dataset.

Original: Then, we sample images from the validation set which are predicted correctly by both two
Revised: Then, we sample images from the validation set which are predicted correctly by both of the

Original: CNNs.
Revised: CNNs.

Original: We make manipulations to the sampled images by setting k ∈ { 2 , 4 , 8 } , feed these images to the networks and check their classiﬁcation accuracy.
Revised: We make manipulations to the sampled images by setting k ∈ { 2 , 4 , 8 } , feed these images to the networks and check their classiﬁcation accuracy.

Original: The results in Table 1, 13 show that when the shape information is destroyed but the local textural information is preserved, Defective CNNs perform consistently worse than standard CNNs, thus verifying out intuition.
Revised: The results in Table 1, 13 show that when the shape information is destroyed but the local textural information is preserved, Defective CNNs perform consistently worse than standard CNNs, thus verifying out intuition.

Original: [Table]
Revised: [Table]

Original: Second, we test on the Stylized-ImageNet (Geirhos et al., 2018), a stylized version of ImageNet, where the local textures of images are changed, while global object shapes remain (See examples in Figure 2).
Revised: Second, we test on the Stylized-ImageNet (Geirhos et al., 2018), a stylized version of ImageNet, where the local textures of images are changed, while global object shapes remain (See examples in Figure 2).

Original: A model that more focuses on the shape cues should achieve higher performance on the Stylized-ImageNet while it is trained on the ImageNet.
Revised: A model that more focuses on the shape cues should achieve higher performance on the Stylized-ImageNet while it is trained on the ImageNet.

Original: We test on the same model used in the randomly shufﬂed experiments by feeding the images from the validation set of Stylized-ImageNet whose corresponding images in ImageNet can be correctly classiﬁed by both two tested models.
Revised: We test on the same model used in the randomly shufﬂed experiments by feeding the images from the validation set of Stylized-ImageNet whose corresponding images in ImageNet can be correctly classiﬁed by both two tested models.

Original: The result in Table 1, 13 show that Defective CNNs achieves consistently higher transferring accuracy than standard CNNs, thus verifying our argument.
Revised: The result in Table 1, 13 show that Defective CNNs achieves consistently higher transferring accuracy than standard CNNs, thus verifying our argument.

Original: [Figure]
Revised: [Figure]

Original: From another perspective, if a model makes predictions relying more on shape information, the manipulation of the shape of objects will play a larger role in generating adversarial examples.
Revised: From another perspective, if a model makes predictions relying more on shape information, the manipulation of the shape of objects will play a larger role in generating adversarial examples.

Original: To verify this, we train defective and standard CNNs on CIFAR-10 and Tiny-ImageNet, and then attack on the validation set.
Revised: To verify this, we train defective and standard CNNs on CIFAR-10 and Tiny-ImageNet, and then attack on the validation set.

Original: Figure 3 shows some examples.
Revised: Figure 3 shows some examples.

Original: We can see that adversarial examples against the defective CNNs change the shape of the objects and may even fool humans as well.
Revised: We can see that adversarial examples against the defective CNNs change the shape of the objects and may even fool humans as well.

Original: Compare with the adversarial examples of Figure 9 in Qin et al.
Revised: Compare with the adversarial examples of Figure 9 in Qin et al.

Original: (2020), our adversarial examples exhibit more salient characteristics of the adversarial classes.
Revised: (2020), our adversarial examples exhibit more salient characteristics of the adversarial classes.

Original: Also, we conduct a user study in Appendix B to show that the adversarial examples generated by Defective CNNs, compared to the standard ones, are more perceptually like the adversarial classes.
Revised: Also, we conduct a user study in Appendix B to show that the adversarial examples generated by Defective CNNs, compared to the standard ones, are more perceptually like the adversarial classes.

Original: The phenomenon not only supports our intuition, but also is consistent with the ﬁndings in Tsipras et al.
Revised: The phenomenon not only supports our intuition, but also is consistent with the ﬁndings in Tsipras et al.

Original: ; Qin et al.
Revised: ; Qin et al.

Original: (2020) that the representations learned by robust models tend to align better with human perception.
Revised: (2020) that the representations learned by robust models tend to align better with human perception.

Original: Furthermore, perturbations generated by standard CNNs and additive Gaussian noises usually hard to affect the shape information (Szegedy et al., 2013; Ford et al., 2019).
Revised: Furthermore, perturbations generated by standard CNNs and additive Gaussian noises usually would not affect the shape information (Szegedy et al., 2013; Ford et al., 2019).

Original: A model is supposed to recognize those adversarial examples better if it relies much on shape information for predictions.
Revised: A model is supposed to recognize those adversarial examples better if it relies much on shape information for predictions.

Original: In Section 4, we show that defective CNNs achieve higher defense performance than standard CNN against the two types of attack.
Revised: In Section 4, we show that defective CNNs achieve higher defense performance than standard CNN against the two types of attack.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In real-world tasks, attackers usually cannot access the parameters of the target models and thus need to transfer adversarial examples generated by their models.
Revised: In real-world tasks, attackers usually cannot access the parameters of the target models and thus need to transfer adversarial examples generated by their models.

Original: This setting of attack is referred to as transfer-based attacks (Liu et al., 2016; Kurakin et al., 2016).
Revised: This setting of attack is referred to as transfer-based attacks (Liu et al., 2016; Kurakin et al., 2016).

Original: Sometimes, attackers can get the ﬁnal model decision and raise the more powerful decision-based attacks (Brendel et al., 2017).
Revised: Sometimes, attackers can get the ﬁnal model decision and raise the more powerful decision-based attacks (Brendel et al., 2017).

Original: Both the two types of black-box attack are available in most real-world scenarios and should be considered.
Revised: Both the two types of black-box attack are available in most real-world scenarios and should be considered.

Original: Recently, Ford et al.
Revised: Recently, Ford et al.

Original: (2019) bridge the adversarial robustness and corruption robustness (Hendrycks & Dietterich, 2018), and points out that a successful adversarial defend method should also effectively defend against additive Gaussian noise.
Revised: (2019) bridge the adversarial robustness and corruption robustness (Hendrycks & Dietterich, 2018), and points out that a successful adversarial defend method should also effectively defend against additive Gaussian noise.

Original: Therefore, to meet the requirements for practical systems, we examine the performance of models against transfer-based attacks, decision-based attacks, and additive Gaussian noise.
Revised: Therefore, to meet the requirements for practical systems, we examine the performance of models against transfer-based attacks, decision-based attacks, and additive Gaussian noise.

Original: In the following sections, we evaluate the robustness of defective CNNs with different architectures and compare with state-of-the-art defense methods against transfer-based attacks, and then make ablation studies on possible design choices of defective CNN.
Revised: In the following sections, we evaluate the robustness of defective CNNs with different architectures and compare with state-of-the-art defense methods against transfer-based attacks, and then make ablation studies on possible design choices of defective CNN.

Original: Due to space limitation, we list the experiments of decision-based attacks, additive Gaussian noise, gray-box attacks, white-box attacks, and more results of transfer-based attacks in Appendix A.
Revised: Due to space limitation, we list the experiments of decision-based attacks, additive Gaussian noise, gray-box attacks, white-box attacks, and more results of transfer-based attacks in Appendix A.

Original: Note that, in this paper, all the successful defense rates except the rates listed in Table 2, 3 are calculated on the samples whose corresponding original images can be classiﬁed correctly by the tested model.
Revised: Note that, in this paper, all the successful defense rates except the rates listed in Table 2, 3 are calculated on the samples whose corresponding original images can be classiﬁed correctly by the tested model.

Original: This can erase the inﬂuence of test accuracy that different models have different test accuracy on clean data, and thus help evaluate the robustness of models.
Revised: This can erase the inﬂuence of test accuracy that different models have different test accuracy on clean data, and thus help evaluate the robustness of models.

Original: 4. T RANSFER - BASED A TTACK 4.1. E XPERIMENTAL S ETTINGS
Revised: 4. T RANSFER - BASED A TTACK 4.1. E XPERIMENTAL S ETTINGS

Original: We evaluate the defense performance of Defective CNNs against transfer-based attacks and compare with state-of-the-art defense methods on CIFAR-10 and Tiny-ImageNet.
Revised: We evaluate the defense performance of Defective CNNs against transfer-based attacks and compare with state-of-the-art defense methods on CIFAR-10 and Tiny-ImageNet.

Original: For CIFAR-10, we follow the setting used in Madry et al.
Revised: For CIFAR-10, we follow the setting used in Madry et al.

Original: (2017), and use a standard ResNet-18 to generate adversarial examples by FGSM (Goodfellow et al., 2015) and PGD (Kurakin et al., 2016).
Revised: (2017), and use a standard ResNet-18 to generate adversarial examples by FGSM (Goodfellow et al., 2015) and PGD (Kurakin et al., 2016).

Original: The two attack methods both have perturbation scale (cid:96) ∞
Revised: The two attack methods both have perturbation scale (cid:96) ∞

Original: = 8 / 255 and PGD runs for 7 gradient descent steps with step size 2 / 255 .
Revised: = 8 / 255 and PGD runs for 7 gradient descent steps with step size 2 / 255 .

Original: For Tiny-ImageNet, we follow the setting used in Mao et al.
Revised: For Tiny-ImageNet, we follow the setting used in Mao et al.

Original: and use a standard ResNet-50 to generate adversarial examples by PGD with (cid:96) ∞
Revised: and use a standard ResNet-50 to generate adversarial examples by PGD with (cid:96) ∞

Original: = 8 / 255 , steps 20 , and step size 2 / 255 .
Revised: = 8 / 255 , steps 20 , and step size 2 / 255 .

Original: We would compare with two types of defense methods including the variants of adversarial training (Madry et al., 2017; Kannan et al., 2018; Mao et al., 2019) and approaches  try to erase the adversarial noise of inputs (Wang & Yu, 2019; Addepalli et al., 2020; Yuan & He, 2020).
Revised: We would compare with two types of defense methods including the variants of adversarial training (Madry et al., 2017; Kannan et al., 2018; Mao et al., 2019) and approaches that try to erase the adversarial noise of inputs (Wang & Yu, 2019; Addepalli et al., 2020; Yuan & He, 2020).

Original: To validate the difference between the proposed method and dropout methods, we also compare with two CNN variants SpatialDropout (Tompson et al., 2015) and DropBlock (Ghiasi et al., 2018).
Revised: To validate the difference between the proposed method and dropout methods, we also compare with two CNN variants SpatialDropout (Tompson et al., 2015) and DropBlock (Ghiasi et al., 2018).

Original: For both methods, we follow the instruction from Ghiasi et al.
Revised: For both methods, we follow the instruction from Ghiasi et al.

Original: (2018) to apply dropout to the { 3 rd , 4 th } block with keep probability p = 0 .
Revised: (2018) to apply dropout to the { 3 rd , 4 th } block with keep probability p = 0 .

Original: The block of DropBlock is set to be a 3 × 3 square.
Revised: The block of DropBlock is set to be a 3 × 3 square.

Original: For our method, we use the corresponding network structure but applying defective convolutional layers to the bottom layers (see illustrations in Appendix C).
Revised: For our method, we use the corresponding network structure but applying defective convolutional layers to the bottom layers (see illustrations in Appendix C).

Original: We use keep probability p = 0 . 1 and train the model with the standard optimization method.
Revised: We use keep probability p = 0 . 1 and train the model with the standard optimization method.

Original: Training details and curves can be found in Appendix D.
Revised: Training details and curves can be found in Appendix D.

Original: Second, we test our proposed method in different architectures on the CIFAR-10 dataset.
Revised: Second, we test our proposed method in different architectures on the CIFAR-10 dataset.

Original: We apply defective convolutional layers, in a way which is similar to the experiment above, to ﬁve popular network architectures:
Revised: We apply defective convolutional layers, in a way which is similar to the experiment above, to ﬁve popular network architectures:

Original: ResNet-18 (He et al., 2016), ResNet-50, DenseNet-121 (Huang et al., 2017),
Revised: ResNet-18 (He et al., 2016), ResNet-50, DenseNet-121 (Huang et al., 2017),

Original: SENet-18 (Hu et al., 2017b) and VGG-19 (Simonyan & Zisserman, 2014).
Revised: SENet-18 (Hu et al., 2017b) and VGG-19 (Simonyan & Zisserman, 2014).

Original: For each architecture, we replace the standard convolutional layer with the defective version on the bottom layers (see illustrations in Appendix C).
Revised: For each architecture, we replace the standard convolutional layer with the defective version on the bottom layers (see illustrations in Appendix C).

Original: We then test the black-box defense performance against transfer-based attacks on 5000 samples from the validation set.
Revised: We then test the black-box defense performance against transfer-based attacks on 5000 samples from the validation set.

Original: Adversarial examples are generated by PGD, which runs for 20 steps with step size 1 and the (cid:96) ∞ perturbation scale is set to 16 / 255 .
Revised: Adversarial examples are generated by PGD, which runs for 20 steps with step size 1 and the (cid:96) ∞ perturbation scale is set to 16 / 255 .

Original: Results on MNIST can be found in Appendix A.
Revised: Results on MNIST can be found in Appendix A.

Original: 4.1. E XPERIMENTAL R ESULTS
Revised: 4.1. E XPERIMENTAL R ESULTS

Original: Table 2, 3 show the results on CIFAR-10 and Tiny-ImageNet, respectively.
Revised: Table 2, 3 show the results on CIFAR-10 and Tiny-ImageNet, respectively.

Original: We can see the proposed method outperforms all the adversarial training variants which need extra training costs and most of the cleaning inputs methods.
Revised: We can see the proposed method outperforms all the adversarial training variants which need extra training costs and most of the cleaning inputs methods.

Original: Although Yuan & He (2020) is competitive with the proposed method in CIFAR-10, it would run loops for erasing noise and  thus  increase timestamps.
Revised: Although Yuan & He (2020) is competitive with the proposed method in CIFAR-10, it need to collect adversarial examples and run inner loops, thus largely increase timestamps.

Original: Also, we can conclude that spatial dropout and drop block do not improve the robustness of standard CNNs.
Revised: Also, we can conclude that spatial dropout and drop block do not improve the robustness of standard CNNs.

Original: The results show the strengths of our proposed method on both robustness and generalization, even though our model is only trained on clean data .
Revised: The results show the strengths of our proposed method on both robustness and generalization, even though our model is only trained on clean data .

Original: Also, it is interesting that the CNNs can maintain such clean accuracy even 90% neurons in bottom layers are dropped.
Revised: Also, it is interesting that the CNNs can maintain such clean accuracy even 90% neurons in bottom layers are dropped

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Second
Revised: on TinyImageNet.

Original: [Table] we list the black-box defense results of applying defective convolutional layers to various architectures in Table 4.
Revised: Second, we list the black-box defense results of applying defective convolutional layers to various architectures in Table 4.

Original: The results show that defective convolutional layers consistently improve the robustness of various network architectures against transfer-based attacks
Revised: The results show that defective convolutional layers consistently improve the robustness of various network architectures against transfer-based attacks.

Revised: We can also see that the trend of robustness increases as the keep probability becomes smaller

Original: [Table]
Revised: [Table]

Original: 4. A BLATION S TUDIES
Revised: 4. A BLATION S TUDIES

Original: There are several design choices of the defective CNN, which include the appropriate positions to apply defective convolutional layers, the chosen of the keep probabilities, the beneﬁt of breaking symmetry, as well as the diversity introduced by randomness.
Revised: There are several design choices of the defective CNN, which include the appropriate positions to apply defective convolutional layers, the choice of the keep probabilities, the beneﬁt of breaking symmetry, as well as the diversity introduced by randomness.

Original: In this subsection, we conduct a series of comparative experiments and use black-box defense performance against transfer-based attacks as the evaluation criterion.
Revised: In this subsection, we conduct a series of comparative experiments and use black-box defense performance against transfer-based attacks as the evaluation criterion.

Original: In our experiments, we found that the performance is not sensitive to the choices on the source model to attack and the target model to defense.
Revised: In our experiments, we found that the performance is not sensitive to the choices on the source model to attack and the target model to defense.

Original: Without loss of generality, we only list the performances using DenseNet-121 as the source model and ResNet-18 as the target model on the CIFAR-10 dataset and leave more experimental results in Appendix A.10.
Revised: Here, we only list the performances using DenseNet-121 as the source model and ResNet-18 as the target model on the CIFAR-10 dataset and leave more experimental results in Appendix A.10.

Original: The results are listed in Table 5.
Revised: The results are listed in Table 5.

Original: Defective Layers on Bottom layers v.s Top Layers, Keep Probabilities.
Revised: Defective Layers on Bottom layers vs. Top Layers, Keep Probabilities.

Original: We apply defective layers with different keep probabilities to the bottom layers and the top layers of the standard CNNs (see illustrations in Appendix C).
Revised: We apply defective layers with different keep probabilities to the bottom layers and the top layers of the standard CNNs (see illustrations in Appendix C).

Original: Comparing the results of the models with the same keep probability but different parts being masked, we ﬁnd that applying defective layers to bottom layers enjoys signiﬁcantly higher success defense rates. Moreover, only applyingdefective layers tobottom layers can achieve better performance than applying defective layers on both thebottom and top layers.
Revised: Comparing the results of the models with the same keep probability but different parts being masked, we ﬁnd that applying defective layers to bottom layers enjoys signiﬁcantly higher success defense rates, while applying to the top layers cannot.

Original: The bottom layers mainly contribute to detect the edges and shape, while the receptive ﬁelds of
Revised: This corroborates the phenomena shown in literature (Zeiler & Fergus, 2014; Mordvintsev et al., 2015), where bottom

Original: [Table]
Revised: [Table]

Original:  neurons in top layers are too large to respond to the location sensitive information.This corroborates the phenomena shown in Zeiler & Fergus (2014); Mordvintsev et al. (2015) where bottom layers respond to edges and top layers respond to entire objects.
Revised: layers mainly contribute to detect the edges and shape, while the receptive ﬁelds of neurons in top layers are too large to respond to the location sensitive information.

Original: Also, we ﬁnd that the defense accuracy monotonically increases as the test accuracy decreases along with the keep probability (See the trend map in Appendix A.9).
Revised: Also, we ﬁnd that the defense accuracy monotonically increases as the test accuracy decreases along with the keep probability (See the trend map in Appendix A.9).

Original: The appropriate value for the keep probability mainly depends on the relative importance of generalization and robustness.
Revised: The appropriate value for the keep probability mainly depends on the relative importance of generalization and robustness.

Original: Another practical way is to ensemble Defective CNNs with different keep probabilities.
Revised: Another practical way is to ensemble Defective CNNs with different keep probabilities.

Original: Defective Neuron v.s. Defective Channel.
Revised: Defective Neuron vs. Defective Channel.

Original: As our method independently selects defective neurons on different channels in a layer, we break the symmetry of the original CNN structure.
Revised: As our method independently selects defective neurons on different channels in a layer, we break the symmetry of the original CNN structure.

Original: To see whether this asymmetric structure would help, we try to directly mask whole channels instead of neurons using the same keep probability as the defective layer and train it to see the performance.
Revised: To see whether this asymmetric structure would help, we try to directly mask whole channels instead of neurons using the same keep probability as the defective layer and train it to see the performance.

Original: This defective channel method does not hurt the symmetry while also leading to the same decrease in the number of convolutional operations.
Revised: This defective channel method does not hurt the symmetry while also leading to the same decrease in the number of convolutional operations.

Original: Table 5 shows that although our defective CNN suffers a small drop in test accuracy due to the low keep probability, we have a great gain in the robustness, compared with the defective-channel CNN.
Revised: Table 5 shows that although our defective CNN suffers a small drop in test accuracy due to the low keep probability, we have a great gain in the robustness, compared with the defective-channel CNN.

Original: Defective Masks are Shared Among Channels or Not.
Revised: Defective Masks are Shared Among Channels or Not.

Original: The randomness in generating masks in different channels and layers allows each convolutional ﬁlter to focus on different input patterns.
Revised: The randomness in generating masks in different channels and layers allows each convolutional ﬁlter to focus on different input patterns.

Original: Also, it naturally involves various topological structures for local feature extraction instead of the expensive learning way (Dai et al., 2017; Chang et al., 2018; Zhu et al., 2019).
Revised: Also, it naturally involves various topological structures for local feature extraction instead of the expensive learning way (Dai et al., 2017; Chang et al., 2018; Zhu et al., 2019).

Original: We show the essentiality of generating various masks per layer via experiments that compare to a method that only randomly generates one mask per layer and uses it in every channel.
Revised: We show the essentiality of generating various masks per layer via experiments that compare to a method that only randomly generates one mask per layer and uses it in every channel.

Original: Table 5 shows that applying the same mask to each channel will decrease the test accuracy.
Revised: Table 5 shows that applying the same mask to each channel will decrease the test accuracy.

Original: This may result from the limitation of expressivity due to the monotone masks at every channel of the defective layer.
Revised: This may result from the limitation of expressivity due to the monotone masks at every channel of the defective layer.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we introduce and experiment on defective CNNs, a modiﬁed version of existing CNNs that makes CNNs capture more information other than local textures, especially the shape.
Revised: In this paper, we introduce and experiment on defective CNNs, a modiﬁed version of existing CNNs that makes CNNs capture more information other than local textures, especially the shape.

Original: We propose four empirical evidence to justify this and also show that Defective CNNs can achieve high robustness against black-box attacks while maintaining high test accuracy.
Revised: We propose four empirical evidence to justify this and also show that Defective CNNs can achieve high robustness against black-box attacks while maintaining high test accuracy.

Original: Another insight resulting from our experiments is that the adversarial perturbations generated against defective CNNs can actually change the semantic information of images and may even “fool” humans.
Revised: Another insight resulting from our experiments is that the adversarial perturbations generated against defective CNNs can actually change the semantic information of images and may even “fool” humans.

Original: We hope that these ﬁndings bring more understanding on adversarial examples and the robustness of neural networks.
Revised: We hope that these ﬁndings bring more understanding on adversarial examples and the robustness of neural networks.

Original: D EFECTIVE C ONVOLUTIONAL N ETWORKS A PPENDIX T ABLE OF C ONTENTS
Revised: D EFECTIVE C ONVOLUTIONAL N ETWORKS A PPENDIX T ABLE OF C ONTENTS

Original: • Appendix A :
Revised: • Appendix A :

Original: More Experimental Results – Decision-based Attack – Additive Gaussian Noise – Transfer-based Attack on Wide-ResNet – Transfer-based Attack from Ensemble Models on CIFAR- – Transfer-based Attack on MNIST – Gray-box Attack – White-box Attack – Randomly Shufﬂed Images and Stylized-ImageNet – Different Keep Probabilities – Experimental Details for Section 4.
Revised: More Experimental Results – Decision-based Attack – Additive Gaussian Noise – Transfer-based Attack on Wide-ResNet – Transfer-based Attack from Ensemble Models on CIFAR- – Transfer-based Attack on MNIST – Gray-box Attack – White-box Attack – Randomly Shufﬂed Images and Stylized-ImageNet – Different Keep Probabilities – Experimental Details for Section 4.

Original: • Appendix B : Adversarial Examples Generated by Defective CNNs • Appendix C : Architecture Illustrations • Appendix D : Training Details on CIFAR-10 and MNIST • Appendix E : Attack approaches
Revised: • Appendix B : Adversarial Examples Generated by Defective CNNs • Appendix C : Architecture Illustrations • Appendix D : Training Details on CIFAR-10 and MNIST • Appendix E : Attack approaches

Original: A M ORE E XPERIMENTAL R ESULTS A. D ECISION - BASED A TTACK A.1. E XPERIMENTAL S ETTINGS
Revised: A M ORE E XPERIMENTAL R ESULTS A. D ECISION - BASED A TTACK A.1. E XPERIMENTAL S ETTINGS

Original: In this subsection, we evaluate the defense performance of networks with defective convolutional layers against the decision-based attack.
Revised: In this subsection, we evaluate the defense performance of networks with defective convolutional layers against the decision-based attack.

Original: Decision-based attack performs based on the prediction of the model.
Revised: Decision-based attack performs based on the prediction of the model.

Original: It needs less information from the model and has the potential to perform better against adversarial defenses based on gradient masking.
Revised: It needs less information from the model and has the potential to perform better against adversarial defenses based on gradient masking.

Original: Boundary attack (Brendel et al., 2017) is one effective decision-based attack.
Revised: Boundary attack (Brendel et al., 2017) is one effective decision-based attack.

Original: The attack will start from a point that is already adversarial by applying a large scale perturbation to the original image and keep decreasing the distance between the original image and the adversarial example by random walks.
Revised: The attack will start from a point that is already adversarial by applying a large scale perturbation to the original image and keep decreasing the distance between the original image and the adversarial example by random walks.

Original: After iterations, we will get the ﬁnal perturbation, which has a relatively small scale.
Revised: After iterations, we will get the ﬁnal perturbation, which has a relatively small scale.

Original: The more robust the model is, the larger the ﬁnal perturbation will be.
Revised: The more robust the model is, the larger the ﬁnal perturbation will be.

Original: In our experiments, we use the implementation of boundary attack in Foolbox (Rauber et al., 2017).
Revised: In our experiments, we use the implementation of boundary attack in Foolbox (Rauber et al., 2017).

Original: It ﬁnds the adversarial initialization by simply adding large scale uniform noise on input images.
Revised: It ﬁnds the adversarial initialization by simply adding large scale uniform noise on input images.

Original: We perform our method on ResNet-18 and test the performance on CIFAR-10 with 500 samples from the validation set.
Revised: We perform our method on ResNet-18 and test the performance on CIFAR-10 with 500 samples from the validation set.

Original: The 5-block structure of ResNet-18 is shown in Appendix Figure 2.
Revised: The 5-block structure of ResNet-18 is shown in Appendix Figure 2.

Original: The blocks are labeled 0 , 1 , 2 , 3 , 4 and the 0 th block is the ﬁrst convolution layer.
Revised: The blocks are labeled 0 , 1 , 2 , 3 , 4 and the 0 th block is the ﬁrst convolution layer.

Original: We apply the defective layer structure with keep probability p = 0 .
Revised: We apply the defective layer structure with keep probability p = 0 .

Original: 1 to the bottom blocks (the 0 th , 1 st , 2 nd blocks).
Revised: 1 to the bottom blocks (the 0 th , 1 st , 2 nd blocks).

Original: For comparison, we implement label smoothing (Szegedy et al., 2016) with smoothing parameter (cid:15)
Revised: For comparison, we implement label smoothing (Szegedy et al., 2016) with smoothing parameter (cid:15)

Original: = 0 .
Revised: = 0 .

Original: 1 on a standard ResNet-18, and spatial dropout and drop block with the setting same as Section 4.1.1.
Revised: 1 on a standard ResNet-18, and spatial dropout and drop block with the setting same as Section 4.1.1.

Original: A.1. E XPERIMENTAL R ESULTS
Revised: A.1. E XPERIMENTAL R ESULTS

Original: We use the median squared to evaluate the performance, which is deﬁned as (cid:96) 2 -distance of ﬁnal perturbation across all samples proposed in (Brendel et al., 2017).
Revised: We use the median squared to evaluate the performance, which is deﬁned as (cid:96) 2 -distance of ﬁnal perturbation across all samples proposed in (Brendel et al., 2017).

Original: The score S ( M ) is deﬁned as
Revised: The score S ( M ) is deﬁned as

Original: Median i (cid:0) 1 N (cid:107) P Mi (cid:107) 22 (cid:1) , where P Mi ∈ R N is the ﬁnal perturbation that the Boundary attack ﬁnds on model M for the i th image.
Revised: Median i (cid:0) 1 N (cid:107) P Mi (cid:107) 22 (cid:1) , where P Mi ∈ R N is the ﬁnal perturbation that the Boundary attack ﬁnds on model M for the i th image.

Original: Before computing P Mi , the images are normalized into [0 , 1] N .
Revised: Before computing P Mi , the images are normalized into [0 , 1] N .

Original: [Table]
Revised: [Table]

Original: The larger value S ( M ) has, the more robust the model is.
Revised: The larger value S ( M ) has, the more robust the model is.

Original: SD, DB, and LS denote spatial dropout, drop block, and label smoothing, respectively.
Revised: SD, DB, and LS denote spatial dropout, drop block, and label smoothing, respectively.

Original: Detailed settings are listed in Section A.1.
Revised: Detailed settings are listed in Section A.1.

Original: From the results in Table 6, we point out that spatial dropout and drop block can not enhance the robustness against the boundary attack.
Revised: From the results in Table 6, we point out that spatial dropout and drop block can not enhance the robustness against the boundary attack.

Original: Neither does the label smoothing technique.
Revised: Neither does the label smoothing technique.

Original: This is consistent with the discovery in Section 4.1.1, and in Papernot et al.
Revised: This is consistent with the discovery in Section 4.1.1, and in Papernot et al.

Original: (2016a) where they point out that label smoothing is a kind of gradient masking method.
Revised: (2016a) where they point out that label smoothing is a kind of gradient masking method.

Original: Also, the defective CNN achieves higher performance over the standard CNN.
Revised: Also, the defective CNN achieves higher performance over the standard CNN.

Original: A. A DDITIVE G AUSSIAN N OISE A.2. E XPERIMENTAL S ETTINGS
Revised: A. A DDITIVE G AUSSIAN N OISE A.2. E XPERIMENTAL S ETTINGS

Original: In this subsection, we evaluate the defense performance of networks with defective convolutional layers against additive Gaussian noise.
Revised: In this subsection, we evaluate the defense performance of networks with defective convolutional layers against additive Gaussian noise.

Original: Recently, Ford et al.
Revised: Recently, Ford et al.

Original: (2019) bridge the adversarial robustness and corruption robustness, and points out that a successful adversarial defense method should also effectively defend against additive Gaussian noise.
Revised: (2019) bridge the adversarial robustness and corruption robustness, and points out that a successful adversarial defense method should also effectively defend against additive Gaussian noise.

Original: Also the Gaussian noises usually do not change the shape of objects, our models should have better defense performance.
Revised: Also the Gaussian noises usually do not change the shape of objects, our models should have better defense performance.

Original: To see whether our structure is more robust in this setting, we feed input images with additive Gaussian noises to both standard and defective CNNs.
Revised: To see whether our structure is more robust in this setting, we feed input images with additive Gaussian noises to both standard and defective CNNs.

Original: To obtain noise of scales similar to the adversarial perturbations, we generate i.i.d.
Revised: To obtain noise of scales similar to the adversarial perturbations, we generate i.i.d.

Original: Gaussian random variables x ∼ N (0 , σ 2 ) , where σ ∈ { 1 , 2 , 4 , 8 , 12 , 16 , 20 , 24 , 28 , 32 } , clip them to the range [ − 2 σ, 2 σ ] and then add them to every pixel of the input image.
Revised: Gaussian random variables x ∼ N (0 , σ 2 ) , where σ ∈ { 1 , 2 , 4 , 8 , 12 , 16 , 20 , 24 , 28 , 32 } , clip them to the range [ − 2 σ, 2 σ ] and then add them to every pixel of the input image.

Original: Note that, the magnitude range of
Revised: Note that, the magnitude range of

Original: Gaussian noises used in our experiments covers all 5 severity levels used in Hendrycks & Dietterich (2018).
Revised: Gaussian noises used in our experiments covers all 5 severity levels used in Hendrycks & Dietterich (2018).

Original: For CIFAR-10, we add Gaussian noises to 5000 samples which are drawn randomly from the validation set and can be classiﬁed correctly by all the tested models.
Revised: For CIFAR-10, we add Gaussian noises to 5000 samples which are drawn randomly from the validation set and can be classiﬁed correctly by all the tested models.

Original: We place the defective layers with keep probability p = 0 .
Revised: We place the defective layers with keep probability p = 0 .

Original: 1 on ResNet-18 in the same way as we did in Section A.1.
Revised: 1 on ResNet-18 in the same way as we did in Section A.1.

Original: A.2.
Revised: A.2.

Original: E XPERIMENTAL R ESULTS
Revised: E XPERIMENTAL R ESULTS

Original: The experimental results are shown in Figure 4.
Revised: The experimental results are shown in Figure 4.

Original: The standard CNN is still robust to small scale
Revised: The standard CNN is still robust to small scale

Original: Gaussian noise such as σ ≤ 8 .
Revised: Gaussian noise such as σ ≤ 8 .

Original: After that, the performance of the standard CNN begins to drop sharply as σ increases.
Revised: After that, the performance of the standard CNN begins to drop sharply as σ increases.

Original: In contrast, defective CNNs show far better robustness than the standard version.
Revised: In contrast, defective CNNs show far better robustness than the standard version.

Original: The defective CNN with keep probability 0 .
Revised: The defective CNN with keep probability 0 .

Original: 1 can maintain high accuracy until σ increase to 16 and have a much slower downward trend as σ increases.
Revised: 1 can maintain high accuracy until σ increase to 16 and have a much slower downward trend as σ increases.

Original: A. T RANSFER - BASED A TTACK ON W IDE -R ES N ET
Revised: A. T RANSFER - BASED A TTACK ON W IDE -R ES N ET

Original: In this subsection, we also evaluate the defense performance of networks with defective convolutional layers against transfer-based attacks.
Revised: In this subsection, we also evaluate the defense performance of networks with defective convolutional layers against transfer-based attacks.

Original: We compare our proposed method with two adversarial training methods (Buckman et al., 2018; Madry et al., 2017).
Revised: We compare our proposed method with two adversarial training methods (Buckman et al., 2018; Madry et al., 2017).

Original: For fair comparisons, we follow Buckman et al.
Revised: For fair comparisons, we follow Buckman et al.

Original: (2018) to generate adversarial examples using wide residual networks (Zagoruyko &
Revised: (2018) to generate adversarial examples using wide residual networks (Zagoruyko &

Original: Komodakis, 2016) with a depth of 32 and a width factor of 4 .
Revised: Komodakis, 2016) with a depth of 32 and a width factor of 4 .

Original: The 4 -block structure of ResNet-32 is shown in Appendix C.
Revised: The 4 -block structure of ResNet-32 is shown in Appendix C.

Original: The blocks are labeled 0 , 1 , 2 , 3 and the 0 th block is the ﬁrst convolution layer.
Revised: The blocks are labeled 0 , 1 , 2 , 3 and the 0 th block is the ﬁrst convolution layer.

Original: Both FGSM (Goodfellow et al., 2015) and PGD (Kurakin et al., 2016) attacks are run on the entire validation set of CIFAR-10 dataset.
Revised: Both FGSM (Goodfellow et al., 2015) and PGD (Kurakin et al., 2016) attacks are run on the entire validation set of CIFAR-10 dataset.

Original: These two methods both have (cid:96) ∞ perturbation scale 8 / 255 and
Revised: These two methods both have (cid:96) ∞ perturbation scale 8 / 255 and

Original: PGD runs for 7 gradient descent steps with step size 2 .
Revised: PGD runs for 7 gradient descent steps with step size 2 .

Original: The generated adversarial examples are used to attack target networks.
Revised: The generated adversarial examples are used to attack target networks.

Original: For the target network, we use the same structure but applying defective convolutional layers to the 0 th and 1 st blocks with keep probability p = 0 .
Revised: For the target network, we use the same structure but applying defective convolutional layers to the 0 th and 1 st blocks with keep probability p = 0 .

Original: 1 and train the model using the standard optimization method.
Revised: 1 and train the model using the standard optimization method.

Original: As is mentioned in Section 3.1, our proposed method is essentially different from dropout, and thus we also take dropout methods as baselines.
Revised: As is mentioned in Section 3.1, our proposed method is essentially different from dropout, and thus we also take dropout methods as baselines.

Original: More speciﬁcally, we test SpatialDropout and DropBlock.
Revised: More speciﬁcally, we test SpatialDropout and DropBlock.

Original: For both methods, we follow the instruction from Ghiasi et al.
Revised: For both methods, we follow the instruction from Ghiasi et al.

Original: (2018) to apply dropout to the 3 rd block with p = 0 .
Revised: (2018) to apply dropout to the 3 rd block with p = 0 .

Original: The block of DropBlock is set to be a 3 × 3 square.
Revised: The block of DropBlock is set to be a 3 × 3 square.

Original: The result is listed in Table 7.
Revised: The result is listed in Table 7.

Original: [Table]
Revised: [Table]

Original: A. T RANSFER - BASED A TTACK FROM E NSEMBLE M ODELS ON CIFAR- 
Revised: A. T RANSFER - BASED A TTACK FROM E NSEMBLE M ODELS ON CIFAR-

Original: In this subsection
Revised: In this subsection

Original: [Table]
Revised: [Table]

Original: we evaluate the defense performance of networks with defective convolutional layers against transfer-based attack from ensemble models on the CIFAR-10 dataset.
Revised: we evaluate the defense performance of networks with defective convolutional layers against transfer-based attack from ensemble models on the CIFAR-10 dataset.

Original: We apply defective convolutional layers to ﬁve popular network architectures ResNet-18, ResNet-50 (He et al., 2016), DenseNet-121, SENet-18 (Hu et al., 2017a), VGG-19 (Simonyan & Zisserman, 2014)
Revised: We apply defective convolutional layers to ﬁve popular network architectures ResNet-18, ResNet-50 (He et al., 2016), DenseNet-121, SENet-18 (Hu et al., 2017a), VGG-19 (Simonyan & Zisserman, 2014)

Original: [Table]
Revised: [Table]

Original: and test the black-box defense performance against transfer-based attacks from ensemble models on the CIFAR-10 dataset.
Revised: and test the black-box defense performance against transfer-based attacks from ensemble models on the CIFAR-10 dataset.

Original: For each architecture, we replace the standard convolutional layer with the defective version on the bottom layers of different architectures.
Revised: For each architecture, we replace the standard convolutional layer with the defective version on the bottom layers of different architectures.

Original: Illustrations of defective layers applied to these network architectures can be found in Appendix C. We test the black-box defense performance against transfer-based attacks on 5000 samples from the validation set.
Revised: Illustrations of defective layers applied to these network architectures can be found in Appendix C. We test the black-box defense performance against transfer-based attacks on 5000 samples from the validation set.

Original: Adversarial examples are generated by PGD, which runs for 7 steps with step size 2 and the (cid:96) ∞ perturbation scale is set to 8 / 255 .
Revised: Adversarial examples are generated by PGD, which runs for 7 steps with step size 2 and the (cid:96) ∞ perturbation scale is set to 8 / 255 .

Original: We generate ﬁve ensemble models as the source model by fusing every four models in all ﬁve models.
Revised: We generate ﬁve ensemble models as the source model by fusing every four models in all ﬁve models.

Original: The results can be found in Table 8.
Revised: The results can be found in Table 8.

Original: These results show that defective convolutional layers can consistently improve the black-box defense performance of various network architectures against transfer-based attacks from ensemble models on the CIFAR-10 dataset
Revised: These results show that defective convolutional layers can consistently improve the black-box defense performance of various network architectures against transfer-based attacks from ensemble models on the CIFAR-10 dataset

Original: [Table]
Revised: [Table]

Original: A. T RANSFER - BASED A TTACK ON MNIST
Revised: A. T RANSFER - BASED A TTACK ON MNIST

Original: In this subsection
Revised: In this subsection

Original: [Table]
Revised: [Table]

Original: we evaluate the defense performance of networks with defective convolutional layers against trasfer-based attack on the MINST dataset.
Revised: we evaluate the defense performance of networks with defective convolutional layers against trasfer-based attack on the MINST dataset.

Original: We apply defective convolutional layers to ﬁve popular network architectures ResNet-18, ResNet-50, DenseNet-121, SENet-18, VGG-
Revised: We apply defective convolutional layers to ﬁve popular network architectures ResNet-18, ResNet-50, DenseNet-121, SENet-18, VGG-

Original: [Table]
Revised: [Table]

Original: and test the black-box defense performance against transfer-based attacks on MNIST dataset.
Revised: and test the black-box defense performance against transfer-based attacks on MNIST dataset.

Original: For each architecture, we replace the standard convolutional layer with the defective version on bottom layers of different architectures.
Revised: For each architecture, we replace the standard convolutional layer with the defective version on bottom layers of different architectures.

Original: Illustrations of defective layers applied to these network architectures can be found in Appendix C. We test the black-box defense performance against transfer-based attacks on 5000 samples from the validation set.
Revised: Illustrations of defective layers applied to these network architectures can be found in Appendix C. We test the black-box defense performance against transfer-based attacks on 5000 samples from the validation set.

Original: Adversarial examples are generated by PGD which runs for 40 steps with step size 0 .
Revised: Adversarial examples are generated by PGD which runs for 40 steps with step size 0 .

Original: 01 × 255 and perturbation scale 0 .
Revised: 01 × 255 and perturbation scale 0 .

Original: 3 × 255 .
Revised: 3 × 255 .

Original: The results can be found in Table 9.
Revised: The results can be found in Table 9.

Original: These results show that defective convolutional layers can consistently improve the black-box defense performance of various network architectures against transfer-based attacks on the MNIST dataset
Revised: These results show that defective convolutional layers can consistently improve the black-box defense performance of various network architectures against transfer-based attacks on the MNIST dataset

Original: [Table]
Revised: [Table]

Original: Numbers in the middle mean the success defense rates.
Revised: Numbers in the middle mean the success defense rates.

Original: Networks in the ﬁrst row are the source models for generating adversarial examples by PGD.
Revised: Networks in the ﬁrst row are the source models for generating adversarial examples by PGD.

Original: 0.5-Bottom and 0.3- mean applying defective convolutional layers with keep probability 0.5 and 0.3 to the bottom layers of the network whose name lies just above them.
Revised: 0.5-Bottom and 0.3- mean applying defective convolutional layers with keep probability 0.5 and 0.3 to the bottom layers of the network whose name lies just above them.

Original: The source and target networks are initialized differently if they share the same architecture.
Revised: The source and target networks are initialized differently if they share the same architecture.

Original: A. G RAY - BOX A TTACK
Revised: A. G RAY - BOX A TTACK

Original: In this subsection, we show the gray-box defense performance of defective CNNs on CIFAR-10.
Revised: In this subsection, we show the gray-box defense performance of defective CNNs on CIFAR-10.

Original: We use gray-box attacks in the following two ways.
Revised: We use gray-box attacks in the following two ways.

Original: One way is to generate adversarial examples against one trained neural network and test those images on a network with the same structure but different initializations.
Revised: One way is to generate adversarial examples against one trained neural network and test those images on a network with the same structure but different initializations.

Original: The other way is speciﬁc to our defective models.
Revised: The other way is speciﬁc to our defective models.

Original: We generate adversarial examples on one trained defective CNN and test them on a network with the same keep probability but different sampling of defective neurons.
Revised: We generate adversarial examples on one trained defective CNN and test them on a network with the same keep probability but different sampling of defective neurons.

Original: In both of these two ways, the adversarial knows some information on the structure of the network but does not know the speciﬁc parameters of it.
Revised: In both of these two ways, the adversarial knows some information on the structure of the network but does not know the speciﬁc parameters of it.

Original: [Table]
Revised: [Table]

Original: From the results listed in Table 10, we ﬁnd that defective CNNs have similar performance on adversarial examples generated by our two kinds of gray-box attacks.
Revised: From the results listed in Table 10, we ﬁnd that defective CNNs have similar performance on adversarial examples generated by our two kinds of gray-box attacks.

Original: This phenomenon indicates that defective CNNs with the same keep probability would catch similar information which is insensitive to the selections of defective neurons.
Revised: This phenomenon indicates that defective CNNs with the same keep probability would catch similar information which is insensitive to the selections of defective neurons.

Original: Also, comparing with the gray-box performance of standard
Revised: Also, comparing with the gray-box performance of standard

Original: CNNs (See Table 11), defective CNNs show stronger defense ability.
Revised: CNNs (See Table 11), defective CNNs show stronger defense ability.

Original: [Table]
Revised: [Table]

Original: A. W HITE - BOX A TTACK
Revised: A. W HITE - BOX A TTACK

Original: In this subsection
Revised: In this subsection

Original: [Table]
Revised: [Table]

Original: we show the white-box defense performance of defective CNNs.
Revised: we show the white-box defense performance of defective CNNs.

Original: Table 12 shows the results of ResNet-18 on the CIFAR-10 dataset.
Revised: Table 12 shows the results of ResNet-18 on the CIFAR-10 dataset.

Original: The performance on other network architectures is similar.
Revised: The performance on other network architectures is similar.

Original: Note that, the proposed method would not involve any obfuscated gradients (Athalye et al., 2018).
Revised: Note that, the proposed method would not involve any obfuscated gradients (Athalye et al., 2018).

Original: Also, We study the combination of the proposed method and adversarial training.
Revised: Also, We study the combination of the proposed method and adversarial training.

Original: We adversarially train a defective CNN under the same setting described in Madry et al.
Revised: We adversarially train a defective CNN under the same setting described in Madry et al.

Original: and reach 51.6% successful defense rate against the default PGD attack ( (cid:96)
Revised: and reach 51.6% successful defense rate against the default PGD attack ( (cid:96)

Original: ∞
Revised: ∞

Original: = 8 / 255 and 7 steps) used in training, which outperforms the standard CNN (50.0%).
Revised: = 8 / 255 and 7 steps) used in training, which outperforms the standard CNN (50.0%).

Original: [Table]
Revised: [Table]

Original: are
Revised: are

Original: We want to emphasize that the adversarial examples generated by defective CNNs appear to have semantic shapes and may even fool humans as well (see Figure 3 and Appendix B).
Revised: We want to emphasize that the adversarial examples generated by defective CNNs appear to have semantic shapes and may even fool humans as well (see Figure 3 and Appendix B).

Original: This indicates that small perturbations can actually change the semantic meaning of images for humans.
Revised: This indicates that small perturbations can actually change the semantic meaning of images for humans.

Original: Those samples should probably not be categorized into adversarial examples and used to evaluate whitebox robustness.
Revised: Those samples should probably not be categorized into adversarial examples and used to evaluate whitebox robustness.

Original: This is also aligned with Ilyas et al.
Revised: This is also aligned with Ilyas et al.

Original: A. R ANDOMLY S HUFFLED I MAGES AND S TYLIZED -I MAGE N ET
Revised: A. R ANDOMLY S HUFFLED I MAGES AND S TYLIZED -I MAGE N ET

Original: In this subsection, we show more results on randomly shufﬂed images and StylizedImageNet (Geirhos et al., 2018).
Revised: In this subsection, we show more results on randomly shufﬂed images and StylizedImageNet (Geirhos et al., 2018).

Original: As shown in Section 3.2, shape information in randomly shufﬂed images is destroyed while textural information preserving, and Stylized-ImageNet has the opposite situation.
Revised: As shown in Section 3.2, shape information in randomly shufﬂed images is destroyed while textural information preserving, and Stylized-ImageNet has the opposite situation.

Original: If a CNN make predictions relying less on textural information but more on shape information, it should have worse performance on randomly shufﬂed images but better performance on
Revised: If a CNN make predictions relying less on textural information but more on shape information, it should have worse performance on randomly shufﬂed images but better performance on

Original: Stylized-ImageNet.
Revised: Stylized-ImageNet.

Original: We construct defective CNNs by applying defective convolutional layers to the bottom layers of standard ResNet-18, ResNet-50, DenseNet-121, SENet-18, and VGG-19.
Revised: We construct defective CNNs by applying defective convolutional layers to the bottom layers of standard ResNet-18, ResNet-50, DenseNet-121, SENet-18, and VGG-19.

Original: We train all defective
Revised: We train all defective

Original: CNNs and their plain counterparts on the ImageNet dataset.
Revised: CNNs and their plain counterparts on the ImageNet dataset.

Original: For each pair of CNNs, we sample images from the validation set, which are predicted correctly by both two kinds of CNNs.
Revised: For each pair of CNNs, we sample images from the validation set, which are predicted correctly by both two kinds of CNNs.

Original: We make manipulations to the sampled images by setting k ∈ { 2 , 4 , 8 } and pick corresponding images from
Revised: We make manipulations to the sampled images by setting k ∈ { 2 , 4 , 8 } and pick corresponding images from

Original: Stylized-ImageNet.
Revised: Stylized-ImageNet.

Original: We check the accuracy of all models on the these images.
Revised: We check the accuracy of all models on the these images.

Original: The results are shown in Table 13.
Revised: The results are shown in Table 13.

Original: We can see the defective CNNs perform consistently worse than the standard CNNs on the randomly shufﬂed images, and perform consistently better than the standard CNNs on StylizedImageNet.
Revised: We can see the defective CNNs perform consistently worse than the standard CNNs on the randomly shufﬂed images, and perform consistently better than the standard CNNs on StylizedImageNet.

Original: This justiﬁes our argument that defective CNNs make predictions relying less on textural information but more on shape information.
Revised: This justiﬁes our argument that defective CNNs make predictions relying less on textural information but more on shape information.

Original: [Table]
Revised: [Table]

Original: A. D IFFERENT K EEP P ROBABILITIES
Revised: A. D IFFERENT K EEP P ROBABILITIES

Original: In this subsection
Revised: In this subsection

Original: [Table]
Revised: [Table]

Original: we show the trade-off between robustness and generalization performance in defective CNNs with different keep probabilities.
Revised: we show the trade-off between robustness and generalization performance in defective CNNs with different keep probabilities.

Original: We use DenseNet-121 (Huang et al., 2017) as the source model to generate adversarial examples from CIFAR-10 with PGD (Kurakin et al., 2016), which runs for 20 steps with step size 1 and perturbation scale 16.
Revised: We use DenseNet-121 (Huang et al., 2017) as the source model to generate adversarial examples from CIFAR-10 with PGD (Kurakin et al., 2016), which runs for 20 steps with step size 1 and perturbation scale 16.

Original: The defective convolutional layers are applied to the bottom layers of ResNet-18 (He et al., 2016).
Revised: The defective convolutional layers are applied to the bottom layers of ResNet-18 (He et al., 2016).

Original: Figure 5 shows that the defense accuracy monotonically increases as the test accuracy decreases along with the keep probability.
Revised: Figure 5 shows that the defense accuracy monotonically increases as the test accuracy decreases along with the keep probability.

Original: We can see the trade-off between robustness and generalization.
Revised: We can see the trade-off between robustness and generalization.

Original: Therefore, a practical way to use Defective CNNs in the real world is to ensemble defective CNNs with different keep probabilities.
Revised: Therefore, a practical way to use Defective CNNs in the real world is to ensemble defective CNNs with different keep probabilities.

Original: Also, in our experiments, we found that ensemble different defective CNNs with the same p can bring improvements on both accuracy and robustness while ensemble standard CNNs can not.
Revised: Also, in our experiments, we found that ensemble different defective CNNs with the same p can bring improvements on both accuracy and robustness while ensemble standard CNNs can not.

Original: A. E XPERIMENTAL D ETAILS FOR S ECTION 4. 
Revised: A. E XPERIMENTAL D ETAILS FOR S ECTION 4.

Original: In this subsection, we will show more experimental results on defective CNNs using different adversarial examples, different attack methods and different mask settings on ResNet-18.
Revised: In this subsection, we will show more experimental results on defective CNNs using different adversarial examples, different attack methods and different mask settings on ResNet-18.

Original: The networks used to generate adversarial examples including ResNet-18, ResNet-50, DenseNet-121, SENet18, and VGG-19.
Revised: The networks used to generate adversarial examples including ResNet-18, ResNet-50, DenseNet-121, SENet18, and VGG-19.

Original: More speciﬁcally, we choose 5000 samples to generate adversarial examples via
Revised: More speciﬁcally, we choose 5000 samples to generate adversarial examples via

Original: FGSM and PGD, and 1000 samples for CW attack.
Revised: FGSM and PGD, and 1000 samples for CW attack.

Original: All samples are drawn from the validation set of
Revised: All samples are drawn from the validation set of

Original: CIFAR-10 dataset and can be correctly classiﬁed correctly by the model used to generate adversarial examples.
Revised: CIFAR-10 dataset and can be correctly classiﬁed correctly by the model used to generate adversarial examples.

Original: For FGSM, we try step size (cid:15) ∈ { 8 , 16 , 32 } , namely FGSM 8 , FGSM 16 , FGSM 32 , to generate adversarial examples.
Revised: For FGSM, we try step size (cid:15) ∈ { 8 , 16 , 32 } , namely FGSM 8 , FGSM 16 , FGSM 32 , to generate adversarial examples.

Original: For PGD, we have tried more extensive settings.
Revised: For PGD, we have tried more extensive settings.

Original: Let { (cid:15), T, α } be the PGD setting with step size (cid:15) , the number of steps T and the perturbation scale α , then we have tried PGD settings (1 , 8 , 4) , (2 , 4 , 4) , (4 , 2 , 4) , (1 , 12 , 8) , (2 , 6 , 8) , (4 , 3 , 8) , (1 , 20 , 16) , (2 , 10 , 16) , (4 , 5 , 16) , (1 , 40 , 32) , (2 , 20 , 32) , (4 , 10 , 32) to generate PGD adversarial examples.
Revised: Let { (cid:15), T, α } be the PGD setting with step size (cid:15) , the number of steps T and the perturbation scale α , then we have tried PGD settings (1 , 8 , 4) , (2 , 4 , 4) , (4 , 2 , 4) , (1 , 12 , 8) , (2 , 6 , 8) , (4 , 3 , 8) , (1 , 20 , 16) , (2 , 10 , 16) , (4 , 5 , 16) , (1 , 40 , 32) , (2 , 20 , 32) , (4 , 10 , 32) to generate PGD adversarial examples.

Original: From the experimental results, we observe the following phenomena.
Revised: From the experimental results, we observe the following phenomena.

Original: First, we ﬁnd that the larger the perturbation scale is, the stronger the adversarial examples are.
Revised: First, we ﬁnd that the larger the perturbation scale is, the stronger the adversarial examples are.

Original: Second, for a ﬁxed perturbation scale, the smaller the step size is, the more successful the attack is, as it searches the adversarial examples in a more careful way around the original image.
Revised: Second, for a ﬁxed perturbation scale, the smaller the step size is, the more successful the attack is, as it searches the adversarial examples in a more careful way around the original image.

Original: Based on these observation, we only show strong PGD attack results in the Appendix, namely the settings (1 , 20 , 16)
Revised: Based on these observation, we only show strong PGD attack results in the Appendix, namely the settings (1 , 20 , 16)

Original: ( PGD 16 ), (2 , 10 , 16) ( PGD 2 , 16 ) and (1 , 40 , 32) ( PGD 32 ).
Revised: ( PGD 16 ), (2 , 10 , 16) ( PGD 2 , 16 ) and (1 , 40 , 32) ( PGD 32 ).

Original: Nonetheless, our models also perform much better on weak PGD attacks.
Revised: Nonetheless, our models also perform much better on weak PGD attacks.

Original: For the CW attack, we have also tried different conﬁdence parameters κ .
Revised: For the CW attack, we have also tried different conﬁdence parameters κ .

Original: However, we ﬁnd that for large κ , the algorithm is hard to ﬁnd adversarial examples for some neural networks such as VGG because of its logit scale.
Revised: However, we ﬁnd that for large κ , the algorithm is hard to ﬁnd adversarial examples for some neural networks such as VGG because of its logit scale.

Original: For smaller κ , the adversarial examples have weak transferability, which means they can be easily defended even by standard CNNs.
Revised: For smaller κ , the adversarial examples have weak transferability, which means they can be easily defended even by standard CNNs.

Original: Therefore, in order to balance these two factors, we choose κ = 40 ( CW 40 ) for DenseNet-121, ResNet-50, SENet-18 and κ = 20 ( CW 20 ) for ResNet-18 as a good choice to compare our models with standard ones.
Revised: Therefore, in order to balance these two factors, we choose κ = 40 ( CW 40 ) for DenseNet-121, ResNet-50, SENet-18 and κ = 20 ( CW 20 ) for ResNet-18 as a good choice to compare our models with standard ones.

Original: The step number for choosing the parameter c is set to 30 .
Revised: The step number for choosing the parameter c is set to 30 .

Original: Note that the noises of FGSM and PGD are considered in the sense of (cid:96) ∞ norm and the noise of CW is considered in the sense of (cid:96) 2 norm.
Revised: Note that the noises of FGSM and PGD are considered in the sense of (cid:96) ∞ norm and the noise of CW is considered in the sense of (cid:96) 2 norm.

Original: All adversarial examples used to evaluate can fool the original network.
Revised: All adversarial examples used to evaluate can fool the original network.

Original: Table 14,15,16,17 and 18 list our experimental results.
Revised: Table 14,15,16,17 and 18 list our experimental results.

Original: DC means we replace defective neurons with defective channels in the corresponding blocks to achieve the same keep probability.
Revised: DC means we replace defective neurons with defective channels in the corresponding blocks to achieve the same keep probability.

Original: SM means we use the same defective mask on all the channels in a layer.
Revised: SM means we use the same defective mask on all the channels in a layer.

Original: × n means we multiply the number of the channels in the defective blocks by n times.
Revised: × n means we multiply the number of the channels in the defective blocks by n times.

Original: EN means we ensemble ﬁve models with different defective masks of the same keep probability.
Revised: EN means we ensemble ﬁve models with different defective masks of the same keep probability.

Original: B A DVERSARIAL E XAMPLES G ENERATED BY D EFECTIVE CNN S
Revised: B A DVERSARIAL E XAMPLES G ENERATED BY D EFECTIVE CNN S

Original: In this subsection, we show more adversarial examples generated by defective CNNs.
Revised: In this subsection, we show more adversarial examples generated by defective CNNs.

Original: Figureshows some adversarial examples generated on the CIFAR-10 dataset along with the corresponding original images.
Revised: Figureshows some adversarial examples generated on the CIFAR-10 dataset along with the corresponding original images.

Original: These examples are generated from CIFAR-10 against a defective ResNet-18 of keep probability 0 .
Revised: These examples are generated from CIFAR-10 against a defective ResNet-18 of keep probability 0 .

Original: 2 on the 0 th , 1 st , 2 nd blocks, a defective ResNet-18 of keep probability 0 .
Revised: 2 on the 0 th , 1 st , 2 nd blocks, a defective ResNet-18 of keep probability 0 .

Original: 1 on the 1 st , 2 nd blocks, and a standard ResNet-18.
Revised: 1 on the 1 st , 2 nd blocks, and a standard ResNet-18.

Original: We use attack method MIFGSM
Revised: We use attack method MIFGSM

Original: (Dong et al., 2017) with perturbation scale α = 16 and α = 32 .
Revised: (Dong et al., 2017) with perturbation scale α = 16 and α = 32 .

Original: We also show some adversarial examples generated from Tiny-ImageNet 3 along with the corresponding original images in Figure 7.
Revised: We also show some adversarial examples generated from Tiny-ImageNet 3 along with the corresponding original images in Figure 7.

Original: These examples are generated from Tiny-ImageNet against a defective ResNet-18 of keep probability of the keep probability 0 .
Revised: These examples are generated from Tiny-ImageNet against a defective ResNet-18 of keep probability of the keep probability 0 .

Original: 1 on the 1 st , 2 nd blocks and a standard ResNet-18.
Revised: 1 on the 1 st , 2 nd blocks and a standard ResNet-18.

Original: The attack methods are MIFGSM with scale 64 and 32 , step size 1 and step number 40 and 80 respectively.
Revised: The attack methods are MIFGSM with scale 64 and 32 , step size 1 and step number 40 and 80 respectively.

Original: The adversarial examples generated by defective CNNs exhibit more semantic shapes of their fooled classes, such as the mouth of the frog in Figure 6.
Revised: The adversarial examples generated by defective CNNs exhibit more semantic shapes of their fooled classes, such as the mouth of the frog in Figure 6.

Original: This also corroborates the point made in Tsipras et al.
Revised: This also corroborates the point made in Tsipras et al.

Original: (2018) that more robust models will be more aligned with human perception.
Revised: (2018) that more robust models will be more aligned with human perception.

Original: To further verify the adversarial examples generated by defective CNNs align better with human perception than standard CNNs, we conduct a user study.
Revised: To further verify the adversarial examples generated by defective CNNs align better with human perception than standard CNNs, we conduct a user study.

Original: We show users a pair of adversarial examples generated by defective CNNs and standard CNNs, respectively.
Revised: We show users a pair of adversarial examples generated by defective CNNs and standard CNNs, respectively.

Original: The corresponding labels are attached.
Revised: The corresponding labels are attached.

Original: The user will be asked which one of the pair is better aligned with the predicted label.
Revised: The user will be asked which one of the pair is better aligned with the predicted label.

Original: More speciﬁcally, we generate two sets of adversarial examples on CIFAR-10 and Tiny-ImageNet by defective CNNs and standard CNNs, respectively.
Revised: More speciﬁcally, we generate two sets of adversarial examples on CIFAR-10 and Tiny-ImageNet by defective CNNs and standard CNNs, respectively.

Original: For each user, we randomly sample 50 pairs from the two sets and ask him/her to select.
Revised: For each user, we randomly sample 50 pairs from the two sets and ask him/her to select.

Original: A total of 13 people are involved in our study.
Revised: A total of 13 people are involved in our study.

Original: The results show that all users select more images generated by defective CNNs than the ones generated by standard CNNs.
Revised: The results show that all users select more images generated by defective CNNs than the ones generated by standard CNNs.

Original: On average, the number of defective CNNs ones is 14 more than the number of standard CNNs ones.
Revised: On average, the number of defective CNNs ones is 14 more than the number of standard CNNs ones.

Original: This supports our arguments.
Revised: This supports our arguments.

Original: [Figure]
Revised: [Figure]

Original: Mushroom
Revised: Mushroom

Original: Monarch Butter-
Revised: Monarch Butter-

Original: ﬂy
Revised: ﬂy

Original: Black Widow
Revised: Black Widow

Original: Mushroom
Revised: Mushroom

Original: Sulphur Butterﬂy
Revised: Sulphur Butterﬂy

Original: Teddy
Revised: Teddy

Original: Ladybug
Revised: Ladybug

Original: Tarantula
Revised: Tarantula

Original: Black Widow
Revised: Black Widow

Original: Ladybug
Revised: Ladybug

Original: Goldﬁsh
Revised: Goldﬁsh

Original: Egyptian Cat
Revised: Egyptian Cat

Original: Brain Coral
Revised: Brain Coral

Original: Fly
Revised: Fly

Original: Standard Poodle
Revised: Standard Poodle

Original: Dragonﬂy
Revised: Dragonﬂy

Original: Roach
Revised: Roach

Original: Mushroom
Revised: Mushroom

Original: Mashed Potato
Revised: Mashed Potato

Original: Teddy drumstick
Revised: Teddy drumstick

Original: C A RCHITECTURE I LLUSTRATIONS
Revised: C A RCHITECTURE I LLUSTRATIONS

Original: In this subsection, we brieﬂy introduce the network architectures used in our experiments.
Revised: In this subsection, we brieﬂy introduce the network architectures used in our experiments.

Original: Generally, we apply defective convolutional layers to the bottom layers of the networks and we have tried six different architectures, namely ResNet-18 , ResNet-50 , DenseNet-121 , SENet-18 , VGG-19 and
Revised: Generally, we apply defective convolutional layers to the bottom layers of the networks and we have tried six different architectures, namely ResNet-18 , ResNet-50 , DenseNet-121 , SENet-18 , VGG-19 and

Original: WideResNet-32 .
Revised: WideResNet-32 .

Original: We next illustrate these architectures and show how we apply defective convolutional layers to them.
Revised: We next illustrate these architectures and show how we apply defective convolutional layers to them.

Original: In our experiments, applying defective convolutional layers to a block means randomly selecting defective neurons in every layer of the block.
Revised: In our experiments, applying defective convolutional layers to a block means randomly selecting defective neurons in every layer of the block.

Original: C. R ES N ET - 
Revised: C. R ES N ET -

Original: ResNet-18 (He et al., 2016) contains 5 blocks: the 0 th block is one single 3 × 3 convolutional layer, and each of the rest contains four 3 × 3 convolutional layers.
Revised: ResNet-18 (He et al., 2016) contains 5 blocks: the 0 th block is one single 3 × 3 convolutional layer, and each of the rest contains four 3 × 3 convolutional layers.

Original: Figure 8 shows the whole structure of ResNet-18.
Revised: Figure 8 shows the whole structure of ResNet-18.

Original: In our experiments, we apply defective convolutional layers to the 0 th , 1 st , 2 nd blocks which are the bottom layers.
Revised: In our experiments, we apply defective convolutional layers to the 0 th , 1 st , 2 nd blocks which are the bottom layers.

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: Similar to ResNet-18, ResNet-50 (He et al., 2016) contains 5 blocks and each block contains several 1 × 1 and 3 × 3 convolutional layers (i.e. Bottlenecks).
Revised: Similar to ResNet-18, ResNet-50 (He et al., 2016) contains 5 blocks and each block contains several 1 × 1 and 3 × 3 convolutional layers (i.e. Bottlenecks).

Original: In our experiment, we apply defective convolutional layers to the 3 × 3 convolutional layers in the ﬁrst three “bottom” blocks.
Revised: In our experiment, we apply defective convolutional layers to the 3 × 3 convolutional layers in the ﬁrst three “bottom” blocks.

Original: The defective layers in the 1 st block are marked by the red arrows in Figure 9.
Revised: The defective layers in the 1 st block are marked by the red arrows in Figure 9.

Original: [Figure]
Revised: [Figure]

Original: DenseNet-121 (Huang et al., 2017) is another popular network architecture in deep learning researches.
Revised: DenseNet-121 (Huang et al., 2017) is another popular network architecture in deep learning researches.

Original: Figure 10 shows the whole structure of DenseNet-121.
Revised: Figure 10 shows the whole structure of DenseNet-121.

Original: It contains 5 Dense-Blocks, each of which contains several 1 × 1 and 3 × 3 convolutional layers.
Revised: It contains 5 Dense-Blocks, each of which contains several 1 × 1 and 3 × 3 convolutional layers.

Original: Similar to what we do for ResNet-50, we apply defective convolutional layers to the 3 × 3 convolutional layers in the ﬁrst three “bottom” blocks.
Revised: Similar to what we do for ResNet-50, we apply defective convolutional layers to the 3 × 3 convolutional layers in the ﬁrst three “bottom” blocks.

Original: The growth rate is set to 32 in our experiments.
Revised: The growth rate is set to 32 in our experiments.

Original: vspace-0.1in
Revised: vspace-0.1in

Original: [Figure]
Revised: [Figure]

Original: SENet (Hu et al., 2017a), a network architecture which won the ﬁrst place in ImageNet contest 2017 , is shown in Figure 11.
Revised: SENet (Hu et al., 2017a), a network architecture which won the ﬁrst place in ImageNet contest 2017 , is shown in Figure 11.

Original: Note that here we use the pre-activation shortcut version of SENet-18 and we apply defective convolutional layers to the convolutional layers in the ﬁrst 3 SE-blocks.
Revised: Note that here we use the pre-activation shortcut version of SENet-18 and we apply defective convolutional layers to the convolutional layers in the ﬁrst 3 SE-blocks.

Original: C. VGG- 
Revised: C. VGG-

Original: VGG-19 (Simonyan & Zisserman, 2014) is a typical neural network architecture with sixteen 3 ×convolutional layers and three fully-connected layers.
Revised: VGG-19 (Simonyan & Zisserman, 2014) is a typical neural network architecture with sixteen 3 ×convolutional layers and three fully-connected layers.

Original: We slightly modiﬁed the architecture by replacing the ﬁnal 3 fully connected layers with 1 fully connected layer as is suggested by recent architectures.
Revised: We slightly modiﬁed the architecture by replacing the ﬁnal 3 fully connected layers with 1 fully connected layer as is suggested by recent architectures.

Original: Figure 12 shows the whole structure of VGG-19.
Revised: Figure 12 shows the whole structure of VGG-19.

Original: We apply defective convolutional layers on the ﬁrst four 3 × 3 convolutional layers.
Revised: We apply defective convolutional layers on the ﬁrst four 3 × 3 convolutional layers.

Original: C.
Revised: C.

Original: W IDE R ES N ETBased on residual networks, Zagoruyko & Komodakis (2016) proposed a wide version of residual networks which have much more channels.
Revised: W IDE R ES N ETBased on residual networks, Zagoruyko & Komodakis (2016) proposed a wide version of residual networks which have much more channels.

Original: In our experiments, we adopt the network with a width factor of 4 and apply defective layers on the 0 th and 1 st blocks.
Revised: In our experiments, we adopt the network with a width factor of 4 and apply defective layers on the 0 th and 1 st blocks.

Original: Figure 13 shows the whole structure of WideResNet-32.
Revised: Figure 13 shows the whole structure of WideResNet-32.

Original: D T RAINING D ETAILS ON CIFAR-10 AND MNIST
Revised: D T RAINING D ETAILS ON CIFAR-10 AND MNIST

Original: To guarantee our experiments are reproducible, here we present more details on the training process in our experiments.
Revised: To guarantee our experiments are reproducible, here we present more details on the training process in our experiments.

Original: When training models on CIFAR-10, we ﬁrst subtract per-pixel mean.
Revised: When training models on CIFAR-10, we ﬁrst subtract per-pixel mean.

Original: Then we apply a zero-padding of width 4 , a random horizontal ﬂip and a random crop of size 32 × 32 on train data.
Revised: Then we apply a zero-padding of width 4 , a random horizontal ﬂip and a random crop of size 32 × 32 on train data.

Original: No other data augmentation method is used.
Revised: No other data augmentation method is used.

Original: We apply SGD with momentum parameter 0 .
Revised: We apply SGD with momentum parameter 0 .

Original: 9 , weight decay parameter 5 × 10 − 4 and mini-batch size 128 to train on the data for 350 epochs.
Revised: 9 , weight decay parameter 5 × 10 − 4 and mini-batch size 128 to train on the data for 350 epochs.

Original: The learning rate starts from 0 .
Revised: The learning rate starts from 0 .

Original: and is divided by 10 when the number of epochs reaches 150 and 250 .
Revised: and is divided by 10 when the number of epochs reaches 150 and 250 .

Original: When training models on MNIST, we ﬁrst subtract per-pixel mean.
Revised: When training models on MNIST, we ﬁrst subtract per-pixel mean.

Original: Then we apply random horizontal ﬂip on train data.
Revised: Then we apply random horizontal ﬂip on train data.

Original: We apply SGD with momentum parameter 0 .
Revised: We apply SGD with momentum parameter 0 .

Original: 9 , weight decay parameter 5 × 10 − 4 and mini-batch size 128 to train on the data for 50 epochs.
Revised: 9 , weight decay parameter 5 × 10 − 4 and mini-batch size 128 to train on the data for 50 epochs.

Original: The learning rate starts from 0 .
Revised: The learning rate starts from 0 .

Original: and is divided by 10 when the number of epochs reaches 20 and 40 .
Revised: and is divided by 10 when the number of epochs reaches 20 and 40 .

Original: Figure 14 shows the train and test curves of standard and defective ResNet-18 on CIFAR-10 and MNIST.
Revised: Figure 14 shows the train and test curves of standard and defective ResNet-18 on CIFAR-10 and MNIST.

Original: Different network structures share similar tendency regarding the train and test curves.
Revised: Different network structures share similar tendency regarding the train and test curves.

Original: [Figure]
Revised: [Figure]

Original: E A TTACK APPROACHES
Revised: E A TTACK APPROACHES

Original: In this subsection, we describe the attack approaches used in our experiments.
Revised: In this subsection, we describe the attack approaches used in our experiments.

Original: We ﬁrst give an overview of how to attack a neural network in mathematical notations.
Revised: We ﬁrst give an overview of how to attack a neural network in mathematical notations.

Original: Let x be the input to the neural network and f θ be the function which represents the neural network with parameter θ .
Revised: Let x be the input to the neural network and f θ be the function which represents the neural network with parameter θ .

Original: The output label of the network to the input can be computed as c = arg max i f θ ( x ) i .
Revised: The output label of the network to the input can be computed as c = arg max i f θ ( x ) i .

Original: In order to perform an adversarial attack, we add a small perturbation δ x to the original image and get an adversarial image x adv = x + δ x .
Revised: In order to perform an adversarial attack, we add a small perturbation δ x to the original image and get an adversarial image x adv = x + δ x .

Original: The new input x adv should look visually similar to the original x .
Revised: The new input x adv should look visually similar to the original x .

Original: Here we use the commonly used (cid:96) ∞ -norm metric to measure similarity, i.e., we require that || δ x || ≤ (cid:15) .
Revised: Here we use the commonly used (cid:96) ∞ -norm metric to measure similarity, i.e., we require that || δ x || ≤ (cid:15) .

Original: The attack is considered successful if the predicted label of the perturbed image c adv = arg max i f θ ( x adv ) i is different from c .
Revised: The attack is considered successful if the predicted label of the perturbed image c adv = arg max i f θ ( x adv ) i is different from c .

Original: Generally speaking, there are two types of attack methods: Targeted Attack , which aims to change the output label of an image to a speciﬁc (and different) one, and Untargeted Attack , which only aims to change the output label and does not restrict which speciﬁc label the modiﬁed example should let the network output.
Revised: Generally speaking, there are two types of attack methods: Targeted Attack , which aims to change the output label of an image to a speciﬁc (and different) one, and Untargeted Attack , which only aims to change the output label and does not restrict which speciﬁc label the modiﬁed example should let the network output.

Original: In this paper, we mainly use the following four gradient-based attack approaches.
Revised: In this paper, we mainly use the following four gradient-based attack approaches.

Original: J denotes the loss function of the neural network and y denotes the ground truth label of x .
Revised: J denotes the loss function of the neural network and y denotes the ground truth label of x .

Original: • Fast Gradient Sign Method (FGSM).
Revised: • Fast Gradient Sign Method (FGSM).

Original: FGSM (Goodfellow et al., 2015) is a one-step untargeted method which generates the adversarial example x adv by adding the sign of the gradients multiplied by a step size (cid:15) to the original benign image x .
Revised: FGSM (Goodfellow et al., 2015) is a one-step untargeted method which generates the adversarial example x adv by adding the sign of the gradients multiplied by a step size (cid:15) to the original benign image x .

Original: Note that FGSM controls the (cid:96) ∞ -norm between the adversarial example and the original one by the parameter (cid:15) .
Revised: Note that FGSM controls the (cid:96) ∞ -norm between the adversarial example and the original one by the parameter (cid:15) .

Original: [Equation]
Revised: [Equation]

Original: • Basic iterative method (PGD).
Revised: • Basic iterative method (PGD).

Original: PGD (Kurakin et al., 2016) is a multiple-step attack method which applies FGSM multiple times.
Revised: PGD (Kurakin et al., 2016) is a multiple-step attack method which applies FGSM multiple times.

Original: To make the adversarial example still stay “close” to the original image, the image is projected to the (cid:96) ∞ -ball centered at the original image after every step.
Revised: To make the adversarial example still stay “close” to the original image, the image is projected to the (cid:96) ∞ -ball centered at the original image after every step.

Original: The radius of the (cid:96) ∞ -ball is called perturbation scale and is denoted by α .
Revised: The radius of the (cid:96) ∞ -ball is called perturbation scale and is denoted by α .

Original: [Equation]
Revised: [Equation]

Original: • Momentum Iterative Fast Gradient Sign Method (MIFGSM).
Revised: • Momentum Iterative Fast Gradient Sign Method (MIFGSM).

Original: MIFGSM (Dong et al., 2017) is a recently proposed multiple-step attack method.
Revised: MIFGSM (Dong et al., 2017) is a recently proposed multiple-step attack method.

Original: It is similar to PGD, but it computes the optimize direction by a momentum instead of the gradients.
Revised: It is similar to PGD, but it computes the optimize direction by a momentum instead of the gradients.

Original: The radius of the (cid:96) ∞ -ball is also called perturbation scale and is denoted by α .
Revised: The radius of the (cid:96) ∞ -ball is also called perturbation scale and is denoted by α .

Original: [Equation]
Revised: [Equation]

Original: • CW Attack.
Revised: • CW Attack.

Original: Carlini & Wagner (2016) shows that constructing an adversarial example can be formulated as solving the following optimization problem:
Revised: Carlini & Wagner (2016) shows that constructing an adversarial example can be formulated as solving the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where c · g ( x (cid:48) ) is the loss function that evaluates the quality of x (cid:48) as an adversarial example and the term
Revised: where c · g ( x (cid:48) ) is the loss function that evaluates the quality of x (cid:48) as an adversarial example and the term

Original: || x (cid:48) − x || 22 controls the scale of the perturbation.
Revised: || x (cid:48) − x || 22 controls the scale of the perturbation.

Original: More speciﬁcally, in the untargeted attack setting, the loss function g ( x ) can be deﬁned as below, where the parameter κ is called conﬁdence.
Revised: More speciﬁcally, in the untargeted attack setting, the loss function g ( x ) can be deﬁned as below, where the parameter κ is called conﬁdence.

Original: [Equation]
Revised: [Equation]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: on CIFAR-
Revised: on CIFAR-

Original: [Table]
Revised: [Table]

Original: on CIFAR-
Revised: on CIFAR-

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: defective ﬁve different masks of respectively.
Revised: defective ﬁve different masks of respectively.

Original: [Table]
Revised: [Table]

Original: on CIFAR-
Revised: on CIFAR-

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: defective ﬁve different masks of keep respectively.
Revised: defective ﬁve different masks of keep respectively.

Original: [Table]
Revised: [Table]