Original: T RANSFERABLE P ERTURBATIONS OF D EEP F EATURE D ISTRIBUTIONS
Revised: T RANSFERABLE P ERTURBATIONS OF D EEP F EATURE D ISTRIBUTIONS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Almost all current adversarial attacks of CNN classiﬁers rely on information derived from the output layer of the network.
Revised: Almost all current adversarial attacks of CNN classiﬁers rely on information derived from the output layer of the network.

Original: This work presents a new adversarial attack based on the modeling and exploitation of class-wise and layer-wise deep feature distributions.
Revised: This work presents a new adversarial attack based on the modeling and exploitation of class-wise and layer-wise deep feature distributions.

Original: We achieve state-of-the-art targeted blackbox transfer-based attack results for undefended ImageNet models.
Revised: We achieve state-of-the-art targeted blackbox transfer-based attack results for undefended ImageNet models.

Original: Further, we place a priority on explainability and interpretability of the attacking process.
Revised: Further, we place a priority on explainability and interpretability of the attacking process.

Original: Our methodology affords an analysis of how adversarial attacks change the intermediate feature distributions of CNNs, as well as a measure of layer-wise and class-wise feature distributional separability/entanglement.
Revised: Our methodology affords an analysis of how adversarial attacks change the intermediate feature distributions of CNNs, as well as a measure of layer-wise and class-wise feature distributional separability/entanglement.

Original: We also conceptualize a transition from task/data-speciﬁc to model-speciﬁc features within a CNN architecture that directly impacts the transferability of adversarial examples.
Revised: We also conceptualize a transition from task/data-speciﬁc to model-speciﬁc features within a CNN architecture that directly impacts the transferability of adversarial examples.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Most recent adversarial attack literature has focused on empirical demonstrations of how classiﬁers can be fooled by the addition of quasi-imperceptible noise to the input (Szegedy et al., 2014; Goodfellow et al., 2015; Carlini &
Revised: Most recent adversarial attack literature has focused on empirical demonstrations of how classiﬁers can be fooled by the addition of quasi-imperceptible noise to the input (Szegedy et al., 2014; Goodfellow et al., 2015; Carlini &

Original: Wagner, 2017; Moosavi-Dezfooli et al., 2016;
Revised: Wagner, 2017; Moosavi-Dezfooli et al., 2016;

Original: Madry et al., 2018; Kurakin et al., 2017).
Revised: Madry et al., 2018; Kurakin et al., 2017).

Original: However, adversarial attacks may be leveraged in other constructive ways to provide insights into how deep learning models learn data representations and make decisions.
Revised: However, adversarial attacks may be leveraged in other constructive ways to provide insights into how deep learning models learn data representations and make decisions.

Original: In this work, we propose a new blackbox transfer-based adversarial attack that outperforms state-of-the-art methods for undefended ImageNet classiﬁers.
Revised: In this work, we propose a new blackbox transfer-based adversarial attack that outperforms state-of-the-art methods for undefended ImageNet classiﬁers.

Original: Importantly, this work provides a broad exploration into how different Deep Neural Network (DNN) models build feature representations and conceptualize classes.
Revised: Importantly, this work provides a broad exploration into how different Deep Neural Network (DNN) models build feature representations and conceptualize classes.

Original: The new attack methodology, which we call the Feature Distribution Attack ( FDA ), leverages class-wise and layerwise deep feature distributions of a substitute DNN to generate adversarial examples that are highly transferable to a blackbox target DNN.
Revised: The new attack methodology, which we call the Feature Distribution Attack ( FDA ), leverages class-wise and layer-wise deep feature distributions of a substitute DNN to generate adversarial examples that are highly transferable to a blackbox target DNN.

Original: One perspective on adversarial attacks is that adversarial noise is a direction in which to “move” the natural data.
Revised: One perspective on adversarial attacks is that adversarial noise is a direction in which to “move” the natural data.

Original: In standard attacks which directly use the classiﬁcation output, the noise points in the direction of the nearest decision boundary at the classiﬁcation layer (Tram`er et al., 2017).
Revised: In standard attacks which directly use the classiﬁcation output, the noise points in the direction of the nearest decision boundary at the classiﬁcation layer (Tram`er et al., 2017).

Original: In this work, our crafted noise points in a direction that makes the data “look like” a sample of another class in intermediate feature space .
Revised: In this work, our crafted noise points in a direction that makes the data “look like” a sample of another class in intermediate feature space .

Original: Intuitively, if we can alter the representation in a layer whose features are representative of the data for the given task, but not speciﬁc to the model, the adversarial example may transfer better (to unobserved architectures) than attacks derived from logit-layer information.
Revised: Intuitively, if we can alter the representation in a layer whose features are representative of the data for the given task, but not speciﬁc to the model, the adversarial example may transfer better (to unobserved architectures) than attacks derived from logit-layer information.

Original: Fig. 1(top) illustrates the feature distribution modeling of a DNN, which is the core mechanism of the attack.
Revised: Figure 1(top) illustrates the feature distribution modeling of a DNN, which is the core mechanism of the attack.

Original: f is a pre-trained substitute whitebox model to which we have full access.
Revised: f is a pre-trained substitute whitebox model to which we have full access.

Original: The true target blackbox model is not shown, but we only assume limited query access and that it has been trained on ImageNet-1k (Deng et al., 2009).
Revised: The true target blackbox model is not shown, but we only assume limited query access and that it has been trained on ImageNet-1k (Deng et al., 2009).

Original: Adversarial examples are then generated on the whitebox model and transferred to the blackbox model.
Revised: Adversarial examples are then generated on the whitebox model and transferred to the blackbox model.

Original: The novelty of the attack comes from the explicit use of class-wise and layer-wise feature distributions.
Revised: The novelty of the attack comes from the explicit use of class-wise and layer-wise feature distributions.

Original: In Fig. 1(top), an auxiliary Neural Network (NN) g l,c learns p ( y = c | f l ( x )) , which is the probability that the layer l features of the whitebox model, extracted from input image x , belong to class c .
Revised: In Figure 1(top), an auxiliary Neural Network (NN) g l,c learns p ( y = c | f l ( x )) , which is the probability that the layer l features of the whitebox model, extracted from input image x , belong to class c .

Original: The attack uses these learned distributions to generate targeted (or untargeted) adversarial examples by maximizing (or minimizing) the probability that the adversarial example is from a particular class’s feature distribution (Fig. 1(bottom)).
Revised: The attack uses these learned distributions to generate targeted (or untargeted) adversarial examples by maximizing (or minimizing) the probability that the adversarial example is from a particular class’s feature distribution (Figure 1(bottom)).

Original: We also use these learned distributions to analyze layer-wise and model-wise transfer properties, and to monitor how perturbations of the input change feature space representations.
Revised: We also use these learned distributions to analyze layer-wise and model-wise transfer properties, and to monitor how perturbations of the input change feature space representations.

Original: Thus, we gain insights on how feature distributions evolve with layer depth and architecture.
Revised: Thus, we gain insights on how feature distributions evolve with layer depth and architecture.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: In blackbox attacks (Narodytska & Kasiviswanathan, 2017; Su et al., 2017; Papernot et al., 2017;
Revised: In blackbox attacks (Narodytska & Kasiviswanathan, 2017; Su et al., 2017; Papernot et al., 2017;

Original: Tram`er et al., 2017; Inkawhich et al., 2019; Dong et al., 2018; Zhou et al., 2018), knowledge of the target model is limited.
Revised: Tram`er et al., 2017; Inkawhich et al., 2019; Dong et al., 2018; Zhou et al., 2018), knowledge of the target model is limited.

Original: In this work, the target model is blackbox in the sense that we do not have access to its gradients and make no assumptions about its architecture (Madry et al., 2018;
Revised: In this work, the target model is blackbox in the sense that we do not have access to its gradients and make no assumptions about its architecture (Madry et al., 2018;

Original: Cheng et al., 2019).
Revised: Cheng et al., 2019).

Original: A popular blackbox technique is transfer-based attacks, in which adversarial examples are constructed on the attackers’ own whitebox model and transferred to the target model.
Revised: A popular blackbox technique is transfer-based attacks, in which adversarial examples are constructed on the attackers’ own whitebox model and transferred to the target model.

Original: Papernot et al.
Revised: Papernot et al.

Original: (2016; 2017) develop special methods for training the attackers’ whitebox model to approximate the target model’s decision boundaries.
Revised: (2016; 2017) develop special methods for training the attackers’ whitebox model to approximate the target model’s decision boundaries.

Original: In this work, we only use models that have been trained under standard conﬁgurations for ImageNet-1k (Deng et al., 2009).
Revised: In this work, we only use models that have been trained under standard conﬁgurations for ImageNet-1k (Deng et al., 2009).

Original: Tram`er et al.
Revised: Tram`er et al.

Original: (2018) and Liu et al.
Revised: (2018) and Liu et al.

Original: (2017) bolster transferability by generating adversarial examples from an ensemble of whitebox models, which helps the noise not overﬁt a single model architecture.
Revised: (2017) bolster transferability by generating adversarial examples from an ensemble of whitebox models, which helps the noise not overﬁt a single model architecture.

Original: Our methods also discourage overﬁtting of the generating architecture, but we instead leverage feature space perturbations at the appropriate layer.
Revised: Our methods also discourage overﬁtting of the generating architecture, but we instead leverage feature space perturbations at the appropriate layer.

Original: In the 2017 NeurIPS blackbox attack competition (Kurakin et al., 2018), the winning method (Dong et al., 2018) used momentum in the optimization step, which helped to speed up the convergence rate and de-noise the gradient directions as to not be overly speciﬁc to the generating architecture.
Revised: In the 2017 NeurIPS blackbox attack competition (Kurakin et al., 2018), the winning method (Dong et al., 2018) used momentum in the optimization step, which helped to speed up the convergence rate and de-noise the gradient directions as to not be overly speciﬁc to the generating architecture.

Original: We also use this approach.
Revised: We also use this approach.

Original: Finally, Tram ` er et al.
Revised: Finally, Tram ` er et al.

Original: (2017) analyze why transferability occurs and ﬁnd that well-trained models have similar decision boundary structures.
Revised: (2017) analyze why transferability occurs and ﬁnd that well-trained models have similar decision boundary structures.

Original: We also analyze transferability, but in the context of how adversarial examples change a model’s internal representations, rather than only making observations at the output layer.
Revised: We also analyze transferability, but in the context of how adversarial examples change a model’s internal representations, rather than only making observations at the output layer.

Original: While all of the above methods generate adversarial examples using information from the classiﬁcation layer of the model, there have been a few recent works delving into the feature space of DNNs for both attacks and defenses.
Revised: While all of the above methods generate adversarial examples using information from the classiﬁcation layer of the model, there have been a few recent works delving into the feature space of DNNs for both attacks and defenses.

Original: Sabour et al.
Revised: Sabour et al.

Original: (2016) show that in whitebox settings, samples can be moved very close together while maintaining their original image-domain representations.
Revised: (2016) show that in whitebox settings, samples can be moved very close together while maintaining their original image-domain representations.

Original: Zhou et al.
Revised: Zhou et al.

Original: (2018) regularize standard untargeted attack objectives to maximize perturbations of (all) intermediate feature maps and increase transferability.
Revised: (2018) regularize standard untargeted attack objectives to maximize perturbations of (all) intermediate feature maps and increase transferability.

Original: However, their primary objective is untargeted and still based on classiﬁcation output information.
Revised: However, their primary objective is untargeted and still based on classiﬁcation output information.

Original: Also, the authors do not consider which layers are affected and how the regularization alters the intermediate representations.
Revised: Also, the authors do not consider which layers are affected and how the regularization alters the intermediate representations.

Original: Inkawhich et al.
Revised: Inkawhich et al.

Original: (2019) show that driving a source sample’s feature representation towards a target sample’s representation at particular layers in deep feature space is an effective method of targeted transfer attack.
Revised: (2019) show that driving a source sample’s feature representation towards a target sample’s representation at particular layers in deep feature space is an effective method of targeted transfer attack.

Original: However, the method is targeted only and relies on the selection of a single (carefully selected) sample of the target class.
Revised: However, the method is targeted only and relies on the selection of a single (carefully selected) sample of the target class.

Original: Also, the attack success rate on ImageNet was empirically low.
Revised: Also, the attack success rate on ImageNet was empirically low.

Original: This work describes a more robust attack, with signiﬁcantly better performance on ImageNet, and provides a more detailed analysis of layer-wise transfer properties.
Revised: This work describes a more robust attack, with signiﬁcantly better performance on ImageNet, and provides a more detailed analysis of layer-wise transfer properties.

Original: For adversarial defenses, Xie et al.
Revised: For adversarial defenses, Xie et al.

Original: Frosst et al.
Revised: Frosst et al.

Original: (2019), and Lin et al.
Revised: (2019), and Lin et al.

Original: (2019) consider the effects of adversarial perturbations in feature space but do not perform a layer-wise analysis of how the internal representations are affected.
Revised: (2019) consider the effects of adversarial perturbations in feature space but do not perform a layer-wise analysis of how the internal representations are affected.

Original:  A TTACK METHODOLOGY
Revised:  A TTACK METHODOLOGY

Original: We assume to have a set of training data and a pre-trained model f from the same task as the target blackbox model (i.e. the ImageNet-1k training set and a pre-trained ImageNet model).
Revised: We assume to have a set of training data and a pre-trained model f from the same task as the target blackbox model (i.e. the ImageNet-1k training set and a pre-trained ImageNet model).

Original: To model the feature distributions for f , we identify a set of classes C
Revised: To model the feature distributions for f , we identify a set of classes C

Original: = { c 1 , ..., c K } and a set of layers
Revised: = { c 1 , ..., c K } and a set of layers

Original: L = { l 1 , ..., l N } that we are keen to probe.
Revised: L = { l 1 , ..., l N } that we are keen to probe.

Original: For each layer in L , we train a small, binary, one-versusall classiﬁer g for each of the classes in C , as shown in Fig. 1 (top).
Revised: For each layer in L , we train a small, binary, one-versusall classiﬁer g for each of the classes in C , as shown in Figure 1 (top).

Original: Each binary classiﬁer is given a unique set of parameters, and referred to as an auxiliary model g l,c .
Revised: Each binary classiﬁer is given a unique set of parameters, and referred to as an auxiliary model g l,c .

Original: The output of an auxiliary model represents the probability that the input feature map is from a speciﬁc class c ∈ C .
Revised: The output of an auxiliary model represents the probability that the input feature map is from a speciﬁc class c ∈ C .

Original: Thus, we say that g l,c ( f l ( x )) outputs p ( y = c
Revised: Thus, we say that g l,c ( f l ( x )) outputs p ( y = c

Original: | f l ( x )) , where f l ( x ) is the layer l feature map of the pre-trained model f given input image x .
Revised: | f l ( x )) , where f l ( x ) is the layer l feature map of the pre-trained model f given input image x .

Original: Once trained, we may leverage the learned feature distributions to create both targeted and untargeted adversarial examples.
Revised: Once trained, we may leverage the learned feature distributions to create both targeted and untargeted adversarial examples.

Original: Here, we focus mostly on targeted attacks, which are considered a harder problem (especially in the blackbox transfer case when we do not have access to the target model’s gradients)
Revised: Here, we focus mostly on targeted attacks, which are considered a harder problem (especially in the blackbox transfer case when we do not have access to the target model’s gradients)

Original: (Kurakin et al., 2018; Sharma et al., 2018).
Revised: (Kurakin et al., 2018; Sharma et al., 2018).

Original: Discussion of untargeted attacks is left to Appendix C. Recall, the goal of a targeted attack is to generate an adversarial noise δ that when added to a clean sample x of class y src , the classiﬁcation result of x + δ is a chosen class y tgt .
Revised: Discussion of untargeted attacks is left to Appendix C. Recall, the goal of a targeted attack is to generate an adversarial noise δ that when added to a clean sample x of class y src , the classiﬁcation result of x + δ is a chosen class y tgt .

Original: The key intuition for our targeted methods is that if a sample has features consistent with the feature distribution of class c at some layer of intermediate feature space, then it will likely be classiﬁed as class c .
Revised: The key intuition for our targeted methods is that if a sample has features consistent with the feature distribution of class c at some layer of intermediate feature space, then it will likely be classiﬁed as class c .

Original: Although not shown in the objective functions for simplicity, for all attacks the adversarial noise δ is constrained by an (cid:96) p norm (i.e. || δ || p ≤ (cid:15) ), and the choice of layer l and target class label y tgt are chosen prior to optimization.
Revised: Although not shown in the objective functions for simplicity, for all attacks the adversarial noise δ is constrained by an (cid:96) p norm (i.e. || δ || p ≤ (cid:15) ), and the choice of layer l and target class label y tgt are chosen prior to optimization.

Original: FDA We propose three targeted attack variants.
Revised: FDA We propose three targeted attack variants.

Original: The most straightforward variant, called FDA , ﬁnds a perturbation δ of the “clean” input image x that maximizes the probability that the layer l features are from the target class y tgt distribution:
Revised: The most straightforward variant, called FDA , ﬁnds a perturbation δ of the “clean” input image x that maximizes the probability that the layer l features are from the target class y tgt distribution:

Original: [Equation]
Revised: [Equation]

Original: We stress that unlike standard attacks that use output layer information to directly cross decision boundaries of the whitebox, our FDA objective leverages intermediate feature distributions which do not implicitly describe these exact boundaries.
Revised: We stress that unlike standard attacks that use output layer information to directly cross decision boundaries of the whitebox, our FDA objective leverages intermediate feature distributions which do not implicitly describe these exact boundaries.

Original: FDA+ms
Revised: FDA+ms

Original: In addition to maximizing the probability that the layer l features are from the target class distribution, the FDA+ms variant also considers minimizing the probability that the layer l features are from the source class y src distribution ( ms = minimize source): 
Revised: In addition to maximizing the probability that the layer l features are from the target class distribution, the FDA+ms variant also considers minimizing the probability that the layer l features are from the source class y src distribution ( ms = minimize source): max

Original: [Equation]
Revised: [Equation]

Original: Here, λ ∈ (0 , 1) weights the contribution of both terms and is a ﬁxed positive value.
Revised: Here, λ ∈ (0 , 1) weights the contribution of both terms and is a ﬁxed positive value.

Original: FDA+fd
Revised: FDA+fd

Original: Similarly, the FDA+fd variant maximizes the probability that the layer l features are from the target class distribution while also maximizing the distance of the perturbed features from the original features ( fd = feature-disruption):
Revised: Similarly, the FDA+fd variant maximizes the probability that the layer l features are from the target class distribution while also maximizing the distance of the perturbed features from the original features ( fd = feature-disruption):

Original: [Equation]
Revised: [Equation]

Original: In other words, the feature-disruption term, with a ﬁxed η ∈ R + , prioritizes making the layer l features of the perturbed sample maximally different from the original sample.
Revised: In other words, the feature-disruption term, with a ﬁxed η ∈ R + , prioritizes making the layer l features of the perturbed sample maximally different from the original sample.

Original: The additional terms in FDA+ms and FDA+fd encourage the adversarial sample to move far away from the starting point, which may intuitively help in generating (targeted) adversarial examples.
Revised: The additional terms in FDA+ms and FDA+fd encourage the adversarial sample to move far away from the starting point, which may intuitively help in generating (targeted) adversarial examples.

Original: Also, notice that FDA+ms requires the modeling of both the source and target class distributions, whereas the others only require the modeling of the target class distribution.
Revised: Also, notice that FDA+ms requires the modeling of both the source and target class distributions, whereas the others only require the modeling of the target class distribution.

Original: Optimization Procedure.
Revised: Optimization Procedure.

Original: The trained auxiliary models afford a way to construct a fully differentiable path for gradient-based optimization of the objective functions.
Revised: The trained auxiliary models afford a way to construct a fully differentiable path for gradient-based optimization of the objective functions.

Original: Speciﬁcally, to compute FDA adversarial noise from layer l , we ﬁrst build a composite model using the truncated whitebox model f l and the corresponding layer’s auxiliary model
Revised: Speciﬁcally, to compute FDA adversarial noise from layer l , we ﬁrst build a composite model using the truncated whitebox model f l and the corresponding layer’s auxiliary model

Original: g l,c = y tgt for the target class y tgt , as shown in Fig. 1(bottom).
Revised: g l,c = y tgt for the target class y tgt , as shown in Figure 1(bottom).

Original: The loss is calculated as the Binary Cross Entropy (BCELoss) between the predicted p ( y = y tgt | f l ( x )) and 1.
Revised: The loss is calculated as the Binary Cross Entropy (BCELoss) between the predicted p ( y = y tgt | f l ( x )) and 1.

Original: Thus, we perturb the input image in the direction that will minimize the loss, in turn maximizing p ( y = y tgt | f l ( x )) .
Revised: Thus, we perturb the input image in the direction that will minimize the loss, in turn maximizing p ( y = y tgt | f l ( x )) .

Original: For optimization, we employ iterative gradient descent with momentum, as the inclusion of a momentum term in adversarial attacks has proven effective (Inkawhich et al., 2019; Dong et al., 2018).
Revised: For optimization, we employ iterative gradient descent with momentum, as the inclusion of a momentum term in adversarial attacks has proven effective (Inkawhich et al., 2019; Dong et al., 2018).

Original: See Appendix D for more details.
Revised: See Appendix D for more details.

Original:  E XPERIMENTAL SETUP
Revised:  E XPERIMENTAL SETUP

Original: ImageNet models.
Revised: ImageNet models.

Original: For evaluation we use popular CNN architectures designed for the ImageNet-1k (Deng et al., 2009) classiﬁcation task: VGG-19 with batch-normalization (VGG19) (Simonyan &
Revised: For evaluation we use popular CNN architectures designed for the ImageNet-1k (Deng et al., 2009) classiﬁcation task: VGG-19 with batch-normalization (VGG19) (Simonyan &

Original: Zisserman, 2015), DenseNet-121 (DN121) (Huang et al., 2017), and ResNet-50 (RN50) (He et al.,
Revised: Zisserman, 2015), DenseNet-121 (DN121) (Huang et al., 2017), and ResNet-50 (RN50) (He et al.,

Original: All models are pre-trained and found in the PyTorch Model Zoo.
Revised: All models are pre-trained and found in the PyTorch Model Zoo.

Original: Note, our methods are in no way speciﬁc to these particular models/architectures.
Revised: Note, our methods are in no way speciﬁc to these particular models/architectures.

Original: We also emphasize transfers across different architectures rather than showing results between models from the same family.
Revised: We also emphasize transfers across different architectures rather than showing results between models from the same family.

Original: Layer decoding scheme .
Revised: Layer decoding scheme .

Original: Given a pre-trained model, we must choose a set of layers L to probe.
Revised: Given a pre-trained model, we must choose a set of layers L to probe.

Original: For each model we subsample the layers such that we probe across the depth.
Revised: For each model we subsample the layers such that we probe across the depth.

Original: For notation we use relative layer numbers, so layer 0 of DN121 ( DN 121 l =0 ) is near the input layer and DN 121 l =12 is closer to the classiﬁcation layer.
Revised: For notation we use relative layer numbers, so layer 0 of DN121 ( DN 121 l =0 ) is near the input layer and DN 121 l =12 is closer to the classiﬁcation layer.

Original: For all models, the deepest layer probed is the logit layer.
Revised: For all models, the deepest layer probed is the logit layer.

Original: Appendix
Revised: Appendix

Original: A decodes the notation for each model.
Revised: A decodes the notation for each model.

Original: Auxiliary model training.
Revised: Auxiliary model training.

Original: We must also choose a set of classes C that we are interested in modeling.
Revised: We must also choose a set of classes C that we are interested in modeling.

Original: Recall, the number of auxiliary models required for a given base model is the number of layers probed multiplied by the number of classes we are interested in modeling.
Revised: Recall, the number of auxiliary models required for a given base model is the number of layers probed multiplied by the number of classes we are interested in modeling.

Original: Attempting to model the feature distributions for all 1000 ImageNet classes for each layer is expensive, so we instead choose to run the majority of tests with a set of 10 randomly chosen classes (which are meant to be representative of the entire dataset): 24:“grey-owl”, 99:“goose”, 245:“bulldog”, 344:“hippo”, 471:“cannon”, 555:“ﬁre-truck”, 661:“Model-T”, 701:“parachute”, 802:“snowmobile”, 919:“streetsign”.
Revised: Attempting to model the feature distributions for all 1000 ImageNet classes for each layer is expensive, so we instead choose to run the majority of tests with a set of 10 randomly chosen classes (which are meant to be representative of the entire dataset): 24:“grey-owl”, 99:“goose”, 245:“bulldog”, 344:“hippo”, 471:“cannon”, 555:“ﬁre-truck”, 661:“Model-T”, 701:“parachute”, 802:“snowmobile”, 919:“streetsign”.

Original: Thus, for each layer of each model we train 10 auxiliary classiﬁers, one for each class.
Revised: Thus, for each layer of each model we train 10 auxiliary classiﬁers, one for each class.

Original: After identifying high performing attack settings, we then produce results for all 1000 classes.
Revised: After identifying high performing attack settings, we then produce results for all 1000 classes.

Original: The architecture of all auxiliary models is the same, regardless of model, layer, or class.
Revised: The architecture of all auxiliary models is the same, regardless of model, layer, or class.

Original: Each is a 2-hidden layer NN with a single output unit.
Revised: Each is a 2-hidden layer NN with a single output unit.

Original: There are 200 neurons in each hidden layer and the number of input units matches the size of the input feature map.
Revised: There are 200 neurons in each hidden layer and the number of input units matches the size of the input feature map.

Original: To train the auxiliary models, unbiased batches from the whole ImageNet-1k training set are pushed through the truncated pretrained model ( f l ), and the extracted features are used to train the auxiliary model parameters.
Revised: To train the auxiliary models, unbiased batches from the whole ImageNet-1k training set are pushed through the truncated pretrained model ( f l ), and the extracted features are used to train the auxiliary model parameters.

Original: Experimental procedure.
Revised: Experimental procedure.

Original: Since we have three pre-trained models, there are 6 blackbox transfer scenarios to evaluate (no self-transfers).
Revised: Since we have three pre-trained models, there are 6 blackbox transfer scenarios to evaluate (no self-transfers).

Original: We use the ImageNet-1K validation set as the test dataset.
Revised: We use the ImageNet-1K validation set as the test dataset.

Original: Because FDA+ms requires both the source and target class distributions, for the targeted attack evaluations we only use source samples from the 10 trained classes, and for each sample, target each of the other 9 classes.
Revised: Because FDA+ms requires both the source and target class distributions, for the targeted attack evaluations we only use source samples from the 10 trained classes, and for each sample, target each of the other 9 classes.

Original: For baseline attacks, we use targeted random-start Projected Gradient Descent (tpgd) (Madry et al., 2018; Kurakin et al., 2018), targeted NeurIPS2017 competition winning momentum iterative method (tmim)
Revised: For baseline attacks, we use targeted random-start Projected Gradient Descent (tpgd) (Madry et al., 2018; Kurakin et al., 2018), targeted NeurIPS2017 competition winning momentum iterative method (tmim)

Original: (Dong et al., 2018), and the Activation Attack (AA) (Inkawhich et al., 2019).
Revised: (Dong et al., 2018), and the Activation Attack (AA) (Inkawhich et al., 2019).

Original: Further, all targeted adversarial examples are constrained by (cid:96) ∞ (cid:15)
Revised: Further, all targeted adversarial examples are constrained by (cid:96) ∞ (cid:15)

Original: = 16 / 255 as described in (Dong et al., 2018; Kurakin et al., 2018).
Revised: = 16 / 255 as described in (Dong et al., 2018; Kurakin et al., 2018).

Original: As experimentally found, λ = 0 .
Revised: As experimentally found, λ = 0 .

Original: 8 in (2) and η = 1 e-
Revised: 8 in (2) and η = 1 e-

Original: 6 in (3).
Revised: 6 in (3).

Original: Finally, as measured over the initially correctly classiﬁed subset of the test dataset (by both the whitebox and blackbox models), attack success is captured in two metrics.
Revised: Finally, as measured over the initially correctly classiﬁed subset of the test dataset (by both the whitebox and blackbox models), attack success is captured in two metrics.

Original: Error is the percentage of examples that the blackbox misclassiﬁes and Targeted Success Rate (tSuc) is the percentage of examples that the blackbox misclassiﬁes as the target label.
Revised: Error is the percentage of examples that the blackbox misclassiﬁes and Targeted Success Rate (tSuc) is the percentage of examples that the blackbox misclassiﬁes as the target label.

Original:  E MPIRICAL RESULTS 5. 10- CLASS I MAGE N ET RESULTS
Revised:  E MPIRICAL RESULTS 5. 10- CLASS I MAGE N ET RESULTS

Original: The primary axis of interest is how attack success rate varies with the layer depth from which the feature distributions are attacked.
Revised: The primary axis of interest is how attack success rate varies with the layer depth from which the feature distributions are attacked.

Original: Fig. 2 shows the transfer results between all pairs of whitebox and blackbox models.
Revised: Figure 2 shows the transfer results between all pairs of whitebox and blackbox models.

Original: Each plot shows a metric of attack success versus relative layer depth of the generated attack.
Revised: Each plot shows a metric of attack success versus relative layer depth of the generated attack.

Original: The notation DN121 → VGG19 indicates adversarial examples were generated with a DN121 whitebox model and transferred to a VGG19 blackbox model.
Revised: The notation DN121 → VGG19 indicates adversarial examples were generated with a DN121 whitebox model and transferred to a VGG19 blackbox model.

Original: Similar to Inkawhich et al.
Revised: Similar to Inkawhich et al.

Original: (2019), transferability trends for FDAs from a given whitebox model appear blackbox model agnostic (e.g. the shape of the curves from DN121 →
Revised: (2019), transferability trends for FDAs from a given whitebox model appear blackbox model agnostic (e.g. the shape of the curves from DN121 →

Original: RN50 are the same as DN121 → VGG19).
Revised: RN50 are the same as DN121 → VGG19).

Original: This is a positive property, as once the optimal transfer layer for a whitebox model is found, evidence shows it will be the same for any blackbox model architecture.
Revised: This is a positive property, as once the optimal transfer layer for a whitebox model is found, evidence shows it will be the same for any blackbox model architecture.

Original: Further, the most powerful transfers come from perturbations of intermediate features, rather than perturbations of classiﬁcation layer information.
Revised: Further, the most powerful transfers come from perturbations of intermediate features, rather than perturbations of classiﬁcation layer information.

Original: Another global trend is that in tSuc, FDA+fd performs best, FDA performs worst, FDA+ms is in-between, and all FDAs signiﬁcantly outperform the other baselines.
Revised: Another global trend is that in tSuc, FDA+fd performs best, FDA performs worst, FDA+ms is in-between, and all FDAs signiﬁcantly outperform the other baselines.

Original: In the error metric, FDA+fd is best in early layers, FDA+ms is best in later layers, and FDA routinely under-performs the AA baseline.
Revised: In the error metric, FDA+fd is best in early layers, FDA+ms is best in later layers, and FDA routinely under-performs the AA baseline.

Original: Although all attacks are targeted, it is relevant to report error as it is still an indication of attack strength.
Revised: Although all attacks are targeted, it is relevant to report error as it is still an indication of attack strength.

Original: Also, it is clearly beneﬁcial for the targeted attack objective to include a term that encourages the adversarial example to move far away from its starting place ( FDA+ms & FDA+fd ) in addition to moving toward the target region.
Revised: Also, it is clearly beneﬁcial for the targeted attack objective to include a term that encourages the adversarial example to move far away from its starting place ( FDA+ms & FDA+fd ) in addition to moving toward the target region.

Original: [Figure]
Revised: [Figure]

Original: We now compare performance across whitebox models.
Revised: We now compare performance across whitebox models.

Original: For a DN121 whitebox, FDA+fd from
Revised: For a DN121 whitebox, FDA+fd from

Original: DN 121 l =7 is the optimal targeted attack with an average tSuc of 34%.
Revised: DN 121 l =7 is the optimal targeted attack with an average tSuc of 34%.

Original: For DN121 → RN50, this attack outperforms the best baseline by 14% and 32% in error and tSuc, respectively.
Revised: For DN121 → RN50, this attack outperforms the best baseline by 14% and 32% in error and tSuc, respectively.

Original: For a VGG whitebox, FDA+fd from V GG 19 l =5 is the optimal targeted attack with an average tSuc of 2.3%.
Revised: For a VGG whitebox, FDA+fd from V GG 19 l =5 is the optimal targeted attack with an average tSuc of 2.3%.

Original: For VGG19 → RN50, this attack outperforms the best baseline by 15% and 2% in error and tSuc, respectively.
Revised: For VGG19 → RN50, this attack outperforms the best baseline by 15% and 2% in error and tSuc, respectively.

Original: For a RN50 whitebox, FDA+fd from RN
Revised: For a RN50 whitebox, FDA+fd from RN

Original: =8 is the optimal targeted attack with an average tSuc of 18%.
Revised: =8 is the optimal targeted attack with an average tSuc of 18%.

Original: , this attack outperforms the best baseline by 27% and 17% 
Revised: For RN50 → DN121, this attack outperforms the best baseline by 27% and 17% in error and tSuc, respectively.

Original: 5. 1000- CLASS I MAGE N ET RESULTS
Revised: 5. 1000- CLASS I MAGE N ET RESULTS

Original: [Table]
Revised: [Table]

Original: Recall, due to the computational complexity of training one auxiliary model per class per layer per model, we ran the previous experiments using 10 randomly sampled ImageNet-1k classes.
Revised: Recall, due to the computational complexity of training one auxiliary model per class per layer per model, we ran the previous experiments using 10 randomly sampled ImageNet-1k classes.

Original: In reality, this may be a realistic attack scenario because an adversary would likely only be interested in attacking certain source-target pairs.
Revised: In reality, this may be a realistic attack scenario because an adversary would likely only be interested in attacking certain source-target pairs.

Original: However, to show that the 10 chosen classes are not special, and the previously identiﬁed optimal transfer layers are still valid, we train all 1000 class auxiliary models for DN 121 l =7 and RN 50 l =8 .
Revised: However, to show that the 10 chosen classes are not special, and the previously identiﬁed optimal transfer layers are still valid, we train all 1000 class auxiliary models for DN 121 l =7 and RN 50 l =8 .

Original: We exclude VGG19 because of its inferior performance in previous tests.
Revised: We exclude VGG19 because of its inferior performance in previous tests.

Original: Table 1 shows results for the four transfer scenarios.
Revised: Table 1 shows results for the four transfer scenarios.

Original: Attack success rates are all averaged over four random 10k splits and the standard deviation of all measurements is less than 1%.
Revised: Attack success rates are all averaged over four random 10k splits and the standard deviation of all measurements is less than 1%.

Original: In these tests, for each source sample, a random target class is chosen.
Revised: In these tests, for each source sample, a random target class is chosen.

Original: As expected, the 1000-class results closely match the previously reported 10-class results.
Revised: As expected, the 1000-class results closely match the previously reported 10-class results.

Original:  A NALYSIS OF TRANSFER PROPERTIES
Revised:  A NALYSIS OF TRANSFER PROPERTIES

Original: We now investigate why a given layer and/or whitebox model is better for creating transferable adversarial examples.
Revised: We now investigate why a given layer and/or whitebox model is better for creating transferable adversarial examples.

Original: We also explore the hypothesis that the layer-wise transfer properties of a
Revised: We also explore the hypothesis that the layer-wise transfer properties of a

Original: DNN implicate the transition of intermediate features from task/data-speciﬁc to model-speciﬁc .
Revised: DNN implicate the transition of intermediate features from task/data-speciﬁc to model-speciﬁc .

Original: Intuitively, early layers of DNNs trained for classiﬁcation may be working to optimally construct a task/data-speciﬁc feature set (Zeiler & Fergus, 2014;
Revised: Intuitively, early layers of DNNs trained for classiﬁcation may be working to optimally construct a task/data-speciﬁc feature set (Zeiler & Fergus, 2014;

Original: Yosinski et al., 2014).
Revised: Yosinski et al., 2014).

Original: However, once the necessary feature hierarchy is built to model the data, further layers may perform extra processing to best suit the classiﬁcation functionality of the model.
Revised: However, once the necessary feature hierarchy is built to model the data, further layers may perform extra processing to best suit the classiﬁcation functionality of the model.

Original: This additional processing may be what makes the features model-speciﬁc.
Revised: This additional processing may be what makes the features model-speciﬁc.

Original: We posit that the peak of the tSuc curve for a given transfer directly encodes the inﬂection point from task/data-speciﬁc to model-speciﬁc features in the whitebox model.
Revised: We posit that the peak of the tSuc curve for a given transfer directly encodes the inﬂection point from task/data-speciﬁc to model-speciﬁc features in the whitebox model.

Original: Instinctively, to achieve targeted attack success, the layer at which the attacks are generated must have captured the concepts of the classes for the general task of classiﬁcation, without being overly speciﬁc to the architecture.
Revised: Instinctively, to achieve targeted attack success, the layer at which the attacks are generated must have captured the concepts of the classes for the general task of classiﬁcation, without being overly speciﬁc to the architecture.

Original: Thus, layers prior to the inﬂection point may not have solidiﬁed the class concepts, whereas layers after the inﬂection point may have established the class concepts and are further processing them for the model output.
Revised: Thus, layers prior to the inﬂection point may not have solidiﬁed the class concepts, whereas layers after the inﬂection point may have established the class concepts and are further processing them for the model output.

Original: This may also be considered an extension of the convergent learning theory of Li et al.
Revised: This may also be considered an extension of the convergent learning theory of Li et al.

Original: and general-to-speciﬁc theory of Yosinski et al.
Revised: and general-to-speciﬁc theory of Yosinski et al.

Original: 6. I NTERMEDIATE DISRUPTION
Revised: 6. I NTERMEDIATE DISRUPTION

Original: One way to measure why and how adversarial attacks work is to observe how the intermediate representations change as a result of perturbations to the input.
Revised: One way to measure why and how adversarial attacks work is to observe how the intermediate representations change as a result of perturbations to the input.

Original: Our trained auxiliary models afford a novel way to monitor the effects of such perturbations in deep feature space.
Revised: Our trained auxiliary models afford a novel way to monitor the effects of such perturbations in deep feature space.

Original: To measure how much a layer’s features have changed as a result of a (targeted) adversarial perturbation, we deﬁne layerwise disruption as the difference between the target class probability before and after perturbation, as measured in layer l of model f : disruption = p ( y =
Revised: To measure how much a layer’s features have changed as a result of a (targeted) adversarial perturbation, we deﬁne layerwise disruption as the difference between the target class probability before and after perturbation, as measured in layer l of model f : disruption = p ( y =

Original: y tgt | f l ( x + δ ))
Revised: y tgt | f l ( x + δ ))

Original: − p ( y = y
Revised: − p ( y = y

Original: tgt
Revised: tgt

Original: | f l ( x )) .
Revised: | f l ( x )) .

Original: Fig. 3 shows the average 
Revised: Figure 3 shows the average disruption caused

Original: [Figure]
Revised: [Figure]

Original: It appears FDAs generated from early layers (e [Figure] g. DN 121 l =0 , V GG 19 l =0 , RN 50 l =0 ) disrupt features the most in early layers and less so in deeper layers.
Revised: It appears FDAs generated from early layers (e.g. DN 121 l =0 , V GG 19 l =0 , RN 50 l =0 ) disrupt features the most in early layers and less so in deeper layers.

Original: Therefore, a sample resembling class y tgt in an early layer does not mean it will ultimately be classiﬁed as y tgt .
Revised: Therefore, a sample resembling class y tgt in an early layer does not mean it will ultimately be classiﬁed as y tgt .

Original: However, recall from
Revised: However, recall from

Original: Fig. 2 that attacks from early layers create very powerful untargeted adversarial examples (error).
Revised: Figure 2 that attacks from early layers create very powerful untargeted adversarial examples (error).

Original: This indicates that early layer perturbations are ampliﬁed as they proceed through the model (Lin et al., 2019), just not in a class-speciﬁc manner.
Revised: This indicates that early layer perturbations are ampliﬁed as they proceed through the model (Lin et al., 2019), just not in a class-speciﬁc manner.

Original: Next, as expected, attacks that use information from the last layers of a whitebox model (e.g. tmim , DN 121 l =13 , V GG 19 l =9 , RN 50 l =12 ) create the largest disruption in the last layers of the whitebox, but not necessarily at the last layers of the blackbox models.
Revised: Next, as expected, attacks that use information from the last layers of a whitebox model (e.g. tmim , DN 121 l =13 , V GG 19 l =9 , RN 50 l =12 ) create the largest disruption in the last layers of the whitebox, but not necessarily at the last layers of the blackbox models.

Original: However, the optimal transfer attacks ( DN 121 l =7 , V GG 19 l =5 , RN 50 l =8 ) have high disruption all throughout the models, not just at the last layer.
Revised: However, the optimal transfer attacks ( DN 121 l =7 , V GG 19 l =5 , RN 50 l =8 ) have high disruption all throughout the models, not just at the last layer.

Original: This is further evidence that perturbations of classiﬁcation-layer features are overly model-speciﬁc and perturbations of optimal transfer-layer features are more speciﬁc to the data/task.
Revised: This is further evidence that perturbations of classiﬁcation-layer features are overly model-speciﬁc and perturbations of optimal transfer-layer features are more speciﬁc to the data/task.

Original: Finally, notice that the maximum disruption caused in any blackbox model layer from VGG19 whitebox transfers is around 40% (row 2).
Revised: Finally, notice that the maximum disruption caused in any blackbox model layer from VGG19 whitebox transfers is around 40% (row 2).

Original: For the DN121 and RN50 whitebox models, the maximum disruption is around 80%.
Revised: For the DN121 and RN50 whitebox models, the maximum disruption is around 80%.

Original: This may explain
Revised: This may explain

Original: VGG19 being an inferior whitebox model to transfer from, as the perturbation of intermediate VGG features does not in-turn cause signiﬁcant disruption of blackbox model features.
Revised: VGG19 being an inferior whitebox model to transfer from, as the perturbation of intermediate VGG features does not in-turn cause signiﬁcant disruption of blackbox model features.

Original: 6. A UXILIARY MODEL CORRELATION WITH FULL MODEL
Revised: 6. A UXILIARY MODEL CORRELATION WITH FULL MODEL

Original: [Figure]
Revised: [Figure]

Original: Another point of analysis is to investigate the correlation/discrepancy between the auxiliary models at a given layer and the output of the whitebox model.
Revised: Another point of analysis is to investigate the correlation/discrepancy between the auxiliary models at a given layer and the output of the whitebox model.

Original: This may also indicate a transition from task/data-speciﬁc to model-speciﬁc features.
Revised: This may also indicate a transition from task/data-speciﬁc to model-speciﬁc features.

Original: We discuss discrepancy as an indication of how different the auxiliary model outputs are from the whitebox model outputs.
Revised: We discuss discrepancy as an indication of how different the auxiliary model outputs are from the whitebox model outputs.

Original: Then correlation is the inverse of discrepancy so that when the auxiliary model outputs align well with the whitebox model outputs, the discrepancy is low and correlation is high.
Revised: Then correlation is the inverse of discrepancy so that when the auxiliary model outputs align well with the whitebox model outputs, the discrepancy is low and correlation is high.

Original: To evaluate discrepancy at a given layer l for input x , we aggregate the logit values (i.e. pre-sigmoid/softmax) for each class in C , as measured by the auxiliary models g l,c and the whitebox model f , into separate vectors.
Revised: To evaluate discrepancy at a given layer l for input x , we aggregate the logit values (i.e. pre-sigmoid/softmax) for each class in C , as measured by the auxiliary models g l,c and the whitebox model f , into separate vectors.

Original: Then, a softmax ( smax ) operation is performed on each vector to establish two proper probability distributions over the classes in C .
Revised: Then, a softmax ( smax ) operation is performed on each vector to establish two proper probability distributions over the classes in C .

Original: Discrepancy is then deﬁned as the Kullback-Leibler divergence ( D KL ) between the two distributions, or discrepancy =
Revised: Discrepancy is then deﬁned as the Kullback-Leibler divergence ( D KL ) between the two distributions, or discrepancy =

Original: D KL (cid:0) smax( (cid:2) g l,c ( f l ( x )) (cid:3) ∀ c ∈C ) (cid:13)(cid:13) smax( (cid:2) f
Revised: D KL (cid:0) smax( (cid:2) g l,c ( f l ( x )) (cid:3) ∀ c ∈C ) (cid:13)(cid:13) smax( (cid:2) f

Original: ( x )[ c ] (cid:3) ∀ c ∈C ) (cid:1) .
Revised: ( x )[ c ] (cid:3) ∀ c ∈C ) (cid:1) .

Original: Here, f ( x )[ c ] is the class c logit value from the whitebox model f
Revised: Here, f ( x )[ c ] is the class c logit value from the whitebox model f

Original: given input x .Fig.
Revised: given input x .

Original:  4 shows the layerwise auxiliary model correlations with the whitebox model outputs as measured from the average discrepancy over 500 input samples of classes in C .
Revised: Figure 4 shows the layer-wise auxiliary model correlations with the whitebox model outputs as measured from the average discrepancy over 500 input samples of classes in C .

Original: Note, the shapes of the curves are more informative than the actual values.
Revised: Note, the shapes of the curves are more informative than the actual values.

Original: Also, VGG19 layers have been shifted in notation by +4 so that layer depth 13 is the logit layer of each model.
Revised: Also, VGG19 layers have been shifted in notation by +4 so that layer depth 13 is the logit layer of each model.

Original: As expected, the auxiliary models in early layers have little correlation with the model output, while auxiliary models in later layers have high correlation with the model output.
Revised: As expected, the auxiliary models in early layers have little correlation with the model output, while auxiliary models in later layers have high correlation with the model output.

Original: Importantly, the optimal transfer layers ( (cid:63) ) mark a transition in the trendlines after which correlation increases sharply.
Revised: Importantly, the optimal transfer layers ( (cid:63) ) mark a transition in the trendlines after which correlation increases sharply.

Original: This effect may directly explain why layers after the optimal layer are suboptimal, because the auxiliary models become highly correlated with the model output and begin to overﬁt the architecture.
Revised: This effect may directly explain why layers after the optimal layer are suboptimal, because the auxiliary models become highly correlated with the model output and begin to overﬁt the architecture.

Original: Since the auxiliary models are not highly-correlated with the output layer at the optimal-transfer-layers, we may surmise that the captured features are still mostly task/data speciﬁc.
Revised: Since the auxiliary models are not highly-correlated with the output layer at the optimal-transfer-layers, we may surmise that the captured features are still mostly task/data speciﬁc.

Original: 6. A UXILIARY MODEL SALIENCY
Revised: 6. A UXILIARY MODEL SALIENCY

Original: For a more qualitative analysis, we may inspect the auxiliary model saliency maps.
Revised: For a more qualitative analysis, we may inspect the auxiliary model saliency maps.

Original: Given an image of class y src , we visualize in Fig. 5 what is salient to the y src auxiliary models at several DN layer depths using SmoothGrad (Smilkov et al., 2017)
Revised: Given an image of class y src , we visualize in Figure 5 what is salient to the y src auxiliary models at several DN layer depths using SmoothGrad (Smilkov et al., 2017)

Original: (see Appendix E for additional saliency examples for RN50).
Revised: (see Appendix E for additional saliency examples for RN50).

Original: Notice, an observable transition occurs at the high-performing transfer layers from Fig. 2 ( DN 121 l =5 , 7 ).
Revised: Notice, an observable transition occurs at the high-performing transfer layers from Figure 2 ( DN 121 l =5 , 7 ).

Original: The salient regions move from large areas around the whole image (e.g. DN 121 l =0 , 3 ) to speciﬁc regions that are also salient in the classiﬁcation layer ( DN 121 l =13 ).
Revised: The salient regions move from large areas around the whole image (e.g. DN 121 l =0 , 3 ) to speciﬁc regions that are also salient in the classiﬁcation layer ( DN 121 l =13 ).

Original: The saliency and correlation transitions together show that the well-transferring layers have learned similar salient features as the classiﬁcation layer while not being overly correlated with the model output.
Revised: The saliency and correlation transitions together show that the well-transferring layers have learned similar salient features as the classiﬁcation layer while not being overly correlated with the model output.

Original: Therefore, perturbations focused on these salient regions signiﬁcantly impact the ﬁnal classiﬁcation without being too speciﬁc to the generating architecture.
Revised: Therefore, perturbations focused on these salient regions signiﬁcantly impact the ﬁnal classiﬁcation without being too speciﬁc to the generating architecture.

Original: 6. C LASS D ISTRIBUTION S EPARABILITY

Original: [Figure]
Revised: [Figure]

Original: Finally, our trained auxiliary models afford new ways of measuring class-wise feature entanglement/separability for the purpose of explaining transfer performance.
Revised: Finally, our trained auxiliary models afford new ways of measuring class-wise feature entanglement/separability for the purpose of explaining transfer performance.

Original: We adopt the definition of entanglement from Frosst et al.
Revised: We adopt the definition of entanglement from Frosst et al.

Original: (2019) which states that highly entangled features have a “lack of separation of class manifolds in representation space,”
Revised: (2019) which states that highly entangled features have a “lack of separation of class manifolds in representation space,”

Original: and deﬁne separability as the inverse of entanglement.
Revised: and deﬁne separability as the inverse of entanglement.

Original: One way to measure the separability between class distributions in a layer using the auxiliary models is to gauge how far a sample has to “move” to enter a region of high class conﬁdence.
Revised: One way to measure the separability between class distributions in a layer using the auxiliary models is to gauge how far a sample has to “move” to enter a region of high class conﬁdence.

Original: We deﬁne intra-class distance as the distance a sample has to move to enter a highconﬁdence region of its source class’s distribution.
Revised: We deﬁne intra-class distance as the distance a sample has to move to enter a highconﬁdence region of its source class’s distribution.

Original: Similarly, we deﬁne inter-class distance as the distance a sample has to move to enter a high-conﬁdence region of a target class distribution, where y src (cid:54) =
Revised: Similarly, we deﬁne inter-class distance as the distance a sample has to move to enter a high-conﬁdence region of a target class distribution, where y src (cid:54) =

Original: y tgt .
Revised: y tgt .

Original: Then, separability in a layer is the difference between average inter-class and intra-class distances.
Revised: Then, separability in a layer is the difference between average inter-class and intra-class distances.

Original: In practice, for a given sample, given model, chosen target class, and chosen layer, we iteratively perturb the sample using FDA with a small step size (e.g. 5 e- 5 ) until the conﬁdence of the target distribution auxiliary network is over a threshold (e.g. 99.9%).
Revised: In practice, for a given sample, given model, chosen target class, and chosen layer, we iteratively perturb the sample using FDA with a small step size (e.g. 5 e- 5 ) until the conﬁdence of the target distribution auxiliary network is over a threshold (e.g. 99.9%).

Original: The number of perturbation steps it takes to reach the conﬁdence threshold encodes the distance of the sample to the targeted class’s distribution.
Revised: The number of perturbation steps it takes to reach the conﬁdence threshold encodes the distance of the sample to the targeted class’s distribution.

Original: Results for the three whitebox models are shown in Fig. 6, where the vertical axis is the separability in units of perturbation steps, and the horizontal axis is the layer-depth of each model.
Revised: Results for the three whitebox models are shown in Figure 6, where the vertical axis is the separability in units of perturbation steps, and the horizontal axis is the layer-depth of each model.

Original: We see that there is some separability in all layers, for all models, indicating that even features very close to the input layer are somewhat class-speciﬁc.
Revised: We see that there is some separability in all layers, for all models, indicating that even features very close to the input layer are somewhat class-speciﬁc.

Original: Further, VGG19’s features are much less separable than
Revised: Further, VGG19’s features are much less separable than

Original: DN121 and RN50, indicating why VGG19 may have performed much worse as a whitebox model in the transferability tests.
Revised: DN121 and RN50, indicating why VGG19 may have performed much worse as a whitebox model in the transferability tests.

Original: In the same vein, DN121 has generally the most separated features which further indicates why it may be a superior whitebox model.
Revised: In the same vein, DN121 has generally the most separated features which further indicates why it may be a superior whitebox model.

Original: Intuitively, if a model/layer has highly class-separable feature distributions, FDA attacks may be more transferable because there is less ambiguity between the target class’s distribution and other class distributions during generation.
Revised: Intuitively, if a model/layer has highly class-separable feature distributions, FDA attacks may be more transferable because there is less ambiguity between the target class’s distribution and other class distributions during generation.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We present a new targeted blackbox transfer-based adversarial attack methodology that achieves state-of-the-art success rates for ImageNet classiﬁers.
Revised: We present a new targeted blackbox transfer-based adversarial attack methodology that achieves state-of-the-art success rates for ImageNet classiﬁers.

Original: The presented attacks leverage learned classwise and layer-wise intermediate feature distributions of modern DNNs.
Revised: The presented attacks leverage learned classwise and layer-wise intermediate feature distributions of modern DNNs.

Original: Critically, the depth at which features are perturbed has a large impact on the transferability of those perturbations, which may be linked to the transition from task/data-speciﬁc to model-speciﬁc features in an architecture.
Revised: Critically, the depth at which features are perturbed has a large impact on the transferability of those perturbations, which may be linked to the transition from task/data-speciﬁc to model-speciﬁc features in an architecture.

Original: We further leverage the learned feature distributions to measure the entanglement/separability of class manifolds in the representation space and the correlations of the intermediate feature distributions with the model output.
Revised: We further leverage the learned feature distributions to measure the entanglement/separability of class manifolds in the representation space and the correlations of the intermediate feature distributions with the model output.

Original: Interestingly, we ﬁnd the optimal attack transfer layers have feature distributions that are class-speciﬁc and highly-separable, but are not overly-correlated with the whitebox model output.
Revised: Interestingly, we ﬁnd the optimal attack transfer layers have feature distributions that are class-speciﬁc and highly-separable, but are not overly-correlated with the whitebox model output.

Original: We also ﬁnd that highly transferable attacks induce large disruptions in the intermediate feature space of the blackbox models. 
Revised: We also ﬁnd that highly transferable attacks induce large disruptions in the intermediate feature space of the blackbox models. A CKNOWLEDGMENTS

Revised: The research was supported in part by AFRL (FA8750-18-2-0057), DARPA, DOE, NIH, NSF and

Revised: ONR.

Original: B. F ULL TARGETED TRANSFER RESULTS
Revised: B. F ULL TARGETED TRANSFER RESULTS

Original: Fig. 7 shows the full targeted attack transfer results from which the Fig. 2 were extracted.
Revised: Figure 7 shows the full targeted attack transfer results from which the Figure 2 were extracted.

Original: These full results include two additional metrics of attack success.
Revised: These full results include two additional metrics of attack success.

Original: Untargeted Transfer Rate (uTR) is the rate at which examples that fool the whitebox also fool the blackbox (encodes likelihood of misclassiﬁcation).
Revised: Untargeted Transfer Rate (uTR) is the rate at which examples that fool the whitebox also fool the blackbox (encodes likelihood of misclassiﬁcation).

Original: Targeted Transfer Rate (tTR) is the rate at which successful targeted examples on the whitebox are also successful targeted examples on the blackbox (encodes likelihood of targeted misclassiﬁcation).
Revised: Targeted Transfer Rate (tTR) is the rate at which successful targeted examples on the whitebox are also successful targeted examples on the blackbox (encodes likelihood of targeted misclassiﬁcation).

Original: Error and tSuc are described in Section
Revised: Error and tSuc are described in Section

Original: [Figure]
Revised: [Figure]

Original: C. U NTARGETED FEATURE DISTRIBUTION ATTACKS
Revised: C. U NTARGETED FEATURE DISTRIBUTION ATTACKS

Original: The goal of an untargeted attack is to generate an adversarial noise δ that when added to a clean sample x of class y src , the classiﬁcation result of x + δ is not y src .
Revised: The goal of an untargeted attack is to generate an adversarial noise δ that when added to a clean sample x of class y src , the classiﬁcation result of x + δ is not y src .

Original: The key intuition for feature distribution-based untargeted attacks is that if a sample’s features are made to be outside of the feature distribution of class y src at some layer of intermediate feature space, then it will likely not be classiﬁed as y src .
Revised: The key intuition for feature distribution-based untargeted attacks is that if a sample’s features are made to be outside of the feature distribution of class y src at some layer of intermediate feature space, then it will likely not be classiﬁed as y src .

Original: uFDA
Revised: uFDA

Original: The ﬁrst untargeted attack variant is uFDA which is described as
Revised: The ﬁrst untargeted attack variant is uFDA which is described as

Original: [Equation]
Revised: [Equation]

Original: uFDA minimizes the probability that the layer l features of the perturbed sample x + δ are from the source class y src distribution.
Revised: uFDA minimizes the probability that the layer l features of the perturbed sample x + δ are from the source class y src distribution.

Original: Unlike the targeted samples which drive towards high conﬁdence regions of a target class feature distribution, this objective drives the sample towards low conﬁdence regions of the source class feature distribution.
Revised: Unlike the targeted samples which drive towards high conﬁdence regions of a target class feature distribution, this objective drives the sample towards low conﬁdence regions of the source class feature distribution.

Original: uFDA+fd
Revised: uFDA+fd

Original: The second untargeted variant uFDA+fd is described as
Revised: The second untargeted variant uFDA+fd is described as

Original: [Equation]
Revised: [Equation]

Original: uFDA+fd also carries a feature disruption term so that the objective drives the perturbed sample towards low conﬁdence regions of the source class feature distribution and maximal distance from the original sample’s feature representation.
Revised: uFDA+fd also carries a feature disruption term so that the objective drives the perturbed sample towards low conﬁdence regions of the source class feature distribution and maximal distance from the original sample’s feature representation.

Original: fd-only The ﬁnal untargeted attack fd-only is described as
Revised: fd-only The ﬁnal untargeted attack fd-only is described as

Original: [Equation]
Revised: [Equation]

Original: Notice, fd-only is simply the feature disruption term and is a reasonable standalone untargeted attack objective because making features maximally different may intuitively cause misclassiﬁcation.
Revised: Notice, fd-only is simply the feature disruption term and is a reasonable standalone untargeted attack objective because making features maximally different may intuitively cause misclassiﬁcation.

Original: To test attack success, we generate untargeted adversarial examples from both DN121 and RN whiteboxes and test transfers to a VGG19 blackbox model.
Revised: To test attack success, we generate untargeted adversarial examples from both DN121 and RN whiteboxes and test transfers to a VGG19 blackbox model.

Original: It is common to evaluate untargeted attacks with a tighter noise constraint (Kurakin et al., 2018; Dong et al., 2018) as the task is simpler, so in these tests we use (cid:96) ∞ (cid:15)
Revised: It is common to evaluate untargeted attacks with a tighter noise constraint (Kurakin et al., 2018; Dong et al., 2018) as the task is simpler, so in these tests we use (cid:96) ∞ (cid:15)

Original: = 4 / 255 and (cid:15)
Revised: = 4 / 255 and (cid:15)

Original: = 8 / 255 (rather than (cid:15) =
Revised: = 8 / 255 (rather than (cid:15) =

Original: 16 / 255 used for targeted tests).
Revised: 16 / 255 used for targeted tests).

Original: For baselines, we use the Madry et al.
Revised: For baselines, we use the Madry et al.

Original: (2018) random start PGD attack ( upgd ) and the Dong et al.
Revised: (2018) random start PGD attack ( upgd ) and the Dong et al.

Original: (2018) competition winning momentum iterative attack ( umim ).
Revised: (2018) competition winning momentum iterative attack ( umim ).

Original: Similar to the layer-wise targeted evaluations, each ”clean” source sample belongs to the same set of 10 previously modeled classes.
Revised: Similar to the layer-wise targeted evaluations, each ”clean” source sample belongs to the same set of 10 previously modeled classes.

Original: Fig. 8 shows the error rates versus layer depth for the attacks.
Revised: Figure 8 shows the error rates versus layer depth for the attacks.

Original: [Figure]
Revised: [Figure]

Original: As expected, the error rate increases with epsilon and the layer depth at which feature-based attacks are generated from has a large impact on attack success rate.
Revised: As expected, the error rate increases with epsilon and the layer depth at which feature-based attacks are generated from has a large impact on attack success rate.

Original: In general, uFDA+fd is the top performer, followed by fd-only , then uFDA .
Revised: In general, uFDA+fd is the top performer, followed by fd-only , then uFDA .

Original: However, uFDA often under-performs the umim baseline, further indicating that for adversarial attacks in feature space it is beneﬁcial to include a term that prioritizes feature disruption (e.g. uFDA+fd & fd-only ).
Revised: However, uFDA often under-performs the umim baseline, further indicating that for adversarial attacks in feature space it is beneﬁcial to include a term that prioritizes feature disruption (e.g. uFDA+fd & fd-only ).

Original: On average across models, at (cid:15) = 4 / 255 , the optimal layer uFDA+fd has an untargeted error rate of 37%, which is 9% higher than the best baseline.
Revised: On average across models, at (cid:15) = 4 / 255 , the optimal layer uFDA+fd has an untargeted error rate of 37%, which is 9% higher than the best baseline.

Original: At (cid:15) = 8 / 255 , the optimal layer uFDA+fd has an untargeted error rate of 79%, which is 27% higher than the best baseline.
Revised: At (cid:15) = 8 / 255 , the optimal layer uFDA+fd has an untargeted error rate of 79%, which is 27% higher than the best baseline.

Original: Also, both whitebox models perform similarly in terms of attack success rate, however the performance of fd-only varies between the two (especially at (cid:15) = 8 / 255 ).
Revised: Also, both whitebox models perform similarly in terms of attack success rate, however the performance of fd-only varies between the two (especially at (cid:15) = 8 / 255 ).

Original: Surprisingly, fd-only which simply disrupts the original feature map is the optimal attack for the DN121 whitebox (by a small margin).
Revised: Surprisingly, fd-only which simply disrupts the original feature map is the optimal attack for the DN121 whitebox (by a small margin).

Original: Finally, note that the optimal transfer layers from the targeted attacks (i.e. DN 121 l =7 and RN 50 l =8 ) are also high performing layers for the untargeted attacks.
Revised: Finally, note that the optimal transfer layers from the targeted attacks (i.e. DN 121 l =7 and RN 50 l =8 ) are also high performing layers for the untargeted attacks.

Original: D. A DVERSARIAL EXAMPLE GENERATION PROCESS
Revised: D. A DVERSARIAL EXAMPLE GENERATION PROCESS

Original: Recall, because the auxiliary models are NNs, the optimization objectives described for both the targeted and untargeted attacks can be solved with an iterative gradient descent procedure.
Revised: Recall, because the auxiliary models are NNs, the optimization objectives described for both the targeted and untargeted attacks can be solved with an iterative gradient descent procedure.

Original: For any version of the FDA attacks, we ﬁrst build a ”composite” model which includes the truncated whitebox model f l and the appropriate auxiliary model g l,c , as shown in Fig. 1(bottom).
Revised: For any version of the FDA attacks, we ﬁrst build a ”composite” model which includes the truncated whitebox model f l and the appropriate auxiliary model g l,c , as shown in Figure 1(bottom).

Original: An attack loss function L FDA is then deﬁned which includes a BCELoss term and any additional term which is trivially incorporated (e.g. the feature disruption term).
Revised: An attack loss function L FDA is then deﬁned which includes a BCELoss term and any additional term which is trivially incorporated (e.g. the feature disruption term).

Original: We then iteratively perturb the source image for K iterations using the sign of a collected momentum term.
Revised: We then iteratively perturb the source image for K iterations using the sign of a collected momentum term.

Original: Similar to Dong et al.
Revised: Similar to Dong et al.

Original: (2018) and Inkawhich et al.
Revised: (2018) and Inkawhich et al.

Original: (2019), momentum is calculated as
Revised: (2019), momentum is calculated as

Original: [Equation]
Revised: [Equation]

Original: where m 

Revised: where m

Original: = 

Revised: =

Original: and I k is the perturbed source image at iteration k .
Revised: and I k is the perturbed source image at iteration k .

Original: The perturbation method for this (cid:96) ∞ constrained attack is then
Revised: The perturbation method for this (cid:96) ∞ constrained attack is then

Original: I [-k +
 =-]
Revised: I [-k +
 =-][Equation]

Original: Clip ( I k − α ∗ sign ( m k +1 ) , 0 , 1) .
Revised: [Equation] k [Equation] sign ( m k +1 ) , 0 , 1) .

Original: In this work, all attacks perturb for K = 10 iterations and α =
Revised: In this work, all attacks perturb for K = 10 iterations and α =

Original: (cid:15)/K .
Revised: (cid:15)/K .

Original: E. A DDITIONAL SALIENCY
Revised: E. A DDITIONAL SALIENCY