Robust Feature-Level Adversaries are Interpretability Tools
Anonymous Author(s)
AfﬁliationAddress email
Abstract
The literature on adversarial attacks in computer vision typically focuses on pixellevel perturbations.
These tend to be very difﬁcult to interpret.
Recent work thatmanipulates the latent representations of image generators to create “feature-level”adversarial perturbations gives us an opportunity to explore perceptible, interpretable adversarial attacks.
We make three contributions.
First, we observe thatfeature-level attacks provide useful classes of inputs for studying the representations in models.
Second, we show that these adversaries are versatile and highlyrobust.
We demonstrate that they can be used to produce targeted, universal, disguised, physically-realizable, and black-box attacks at the ImageNet scale.
Third,we show how these adversarial images can be used as a practical interpretabilitytool for identifying bugs in networks.
We use these adversaries to make predictionsabout spurious associations between features and classes which we then test bydesigning “copy/paste” attacks in which one natural image is pasted into another tocause a targeted misclassiﬁcation.
Our results indicate that feature-level attacksare a promising approach for rigorous interpretability research.
They support thedesign of tools to better understand what a model has learned and diagnose brittlefeature associations.
Introduction
State-of-the-art neural networks are vulnerable to adversarial examples.
Conventionally, adversarialinputs for visual classiﬁers take the form of small-norm perturbations to natural images [64, 17].
These perturbations reliably cause conﬁdent misclassiﬁcations.
However, to a human, they typicallyappear as random or mildly-textured noise.
Consequently, it is difﬁcult to interpret these attacks.
Beyond the observation that such attacks are possible, it is hard to learn about the underlying victimnetwork from these pixel-level perturbations.
In contrast, many real-world failures of biological vision are caused by perceptible, human-describablefeatures.
For instance, the ringlet butterﬂy’s predators are stunned by adversarial "eyespots" on itswings (Appendix A.1, Fig. 7).
This falls outside the scope of conventional adversarial examplesbecause the misclassiﬁcation results from a feature-level change to an object/image.
The adversarialeyespots are robust in the sense that the same attack works across a variety of different backgroundsand viewing conditions.
Furthermore, because the attack relies on high-level features, it is easy for ahuman to describe it.
This work takes inspiration from the ringlet butterﬂy’s eyespots and similar examples in whicha model is fooled in the real world by an interpretable feature (e.g. [45]).
Our goal is to designadversaries that reveal easily-understandable weaknesses of the victim network.
We focus on twodesiderata for adversarial perturbations: attacks must be (1) interpretable (i.e. describable) to ahuman, and (2) robust so that interpretations generalize.
We refer to these types of attacks as “feature-
[Figure]
level” adversarial examples.
Several previous works have created attacks by perturbing the latentrepresentations of an image generator (e.g., [23]), but thus far, approaches have been small in scale,limited in robustness, and not interpretability-driven (See Section 2).
We build on this prior work to propose a method that generates feature-level attacks against computervision models.
Our attack method works on ImageNet scale models and creates robust, feature-leveladversarial examples.
We test three methods of introducing adversarial features into source imageseither by modifying the generator’s latents and/or inserting a generated patch into natural images.
Incontrast to previous works that have enforced the “adversarialness” of attacks only by inserting smallfeatures or restricting the distance between an adversary and a benign input, we also introduce newmethods that regularize the feature to be perceptible yet disguised to resemble something other thanthe target class.
We show that our method produces robust attacks that provide actionable insights into a network’slearned representation.
Fig.
1 demonstrates the interpretability beneﬁts of this type of feature-levelattack.
It compares a conventional, pixel-level, adversarial patch, created using the method from[6], with a feature-level attack using our method.
While both attacks attempt to make a networkmisclassify a bee as a ﬂy, the pixel-level attack exhibits high-frequency patterns and lacks visuallycoherent objects.
On the other hand, the feature-level attacks displays easily describable features: thecolored balls.
We can validate this insight by considering the network performance when a picture ofa trafﬁc light is inserted into the image a bee.
In this example, the image classiﬁcation goes from a55% conﬁdence that the image is a bee to a 97% conﬁdence that the image is of a ﬂy.
Fig.
6 showsseveral of these “copy/paste” attacks and the feature-level attacks that inspired them.
Our contributions are threefold.
Conceptual: We observe that robust feature-level adversaries can used to produce useful
types of inputs for studying the representations of deep networks
Robust Attacks: We introduce methods for generating feature-level adversaries that are
uniquely versatile and able to produce targeted, universal, disguised, physically-realizable,and black-box attacks at the ImageNet scale.
See Table 1.
Interpretability: We generalize from our adversarial examples to design copy/paste attacks,verifying that our adversaries help us understand the network well enough to exploit it.
The following sections contain background, methods, experiments, and discussion.
Appendix A.11has a high-level summary for a lay-audience.
We are making code openly available.
Related Work
Here, we contextualize our approach with others related to improving on conventional adversarial examples [64, 17].
Table 1 summarizes capabilities
[Table]
Copy/Paste =useful for designing attacks in which a natural feature is pasted into a natural image.
Inspiration from Nature: Mimicry is common in nature, and sometimes, rather than holisticallyimitating another species, a mimic will only display particular features.
For example, many animalsuse adversarial eyespots to confuse predators [62] (see Appendix A.1 Fig. 7a).
Another example isthe mimic octopus which imitates the patterning, but not the shape, of a banded sea snake.
We showin Figure 7b that a ResNet50 classiﬁes an image of one as a sea snake.
Generative Modeling: An approach related to ours has been to train a generator or autoencoderto produce small adversarial perturbations that are applied to natural inputs.
This has been done tosynthesize imperceptible attacks that are transferable, universal, or efﬁcient to produce [19, 40, 41,50, 70, 18, 69].
Rather than training a generator, ours and other works have perturbed the latents ofpretrained generative models to produce perceptible alterations.
[37] did this with a differentiableimage renderer.
Others [53, 61, 28, 27, 58, 23] have used deep generative networks, and [67] aimedto create more semantically-understandable attacks by using an autoencoder with a “disentangled”embedding space.
Our work is different in four ways.
(1) These works focus on small classiﬁerstrained on simple datasets (MNIST [35], Fashion MNIST [71], SVHN [43], CelebA [38], BDD [73],
INRIA [8], and MPII [1]) while we work at the ImageNet [51] scale.
(2) We do not simply rely onusing small features or restricting the distance to a benign image to enforce the adversarialness ofattacks.
We introduce techniques that regularize the adversarial feature to be perceptible yet
disguisedto
resemble something other than the target class.
(3) We evaluate three distinct ways of insertingadversarial features into images.
(4) Our work is interpretability-oriented.
[Table]
[Figure]
Attacks in the Physical World: Physical-realizability demonstrates robustness.
We show that ourattacks fool a classiﬁer when printed and photographed.
This directly relates to [33] who foundthat pixel-space adversaries could do this to a limited extent in controlled settings.
More recently,[55, 6, 14, 2, 36, 65, 31, 30] created adversarial clothing, stickers, or objects.
In contrast with these,we also produce attacks in the physical world that are disguised as a non-target class.
Adversaries and Interpretability: Using adversarial examples to better interpret networks has beenproposed by [10] and [66].
We use ours to discover human-describable feature/class associationslearned by a network.
This relates to [16] and [34] who debug networks by searching over featureinsertions, transformations, and textural changes.
More similar to our work are [7, 42, 22], whouse feature visualization [46] and network dissection [3] to interpret the network.
Each use theirinterpretations to design “copy/paste” attacks in which one natural image pasted inside another causesan unrelated misclassiﬁcation.
We add to this work with a new method to identify such adversarialfeatures.
Unlike any previous approach, ours does so in a way that allows for targeted attacks thattake into account an arbitrary distribution of source images.
Methods
We adopt the “unrestricted” adversary paradigm [61] under which an attack is successful if thenetwork’s classiﬁcation differs from an oracle’s
(e.g., a human).
Our adversaries can only changea small, ﬁxed portion of either the generator’s latent or the image.
We use white-box access to thenetwork, though we present black-box attacks based on transfer from an ensemble in Appendix A.6.
Our attacks involve perturbing the latent representation in some layer of an image generator toproduce an adversarial feature-level alteration.
Fig.
2 depicts our approach.
We test three types ofattacks, “patch”, “region” and “generalized patch” (plus a fourth in Appendix A.5 which we call“channel” attacks).
We ﬁnd patch attacks to generally be the most successful.
Patch: We use the generator to produce a square patch that is inserted into a natural image [56].
Region: Starting with some generated image, we randomly select a square column of the latent in agenerator layer which spans the channel dimension and replace it with a learned insertion.
This isanalogous to a square patch in the pixel representation.
We keep insertion location ﬁxed over training.
The modiﬁed latent is passed through the rest of the generator, producing the adversarial image.
Generalized Patch: These patches can be of any shape, hence the name “generalized” patch.
Weﬁrst generate a region attack and then extract a generalized patch from it.
We do this by takingthe absolute-valued pixel-level difference between the original and adversarial image, applying a
Gaussian ﬁlter for smoothing, and creating a binary mask from the top decile of these pixel differences.
We apply this mask to the generated image to isolate the region that the perturbation altered.
We canthen treat this as a patch and overlay it onto an image in any location.
Basic Attacks: For all attacks, we train a perturbation  to the latent of the generator to minimize aloss that optimizes for both fooling the classiﬁer and appearing interpretable:
[Equation]
with X a distribution over source images (e.g., a dataset or generation distribution), T a distributionover transformations, L a distribution over insertion locations (this only applies for patches andgeneralized patches), C the target classiﬁer, A an image-generating function, L x-ent a targetedcrossentropy loss for fooling the classiﬁer, y targ the target class, and L reg a regularization loss.
Theadversary has no control over X , T , or L , so it must learn features that fool the network independent 130 of any particular source image, transformation, or insertion location.
For all of our attacks, L regcontains a total variation loss, TV ( a ) , to discourage high-frequency patterns.
“Disguised”
Attacks: Ideally, a feature-level adversarial example should appear to a human aseasily-describable but should not resemble the attack’s target class.
We call such attacks “disguised.”
Here, the main goal is not to fool a human, but to help them learn about what types of realisticfeatures might cause the model to make a mistake.
To train these disguised attacks, we use additionalterms in L reg as proxies for these two criteria.
We differentiably resize the patch or the extractedgeneralized patch and pass it through a GAN discriminator and auxiliary classiﬁer.
We then addweighted terms to the regularization loss based on the discriminator’s
( D ) logistic loss for classifyingthe input as fake, the output entropy ( H ) of some classiﬁer ( C 0 ), and/or the negative of the classiﬁer’scrossentropy loss for labeling the input as the attack’s target class.
Note that C 0 could either be the same or different from C .
With all of these terms, the regularization objective is
L reg ( a ) =
1 TV ( a ) +  2 L logistic [ D ( P ( a ))]
+  3 H [ C 0 ( P ( a ))]
4 L x-ent
[ C 0 ( P ( a ) , y targ )] | {z } “Disguise” Regularizers .
Here, P ( a ) returns the extracted and resized patch from adversarial image a .
In order, these threenew terms encourage the adversarial feature to (1) look realistic, and (2) look like some speciﬁc class,but (3) not the target class.
The choice of disguise class is left entirely to the training process.
Experiments
We use BigGAN generators from [5, 68], and perturb the post-ReLU outputs of the internal ‘GenBlocks.’
We used the BigGAN discriminator and adversarially trained classiﬁers from [12] fordisguise regularization.
By default, we attacked a ResNet50 [20], restricting patch attacks to 1/16 ofthe image and region and generalized patch attacks to 1/8.
Appendix A.2 has additional details.
4.1 Robust Attacks
[Figure]
the proportion of attacks in which the patch was not classiﬁed by the network as the target class when viewed on its own.
Mean target conﬁdence (y axis) gives the empirical fooling conﬁdences of 250 patch attacks.
Each is an average oversource images.
The proportion of each distribution above 0.5 gives a lower-bound for the top-1 attack success rate.
The mean target class conﬁdence for using randomly-sampled natural images as patches is 0.0008 and is shown as a thin dotted line at the bottom
Figure 3 shows examples of targeted, universal, and disguised feature-level patch (top), region (middle), and generalized patch (bottom)attacks which were each trained with all of the
[Figure]
disguise regularization terms from Eq. 2.
Weﬁnd the disguises to be effective, particularly forthe patches (top row), but imperfect.
Appendix
A.3 discusses this and what it may suggest aboutnetworks, human visual cortex, and size bias.
Performance versus Disguise: Here, we studyour patch attacks in depth to test how effectivethey are at fooling the network and how successfully they can help to identify non-targetclass features that can fool the network.
Wecompared seven different approaches.
The ﬁrstwas our full approach using the generator andall disguise regularization terms from Eq.
The rest were ablation tests in which we omitted the generator (No Gen), the discriminator(No Disc) regularization term (No Reg), the entropy regularization term (No Ent), the crossentropy regularization term (No Patch X-ent), allthree regularization terms (Only Gen), and ﬁnally the discriminator and all three regularization terms (Brown et al. ’17).
This ﬁnal unregularized, pixel-level method resulted in the sameapproach as Brown et al.
For eachtest, all else was kept identical including penalizing total variation, training under transformations, and initializing the patch as a generatoroutput.
For each method, we generated universal attackswith random target classes until we obtained250 successfully “disguised” ones in which theresulting adversarial feature was not classiﬁed by the network as the target class when viewed on itsown.
Fig.
4 plots the success rate versus the distribution of target class mean fooling conﬁdences foreach type of attack.
For all methods, these universal attacks have variable fooling conﬁdences due inlarge part to the random selection of target class.
Some attacks are stochastically Pareto dominatedby others.
For example, the pixel-space Brown et al.
(2017) attacks were the least effective at foolingthe target network and had the third least disguise rate.
In other cases, there is a tradeoff between
See Appendix A.10 Fig.
13 for full-sized versions of fooling performance and disguise which can be controlled using the regularization terms from Eq.2.
We also compare all attacks to a baseline in which we used randomly-sampled natural images aspatches.
These resulted in a mean target class conﬁdence of 0.0008.
Notably, Fig. 4 does not capture everything that one might care about in these attacks.
It does notshow any measure of how “realistic” the resulting patches look.
In Appendix A.4, Fig.
9 plots thesame fooling conﬁdence data from the y axis in Fig.
4 versus the disguise class label conﬁdence froman Inception-v3 which we use as a proxy for how realistic a human would ﬁnd the patch.
It suggeststhat the best attacks for producing patches that appear realistic are the “All” and “No Disc” methods.
In Appendix A.10, Figs. 13 14, and 15 give examples of successful “All”, “Only Gen”, and “Brownet al. ’
17” attacks respectively.
Because they were initialized from generator outputs, some of the“Brown et al. ’
17” attacks have a veneer-like resemblance to non-target class features.
Nonetheless,they contain higher-frequency patterns and less coherent objects in comparison to the two sets offeature-level attacks.
We subjectively ﬁnd the “All” attacks to be the best disguised.
Physical-Realizability:
To test their ability to transfer to the physical world, we generated 100additional targeted, universal, and disguised adversarial patches.
We used the generator and allregularization terms (the “All” condition from above).
We selected the 10 with the best mean foolingconﬁdence, printed them, and photographed each next to 9 different ImageNet classes.
1 We conﬁrmedthat photographs of each object were correctly classiﬁed without a patch.
Figure 5 shows successfulexamples.
Meanwhile, resizable and printable versions of all these patches and others are in Appendix
A.10.
The mean and standard deviation of the fooling conﬁdences for our attacks in the physicalworld were 0.312 and 0.318 respectively ( n = 90 , not i.i.d.).
This means that these patches’ meaneffectiveness dropped by less than 12 when transferring to the physical world.
Black-Box Attacks: In Appendix A.6, we show that our universal attacks can transfer from anensemble to a held-out model.
4.2 Interpretability
If an adversarial feature successfully fools the victim network, this suggests that the network associatesthat feature in context of a source image with the target class.
We ﬁnd that our adversaries can suggestboth beneﬁcial and harmful feature-class associations.
In Appendix A.7, Fig.
11 provides a simpleexample of each.
One challenge in the explainable AI literature is to develop interpretations that go beyond seemingplausible and stand up to scrutiny [72].
Observing robust feature level adversarial patches can easilybe used to develop hypotheses about the network’s behavior, e.g. “The network thinks that beefeatures plus colorful balls implies a ﬂy.”
But are these valid, useful interpretations of the network?
In other words, are our adversaries adversarial because of their interpretable qualities, or is it becauseof hidden motifs?
A rigorous approach to interpretability requires veriﬁcation.
We do this by usingour attacks to make and validate predictions about how to fool the target network with natural objects.
Validating Interpretations with Copy/Paste Attacks: A “copy-paste” attack is created by insertingone natural image into another to cause an unexpected misclassiﬁcation.
They are more restrictedthan patch attacks because the features pasted into an image must be natural objects.
As a result,they are of high interest for physically-realizable attacks because they suggest combinations of real objects that yield unexpected classiﬁcations.
They also have precedent in the real world.
For example,subimage insertions into pornographic images have been used to evade NSFW content detectors [74].
To develop copy/paste attacks, we select a source and target class, generate class-universal adversarialfeatures, and manually analyze them for motifs that resemble natural objects.
Here, we used basic attacks without the disguise regularization terms from Eq.
We then paste images of these objectsinto natural images and pass them through the classiﬁer.
Fig.
6 shows four types of copy/paste attacks.
In each odd row, we show six patch, region, andgeneralized patch adversaries that were used to guide the design of a copy/paste attack.
In eacheven row are the copy/paste adversaries for the 6 (of 50) images for the source class for whichthe insertion resulted in the highest target conﬁdence increase along with the mean target classconﬁdences before and after patch insertion for those 6.
The success of these attacks shows theirusefulness for interpreting the target network because they require that a human understands themistake the model is making like “Bee ^ Trafﬁc Light !
Fly” well enough to manually exploit it.
Given the differences in the adversarial features that are produced in the Bee !
Fly and Trafﬁc Light!
Fly attacks, Fig. 6 also demonstrates how our attacks take the distribution of source images intoaccount.
Three prior works [7, 42, 22] have developed copy/paste attacks, also via interpretability tools.
Unlike[42, 22], our approach allows for targeted attacks.
And unlike all three, rather than simply identifyingfeatures associated with a class, our adversaries generate fooling features for a target class conditionalon any distribution over source images (i.e. the source class) with which the adversaries are trained.
Little work has been done on copy/paste adversaries, and thus far, methods have either not allowedfor targeted attacks or have required a human in the loop.
This makes objective comparisons difﬁcult.
However we provide examples of a feature-visualization based method inspired by [7] in Appendix
A.8 to compare with ours.
We conclude that our attacks’ ability to take the source image distributioninto account can allow it to be more effective at discovering certain weaknesses compared to thebaseline inspired by [7].
Discussion and Broader Impact
Here we use feature-level adversarial examples to fool and interpret deep networks in order tocontribute to a more practical understanding of network vulnerabilities.
As an attack method, ourapproach is versatile.
It can produce targeted, universal, disguised, physically-realizable, black-box,and copy/paste attacks at the ImageNet scale.
Furthermore, this attack method can be used as aninterpretability tool to help diagnose ﬂaws in models.
We ground the notion of interpretability in theability to make predictions about when a model will fail.
Finally, we demonstrate this through thedesign of targeted copy/paste attacks for any distribution over source inputs.
Like any work on adversarial attacks, our methods could be used maliciously to make a system fail,but we emphasize their diagnostic value.
Understanding threats is a prerequisite to avoiding them.
Given the robustness and versatility of our attacks, we argue that they may be valuable for continuedwork to address threats that systems may face in practical applications.
There are at least two ways inwhich these methods can be useful.
The ﬁrst is for adversarial training.
Training networks on adversarial images has been shown toimprove their robustness to the attacks that are used [12].
But this does not guarantee robustness toother types of adversarial inputs (e.g. [21]).
Our feature-level attacks are categorically different fromconventional pixel-level ones, and our copy/paste attacks show how networks can be fooled by novelcombinations of natural objects, failures that are outside the conventional paradigm for adversarialrobustness
(e.g., [12]).
Consequently, we expect that adversarial training on broader classes of attackssuch as the one we propose here will be valuable for designing more robust models.
As a promisingsign, we show in Appendix A.9 that adversarial training is useful for our attacks.
The second is for rigorously diagnosing ﬂaws.
We show that feature-level adversaries aid the discoveryof exploitable spurious feature/class associations (Fig. 6) and a socially-harmful bias (Appendix
A.7 Fig.
Our approach could also be extended beyond what we have demonstrated here.
Forexample, our methods could be used for feature visualization [46] of a network’s internal neurons.
An analogous approach to ours can also be used in Natural Language Processing [60, 49], and weare currently working on a method for this.
Furthermore, it may be valuable to use these adversariesto identify generalizable ﬂaws in networks that humans can easily understand but with minimalhuman involvement.
This would be much more scalable and prevent human priors from inﬂuencinginterpretations.
Toward this goal, we are currently working on methods for the fully-automateddiscovery of
/paste
attacks
A limitation of our approach is that when multiple desiderata are optimized for at the same time(e.g., universality + transformation robustness + disguise), attacks are generally less successful, moretime-consuming, and require more screening to ﬁnd good ones.
This could be a bottleneck forlarge-scale adversarial training.
Ultimately, this type of attack is limited by the efﬁciency and qualityof the generator, so future work should leverage advances in generative modeling.
Our evaluationmethod is also limited to be a proof-of-concept for the design of copy/paste attacks.
Future workshould evaluate this more rigorously.
We are currently working toward developing a benchmark forinterpretability tools based on their ability to aid a human in rediscovering trojans [15] that have beenimplanted into a model.
As AI becomes increasingly capable, it becomes more important to design models that are reliable.
Each of the 11 proposals for building safe AI outlined in [25] explicitly call for adversarial robustnessand/or interpretability tools, and recent work from [76] on high-stakes reliability in AI found thatinterpretability tools strengthened their ability to produce inputs for adversarial training.
Giventhe close relationship between interpretability and adversarial robustness, continued study of theconnections between them will be key for building safer AI systems. Checklist For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See
Section 5.(d)
Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] See
supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Appendix A.2 and supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Fig.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section A.2.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No] We only used open source assets plus
ImageNet.
We are license compliant but do not discuss this explicitly.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] We are license compliant for ImageNet but do not discuss this
explicitly.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A] Data we use is limited to ImageNet.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]