N ON - ROBUST F EATURES THROUGH THE L ENS OF U NIVERSAL P ERTURBATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent work ties adversarial examples to existence of non-robust features: features which are susceptible to small perturbations and believed to be unintelligible to humans, but still useful for prediction.
We study universal adversarial perturbations and demonstrate that the above picture is more nuanced.
Speciﬁcally, even though universal perturbations—similarly to standard adversarial perturbations—do leverage non-robust features, these features tend to be fundamentally different from the “standard” ones and, in particular, non-trivially human-aligned.
Namely, universal perturbations have more human-aligned locality and spatial invariance properties.
However, we also show that these human-aligned non-robust features have much less predictive signal than general non-robust features.
Our ﬁndings thus take a step towards improving our understanding of these previously unintelligible features
I NTRODUCTION
Modern deep neural networks perform extremely well across many prediction tasks, but they largely remain vulnerable to adversarial examples (Szegedy et al., 2014).
Models’ brittleness to these small, imperceptible perturbations highlights one alarming way in which models deviate from humans.
Recent work gives evidence that this deviation is due to the presence of useful non-robust features in our datasets (Ilyas et al., 2019).
These are brittle features that are sensitive to small perturbations—too small to be noticeable to humans, yet capture enough predictive signal to generalize well on the underlying classiﬁcation task.
When models rely on non-robust features, they become vulnerable to adversarial examples, as even small perturbations can ﬂip the features’ signal.
While prior work gives evidence of non-robust features in natural datasets, we lack a more ﬁnegrained understanding of their properties.
In general, we do not understand well how models make decisions, so it is unclear how much we can understand about these features that are believed to be imperceptible.
A number of works suggest that these features may exploit certain properties of the dataset that are misaligned with human perception (Ilyas et al., 2019), such as high-frequency information (Yin et al., 2019), but much remains unknown.
[Figure]
In this work, we illustrate how we can isolate more human-aligned non-robust features by imposing additional constraints on adversarial perturbations.
In particular, we revisit universal adversarial perturbations (Moosavi-Dezfooli et al., 2017a), i.e. adversarial perturbations that generalize across many inputs. Prior works have observed that these perturbations appear to be semantic (Hayes & Danezis, 2019; Khrulkov & Oseledets, 2018; Liu et al., 2019).
We demonstrate that universal perturbations possess additional human-aligned properties different from standard adversarial perturbations, and analyze the non-robust features leveraged by these perturbations.
Concretely, our ﬁndings are:
Universal perturbations have more human-aligned properties.
We show that universality adversarial perturbations have additional human-aligned properties that distinguish them from standard adversarial perturbations (e.g., Figure 1).
In particular, (1) the most semantically identiﬁable local patches inside universal perturbations also contain the most signal; and (2) universal perturbations are approximately spatially invariant , in that they are still effective after translations.
Non-robust features can be semantically meaningful.
We show that universal perturbations are primarily relying on non-robust features rather than robust ones.
Speciﬁcally, we compare the sensitivity of natural and (adversarially) robust models to rescalings of these perturbations to demonstrate that universal perturbations likely rely on non-robust features.
Together with our ﬁrst ﬁnding, this shows that some non-robust features can be human-aligned.
Universal perturbations contain less non-robust signal.
We ﬁnd that the non-robust features leveraged by universal perturbations have less predictive signal than those leveraged by standard adversarial perturbations, despite being more human-aligned.
We measure both (1) generalizability to the original test set and (2) transferability of perturbations across independent models, following the methodology of Ilyas et al.
Under these metrics, universal perturbations consistently obtain non-trivial but substantially worse performance than standard adversarial perturbations.
P RELIMINARIES
We consider a standard classiﬁcation task: given input-label samples ( x, y ) ∈
X × Y from a data distribution D , the goal is to to learn a classiﬁer C : X → Y that generalizes to new data. Non-robust vs. robust features.
Following Ilyas et al.
(2019), we introduce the following terminology.
A useful feature for classiﬁcation is a function that is (positively) correlated with the correct label in expectation.
A feature is robustly useful if, even under adversarial perturbations (within a speciﬁed set of valid perturbations ∆ ) the feature is still useful.
Finally, a useful, non-robust feature is a feature that is useful but not robustly useful.
These features are useful for classiﬁcation in the standard setting, but can hurt accuracy in the adversarial setting (since their correlation with the label can be reversed).
For conciseness, throughout this paper we will refer to such features simply as non-robust features .
Universal perturbations.
A universal adversarial perturbation (or just universal perturbation for short), as introduced in Moosavi-Dezfooli et al. (2017a), is a perturbation δ that causes the classiﬁer
C to predict the wrong label on a large fraction of inputs from D .
We focus on targeted universal perturbations that fool C into predicting a speciﬁc (usually incorrect) target label t .
Thus, a (targeted) universal perturbation δ ∈ ∆ satisﬁes P ( x,y )
∼D [ C ( x + δ )
=
t ] = ρ , where ρ is the attack success rate (ASR) of the universal perturbation.
The only technical difference between universal perturbations and (standard) adversarial perturbations is the use of a single perturbation vector δ that is applied to all inputs.
Thus, one can think of universality as a constraint for the perturbation δ to be inputindependent.
Universal perturbations can leverage non-robust features in data, as we show here; we refer to these simply as universal non-robust features .
(cid:96) p perturbations.
We study the case where ∆ is the set of (cid:96) p -bounded perturbations, i.e. ∆ = { δ ∈ R d | || δ || p ≤ (cid:15) } for p = 2 , ∞ .
This is the most widely studied setting for research on adversarial examples and has proven to be an effective benchmark (Carlini et al., 2019).
Additionally, (cid:96) p -robustness appears to be aligned to a certain degree with the human visual system (Tsipras et al., 2019).
2. C OMPUTING UNIVERSAL PERTURBATIONS
We compute universal perturbations by using projected gradient descent (PGD) on the following optimization problem:
[Equation]
where L is the standard cross-entropy loss function for classiﬁcation, and f are the logits prior to classiﬁcation. and t is the target label.
While many different algorithms have been developed for computing universal perturbations with varying attack success rates, we do not consider them as achieving the highest rate is not necessary for our investigations.
As observed in Moosavi-Dezfooli et al. (2017a), universal perturbations trained on only a fraction of the dataset still generalize to the rest of the dataset, so in practice it sufﬁces to approximate the expectation in equation 1 using a relatively small batch of data drawn from D .
We call the batch we optimize over the base set , and use K to refer to its size.
We describe the selection of all hyperparameters in Appendix B.
2. E XPERIMENTAL SETTINGS
Datasets.
We conduct our experiments on a subset of ImageNet (Deng et al., 2009) and CIFAR- (Krizhevsky et al., 2009).
The Mixed10 ImageNet dataset (Engstrom et al., 2019), which we refer to as ImageNet-M10, is formed by sub-selecting and grouping together semantically similar classes from ImageNet; it provides a more computationally efﬁcient alternative to the full dataset, while retaining some of ImageNet’s complexity (see Appendix B.1 for more details about the dataset).
In the main text, we focus our results on ImageNet-M10, and in the Appendices we provide additional results on CIFAR-10 as well as visualizations from a full ImageNet model.
Models.
We use the standard ResNet-18 architecture (He et al., 2016) unless mentioned otherwise.
P ROPERTIES OF U NIVERSAL P ERTURBATIONS
We ﬁnd that universal perturbations, in contrast to adversarial perturbations, are much more aligned with human perception.
Beyond visualizations, we demonstrate this in the following ways: (1) local patches that are most semantically identiﬁable with the target class also contain the most predictive signal; and (2) universal perturbations are approximately spatially invariant, whereas standard perturbations are not.
3. P ERCEPTUAL ALIGNMENT
Visual comparison with standard adversarial perturbations.
Standard (cid:96) p adversarial perturbations are often thought to be incomprehensible to humans (Szegedy et al., 2014; Ilyas et al., 2019).
Even when magniﬁed for visualization, these perturbations are not identiﬁable with their target class.
In contrast, universal perturbations are visually more interpretable: when ampliﬁed, they contain local regions that we can identify with the target class (see Figure 2 for example universal perturbations, and Figure 1 for a comparison to standard perturbations).
Prior works have also observed that universal perturbations resemble their target class (Hayes & Danezis, 2019) and contain meaningful texture patterns (Khrulkov & Oseledets, 2018; Liu et al., 2019). Next, we go beyond simple visualizations by identifying two additional human-aligned properties of universal perturbations. [Figure]
Predictive signals are localized to most semantically identiﬁable patches.
While universal perturbations contain parts that are semantically identiﬁable with the target class, they also contain less identiﬁable regions; a priori, it is unclear which parts inﬂuencing the model.
To investigate this, we randomly select different local patches of the perturbation, evaluate their ASR in isolation, and inspect them visually.
For (cid:96) 2 perturbations, the patches vary widely in norm, so a possible concern is that the model is only reacting to the patches with the highest norm; to correct for this, we linearly scale up all patches to have the same norm as the largest norm patch.
Even after normalization, the patches with the highest ASR are more semantically identiﬁable with the target class (Figure 3).
For (cid:96) ∞ perturbations, where all patches have similar norms (so no further normalization is done), we also observe that their ASR is proportional to their visual saliency.
This demonstrates that the model is indeed reacting primarily to the most salient parts of the perturbation.
[Figure]
3. S PATIAL INVARIANCE
Adversarial perturbations are not only unintelligible but also extremely brittle to translations, as we demonstrate.
In contrast, we ﬁnd that universal perturbations are translationally invariant to a large degree.
As spatial invariance is one of the key properties of the human visual system (Hubel & Wiesel, 1968), this illustrates another way that universal perturbations are more human-aligned than standard perturbations.
We quantify spatial invariance by measuring the ASR of translated perturbations.
Speciﬁcally, we compute targeted standard adversarial perturbations on a sample of 256 images from the ImageNetM10 test set.
Then, we evaluate the ASR of different translated copies of these perturbations over the test set (copies of each perturbation are evaluated only on its corresponding image).
For comparison, we take a precomputed set of universal perturbations of the same norm, and evaluate them across all images.
Figure 16 shows that universal perturbations still achieve non-trivial ASR after translations of varying magnitudes.
In contrast, standard adversarial perturbations achieve a chance-level 10% ASR when shifted by more than eight pixels (two grid cells in Figure 16).
This illustrates that the model reacts differently to universal perturbations than to standard adversarial perturbations.
U NIVERSAL P ERTURBATIONS L EVERAGE N ON -R OBUST F EATURES
The predominant belief about non-robust features is that they are incomprehensible to humans.
We saw, however, that universal perturbations have human-aligned properties. And since universal perturbations are bounded in norm, they likely rely on non-robust features to fool the model. We might then conclude that there exist non-robust features that are human-aligned. However, there is a possibility of robust feature leakage (Goh, 2019): while a small perturbation cannot entirely ﬂip the correlation of a robust feature, it can still induce a small correlation by perturbing the feature.
If universal perturbations primarily rely on robust features, we cannot necessarily conclude that the nice properties we observed earlier are tied to non-robust features.
In -  111 x shifty s h
i f t adversarial ( 2 , =6.0) -  111 x shiftuniversal ( 2 , =6.0) -  111 x shiftuniversal ( , =8/255) 0.
A tt a c k S
u cc e ss R a t e this section, we provide evidence that robust leakage is unlikely to be a primary contribution to the signal. Scaling analysis on natural vs. robust model Our analysis based on the following premise: if the universal perturbations are primarily relying on small perturbations of robust features, then a robust model should eventually react when we amplify the signal in the universal perturbations.
Below, we ﬁrst show that simply linearly scaling the perturbation can effectively tune the strength of the signal.
More precisely, given a universal perturbation δ , we evaluate the ASR of t · δ , where t is a scaling parameter.
2 When we perform this scaling on a natural (non-robust) model, the ASR increases as we increase t (Figure 5).
(This holds even if networks used to train the universal perturbation and the one used to evaluate its ASR differ.)
If there was signiﬁcant robust leakage, we would expect a similar qualitative behavior for the robust model.
Instead, we ﬁnd that the robust model does not react to any scaling of the perturbation.
On the other hand, a universal perturbation of norm (cid:15) = 30 . 0 3 is computed on an adversarially-trained robust model ( (cid:15) = 6 . 0 ) still fools the natural model, even when scaled down.
Together, these ﬁndings suggest that universal perturbations computed on the natural model are unlikely to leverage robust features used by the robust model.
In fact, a more ﬁne-grained scaling analysis shows that the signal is indeed primarily ampliﬁed in the semantic parts of the image.
In Section 3 we already observed that models are most sensitive to the most semantic parts of universal perturbations; here we check that the order remains consistent across various scales.
We repeat the above scaling evaluation on natural vs. robust models, except we scale only patches of the perturbation, of varying levels of semantic content (Figure 5b).
As before, we ﬁnd that none of the patches, even after scaling, trigger the robust network.
On the non-robust models, the relative effectiveness of local patches is consistent with their semantic content: patches that are more semantically identiﬁable have more signal at all scales.
The above analysis shows that universal perturbations are likely primarily targeting non-robust features.
Combined with our ﬁndings earlier in Section 3, this shows that non-robust features may not be entirely unintelligible to humans.
Our visualizations (Figure 5c) suggests that universal non-robust features may be capturing intuitive properties, such as the presence of a faint bird head.
Thus, non-robustness in networks may partially arise from relying on cues that are small in magnitude but still human-aligned. [Figure]
U NIVERSAL N ON -R OBUST F EATURES C ONVEY L ESS S IGNAL
So far we demonstrated that universal perturbations tend to leverage highly semantic non-robust features.
We now show that these universal non-robust features have less generalizable signal than standard non-robust features.
As in Ilyas et al.
(2019), we measure the usefulness of non-robust features based on: (1) how well a model generalizes to the original test set when trained on a dataset where the only useful features are those introduced by universal perturbations; and (2) the transferability of universal perturbations across models.
(1) directly measures the usefulness of universal non-robust features for learning, while (2) measures the degree to which different models learn the same set of features.
5. G ENERALIZATION FROM UNIVERSAL NON - ROBUST FEATURES
We construct two datasets, (cid:98) D univ and (cid:98) D adv , where the only useful features are universal non-robust features and non-robust features, respectively.
Constructing (cid:98) D adv and (cid:98) D univ .
We follow the procedure outlined in Ilyas et al.
(2019) to construct (cid:98) D adv (which corresponds to (cid:98) D rand from their paper): 1) for each input-label pair ( x, y ) , select a target class t uniformly at random; then 2) compute an (cid:96) 2 adversarial perturbation δ on x targeted towards class t , and replace the original input-label pair ( x, y ) with ( x + δ, t ) .
By construction, x and t are decorrelated and the only features correlated with t are those leveraged by δ .
Because δ is constrained to be within an (cid:96) 2 ball, it can only yield non-robust features.
4 Next, we adapt the above procedure to construct (cid:98) D univ .
To this end, in step 2, we use a universal perturbation in place of an adversarial perturbation.
Comparing the signal in (cid:98) D adv and (cid:98) D univ .
We train new ResNet-18 models on the (cid:98) D
adv and (cid:98) D univ datasets and evaluate them on the original test set. The best generalization accuracies from training on (cid:98) D univ and (cid:98) D adv were 23.2% and 74.5%, respectively.
This indicates that universal non-robust features have much less generalizable signal than standard non-robust features. [Table] [Table]
5. T RANSFERABILITY OF UNIVERSAL PERTURBATIONS
Ilyas et al.
(2019) suggest that the pervasive transferability of adversarial perturbations (Papernot et al., 2016; Moosavi-Dezfooli et al., 2017a) can be attributed to different models relying on similar non-robust features.
Consequently, perturbations that leverage more shared non-robust features should transfer better.
Here we are primarily interested in transferability as a proxy for detecting shared non-robust features and we compare the transferability of adversarial and universal perturbations
[Figure]
r
[Figure]
r a t e
[Figure]
To measure transferability, we (1) perturb examples using either an adversarial perturbation or a universal perturbation on the source model, and (2) measure the probability that the perturbed input is classiﬁed as the target class on a new target model (a ResNet-18 trained independently).
To make standard and universal perturbations comparable, we only consider perturbed images that are misclassiﬁed by the source model, and also evaluate transfer on images that have labels different from the target label.
Figure 6 shows that universal perturbations transfer much less, suggesting that models share fewer universal non-robust features.
This is consistent with results from Section 5.1.
5. I NTERPOLATING UNIVERSALITY
The above experiments demonstrate that while universal non-robust features are more human-aligned, they also have less generalizable signal.
To explore the underlying trade-off, we investigate to what extent one can interpolate between the properties of universal and standard non-robust features.
To this end, we explore two different methods of “interpolating” universality:
Interpolating via changing the base set size.
One natural way to interpolate universality is to vary K , the size of the base set (cf.
Section 2.1).
K = 1 corresponds to adversarial perturbations, and large enough K corresponds to fully universal perturbations.
Table 1 shows that generalization behavior generally decreases with the size of the base set.
Notably, generalization begins to suffer even for relatively small values of K .
In contrast, the semantics of universal perturbations improves as the base set becomes larger, in line with the hypothesized trade-off (cf.
Appendix A.5).
Interpolating via restricting the source class.
We expect that examples from the same class share similar features, including non-robust ones.
This motivates us to interpolate universality along a different axis: the level of semantic hierarchy.
We consider class-universal and subclass-universal perturbations, where we restrict the base set to examples from a single class (e.g., dog) or subclass (e.g., Terrier).
If certain non-robust features exist only in examples from the same class, we expect class-universal perturbations to leverage those features.
To examine this, we repeat the experiments from Section 5.1 using these perturbations (cf.
Table 2) ; the generalization on the original test set improves modestly.
This suggests that if non-robust features are speciﬁc to small subpopulations of examples, then these subpopulations must be more ﬁne-grained than division into different (sub) classes.
Discussion Above results show that we can interpolate the signal present in universal non-robust features by varying the base set’s size, and to a smaller extent, its class diversity.
This can be intuitively captured by the following toy model:
Each example comes with a set of non-robust features, and different examples share a varying subset of features depending on their semantic distance.
By computing perturbations over a set of images, we limit the set of non-robust features to their intersection; it follows that the number of features in this intersection, and hence signal, decreases with more images and as the images become further apart.
Existing theoretical models (Tsipras et al., 2019; Allen-Zhu & Li, 2020) fail to distinguish between standard and universal perturbations, and hence fail to capture the phenomena observed here.
We believe that more empirical observations are necessary to identify a precise theoretical model.
In Appendix A, we provide additional analyses for some results in this section.
R ELATED W ORK
Moosavi-Dezfooli et al.
(2017a)
ﬁrst introduce universal adversarial perturbations.
Many followup works explore its various aspects, including possible theoretical origins, different methods of generation, and defenses (Moosavi-Dezfooli et al., 2017b; Poursaeed et al., 2018;
Khrulkov &
Oseledets, 2018; Wu & Fu, 2019; Gupta et al., 2019; Hayes & Danezis, 2019; Khrulkov & Oseledets, 2018; Liu et al., 2019; Shafahi et al., 2020).
Using ideas from analysis of universal perturbations, Jetley et al. ﬁnd that class-speciﬁc patterns unintelligible to humans can induce misclassiﬁcation while also being essential for classiﬁcation.
Ilyas et al.
ﬁrst formalize non-robust features, and demonstrate that they alone are sufﬁcient for generalization. In particular, adversarial examples are viewed as a consequence of misalignment between non-robust features and human priors. Conversely, several works discuss the usefulness and perceptual alignment of robust features (Tsipras et al., 2019; Santurkar et al., 2019).
Recent works show how models use certain signals humans do not (primarily) rely on: examples include texture, high-frequency content, and background (Geirhos et al., 2018; Yin et al., 2019; Xiao et al., 2020).
C ONCLUSION
In this work, we revisit universal perturbations and use them to analyze non-robust features.
We ﬁnd that universal perturbations are highly semantic and spatially invariant (in contrast to adversarial perturbations), and give ﬁrst evidence of perceptually aligned, interpretable non-robust features.
However, we also show that they account for only a small part of all non-robust features.
Understanding the structure and properties of the remaining bulk of non-robust features is a natural direction for future work.
More broadly, developing a framework for studying features in more ﬁne-grained ways—beyond their correlation and robustness—would be helpful for understanding our models better.
A PPENDICES
In Appendix A, we provide deferred details and analyses for results in Section 5.
In Appendix B, we describe details of the experimental setup and all hyperparameters used in the experiments.
In Appendices C to E, we give a selection of additional results for experiments in the main paper across different settings.
In Appendix F, we discuss the diversity of universal perturbations.
A D EFERRED D ETAILS FROM S ECTIONA. D ETAILS OF (cid:98) D univ CONSTRUCTION
We provide additional details for constructing the (cid:98) D univ dataset in Section 5.1.
As it is rather expensive to compute a separate universal perturbation for each input, we compute and use the same universal perturbation per batch of inputs.
One possible concern is that re-using same perturbations across the batch results in a reduction in “diversity” of features considered.
However, the total number of unique universal perturbations used does not appear to have a large impact from our experience.
In particular, using a different universal perturbation for every input does not noticeably affect the generalization of the resulting datasets.
Another complication is that for the robustness threshold (cid:15) considered here, universal perturbations fool models only on a fraction ρ of all training examples (while adversarial perturbations at the same threshold succeed on all examples).
Hence, we only include examples on which the universal perturbation succeeds (i.e., a pre-trained model classiﬁes the perturbed input as t ).
This ﬁltering step reduces the size of the dataset, so we decrease the number of examples in (cid:98) D adv accordingly in our evaluation.
We also ensure that the distribution of new and original labels is uniform and independent so that the original label is uncorrelated with the new label.
A. A LL R ESULTS ON G ENERALIZATION ON N ON -R OBUST D ATASETS
Here we collect additional results for the generalization experiment across various settings: (cid:96) 2 , (cid:96) ∞ perturbations in ImageNet-M10 and (cid:96) 2 perturbations on on CIFAR-10.
We did not conduct experiments on ImageNet, as the constructions of the datasets is computationally too intensive.
Table 3 shows that there is consistently much lower generalization from (cid:98) D univ than from (cid:98) D adv .
[Table]
A. B OUNDING R OBUST F EATURE L EAKAGE
The accuracies on the (cid:98) D univ datasets are very low (for comparison, they are even lower compared to
ResNet-18 model trained on (ﬁxed) randomly initialized features
[Table]
which achieve an accuracy greater than 30%)
Given this, one concern is that the small signal we observe can be entirely accounted for by leakage of robust features into these datasets (Ilyas et al., 2019; Goh, 2019)
[Table]
We run several additional experiments to bound the amount of leakage (we focus on (cid:96) ∞ perturbations on ImageNet-M10):
• We construct a dataset similar to (cid:98) D det (Ilyas et al., 2019) but instead using universal perturbations, where we choose new corrupted labels by cyclically shifting original labels, rather than choosing them randomly; the robust features now point away from the label.
Models trained on this new dataset achieve an accuracy up to 19.1% on the original test set, which is less than 26.5% from (cid:98) D univ , but still well above chance-level of 10%.
This demonstrates that there is residual signal in universal perturbations that cannot be entirely accounted for by robust leakage.
• We can also probe the original dataset by seeing how well different features 9 can ﬁt the dataset.
Adapting the procedure in Goh (2019), we take pre-trained features from different natural and (cid:96) ∞ robust models on ImageNet-M10, and train a linear classiﬁer over those features on (cid:98) D univ .
The results in Table 4 show that features from natural models, even sourced from a different model than the one used to generate (cid:98) D univ or from a different architecture, capture more of the signal in (cid:98) D univ than robust features.
For comparison, the model trained on (cid:98) D univ from scratch, as we saw earlier, achieves an accuracy of 26.5%, which is higher than 22.3%.
This demonstrates again that while some leakage is happening, it cannot account for all of the signal in universal perturbations.
[Table]
• The scaling analysis from Section 4 also gives supporting evidence, from a different angle.
These results consistently show that while some robust leakage occurs, it cannot account for all of the signal in universal perturbations.
A small but non-trivial amount of signal comes from the universal non-robust features.
One plausible hypothesis for the degradation of signal quality between K = 1 and higher K is that optimizing the perturbation over more images splits the effective perturbation norm budget, and is akin to optimizing over each image separately with a reduced budget, say (cid:15)/K .
To test this hypothesis, we constructed (cid:98) D adv with a reduced budget of (cid:15)
= 3 .
We compare the generalization [-is on par with (cid:15) = 6 . 
 (and is much higher than-]from this new dataset to that from the K = 2 , (cid:15) = 6 . 0 case in Table 5 (latter is the same value as in Table 1). Given the large gap, it seems unlikely that this simple norm budget sharing can explain the above interpolation phenomena.
This suggests that when universal perturbations are computed over multiple images, different non-robust features in different images are interacting in a non-trivial manner. [Table] [Figure]
[Figure]
Table 1 shows that generalization decreases with the size of the base set.
In contrast, Figure 7 shows that the semantic quality of universal perturbations improves as the base set becomes larger.
This suggests a possible trade-off between semantic quality and generalizable signal, at least along this particular axis.
However, a more careful examination suggests that the picture might be more nuanced: better semantics only become obvious at higher values of K ( K (cid:38) 64 ), whereas generalization suffers even at relatively small values of K ( K ∼ 16 ).
Understanding whether these trends are both manifestations of the same underlying phenomena require further work.
A. A LTERNATE B ASE S ETS
In this paper, we focused on computing universal perturbations over base sets (cf. Section 2.1) consisting of different images (from possibly a restricted set of classes). To study the importance of sample diversity in the base set, we computed universal perturbations over the base set consisting of K different augmentations of a single image.
The resulting universal perturbations still have similar (but less salient) visual characteristics, and are transferable but to a lesser degree than perturbations computed over base set of different images.
B E XPERIMENTAL S ETUP B. D ATASETS
{+Our experiments use CIFAR-10 and ImageNet-M10, a subset of ImageNet ILSVRC
+}
(Russakovsky et al., 2015); we focus primarily on ImageNet-M10.
ImageNet-M10 consists of ten super-classes, each corresponding a WordNet (Miller, 1995) ID in the hierarchy.
Different super-classes contain varying number of ImageNet classes, so the dataset is balanced by choosing six classes within each super-class.
The class numbers of the corresponding
{+ImageNet classes are shown in Table 
+}
For most experiments, the super-classes correspond to the ten labels; for experiments using ﬁne-grained classes, we use the 60 (sub) classes as labels.
A sample image from each class is shown in Figure
The ImageNet-M10 dataset can be used easily on top of a standard ImageNet directory by loading the mixed_10 dataset from the robustness library (Engstrom et al., 2019).
ImageNet-M10 is also similar to restricted ImageNet, which has been used for adversarial robustness research (Ilyas et al., 2019;
Santurkar et al., 2019).
[Figure]
insect monkey car feline truck fruit fungus boat
[Table]
B. H YPERPARAMETERS
All experiments in the paper involve a combination of: training a model, computing adversarial perturbations, and computing universal perturbations.
Below, we provide details for all the hyperparameters and their selection.
Robustness threshold We use the same robustness threshold throughout: (cid:96) 2 , (cid:15) = 6 .
and (cid:96) ∞ , (cid:15) = 8 / 255 for ImageNet datasets, and (cid:96) 2 , (cid:15) = 1 . 0 for CIFAR-10.
B.2. T RAINING M ODELS
Training base models All models are trained with SGD with momentum and standard data augmentation.
For corresponding robust models, the same optimization parameters are used with PGD training (Madry et al., 2017) using 3 steps and attack step size 23 (cid:15) (for both (cid:96) 2 and (cid:96) ∞ ).
Weight decay of 5e-4 was used in all cases.
The accuracies of these models are shown in Table 7. experiments.
[Table]
Training models on constructed datasets The models on the constructed datasets, including (cid:98) D univ and (cid:98)
D adv (Section 5.1), are trained over the following grid of hyperparameters: three learning rates (0.01, 0.05, 0.1), two batch sizes (128, 256), including/ not including a single learning rate drop by a factor of 10.
All models are trained with 400 epochs, standard data augmentation, and weight decay 5 · 10 − 4 .
10 We report the setting corresponding to the highest test accuracies in Table 8.
As an exception, for the K -interpolated datasets (Section 5.3), we ﬁxed a single set of hyperparameters to train models.
B.2. C OMPUTING ADVERSARIAL PERTURBATIONS
Adversarial perturbations For a given (cid:15) , PGD with an attack step size of (cid:15) 3 and 10 steps is used (no random restarts.) [Table]
Universal perturbations The hyperparameters of the PGD algorithm for universal perturbations are: learning rate schedule 11 , number of epochs [Table] and batch size.
In order to see the impact of the choice of these parameters, we ran a grid search over the parameters; a sample of these selections is visible in Figure 9.
Overall, we observe that the exact choice of these parameters did not have a large impact on the ﬁnal training accuracy (and the quality of the resulting perturbations, both in terms of visuals and generalization to other images) as long as (1) the initial learning rate is not too low or too high, and (2) a sufﬁcient number of epochs is used.
So in all of our experiments, we ﬁx a learning rate of 2.0 and a batch size of 128 or 256, and trained a sufﬁcient number of epochs (max of 100).
The use of a learning rate decay did not seem to have much impact, so we use the same learning rate throughout.
C V ISUALIZATIONS
We show visualizations of universal perturbations across different datasets (CIFAR-10, ImageNet-M, and ImageNet), architectures (VGG for Mixed10), and norm constraint ( (cid:96) ∞ for Mixed10).
Normalization for visualization.
For all visualizations, perturbations were rescaled as follows (for each input channel): ﬁrst truncated to [ − 3 σ, 3 σ ] (where σ is the standard deviation across all channels and pixels), then rescaled and shifted to lie in [0 , 1] .
C. D IFFERENT DATASETS
[Figure]
T r a i n i n g
A cc u r a c y
[Figure] [Figure] [Figure]
[Figure] [Figure]
D L OCALITY A NALYSIS
We provide full results of randomized locality analysis on ImageNet-M10 in Figures 14 and 15.
We did not perform the analysis on CIFAR-10 perturbations, as the perturbations are more “global” by nature, e.g. the semantic part seems to already occupy most of the perturbation.
Exploring properties of (cid:96) 2 vs (cid:96) ∞ perturbations. For both (cid:96) 2 and (cid:96) ∞ universal perturbations, the patches that are most identiﬁable with their classes have much higher ASR. A major difference is that for (cid:96) 2 perturbations, the patches with low ASR have much smaller norms, whereas for (cid:96) ∞ perturbations, the patches with low ASR have the similar norms (measured in either (cid:96) 2 or (cid:96) ∞ ). This is expected, as for (cid:96) 2 every local patch contributes to the total (cid:96) 2 norm and it is unsurprising that optimization avoid putting weight on patches with little signal; in contrast, for (cid:96) ∞ perturbations, different patches are independent in meeting the norm bound, so the perturbation can have patches
[Figure] [Figure] [Figure]
with low signal that are similar in norm.
However, it is not clear whether these low signal patches found by optimization serve some purpose.
To test this, we experimented with zeroing-out these patches or replacing them with copies of patches with the highest ASR.
Evaluation of the resulting perturbations show that the ASR is highest with these patches in place, and are in fact lowered when we replace them with patches with higher ASR.
This suggests that these low signal patches have a non-linear interaction with rest of the perturbation, and are providing a small boost in signal when combined with other patches.
Understanding this phenomena in more depth may shed more light on the differences between properties of (cid:96) ∞ and (cid:96) 2 universal perturbations.
E S PATIAL I NVARIANCE
Section 3.2 shows a sample of translation invariance evaluation for perturbations on ImageNet-M10.
We include additional samples from evaluation using (cid:96) 2 perturbations on CIFAR-10, to check that the phenomena is general.
[Figure]
We used a subsampled grid of all the possible offsets with strides of four pixels.
As before, the value at coordinate ( i, j ) represents the average attack success rate when the perturbations are shifted by i pixels along the x -axis and j pixels along the y -axis, with wrap around to preserve information. Evaluation of adversarial perturbation is only shown for one target as it is redundant.
F P ERTURBATION D IVERSITY
Throughout our investigations, we observed that the computed universal perturbations seemed to lack in diversity.
We see in Figure 17 (ﬁrst row) that generating multiple different universal perturbations for a given target leads to visually similar perturbations.
14 Prior works study generating more diverse universal perturbations (Reddy Mopuri et al., 2018).
To increase diversity, we generated universal perturbations with an additional penalty to encourage orthogonality of perturbed representations at different layers of the network.
Even with the added regularization, the resulting perturbations appear to be remain similar, but with changes in spatial arrangements of the semantic patterns (second row).
One effective way we found to increase the diversity of these perturbations is to compute universal perturbations for targeting speciﬁc sub-classes of Mixed10.
To do so, we ﬁrst train a ﬁne-grained model using Mixed10’s subclass labels, and generate universal perturbations for each of the six subclasses per class.
In Figure 18, we show the universal perturbations targeting individual sub-classes of the bird class.
We observe distinct patterns characteristic of each species of birds, an increase in diversity from before.
However, generation of these perturbations requires a model trained on the ﬁne-grained labels.
An interesting question is whether one can recover such diversity using just the model trained on coarse-grained labels without injection of additional label information.
d i ff e r e n t i n i t i a li z a t i o n o r t h
o g o n a li t y a t 2 n d l a y e r e x a m p l e un i v e r s a l p e r t u r b a t i o n Observe the distinctive visual characteristics of each class, for instance the outline of a chicken (column 2), the neck of an ostrich (column 3), and the distinctive colors of different species (columns 3-5).