L INKING AVERAGE - AND WORST - CASE PERTURBATION ROBUSTNESS VIA CLASS SELECTIVITY AND DIMENSIONALITY
Anonymous authors
Paper under double-blind review
A BSTRACT
Representational sparsity is known to affect robustness to input perturbations in deep neural networks (DNNs), but less is known about how the semantic content of representations affects robustness.
Class selectivity—the variability of a unit’s responses across data classes or dimensions—is one way of quantifying the sparsity of semantic representations.
Given recent evidence that class selectivity may not be necessary for, and in some cases can impair generalization, we sought to investigate whether it also confers robustness (or vulnerability) to perturbations of input data.
We found that class selectivity leads to increased vulnerability to average-case (naturalistic) perturbations in ResNet18, ResNet50, and ResNet20, as measured using Tiny ImageNetC (ResNet18 and ResNet50) and CIFAR10C (ResNet20).
Networks regularized to have lower levels of class selectivity are more robust to average-case perturbations, while networks with higher class selectivity are more vulnerable.
In contrast, we found that class selectivity increases robustness to multiple types of worst-case (i.e. white box adversarial) perturbations, suggesting that while decreasing class selectivity is helpful for average-case perturbations, it is harmful for worst-case perturbations.
To explain this difference, we studied the dimensionality of the networks’ representations: we found that the dimensionality of early-layer representations is inversely proportional to a network’s class selectivity, and that adversarial samples cause a larger increase in early-layer dimensionality than corrupted samples.
We also found that the input-unit gradient was more variable across samples and units in high-selectivity networks compared to low-selectivity networks.
These results lead to the conclusion that units participate more consistently in low-selectivity regimes compared to high-selectivity regimes, effectively creating a larger attack surface and hence vulnerability to worst-case perturbations.
I NTRODUCTION
Methods for understanding deep neural networks (DNNs) often attempt to ﬁnd individual neurons or small sets of neurons that are representative of a network’s decision (Erhan et al., 2009; Zeiler and Fergus, 2014; Karpathy et al., 2016;
Amjad et al., 2018; Lillian et al., 2018; Dhamdhere et al., 2019; Olah et al., 2020).
Selectivity in individual units (i.e. variability in a neuron’s activations across semantically-relevant data features) has been of particular interest to researchers trying to better understand deep neural networks (DNNs)
(Zhou et al., 2015; Olah et al., 2017; Morcos et al., 2018;
Zhou et al., 2018; Meyes et al., 2019;
Na et al., 2019;
Zhou et al., 2019; Rafegas et al., 2019;
Bau et al., 2020; Leavitt and Morcos, 2020).
However, recent work has shown that selective neurons can be irrelevant, or even detrimental to network performance, emphasizing the importance of examining distributed representations for understanding DNNs (Morcos et al., 2018; Donnelly and Roegiest, 2019; Dalvi et al., 2019b; Leavitt and Morcos, 2020).
In parallel, work on robustness seeks to build models that are robust to perturbed inputs (Szegedy et al., 2013; Carlini and Wagner, 2017a;b; Vasiljevic et al., 2016; Kurakin et al., 2017; Gilmer et al., 2018; Zheng et al., 2016).
Hendrycks and Dietterich (2019) distinguish between two types of robustness: corruption robustness, which measures a classiﬁer’s performance on low-quality or naturalistically-perturbed inputs—and thus is an "average-case" measure—and adversarial robustness, which measures a classiﬁer’s performance on small, additive perturbations that are tailored to the classiﬁer—and thus is a "worst-case" measure.
Research on robustness has been predominantly focused on worst-case perturbations, which is affected by weight and activation sparsity (Madry et al., 2018;
Balda et al., 2020; Ye et al., 2018; Guo et al., 2018; Dhillon et al., 2018) and representational dimensionality (Langeberg et al., 2019; Sanyal et al., 2020; Nayebi and Ganguli, 2017).
But less is known about the mechanisms underlying average-case perturbation robustness and its common factors with worst-case robustness.
Some techniques for improving worst-case robustness also improve average-case robustness (Hendrycks and Dietterich, 2019; Ford et al., 2019; Yin et al., 2019), thus it is possible that sparsity and representational dimensionality also contribute to average-case robustness.
Selectivity in individual units can be also be thought of a measure of the sparsity with which semantic information is represented. And because class selectivity regularization provides a method for controlling selectivity, and class selectivity regularization has been shown to improve test accuracy on unperturbed data (Leavitt and Morcos, 2020), we sought to investigate whether it could be utilized to improve perturbation robustness and elucidate the factors underlying it.
In this work we pursue a series of experiments investigating the causal role of selectivity in robustness to worst-case and average-case perturbations in DNNs.
To do so, we used a recently-developed class selectivity regularizer (Leavitt and Morcos, 2020) to directly modify the amount of class selectivity learned by DNNs, and examined how this affected the DNNs’ robustness to worst-case and average-case perturbations.
Our ﬁndings are as follows:
• Networks regularized to have lower levels of class selectivity are more robust to average-case perturbations, while networks with higher class selectivity are generally less robust to average-case perturbations, as measured in ResNets using the Tiny ImageNetC and CIFAR10C datasets.
The corruption robustness imparted by regularizing against class selectivity was consistent across nearly all tested corruptions.
• In contrast to its impact on average-case perturbations, decreasing class selectivity reduces robustness to worst-case perturbations in both tested models, as assessed using gradient-based white-box attacks.
• The variability of the input-unit gradient across samples and units is proportional to a network’s overall class selectivity, indicating that high variability in perturbability within and across units may facilitate worst-case perturbation robustness.
• The dimensionality of activation changes caused by corruption markedly increases in early layers for both perturbation types, but is larger for worst-case perturbations and low-selectivity networks.
This implies that representational dimensionality may present a trade-off between worst-case and average-case perturbation robustness.
Our results demonstrate that changing class selectivity, and hence the sparsity of semantic representations, can confer robustness to average-case or worst-case perturbations, but not both simultaneously.
They also highlight the roles of input-unit gradient variability and representational dimensionality in mediating this trade-off.
R ELATED W ORK 2. P ERTURBATION R OBUSTNESS
The most commonly studied form of robustness in DNNs is robustness to adversarial attacks, in which an input is perturbed in a manner that maximizes the change in the network’s output while of sparsity. For example, if a network has high test accuracy on a classiﬁcation task, it is necessarily representing class (semantic) information. But if the mean class selectivity across units is low, then the individual units do not contain much class information, thus the class information must be distributed across units; the semantic representation in this case is not sparse, it is distributed. attempting to minimize or maintain below some threshold the magnitude of the change to the input (Serban et al., 2019; Warde-Farley and Goodfellow, 2017) .
Because white-box adversarial attacks are optimized to best confuse a given network, robustness to adversarial attacks are a "worst-case" measure of robustness.
Two factors that have been proposed to account for DNN robustness to worst-case perturbations are particularly relevant to the present study: sparsity and dimensionality.
Multiple studies have linked activation and weight sparsity with robustness to worst-case perturbations.
Adversarial training improves worst-case robustness Goodfellow et al.
; Huang et al.
and results in sparser weight matrices (Madry et al., 2018; Balda et al., 2020).
Methods for increasing the sparsity of weight matrices (Ye et al., 2018; Guo et al., 2018) and activations (Dhillon et al., 2018) likewise improve worst-case robustness, indicating that the weight sparsity caused by worst-case perturbation training is not simply a side-effect.
Researchers have also attempted to understand the nature of worst-case robustness from a perspective complementary to that of sparsity: dimensionality.
Like sparsity, worst-case perturbation training reduces the rank of weight matrices and representations, and regularizing weight matrices and representations to be low-rank can improve worst-case perturbation robustness (Langeberg et al., 2019; Sanyal et al., 2020; Nayebi and Ganguli, 2017).
Taken together, these studies support the notion that networks with low-dimensional representations are more robust to worst-case perturbations.
Comparatively less research has been conducted to understand the factors underlying averagecase robustness.
Certain techniques for improving worst-case perturbation robustness also help against average-case perturbations (Hendrycks and Dietterich, 2019;
Geirhos et al., 2018; Ford et al., 2019).
Examining the frequency domain has elucidated one mechanism: worst-case perturbations for "baseline" models tend to be in the high frequency domain, and improvements in averagecase robustness resulting from worst-case robustness training are at least partially ascribable to models becoming less reliant on high-frequency information (Yin et al., 2019; Tsuzuku and Sato, 2019; Geirhos et al., 2018).
But it remains unknown whether other factors such as sparsity and dimensionality link these two forms of robustness.
2. C LASS SELECTIVITY
One technique that has been of particular interest to researchers trying to better understand deep (and biological) neural networks is examining the selectivity of individual units (Zhou et al., 2015;
Olah et al., 2017; Morcos et al., 2018;
Zhou et al., 2018; Meyes et al., 2019;
Na et al., 2019;
Zhou et al., 2019; Rafegas et al., 2019;
Bau et al., 2020; Leavitt and Morcos, 2020; Sherrington, 1906;
Kandel et al., 2000).
Evidence regarding the importance of selectivity has mostly relied on single unit ablation, and has been equivocal (Radford et al., 2017; Morcos et al., 2018; Amjad et al., 2018;
Zhou et al., 2018; Donnelly and Roegiest, 2019; Dalvi et al., 2019a).
However Leavitt and Morcos (2020) examined the role of single unit selectivity in network performance by regularizing for or against class selectivity in the loss function, which sidesteps the limitations of single unit ablation and correlative approaches and allowed them to investigate the causal effect of class selectivity.
They found that reducing class selectivity has little negative impact on—and can even improve—test accuracy in CNNs trained on image recognition tasks, but that increasing class selectivity has signiﬁcant negative effects on test accuracy.
However, their study focused on examining the effects of class selectivity on test accuracy in unperturbed (clean) inputs.
Thus it remains unknown how class selectivity affects robustness to perturbed inputs, and whether class selectivity can serve as or elucidate a link between worst-case and average-case robustness.
A PPROACH
A detailed description of our approach is provided in Appendix A.1.
Models and training protocols Our experiments were performed on ResNet18 and ResNet50 (He et al., 2016) trained on Tiny ImageNet (Fei-Fei et al., 2015), and ResNet20 (He et al., 2016) trained on CIFAR10 (Krizhevsky, 2009).
We focus primarily on the results for ResNet18 trained on Tiny ImageNet in the main text for space, though results were qualitatively similar for ResNet50, and ResNet20 trained on CIFAR10.
Experimental results were obtained with model parameters from the epoch that achieved the highest validation set accuracy over the training epochs, and 20 replicate models (ResNet18 and ResNet20) or 5 replicate models (Resnet50) with different random seeds were run for each hyperparameter set.
Class selectivity index Following (Leavitt and Morcos, 2020).
A unit’s class selectivity index is calculated as follows:
At every ReLU, the activation in response to a single sample was averaged across all elements of the ﬁlter map (which we refer to as a "unit").
The class-conditional mean activation was then calculated across all samples in the clean test set, and the class selectivity index ( SI ) was calculated as follows:
[Equation]
where µ max is the largest class-conditional mean activation and µ − max is the mean response to the remaining (i.e. non- µ max ) classes.
The selectivity index ranges from 0 to 1.
A unit with identical average activity for all classes would have a selectivity of 0, and a unit that only responds to a single class would have a selectivity of 1.
As Morcos et al.
(2018) note, the selectivity index is not a perfect measure of information content in single units.
For example, a unit with a litte bit of information about many classes would have a low selectivity index.
However, it identiﬁes units that are class-selective similarly to prior studies (Zhou et al., 2018).
Most importantly, it is differentiable with respect to the model parameters.
Class selectivity regularization We used (Leavitt and Morcos, 2020)’s class selectivity regularizer to control the levels of class selectivity learned by units in a network during training.
Class selectivity regularization is achieved by minimizing the following loss function during training:
[Equation]
The left-hand term in the loss function is the standard classiﬁcation cross-entropy, where c is the class index, C is the number of classes, y c is the true class label, and ˆ y c is the predicted class probability.
The right-hand component of the loss function, − αµ SI , is the class selectivity regularizer.
The regularizer consists of two terms: the selectivity term,
[Equation]
where l is a convolutional layer, L is number of layers, u is a unit, U is the number of units in a given layer, and SI u is the class selectivity index of unit u .
The selectivity term of the regularizer is obtained by computing the selectivity index for each unit in a layer, then computing the mean selectivity index across units within each layer, then computing the mean selectivity index across layers.
Computing the mean within layers before computing the mean across layers (as compared to computing the mean across all units in the network) mitigates the biases induced by the larger numbers of units in deeper layers.
The other term in the regularizer is α , the regularization scale, which determines whether class selectivity is promoted or discouraged.
Negative values of α discourage class selectivity in individual units and positive values encourage it.
The magnitude of α controls the contribution of the selectivity term to the overall loss.
During training, the class selectivity index was computed for each minibatch.
The ﬁnal (logit) layer was not subject to selectivity regularization or included in our analyses because by deﬁnition, the logit layer must be class selective in a classiﬁcation task.
Measuring average-case robustness To evaluate robustness to average-case perturbations, we tested our networks on CIFAR10C and Tiny ImageNetC, two benchmark datasets consisting of the
CIFAR10 or Tiny ImageNet data, respectively, to which a set of naturalistic corruptions have been applied (Hendrycks and Dietterich, 2019, examples in Figure A1).
We average across all corruption types and severities (see Appendix A.1.2 for details) when reporting corrupted test accuracy.
Measuring worst-case robustness We tested our models’ worst-case (i.e. adversarial) robustness using two methods.
The fast gradient sign method (FGSM) (Goodfellow et al., 2015) is a simple attack that computes the gradient of the loss with respect to the input image, then scales the image
’s pixels (within some bound) in the direction that increases the loss.
The second method, projected gradient descent (PGD) (Kurakin et al., 2016;
Madry et al., 2018), is an iterated version of FGSM.
We used a step size of 0.0001 and an l ∞ norm perturbation budget ( (cid:15) ) of 16/255.
Computing the stability of units and layers To quantify variation in networks’ perturbability, we ﬁrst computed the l 2 norm of the input-unit gradient for each unit u in a network.
We then computed the mean ( µ u ) and standard deviation ( σ u ) of the norm across samples for each unit. σ u /µ u yields [Figure]
[Figure] the coefﬁcient of variation (Everitt, 2002) for a unit ( CV u ), a measure of variation in perturbability for individual units.
We also quantiﬁed the variation across units in a layer by computing the standard deviation of µ u across units in a layer l , σ ( µ u ) =
σ l , and dividing this by the corresponding mean across units µ ( µ u ) = µ l , to yield the CV across units σ l /µ l = CV l .
R ESULTS 4. A VERAGE - CASE ROBUSTNESS IS INVERSELY PROPORTIONAL TO CLASS SELECTIVITY
Certain kinds of sparsity—including reliance on single directions (Morcos et al., 2018), and the semantic sparsity measured by class selectivity (Leavitt and Morcos, 2020)—have been shown to impair network performance.
We sought to extend this question to robustness: how does the sparsity of semantic representations affect robustness to average-case perturbations of the input data?
We used a recently-introduced method (Leavitt and Morcos (2020); Approach 3) to modulate the amount of class selectivity learned by DNNs (Figure A2 demonstrates effects of selectivity regularization).
We then examined how this affected performance on Tiny ImageNetC and CIFAR10C, two benchmark datasets for average-case corruptions (Approach 3; example images in Figure A1).
Changing the level of class selectivity across neurons in a network could one of the following effects on corruption robustness: If concentrating semantic representations into fewer neurons (i.e. promoting semantic sparsity) provides fewer potent dimensions on which perturbed inputs can act, then increasing class selectivity should confer networks with robustness to average-case perturbations, while reducing class selectivity should render networks more vulnerable.
Alternatively, if distributing semantic representations across more units (i.e. reducing sparsity) dilutes the changes induced by perturbed inputs, then reducing class selectivity should increase a network’s robustness to averagecase perturbations, while increasing class selectivity should reduce robustness.
We found that decreasing class selectivity leads to increased robustness to average-case perturbations for both ResNet18 tested on Tiny ImageNetC (Figure 1) and ResNet20 tested on CIFAR10C (Figure
A6).
In ResNet18, we found that mean test accuracy on corrupted inputs increases as class selectivity decreases (Figure 1), with test accuracy reaching a maximum at regularization scale α = − 2 {+. 
+}
(mean test accuracy across corruptions and severities at α − 2 .
= 17), representing a 3.5 percentage point (pp) increase relative to no selectivity regularization (i.e. α 0 ; test accuracy at α 0 = 13 . 5 ).
In contrast, regularizing to increase class selectivity has either no effect or a negative impact on corruption robustness.
Corrupted test accuracy remains relatively stable until α = 1 . 0 , after which point it declines.
The results are qualitatively similar for ResNet50 tested on Tiny ImageNetC (Figure A9), and for ResNet20 tested on CIFAR10C (Figure A6), except the vulnerability to corruption caused by increasing selectivity is even more dramatic in ResNet20.
We also found similar results when controlling for the difference in clean accuracy for models with different α (Appendix A.3). We observed that regularizing to decrease class selectivity causes robustness to average-case perturbations.
But it’s possible that the causality is unidirectional, leading to the question of whether the
[Figure] converse is also true: does increasing robustness to average-case perturbations cause class selectivity to decrease?
We investigated this question by training with AugMix, a technique known to improve worst-case robustness (Hendrycks et al., 2020a). We found that AugMix does indeed decrease the mean level of class selectivity across neurons in a network (Appendix A.4; Figure A11). AugMix decreases overall levels of selectivity similarly to training with a class selectivity regularization scale of approximately α = − 0 .
1 or α = − 0 . 2 in both ResNet18 trained on Tiny ImageNet (Figures A11a and A11b) and ResNet20 trained on CIFAR10 (Figures A11c and A11d).
These results indicate that the causal relationship between average-case perturbation robustness and class selectivity is bidirectional: not only does decreasing class selectivity improve average-case perturbation robustness, but improving average-case perturbation-robustness also causes class selectivity to decrease.
We also found that the effect of class selectivity on perturbed robustness is consistent across corruption types.
Regularizing against selectivity improves perturbation robustness in all 15 Tiny ImageNetC corruption types for ResNet18 (Figure A4) and 14 of 15 Tiny ImageNetC corruption types in ResNet (Figure A10), and 14 of 19 corruption types in CIFAR10C for ResNet20 (Figure A7). Together these results demonstrate that reduced class selectivity confers robustness to average-case perturbations, implying that distributing semantic representations across neurons—i.e. low sparsity—may dilute the changes induced by average-case perturbations.
4. C LASS SELECTIVITY IMPARTS WORST - CASE PERTURBATION ROBUSTNESS
We showed that the sparsity of a network’s semantic representations, as measured with class selectivity, is causally related to a network’s robustness to average-case perturbations.
But how does the sparsity of semantic representations affect worst-case robustness?
We addressed this question by testing our class selectivity-regularized networks on inputs that had been perturbed using using one of two gradient-based methods (see Approach 3).
If distributing semantic representations across units provides more dimensions upon which a worstcase perturbation is potent, then worst-case perturbation robustness should be proportional to class selectivity.
However, if increasing the sparsity of semantic representations creates more responsive individual neurons, then worst-case robustness should be inversely proportional to class selectivity.
Unlike average-case perturbations, decreasing class selectivity decreases robustness to worst-case perturbations for ResNet18 (Figure 2) and ResNet50 (Figure A13) trained on Tiny ImageNet, and ResNet20 trained on CIFAR10 (Figures A12).
For small perturbations (i.e. close to x=0), the effects of class selectivity regularization on test accuracy (class selectivity is inversely correlated with unperturbed test accuracy) appear to overwhelm the effects of perturbations.
But as the magnitude of perturbation increases, a stark ordering emerges: test accuracy monotonically decreases as a function of class selectivity in ResNet18 and ResNet50 for both FGSM and PGD attacks (ResNet18: Figures 2a and 2b; ResNet50: Figures A13a and A13b).
The ordering is also present for ResNet20, though less consistent for the two networks with the highest class selectivity ( α = 0 .
7 and α = 1 . 0 ).
However, increasing class selectivity is much more damaging to test accuracy in ResNet20 trained on CIFAR10 compared to ResNet18 trained on Tiny ImageNet (Leavitt and Morcos, 2020, Figure
A2), so
the the substantial performance deﬁcits of extreme selectivity in ResNet20 likely mask the perturbation-robustness.
This result demonstrates that networks with sparse semantic representations are less vulnerable to worst-case perturbation than networks with distributed semantic representations.
We also veriﬁed that the worst-case robustness of high-selectivity networks is not fully explained by gradient-masking (Athalye et al., 2018, Appendix A.5).
Interestingly, class selectivity regularization does not appear to affect robustness to "natural" adversarial examples (Appendix A.6), which are "unmodiﬁed, real-world examples...selected to cause a model to make a mistake" (Hendrycks et al., 2020b).
Performance on ImageNet-A, a benchmark of natural adversarial examples (Hendrycks et al., 2020b), was similar across all tested values of α for both ResNet18 (Figure A15a) and ResNet50 (Figure A15b), indicating that class selectivity regularization may share some limitations with other methods for improving both worst-case and average-case robustness, many of which also fail to yield signiﬁcant robustness improvements against
ImageNet-A (Hendrycks et al., 2020b).
We found that regularizing to increase class selectivity causes robustness to worst-case perturbations.
But is the converse true? Does increasing robustness to worst-case perturbations also cause class selectivity to increase? We investigated this by training networks with a commonly-used technique to improve worst-case perturbation robustness, PGD training. We found that PGD training does indeed increase the mean level of class selectivity across neurons in a network, and this effect is proportional to the strength of PGD training: networks trained with more strongly-perturbed samples have higher class selectivity (Appendix A.7).
This effect was present in both ResNet18 trained on
Tiny ImageNet (Figure A16c) and ResNet20 trained on CIFAR10 (Figure A16f), indicating that the causal relationship between worst-case perturbation robustness and class selectivity is bidirectional.
Networks whose outputs are more stable to small input perturbations are known to have improved generalization performance and worst-case perturbation robustness (Drucker and Le Cun, 1992;
Novak et al., 2018; Sokolic et al., 2017; Rifai et al., 2011; Hoffman et al., 2019).
To examine whether increasing class selectivity improves worst-case perturbation robustness by increasing network stability, we analyzed each network’s input-output Jacobian, which is proportional to its stability—a large-magnitude Jacobian means that a small change to the network’s input will cause a large change to its output.
If class selectivity induces worst-case robustness by increasing network stability, then networks with higher class selectivity should have smaller Jacobians.
But if increased class selectivity induces adversarial robustness through alternative mechanisms, then class selectivity should have no effect on the Jacobian.
We found that the l 2 norm of the input-output Jacobian is inversely proportional to class selectivity for ResNet18 (Figure 2c), ResNet50 (Figure A13c), and ResNet (Figure A12c), indicating that distributed semantic representations are more vulnerable to worst-case perturbation because they are less stable than sparse semantic representations.
4. V ARIABILITY OF THE INPUT - UNIT GRADIENT ACROSS SAMPLES AND UNITS
We observed that the input-output Jacobian is proportional to worst-case vulnerability and inversely proportional to class selectivity, but focusing on input-output stability potentially overlooks phenomena present in hidden layers and units.
If class selectivity imparts worst-case robustness by making individual units less reliably perturbable—because each unit is highly tuned to a particular subset of images—then we should expect to see more variation across input-unit gradients for units in high-selectivity networks compared to units in low-selectivity networks.
Alternatively, worst-case robustness in high-selectivity networks could be achieved by reducing both the magnitude and variation of units’ perturbability, in which case we would expect to observe lower variation across input-unit gradients for units in high-selectivity networks compared to low-selectivity networks.
We quantiﬁed variation in unit perturbability using the coefﬁcient of variation of the input-unit gradient across samples for each unit ( CV u ; Approach 3).
The CV is a measure of variability that normalizes the standard deviation of a quantity by the mean.
A large CV indicates high variability, a small CV indicates low variability.
To quantify variation in perturbability across units, we computed the CV across units in each layer, ( CV l ; Approach 3).
We found that units in high-selectivity networks exhibited greater variation in their perturbability than units in low-selectivity networks, both within individual units and across units in each layer.
This effect was present in both ResNet18 trained on Tiny ImageNet (Figure 3) and ResNet20 trained on CIFAR10 (Figure A18), although the effect was less consistent for across-unit variability in later layers in ResNet18 (Figure 3b).
Interestingly, class selectivity affects both the numerator ( σ ) and denominator ( µ ) of the CV calculation for both the CV across samples and CV across units (Appendix A.8).
These results indicate that that high class selectivity imparts worst-case robustness by increasing the variation in perturbability within and across units, while the worst-case vulnerability associated with low class selectivity results from more consistently perturbable units.
It is worth noting that the inverse can be stated with regards to average-case robustness: low variation in perturbability both within and across units in low-selectivity networks is associated with robustness to average-case perturbations, despite the these units (and networks) being more perturbable on average.
4. D IMENSIONALITY IN EARLY LAYERS PREDICTS PERTURBATION VULNERABILITY
[Figure] i [Figure] Prior research has elucidated the mechanisms of worstcase perturbations using the framework of dimensionality (Langeberg et al., 2019;
Nayebi and Ganguli, 2017; Sanyal et al., 2020). Investigating the dimensionality of the changes induced by average-case and worst-case perturbations could reveal a common factor linking them to each other and to class selectivity [Figure]
One possible explanation for the discrepancy between class selectivity’s impact on worst-case and average-case corruptions is that different corruption types impact representations with varying dimensionalities.
For example, if only a few neurons are needed to change a network’s decision, the dimensionality of the change in the representation due to perturbation might be very low, as only a few units need to be modiﬁed.
We thus measured dimensionality using a straightforward, linear method: we applied Principal Component Analysis (PCA) to the activation matrices of each layer in our networks and computed the number of components necessary to explain 95% of the variance (and replicated our results with different variance thresholds; Appendix A.1.4; Appendix A.9).
We ﬁrst examined the dimensionality of the representations of the clean test data.
If the sparsity of semantic representations is reﬂected in dimensionality, then networks with more class selectivity should have lower-dimensional representations than networks with less class selectivity.
Alternatively, if high-selectivity representations are of similar dimensionality to low-selectivity representations—though perhaps occupying different sub-spaces—then dimensionality would be unaffected by class selectivity.
We found that the sparsity of a DNN’s semantic representations corresponds directly to the dimensionality of those representations.
Dimensionality is inversely proportional to class selectivity in early
ResNet18 layers ( ≤ layer 9; Figure 4a), and across all of ResNet20 (Figure [-A15d). Networks with a) 
 8 10 12 14 16 Layer 0.-]A21d).
[-b) 0 2 4 6 8 10 12 14 16 Layer 0. F r a c t i o n o f T o t a l D i m e n s i o n a li t y c) 
 2 4 6 8 10 12 14 16 Layer 0. F r a c t i o n o f T o t a l D i m e n s i o n a li t y-]Networks with higher class selectivity tend to have lower dimensionality, and networks with lower class selectivity tend to have higher dimensionality.
These results show that the sparsity of a network’s semantic representations is indeed reﬂected in those representations’ dimensionality.
We next examined the dimensionality of perturbation-induced changes in representations by subtracting the perturbed activation matrix from the clean activation matrix and computing the dimensionality of this "difference matrix" (see Appendix A.1.4).
Intuitively, this metric quantiﬁes the dimensionality of the change in the representation caused by perturbing the input.
If it is small, the perturbation impacts fewer units, while if it is large, more units are impacted.
Interestingly, we found that the dimensionality of the changes in activations induced by both average-case (Figure 4b) and worst-case perturbations (Figure 4c) was notably higher for networks with reduced class-selectivity, suggesting that decreasing class selectivity causes changes in input to become more distributed.
We found that the activation changes caused by average-case perturbations are higher-dimensional than the representations of the clean data in both ResNet18 (compare Figures 4b and 4a) and ResNet (Figures A21e and A21d), and that this effect is inversely proportional to class selectivity (Figures 4b and A21e); the increase in dimensionality from average-case perturbations was more pronounced in low-selectivity networks than in high-selectivity networks.
These results indicate that class selectivity not only predicts the dimensionality of a representation, but also the change in dimensionality induced by an average-case perturbation.
Notably, however, the increase in early-layer dimensionality was much larger for worst-case perturbations than average-case perturbations (Figure 4c; Figure A21f) .
These results indicate that, while the changes in dimensionality induced by both naturalistic and adversarial perturbations are proportional {+a) 0 2 4 6 8 10 12 14 16 Layer 0. F r a c t i o n o f T o t a l D i m e n s i o n a li t y Class Selectivity Regularization Scale (α) -2.0-1. b) 
 2 4 6 8 10 12 14 16 Layer 0.+}
{+F r a c t i o n o f T o t a l D i m e n s i o n a li t y c) 
 2 4 6 8 10 12 14 16 Layer 0. F r a c t i o n o f T o t a l D i m e n s i o n a li t y to the dimensionality of the network’s representations, these changes do not consistently project onto coding-relevant dimensions of the representations.+} Indeed, the larger change in early-layer dimensionality caused by worst-case perturbations likely reﬂects targeted projection onto codingrelevant dimensions and provides intuition as to why low-selectivity networks are more susceptible to worst-case perturbations.
Hidden layer representations in DNNs are known to lie on non-linear manifolds that are of lower dimensionality than the space in which they’re embedded (Goodfellow et al., 2016; Ansuini et al., 2019). Consequently, linear methods such as PCA can provide misleading estimates of hidden layer dimensionality. Thus we also quantiﬁed the intrinsic dimensionality (ID) of each layer’s representations (see Appendix A.1.4).
Interestingly, the results were qualitatively similar to what we observed when examining linear dimensionality (Figure A22) in both ResNet18 trained on Tiny
ImageNet (Figure A22a-A22c) and ResNet20 trained on CIFAR10 (Figure A22d-A22f).
Thus both linear and non-linear measures of dimensionality imply that representational dimensionality may present a trade-off between worst-case and average-case perturbation robustness.  D ISCUSSION Our results demonstrate that changes in the sparsity of semantic representations, as measured with class selectivity, induce a trade-off between robustness to average-case vs. worst-case perturbations: highly-distributed semantic representations confer robustness to average-case perturbations, but their increased dimensionality and consistent perturbability result in vulnerability to worst-case perturbations.
In contrast, sparse semantic representations yield low-dimensional representations and inconsistently-perturbable units, imparting worst-case robustness.
Furthermore, the dimensionality of the difference in early-layer activations between clean and perturbed samples is larger for worst-case perturbations than for average-case perturbations.
More generally, our results link average-case and worst-case perturbation robustness through class selectivity and representational dimensionality.
We hesitate to generalize too broadly about our ﬁndings, as they are limited to CNNs trained on image classiﬁcation tasks.
It is possible that the results we report here are speciﬁc to our models and/or datasets, and also may not extend to other tasks.
Scaling class selectivity regularization to datasets with large numbers of classes also remains an open problem (Leavitt and Morcos, 2020).
Our ﬁndings could be utilized for practical ends and to clarify ﬁndings in prior work.
Relevant to both of these issues is the task of adversarial example detection.
There is conﬂicting evidence that intrinsic dimensionality can be used to characterize or detect adversarial (worst-case) samples (Ma et al., 2018; Lu et al., 2018).
The ﬁnding that worst-case perturbations cause a marked increase in both intrinsic and linear dimensionality indicates that there may be merit in continuing to study these quantities for use in worst-case perturbation detection. And the observation that the causal relationship between class-selectivity and worst- and average-case robustness is bidirectional helps clarify the known beneﬁts of sparsity (Madry et al., 2018; Balda et al., 2020; Ye et al., 2018; Guo et al., 2018; Dhillon et al., 2018) and dimensionality (Langeberg et al., 2019; Sanyal et al., 2020; Nayebi and Ganguli, 2017) on worst-case robustness.
It furthermore raises the question of whether enforcing low-dimensional representations also causes class selectivity to increase.
Our work may also hold practical relevance to developing robust models: class selectivity could be used as both a metric for measuring model robustness and a method for achieving robustness (via regularization).
We hope future work will more comprehensively assess the utility of class selectivity as part of the deep learning toolkit for these purposes.
A A PPENDIX A. D ETAILED APPROACH
Unless otherwise noted: all experimental results were derived from the corrupted or adversarial test set with the parameters from the epoch that achieved the highest clean validation set accuracy over the training epochs; 20 replicates with different random seeds were run for each hyperparameter set; error bars and shaded regions denote bootstrapped 95% conﬁdence intervals; selectivity regularization was not applied to the ﬁnal (output) layer, nor was the ﬁnal layer included in any of our analyses.
A.1. M ODELS
All models were trained using stochastic gradient descent (SGD) with momentum = 0.9 and weight decay = 0.0001.
The maxpool layer after the ﬁrst batchnorm layer in ResNet18 (see He
et al.
(2016)) was removed because of the smaller size of Tiny ImageNet images compared to standard ImageNet images (64x64 vs. 256x256, respectively).
ResNet18 and ResNet50 were trained for 90 epochs with a minibatch size of 4096 (ResNet18) or 1400 (ResNet50) samples with a learning rate of 0.1, multiplied (annealed) by 0.1 at epochs 35, 50, 65, and 80.
ResNet20 (code modiﬁed from Idelbayev (2020)) were trained for 200 epochs using a minibatch size of 256 samples and a learning rate of 0.1, annealed by 0.1 at epochs 100 and 150.
A.1. D ATASETS
Tiny Imagenet (Fei-Fei et al., 2015) consists of 500 training images and 50 images for each of itsclasses.
We used the validation set for testing and created a new validation set by taking 50 images per class from the training set, selected randomly for each seed.
We split the 50k CIFAR10 training samples into a 45k sample training set and a 5k validation set, similar to our approach with Tiny
Imagenet.
All experimental results were derived from the test set with the parameters from the epoch that achieved the highest validation set accuracy over the training epochs.
20 replicates with different random seeds were run for each hyperparameter set.
Selectivity regularization was not applied to the ﬁnal (output) layer, nor was the ﬁnal layer included any of our analyses.
CIFAR10C consists of a dataset in which 19 different naturalistic corruptions have been applied to the CIFAR10 test set at 5 different levels of severity.
Tiny ImageNetC also has 5 levels of corruption severity, but consists of 15 corruptions.
We would like to note that Tiny ImageNetC does not use the Tiny ImageNet test data.
While the two datasets were created using the same data generation procedure—cropping and scaling images from the same 200 ImageNet classes—they differ in the speciﬁc ImageNet images they use.
It is possible that the images used to create Tiny ImageNetC are out-of-distribution with regards to the
Tiny ImageNet training data, in which case our results from testing on Tiny ImageNetC actually underestimate the corruption robustness of our networks.
The creators of Tiny ImageNetC kindly provided the clean (uncorrupted) Tiny ImageNetC data necessary for the dimensionality analysis, which relies on matches corrupted and clean data samples.
A.1. S OFTWARE
Experiments were conducted using PyTorch (Paszke et al., 2019), analyzed using the SciPy ecosystem (Virtanen et al., 2019), and visualized using Seaborn (Waskom et al., 2017).
A.1. Q UANTIFYING D IMENSIONALITY
We quantiﬁed the dimensionality of a layer’s representations by applying PCA to the layer’s activation matrix for the clean test data and counting the number of dimensions necessary to explain 95% of the variance, then dividing by the total number of dimensions (i.e. the fraction of total dimensionality; we also replicated our results using the fraction of total dimensionality necessary to explain 90% and 99% of the variance).
The same procedure was applied to compute the dimensionality of perturbationinduced changes in representations, except the activations for a perturbed data set were subtracted
[Figure]
from the corresponding clean activations prior to applying PCA.
For average-case perturbations, we performed this analysis for every corruption type and severity, and for the worst-case perturbations we used PGD with 40 steps.
Hidden layer representations in DNNs are known to lie on non-linear manifolds that are of lower dimensionality than the space in which they’re embedded (Goodfellow et al., 2016; Ansuini et al., 2019). Consequently, linear methods such as PCA can fail to capture the "intrinsic" dimensionality of hidden layer representations. Thus we also quantiﬁed the intrinsic dimensionality (ID) of each layer’s representations using the method of (Facco et al., 2017).
The method, based on that of Levina and Bickel (2005), estimates ID by computing the ratio between the distances to the second and ﬁrst nearest neighbors of each data point.
We used the implementation of Ansuini et al.
Our procedure was otherwise identical as when computing the linear dimensionality: we computed the dimensionality across all test data for each layer, then divided by the number of units per layer.
We then computed the dimensionality of perturbation-induced changes in representations, except the activations for a perturbed data set were subtracted from the corresponding clean activations prior to computed ID.
For average-case perturbations, we performed this analysis for every corruption type and severity, and for the worst-case perturbations we used PGD with 40 steps.
A. E FFECTS OF CLASS SELECTIVITY REGULARIZATION ON TEST ACCURACY
[Figure]
A. A DDITIONAL RESULTS FOR AVERAGE - CASE PERTURBATION ROBUSTNESS
Because modifying class selectivity can affect performance on clean (unperturbed) inputs (Leavitt and Morcos (2020); Figure A2) [Figure] it is possible that the effects we observe of class selectivity on perturbed test accuracy are not caused by changes in perturbation robustness per se [Figure]
but simply by changes in baseline model accuracy. We controlled for this by normalizing each model’s perturbed test accuracy by its clean (unperturbed) test accuracy.
The results are generally consistent even after controlling for clean test accuracy, although increasing class selectivity does not cause the same deﬁcits in as measured using non-normalized perturbed test accuracy in ResNet18 trained on Tiny
ImageNet (Figure A3a).
Interestingly, in ResNet20 trained on CIFAR [Figure] normalizing perturbed test accuracy reveals a more dramatic improvement in perturbation robustness caused by reducing class selectivity (Figure A6c). The results for Resnet50 trained on Tiny ImageNet are entirely consistent between raw vs. normalized measures (Figure A9b vs. Figure A9c)
[Figure]
[Figure]
[Figure]
[Figure] [Figure] [Figure] (cid:14)(cid:21)(cid:25)(cid:25)(cid:28)(cid:13)(cid:22)(cid:24)(cid:21)(cid:20)(cid:19)(cid:30)(cid:12)(cid:30)(cid:25)(cid:24)(cid:22)(cid:11)
[Figure]
[Figure] [Figure] A. T HE CAUSAL RELATIONSHIP BETWEEN CLASS SELECTIVITY AND AVERAGE - CASE ROBUSTNESS IS BIDIRECTIONAL We found that regularizing to decrease class selectivity causes robustness to average-case perturbations.
But is the converse is also true? Does increasing robustness to average-case perturbations also cause class selectivity to increase? We investigated this question by training with AugMix, a technique known to improve worst-case robustness (Hendrycks et al., 2020a). Brieﬂy, AugMix stochastically applies a diverse set of image augmentations and uses a Jensen-Shannon Divergence consistency loss.
Our AugMix parameters were as follows: mixture width: 3; mixture depth: stochastic; augmentation probability: 1; augmentation severity: 2. We found that AugMix does indeed decrese the mean level of class selectivity across neurons in a network (Figure A11). AugMix decreases overall levels of selectivity similarly to training with a class selectivity regularization scale of approximately α = − 0 .or α = − 0 .
2 in both ResNet18 trained on Tiny ImageNet (Figures A11a and A11b) and ResNet trained on CIFAR10 (Figures A11c and A11d).
These results indicate that the causal relationship between average-case perturbation robustness and class selectivity is bidirectional: not only does decreasing class selectivity cause average-case perturbation robustness to increase, but increasing average-case perturbation-robustness also causes class selectivity to decrease.
[Figure]
t
i
[Figure]
i t y I n
[Figure]
A. W ORST - CASE PERTURBATION ROBUSTNESS
We also conﬁrmed that the worst-case robustness of high-selectivity ResNet18 and ResNet20 networks was not simply due to gradient-masking (Athalye et al., 2018) by generating worst-case perturbations using each of the replicate models trained with no selectivity regularization ( α = 0 ), then testing selectivity-regularized models on these samples.
We found that high-selectivity models were less vulnerable to the α = 0 samples than low-selectivity models for high-intensity perturbations (Appendix A14, indicating that gradient-masking does not fully account for the worst-case robustness of high-selectivity models. [Table] [Figure] n p
[Figure]
t p
[Figure]
t J a c
[Figure]
i
[Figure]
n
[Figure]
[Figure]
[Figure]
A. C LASS SELECTIVITY REGULARIZATION DOES NOT AFFECT ROBUSTNESS TO NATURAL ADVERSARIAL EXAMPLES
We also examined whether class selectivity regularization affects robustness to "natural" adversarial examples, images that are "natural, unmodiﬁed, real-world examples...selected to cause a ﬁxed model to make a mistake" (Hendrycks et al., 2020b).
We tested robustness to natural adversarial examples using ImageNet-A, a dataset of natural adversarial examples that belong to ImageNet classes but consistently cause misclassiﬁcation errors with high conﬁdence (Hendrycks et al., 2020b).
We adapted ImageNet-A to our models trained on Tiny ImageNet (ResNet18 and ResNet50) by only testing on the 74 image classes that overlap between ImageNet-A and Tiny ImageNet (yielding a total of 2957 samples), and downsampling the images to 64 x 64.
Test accuracy was similar across all tested values of α for both ResNet18 (Figure A15a) and ResNet50 (Figure A15b), indicating that class selectivity regularization may share some limitations with other methods for improving robustness, many of which also fail to yield signiﬁcant robustness against ImageNet-A (Hendrycks et al., 2020b).
[Figure]
[Figure]
T e s t
A cc u r a c y
[Figure]
A. T HE CAUSAL RELATIONSHIP BETWEEN CLASS SELECTIVITY AND WORST - CASE ROBUSTNESS IS BIDIRECTIONAL
We observed that regularizing to increase class selectivity causes robustness to worst-case perturbations.
But is the converse is also true?
Does increasing robustness to worst-case perturbations cause class selectivity to increase?
We investigated this question using PGD training, a common technique for improving worst-case robustness.
PGD training applies the PGD method of sample perturbation (see Approach 3) to samples during training.
We used the same parameters for PGD sample generation when training our models as when testing (Approach 3).
The number of PGD iterations controls the intensity of the perturbation, and the degree of perturbation-robustness in the trained model (Madry et al., 2018).
We found that PGD training does indeed increase the mean level of class selectivity across neurons in a network, and this effect is proportional to the strength of
PGD training: networks trained with more strongly-perturbed samples have higher class selectivity (Figure A16).
Interestingly, PGD training also appear to cause units to die (Lu et al., 2019), and the number of dead untis is proportional to the intensity of PGD training (Figures A16b and A16e).
Removing dead units, which have a class selectivity index of 0, from the calculation of mean class selectivity results in a clear, monotonic effect of PGD training intensity on class selectivity in both
ResNet18 trained on Tiny ImageNet (Figure A16c) and ResNet20 trained on CIFAR10 (Figure A16f).
These results indicate that the causal relationship between worst-case perturbation robustness and class selectivity is bidirectional: increasing class selectivity not only causes increased worst-case perturbation robustness, but increasing worst-case perturbation-robustness also causes increased class selectivity. [Figure] C l a ss S e l c t i v i
t y I n
d
[Figure] [Figure] [Figure]
I n p u t - U n i t
G r a d i e n t ( µ u )
[Figure]
r [Figure]
D e v i a t i
[Figure]
n o f
I n p u t - U n i t
G r a d i e n t
[Figure]
U n i t ( σ
[Figure] [Figure]
I n p u t - U n i t G r a d
i e n t
V a r i a b ili t y , U n i t ( C V u )
[Figure] [Figure]
n
[Figure]
t - U n t
[Figure]
n t
[Figure]
u )
[Figure]
S t a n d a r d
D e v i a t i o n o f
I n p u t - U n i t
G r a d i e n t ,
U n i t ( σ u )
[Figure]
I n p u t - U n i t G r a d i e n t
V a r i a b ili t y , L a y e r ( C
V l )
[Figure]
F r a c t i o n o f T o t a l D i m e n s i o n a li t y
[Figure]
r
[Figure]
c t i
[Figure]
n o f T o t a l D
i
[Figure]
e n s i o n a li t y
[Figure]
[Figure]
a l D
i
[Figure] [Figure]
t y
[Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] a c t i o n o f T o t a l D i m n s i o n a li t y
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] r a c t i o n o f T o t a l D i m e n s i o n a li t y
[Figure] [Table]
t o [Table] l m e n s i o n a li t y I n t r i n s i c
[Table]
I n t r i n s i c f) 0 2 4 6 8 10 12 14 16 18 Layer 0.
F r a c t i o n o f T o t a l D i m e n s i o n a li t
y , I n t r i n s i c