Original: S ELECTIVITY CONSIDERED HARMFUL : EVALUATING THE CAUSAL IMPACT OF CLASS SELECTIVITY IN DNN S
Revised: S ELECTIVITY CONSIDERED HARMFUL : EVALUATING THE CAUSAL IMPACT OF CLASS SELECTIVITY IN DNN S

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The properties of individual neurons are often analyzed in order to understand the biological and artiﬁcial neural networks in which they’re embedded.
Revised: The properties of individual neurons are often analyzed in order to understand the biological and artiﬁcial neural networks in which they’re embedded.

Original: Class selectivity—typically deﬁned as how different a neuron’s responses are across different classes of stimuli or data samples—is commonly used for this purpose.
Revised: Class selectivity—typically deﬁned as how different a neuron’s responses are across different classes of stimuli or data samples—is commonly used for this purpose.

Original: However, it remains an open question whether it is necessary and/or sufﬁcient for deep neural networks (DNNs) to learn class selectivity in individual units.
Revised: However it remains an open question whether it is necessary and/or sufﬁcient for deep neural networks (DNNs) to learn class selectivity in individual units.

Original: We investigated the causal impact of class selectivity on network function by directly regularizing for or against class selectivity.
Revised: We investigated the causal impact of class selectivity on network function by directly regularizing for or against class selectivity.

Original: Using this regularizer to reduce class selectivity across units in convolutional neural networks increased test accuracy by over 2% for ResNet18  trained on Tiny ImageNet.For ResNet20 trained on
Revised: Using this regularizer to reduce class selectivity across units in convolutional neural networks increased test accuracy by over 2% for ResNet18 and 1% ResNet50 trained on Tiny ImageNet.

Original:  CIFAR10 we could reduce class selectivity by a factor of 2.5 with no impact on test accuracy, and reduce it nearly to zero with only a small ( ∼ 2%) drop in test accuracy.
Revised: For ResNet trained on CIFAR10 we could reduce class selectivity by a factor of 2.5 with no impact on test accuracy and reduce it nearly to zero with only a small ( ∼ 2%) drop in test accuracy.

Original: In contrast, regularizing to increase class selectivity signiﬁcantly decreased test accuracy across all models and datasets.
Revised: In contrast, regularizing to increase class selectivity signiﬁcantly decreased test accuracy across all models and datasets

Original: These results indicate that class selectivity in individual units is neither sufﬁcient nor strictly necessary, and can even impair DNN performance.
Revised: These results indicate that class selectivity in individual units is neither sufﬁcient nor strictly necessary and can even impair DNN performance.

Original: They also encourage caution when focusing on the properties of single units as representative of the mechanisms by which DNNs function.
Revised: They also encourage caution when focusing on the properties of single units as representative of the mechanisms by which DNNs function.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Our ability to understand deep learning systems lags considerably behind our ability to obtain practical outcomes with them.
Revised: Our ability to understand deep learning systems lags considerably behind our ability to obtain practical outcomes with them.

Original: A breadth of approaches have been developed in attempts to better understand deep learning systems and render them more comprehensible to humans (Yosinski et al., 2015;
Revised: A breadth of approaches have been developed in attempts to better understand deep learning systems and render them more comprehensible to humans (Yosinski et al., 2015;

Original: Bau et al., 2017; Olah et al., 2018; Hooker et al., 2019).
Revised: Bau et al., 2017; Olah et al., 2018; Hooker et al., 2019).

Original: Many of these approaches examine the properties of single neurons and treat them as representative of the networks in which they’re embedded (Erhan et al., 2009; Zeiler and Fergus, 2014;
Revised: Many of these approaches examine the properties of single neurons and treat them as representative of the networks in which they’re embedded (Erhan et al., 2009; Zeiler and Fergus, 2014;

Original: Karpathy et al., 2016; Amjad et al., 2018; Lillian et al., 2018;
Revised: Karpathy et al., 2016; Amjad et al., 2018; Lillian et al., 2018;

Original: Dhamdhere et al., 2019; Olah et al., 2020).
Revised: Dhamdhere et al., 2019; Olah et al., 2020).

Original: The selectivity of individual units (i.e. the variability in a neuron’s responses across data classes or dimensions) is one property that has been of particular interest to researchers trying to better understand deep neural networks (DNNs)
Revised: The selectivity of individual units (i.e. the variability in a neuron’s responses across data classes or dimensions) is one property that has been of particular interest to researchers trying to better understand deep neural networks (DNNs)

Original: (Zhou et al., 2015; Olah et al., 2017; Morcos et al., 2018b;
Revised: (Zhou et al., 2015; Olah et al., 2017; Morcos et al., 2018b;

Original: Zhou et al., 2018; Meyes et al., 2019;
Revised: Zhou et al., 2018; Meyes et al., 2019;

Original: Na et al., 2019;
Revised: Na et al., 2019;

Original: Zhou et al., 2019; Rafegas et al., 2019;
Revised: Zhou et al., 2019; Rafegas et al., 2019;

Original: Bau et al., 2020).
Revised: Bau et al., 2020).

Original: This focus on individual neurons makes intuitive sense, as the tractable, semantic nature of selectivity is extremely alluring; some measure of selectivity in individual units is often provided as an explanation of "what" a network is "doing".
Revised: This focus on individual neurons makes intuitive sense, as the tractable, semantic nature of selectivity is extremely alluring; some measure of selectivity in individual units is often provided as an explanation of "what" a network is "doing".

Original: One notable study highlighted a neuron selective for sentiment in an LSTM network trained on a word prediction task (Radford et al., 2017).
Revised: One notable study highlighted a neuron selective for sentiment in an LSTM network trained on a word prediction task (Radford et al., 2017).

Original: Another attributed visualizable, semantic features to the activity of individual neurons across GoogLeNet trained on ImageNet (Olah et al., 2017).
Revised: Another attributed visualizable, semantic features to the activity of individual neurons across GoogLeNet trained on ImageNet (Olah et al., 2017).

Original: Both of these examples inﬂuenced many subsequent studies, demonstrating the widespread, intuitive appeal of "selectivity" (Amjad et al., 2018;
Revised: Both of these examples inﬂuenced many subsequent studies, demonstrating the widespread, intuitive appeal of "selectivity" (Amjad et al., 2018;

Original: Meyes et al., 2019; Morcos et al., 2018b; Zhou et al., 2015; 2018;
Revised: Meyes et al., 2019; Morcos et al., 2018b; Zhou et al., 2015; 2018;

Original: Bau et al., 2017; Karpathy et al., 2016;
Revised: Bau et al., 2017; Karpathy et al., 2016;

Original: Na et al., 2019; Radford et al., 2017; Rafegas et al., 2019; Morcos et al., 2018b; Olah et al., 2017; 2018; 2020).
Revised: Na et al., 2019; Radford et al., 2017; Rafegas et al., 2019; Morcos et al., 2018b; Olah et al., 2017; 2018; 2020).

Original: Finding intuitive ways of representing the workings of DNNs is essential for making them understandable and accountable, but we must ensure that our approaches are based on meaningful properties of the system.
Revised: Finding intuitive ways of representing the workings of DNNs is essential for making them understandable and accountable, but we must ensure that our approaches are based on meaningful properties of the system.

Original: Recent studies have begun to address this issue by investigating the relationships between selectivity and measures of network function such as generalization and robustness to perturbation
Revised: Recent studies have begun to address this issue by investigating the relationships between selectivity and measures of network function such as generalization and robustness to perturbation

Original: (Morcos et al., 2018b; Zhou et al., 2018; Dalvi et al., 2019).
Revised: (Morcos et al., 2018b; Zhou et al., 2018; Dalvi et al., 2019).

Original: Selectivity has also been used as the basis for targeted modulation of neural network function through individual units (Bau et al., 2019a;b).
Revised: Selectivity has also been used as the basis for targeted modulation of neural network function through individual units (Bau et al., 2019a;b).

Original: However there is also growing evidence from experiments in both deep learning (Fong and Vedaldi, 2018; Morcos et al., 2018b; Gale et al., 2019; Donnelly and Roegiest, 2019) and neuroscience (Leavitt et al., 2017; Zylberberg, 2018; Insanally et al., 2019) that single unit selectivity may not be as important as once thought.
Revised: However there is also growing evidence from experiments in both deep learning (Fong and Vedaldi, 2018; Morcos et al., 2018b; Gale et al., 2019; Donnelly and Roegiest, 2019) and neuroscience (Leavitt et al., 2017; Zylberberg, 2018; Insanally et al., 2019) that single unit selectivity may not be as important as once thought.

Original: Previous studies examining the functional role of selectivity in DNNs have often measured how selectivity mediates the effects of ablating single units, or used indirect, correlational approaches that modulate selectivity indirectly (e.g. batch norm)
Revised: Previous studies examining the functional role of selectivity in DNNs have often measured how selectivity mediates the effects of ablating single units, or used indirect, correlational approaches that modulate selectivity indirectly (e.g. batch norm)

Original: (Morcos et al., 2018b;
Revised: (Morcos et al., 2018b;

Original: Zhou et al., 2018; Lillian et al., 2018; Meyes et al., 2019).
Revised: Zhou et al., 2018; Lillian et al., 2018; Meyes et al., 2019; Kanda et al., 2020).

Original: But single unit ablation in trained networks has two critical limitations: it cannot address whether the presence of selectivity is beneﬁcial, nor whether networks need to learn selectivity to function properly.
Revised: But single unit ablation in trained networks has two critical limitations: it cannot address whether the presence of selectivity is beneﬁcial, nor whether networks need to learn selectivity to function properly.

Original: It can only address the effect of removing a neuron from a network whose training process assumed the presence of that neuron.
Revised: It can only address the effect of removing a neuron from a network whose training process assumed the presence of that neuron.

Original: And even then, the observed effect might be misleading.
Revised: And even then, the observed effect might be misleading.

Original: For example, a property that is critical to network function may be replicated across multiple neurons.
Revised: For example, a property that is critical to network function may be replicated across multiple neurons.

Original: This redundancy means that ablating any one of these neurons would show little effect, and could thus lead to the erroneous conclusion that the examined property has little impact on network function.
Revised: This redundancy means that ablating any one of these neurons would show little effect, and could thus lead to the erroneous conclusion that the examined property has little impact on network function.

Original: We were motivated by these issues to pursue a series of experiments investigating the causal importance of class selectivity in artiﬁcial neural networks.
Revised: We were motivated by these issues to pursue a series of experiments investigating the causal importance of class selectivity in artiﬁcial neural networks.

Original: To do so, we introduced a term to the loss function that allows us to directly regularize for or against class selectivity, giving us a single knob to control class selectivity in the network.
Revised: To do so, we introduced a term to the loss function that allows us to directly regularize for or against class selectivity, giving us a single knob to control class selectivity in the network.

Original: The selectivity regularizer sidesteps the limitations of single unit ablation and other indirect techniques, allowing us to conduct a series of experiments evaluating the causal impact of class selectivity on DNN performance.
Revised: The selectivity regularizer sidesteps the limitations of single unit ablation and other indirect techniques, allowing us to conduct a series of experiments evaluating the causal impact of class selectivity on DNN performance.

Original: Our ﬁndings are as follows:
Revised: Our ﬁndings are as follows:

Original: • Performance can be improved by reducing class selectivity, suggesting that naturally-learned levels of class selectivity can be detrimental.
Revised: • Performance can be improved by reducing class selectivity, suggesting that naturally-learned levels of class selectivity can be detrimental.

Original: Reducing class selectivityin ResNet18 trained on Tiny ImageNet could improve test accuracy by over 2%.
Revised: Reducing class selectivity could improve test accuracy by over 2% in ResNet18 and 1% in ResNet50 trained on Tiny ImageNet.

Original: • Even when class selectivity isn’t detrimental to network function, it remains largely unnecessary.
Revised: • Even when class selectivity isn’t detrimental to network function, it remains largely unnecessary.

Original: We reduced the mean class selectivity of units in ResNet20 trained on CIFAR10 by a factor of ∼ 2.5 with no impact on test accuracy, and by a factor of ∼ 20—nearly to a mean of 0—with only a 2% change in test accuracy.
Revised: We reduced the mean class selectivity of units in ResNet20 trained on CIFAR10 by a factor of ∼ 2.5 with no impact on test accuracy, and by a factor of ∼ 20—nearly to a mean of 0—with only a 2% change in test accuracy.

Original: • Our regularizer does not simply cause networks to preserve class-selectivity by rotating it off of unit-aligned axes (i
Revised: • Our regularizer does not simply cause networks to preserve class-selectivity by rotating it off of unit-aligned axes (i

Original: e. by distributing selectivity linearly across units), but rather seems to suppress selectivity more generally, even when optimizing for high-selectivity basis sets .
Revised: e. by distributing selectivity linearly across units), but rather seems to suppress selectivity more generally, even when optimizing for high-selectivity basis sets .

Original: This demonstrates the viability of low-selectivity representations distributed across units.
Revised: This demonstrates the viability of low-selectivity representations distributed across units.

Original: • We show that regularizing to increase class selectivity, even by small amounts, has signiﬁcant negative effects on performance.
Revised: • We show that regularizing to increase class selectivity, even by small amounts, has signiﬁcant negative effects on performance.

Original: Trained networks seem to be perched precariously at a performance cliff with regard to class selectivity.
Revised: Trained networks seem to be perched precariously at a performance cliff with regard to class selectivity.

Original: These results indicate that the levels of class selectivity learned by individual units in the absence of explicit regularization are at the limit of what will impair the network.
Revised: These results indicate that the levels of class selectivity learned by individual units in the absence of explicit regularization are at the limit of what will impair the network.

Original: Our ﬁndings collectively demonstrate that class selectivity in individual units is neither necessary nor sufﬁcient for convolutional neural networks (CNNs) to perform image classiﬁcation tasks, and in some cases can actually be detrimental.
Revised: Our ﬁndings collectively demonstrate that class selectivity in individual units is neither necessary nor sufﬁcient for convolutional neural networks (CNNs) to perform image classiﬁcation tasks, and in some cases can actually be detrimental.

Original: This alludes to the possibility of class selectivity regularization as a technique for improving CNN performance.
Revised: This alludes to the possibility of class selectivity regularization as a technique for improving CNN performance.

Original: More generally, our results encourage caution when focusing on the properties of single units as representative of the mechanisms by which CNNs function, and emphasize the importance of analyses that examine properties across neurons (i.e. distributed representations).
Revised: More generally, our results encourage caution when focusing on the properties of single units as representative of the mechanisms by which CNNs function, and emphasize the importance of analyses that examine properties across neurons (i.e. distributed representations).

Original: Most importantly, our results are a reminder to verify that the properties we do focus on are actually relevant to CNN function.
Revised: Most importantly, our results are a reminder to verify that the properties we do focus on are actually relevant to CNN function.

Original:  R ELATED WORK 2. S ELECTIVITY IN DEEP LEARNING
Revised:  R ELATED WORK 2. S ELECTIVITY IN DEEP LEARNING

Original: Examining some form of selectivity in individual units constitutes the bedrock of many approaches to understanding DNNs.
Revised: Examining some form of selectivity in individual units constitutes the bedrock of many approaches to understanding DNNs.

Original: Sometimes the goal is simply to visualize selectivity, which has been pursued using a breadth of methods.
Revised: Sometimes the goal is simply to visualize selectivity, which has been pursued using a breadth of methods.

Original: These include identifying the input sample(s)
Revised: These include identifying the input sample(s)

Original: (e.g. images) or sample subregions that maximally activate a given neuron (Zhou et al., 2015; Rafegas et al., 2019), and numerous optimization-based techniques for generating samples that maximize unit activations (Erhan et al., 2009; Zeiler and Fergus, 2014;
Revised: (e.g. images) or sample subregions that maximally activate a given neuron (Zhou et al., 2015; Rafegas et al., 2019), and numerous optimization-based techniques for generating samples that maximize unit activations (Erhan et al., 2009; Zeiler and Fergus, 2014;

Original: Simonyan et al., 2014;
Revised: Simonyan et al., 2014;

Original: Yosinski et al., 2015; Nguyen et al., 2016; Olah et al., 2017; 2018).
Revised: Yosinski et al., 2015; Nguyen et al., 2016; Olah et al., 2017; 2018).

Original: While the different methods for quantifying single unit selectivity are often conceptually quite similar (measuring how variable are a neuron’s responses across different classes of data samples), they have been applied across a broad range of contexts (Amjad et al., 2018;
Revised: While the different methods for quantifying single unit selectivity are often conceptually quite similar (measuring how variable are a neuron’s responses across different classes of data samples), they have been applied across a broad range of contexts (Amjad et al., 2018;

Original: Meyes et al., 2019; Morcos et al., 2018b; Zhou et al., 2015; 2018;
Revised: Meyes et al., 2019; Morcos et al., 2018b; Zhou et al., 2015; 2018;

Original: Bau et al., 2017; Karpathy et al., 2016;
Revised: Bau et al., 2017; Karpathy et al., 2016;

Original: Na et al., 2019; Radford et al., 2017; Rafegas et al., 2019).
Revised: Na et al., 2019; Radford et al., 2017; Rafegas et al., 2019).

Original: For example, Bau et al.
Revised: For example, Bau et al.

Original: (2017) quantiﬁed single unit selectivity for "concepts" (as annotated by humans) in networks trained for object and scene recognition.
Revised: (2017) quantiﬁed single unit selectivity for "concepts" (as annotated by humans) in networks trained for object and scene recognition.

Original: Olah et al.
Revised: Olah et al.

Original: (2018; 2020) have pursued a research program examining single unit selectivity as a building block for understanding DNNs.
Revised: (2018; 2020) have pursued a research program examining single unit selectivity as a building block for understanding DNNs.

Original: And single units in models trained to solve natural language processing tasks have been found to exhibit selectivity for syntactical and semantic features (Karpathy et al., 2016;
Revised: And single units in models trained to solve natural language processing tasks have been found to exhibit selectivity for syntactical and semantic features (Karpathy et al., 2016;

Original: Na et al., 2019), of which the "sentiment-selective neuron" reported by Radford et al.
Revised: Na et al., 2019), of which the "sentiment-selective neuron" reported by Radford et al.

Original: (2017) is a particularly recognized example.
Revised: (2017) is a particularly recognized example.

Original: The relationship between individual unit selectivity and various measures of DNN performance have been examined in prior studies, but the conclusions have not been concordant.
Revised: The relationship between individual unit selectivity and various measures of DNN performance have been examined in prior studies, but the conclusions have not been concordant.

Original: Morcos et al.
Revised: Morcos et al.

Original: (2018b), using single unit ablation and other techniques, found that a network’s test set generalization is negatively correlated (or uncorrelated) with the class selectivity of its units.
Revised: (2018b), using single unit ablation and other techniques, found that a network’s test set generalization is negatively correlated (or uncorrelated) with the class selectivity of its units, a ﬁnding replicated by Kanda et al.

Original: In contrast, though Amjad et al.
Revised: In contrast, though Amjad et al.

Original: conﬁrmed these results for single unit ablation, they also performed cumulative ablation analyses which suggested that selectivity is beneﬁcial, suggesting that redundancy across units may make it difﬁcult to interpret single unit ablation studies.
Revised: conﬁrmed these results for single unit ablation, they also performed cumulative ablation analyses which suggested that selectivity is beneﬁcial, suggesting that redundancy across units may make it difﬁcult to interpret single unit ablation studies.

Original: In a follow-up study, Zhou et al.
Revised: In a follow-up study, Zhou et al.

Original: (2018) found that ablating class-selective units impairs classiﬁcation accuracy for speciﬁc classes (though interestingly, not always the same class the unit was selective for), but a compensatory increase in accuracy for other classes can often leave overall accuracy unaffected.
Revised: (2018) found that ablating class-selective units impairs classiﬁcation accuracy for speciﬁc classes (though interestingly, not always the same class the unit was selective for), but a compensatory increase in accuracy for other classes can often leave overall accuracy unaffected.

Original: Ukita (2018) found that orientation selectivity in individual units is correlated with generalization performance in convolutional neural networks (CNNs), and that ablating highly orientation-selective units impairs classiﬁcation accuracy more than ablating units with low orientation-selectivity.
Revised: Ukita (2018) found that orientation selectivity in individual units is correlated with generalization performance in convolutional neural networks (CNNs), and that ablating highly orientation-selective units impairs classiﬁcation accuracy more than ablating units with low orientation-selectivity.

Original: But while orientation selectivity and class selectivity can both be considered types of feature selectivity, orientation selectivity is far less abstract and focuses on speciﬁc properties of the image (e.g., oriented edges) rather than semantically meaningful concepts and classes.
Revised: But while orientation selectivity and class selectivity can both be considered types of feature selectivity, orientation selectivity is far less abstract and focuses on speciﬁc properties of the image (e.g., oriented edges) rather than semantically meaningful concepts and classes.

Original: Nevertheless, this study still demonstrates the importance of some types of selectivity.
Revised: Nevertheless, this study still demonstrates the importance of some types of selectivity.

Original: Results are also variable for models trained on NLP tasks.
Revised: Results are also variable for models trained on NLP tasks.

Original: Dalvi et al.
Revised: Dalvi et al.

Original: (2019) found that ablating units selective for linguistic features causes greater performance deﬁcits than ablating less-selective units, while Donnelly and Roegiest (2019) found that ablating the "sentiment neuron" of Radford et al.
Revised: (2019) found that ablating units selective for linguistic features causes greater performance deﬁcits than ablating less-selective units, while Donnelly and Roegiest (2019) found that ablating the "sentiment neuron" of Radford et al.

Original: (2017) has equivocal effects on performance.
Revised: (2017) has equivocal effects on performance.

Original: These ﬁndings seem challenging to reconcile.
Revised: These ﬁndings seem challenging to reconcile.

Original: All of these studies examining class selectivity in single units are hamstrung by their reliance on single unit ablation, which could account for their conﬂicting results.
Revised: All of these studies examining class selectivity in single units are hamstrung by their reliance on single unit ablation, which could account for their conﬂicting results.

Original: As discussed earlier, single unit ablation can only address whether class selectivity affects performance in trained networks, and not whether individual units to need to learn class selectivity for optimal network function.
Revised: As discussed earlier, single unit ablation can only address whether class selectivity affects performance in trained networks, and not whether individual units to need to learn class selectivity for optimal network function.

Original: And even then, the conclusions obtained from single neuron ablation analyses can be misleading due to redundancy across units (Amjad et al., 2018; Meyes et al., 2019).
Revised: And even then, the conclusions obtained from single neuron ablation analyses can be misleading due to redundancy across units (Amjad et al., 2018; Meyes et al., 2019).

Original: 2. S ELECTIVITY IN NEUROSCIENCE
Revised: 2. S ELECTIVITY IN NEUROSCIENCE

Original: Measuring the responses of single neurons to a relevant set of stimuli has been the canonical ﬁrst-order approach for understanding the nervous system (Sherrington, 1906; Adrian, 1926; Granit, 1955;
Revised: Measuring the responses of single neurons to a relevant set of stimuli has been the canonical ﬁrst-order approach for understanding the nervous system (Sherrington, 1906; Adrian, 1926; Granit, 1955;

Original: Hubel and Wiesel, 1959; Barlow, 1972; Kandel et al., 2000); its application has yielded multiple
Revised: Hubel and Wiesel, 1959; Barlow, 1972; Kandel et al., 2000); its application has yielded multiple

Original: Nobel Prizes (Hubel and Wiesel, 1959; 1962; Hubel, 1982; Wiesel, 1982; O’Keefe and Dostrovsky, 1971; Fyhn et al., 2004).
Revised: Nobel Prizes (Hubel and Wiesel, 1959; 1962; Hubel, 1982; Wiesel, 1982; O’Keefe and Dostrovsky, 1971; Fyhn et al., 2004).

Original: But recent experimental ﬁndings have raised doubts about the necessity of selectivity for high-ﬁdelity representations in neuronal populations (Leavitt et al., 2017; Insanally et al., 2019; Zylberberg, 2018), and neuroscience research seems to be moving beyond characterizing neural systems at the level of single neurons, towards population-level phenomena (Shenoy et al., 2013; Raposo et al., 2014; Fusi et al., 2016; Morcos and Harvey, 2016;
Revised: But recent experimental ﬁndings have raised doubts about the necessity of selectivity for high-ﬁdelity representations in neuronal populations (Leavitt et al., 2017; Insanally et al., 2019; Zylberberg, 2018), and neuroscience research seems to be moving beyond characterizing neural systems at the level of single neurons, towards population-level phenomena (Shenoy et al., 2013; Raposo et al., 2014; Fusi et al., 2016; Morcos and Harvey, 2016;

Original: Pruszynski and Zylberberg, 2019; Heeger and Mackey, 2019; Saxena and Cunningham, 2019).
Revised: Pruszynski and Zylberberg, 2019; Heeger and Mackey, 2019; Saxena and Cunningham, 2019).

Original: Single unit selectivity-based approaches are ubiquitous in attempts to understand artiﬁcial and biological neural systems, but growing evidence has led to questions about the importance of focusing on selectivity and its role in DNN function.
Revised: Single unit selectivity-based approaches are ubiquitous in attempts to understand artiﬁcial and biological neural systems, but growing evidence has led to questions about the importance of focusing on selectivity and its role in DNN function.

Original: These factors, combined with the limitations of prior approaches, lead to the question: is class selectivity necessary and/or sufﬁcient for DNN function?
Revised: These factors, combined with the limitations of prior approaches, lead to the question : is class selectivity necessary and or sufﬁcient for DNN function

Original:  A PPROACH
Revised:  A PPROACH

Original: Networks naturally seem to learn solutions that result in class-selective individual units (Zhou et al., 2015;
Revised: Networks naturally seem to learn solutions that result in class-selective individual units (Zhou et al., 2015;

Original: Olah et al., 2017; Morcos et al., 2018b; Zhou et al., 2018; Meyes et al., 2019;
Revised: Olah et al., 2017; Morcos et al., 2018b; Zhou et al., 2018; Meyes et al., 2019;

Original: Na et al., 2019;
Revised: Na et al., 2019;

Original: Zhou et al., 2019; Rafegas et al., 2019; Amjad et al., 2018; Meyes et al., 2019;
Revised: Zhou et al., 2019; Rafegas et al., 2019; Amjad et al., 2018; Meyes et al., 2019;

Original: Bau et al., 2017;
Revised: Bau et al., 2017;

Original: Karpathy et al., 2016; Radford et al., 2017;
Revised: Karpathy et al., 2016; Radford et al., 2017;

Original: Olah et al., 2018; 2020).
Revised: Olah et al., 2018; 2020).

Original: We examined whether learning class-selective representations in individual units is actually necessary for networks to function properly.
Revised: We examined whether learning class-selective representations in individual units is actually necessary for networks to function properly.

Original: Motivated by the limitations of single unit ablation techniques and the indirectness of using batch norm or dropout to modulate class selectivity (e.g. Morcos et al.
Revised: Motivated by the limitations of single unit ablation techniques and the indirectness of using batch norm or dropout to modulate class selectivity (e.g. Morcos et al.

Original: (2018b)
Revised: (2018b)

Original: ; Zhou et al.
Revised: ; Zhou et al.

Original: Lillian et al.
Revised: Lillian et al.

Original: ; Meyes et al.
Revised: ; Meyes et al.

Original: , we developed an alternative approach for examining the necessity of class selectivity for network performance.
Revised: , we developed an alternative approach for examining the necessity of class selectivity for network performance.

Original: By adding a term to the loss function that serves as a regularizer to suppress (or increase) class selectivity, we demonstrate that it is possible to directly modulate the amount of class selectivity in all units in aggregate.
Revised: By adding a term to the loss function that serves as a regularizer to suppress (or increase) class selectivity, we demonstrate that it is possible to directly modulate the amount of class selectivity in all units in aggregate.

Original: We then used this approach as the basis for a series of experiments in which we modulated levels of class selectivity across individual units and measured the resulting effects on the network.
Revised: We then used this approach as the basis for a series of experiments in which we modulated levels of class selectivity across individual units and measured the resulting effects on the network.

Original: Critically, the selectivity regularizer sidesteps the limitations of single unit ablation-based approaches, allowing us to answer otherwise-inaccessible questions such as whether single units actually need to learn class selectivity, and whether increased levels of class selectivity are beneﬁcial.
Revised: Critically, the selectivity regularizer sidesteps the limitations of single unit ablation-based approaches, allowing us to answer otherwise-inaccessible questions such as whether single units actually need to learn class selectivity, and whether increased levels of class selectivity are beneﬁcial.

Original: Unless otherwise noted: all experimental results were derived from the test set with the parameters from the epoch that achieved the highest validation set accuracy over the training epochs; 20 replicates with different random seeds were run for each hyperparameter set; error bars and shaded regions denote bootstrapped 95% conﬁdence intervals; selectivity regularization was not applied to the ﬁnal (output) layer, nor was the ﬁnal layer included in any of our analyses because by deﬁnition the outputlayer must be class selective in a classiﬁcation task. 
Revised: Unless otherwise noted: all experimental results were derived from the test set with the parameters from the epoch that achieved the highest validation set accuracy over the training epochs; 20 replicates with different random seeds were run for each hyperparameter set; error bars and shaded regions denote bootstrapped 95% conﬁdence intervals; selectivity regularization was not applied to the output (logits), nor was the output included in any of our analyses because by deﬁnition the output must be class selective in a classiﬁcation task. Selectivity regularization was only applied to intermediate (hidden) layers with non-linearities.

Original: 3. M ODELS AND DATASETS
Revised: 3. M ODELS AND DATASETS

Original: Our experiments were performed on ResNet18  (He et al., 2016) trained on Tiny ImageNet (Fei-Fei et al., 2015), and ResNet20 (He et al., 2016) and a VGG16-like network (Simonyan and Zisserman, 2015), both trained on CIFAR10 (Krizhevsky, 2009).
Revised: Our experiments were performed on ResNet18 and ResNet50 (He et al., 2016) trained on Tiny ImageNet (Fei-Fei et al., 2015), and ResNet20 (He et al., 2016) and a VGG16-like network (Simonyan and Zisserman, 2015), both trained on CIFAR10 (Krizhevsky, 2009).

Original: Additional details about hyperparameters, data, training, and software are in Appendix A.1.
Revised: Additional details about hyperparameters, data, training, and software are in Appendix A.1.

Original: We focus on  Tiny ImageNet in the main text, but results were qualitatively similar across models and datasets except where noted.
Revised: We focus on ResNet18 trained on Tiny ImageNet in the main text, but results were qualitatively similar across models and datasets except where noted.

Original: 3. D EFINING CLASS SELECTIVITY
Revised: 3. D EFINING CLASS SELECTIVITY

Original: There are a breadth of approaches for quantifying class selectivity in individual units (Moody et al., 1998;
Revised: There are a breadth of approaches for quantifying class selectivity in individual units (Moody et al., 1998;

Original: Zhou et al., 2015; Li et al., 2015; Zhou et al., 2018; Gale et al., 2019).
Revised: Zhou et al., 2015; Li et al., 2015; Zhou et al., 2018; Gale et al., 2019).

Original: We chose the neuroscience-inspired approach of Morcos et al.
Revised: We chose the neuroscience-inspired approach of Morcos et al.

Original: (2018b) because it is similar to many widely-used metrics, easy to compute, and most importantly, differentiable (the utility of this is addressed in the next section).
Revised: (2018b) because it is similar to many widely-used metrics, easy to compute, and most importantly, differentiable (the utility of this is addressed in the next section).

Original: We also conﬁrmed the efﬁcacy of our regularizer on a different, non-differentiable selectivity metric (see Appendix A.13).
Revised: We also conﬁrmed the efﬁcacy of our regularizer on a different, non-differentiable selectivity metric (see Appendix A.13).

Original: For a single convolutional feature map (which we refer to as a "unit"), we computed the mean activation across elements of the ﬁlter map in response to a single sample, after the non-linearity.
Revised: For a single convolutional feature map (which we refer to as a "unit"), we computed the mean activation across elements of the ﬁlter map in response to a single sample, after the non-linearity.

Original: Then the class-conditional mean activation (i.e. the mean activation for each class) was calculated across all samples in the test set, and the class selectivity index ( SI ) was calculated as follows:
Revised: Then the class-conditional mean activation (i.e. the mean activation for each class) was calculated across all samples in the test set, and the class selectivity index ( SI ) was calculated as follows:

Original: [Equation]
Revised: [Equation]

Original:  where µ max is the largest class-conditional mean activation, µ − max is the mean response to the remaining (i.e. non- µ max ) classes, and (cid:15) is a small value to prevent division by zero (we used 10 − 7 ) in the case of a dead unit.
Revised: ) where µ max is the largest class-conditional mean activation, µ − max is the mean response to the remaining (i.e. non- µ max ) classes, and (cid:15) is a small value to prevent division by zero (we used 10 − 7 ) in the case of a dead unit.

Original: The selectivity index can range from 0 to 1.
Revised: The selectivity index can range from 0 to 1.

Original: A unit with identical average activity for all classes would have a selectivity of 0, and a unit that only responded to a single class would have a selectivity of 1. 
Revised: A unit with identical average activity for all classes would have a selectivity of 0, and a unit that only responded to a single class would have a selectivity of 1. [Figure]

Original: As Morcos et al.
Revised: As Morcos et al.

Original: (2018b) note, this selectivity index is not a perfect measure of information content in single units.
Revised: (2018b) note, this selectivity index is not a perfect measure of information content in single units.

Original: For example, a unit with some information about many classes would have a low selectivity index.
Revised: For example, a unit with some information about many classes would have a low selectivity index.

Original: But it achieves the goal of identifying units that are class-selective in a similarly intuitive way as prior studies (Zhou et al., 2018), while also being differentiable with respect to the model parameters.[Figure]
Revised: But it achieves the goal of identifying units that are class-selective in a similarly intuitive way as prior studies (Zhou et al., 2018), while also being differentiable with respect to the model parameters.

Original: 3. A SINGLE KNOB TO CONTROL CLASS SELECTIVITY
Revised: 3. A SINGLE KNOB TO CONTROL CLASS SELECTIVITY

Original: Because the class selectivity index is differentiable, we can insert it into the loss function, allowing us to directly regularize for or against class selectivity.
Revised: Because the class selectivity index is differentiable, we can insert it into the loss function, allowing us to directly regularize for or against class selectivity.

Original: Our loss function, which we seek to minimize, thus takes the following form:
Revised: Our loss function, which we seek to minimize, thus takes the following form:

Original: [Equation]
Revised: [Equation]

Original: The left-hand term in the loss function is the traditional cross-entropy between the softmax of the output units and the true class labels, where c is the class index, C is the number of classes, y c is the true class label, and ˆ y c is the predicted class probability.
Revised: The left-hand term in the loss function is the traditional cross-entropy between the softmax of the output units and the true class labels, where c is the class index, C is the number of classes, y c is the true class label, and ˆ y c is the predicted class probability.

Original: We refer to the right-hand component of the loss function, − αµ SI , as the class selectivity regularizer (or regularizer, for brevity).
Revised: We refer to the right-hand component of the loss function, − αµ SI , as the class selectivity regularizer (or regularizer, for brevity).

Original: The regularizer consists of two terms: the selectivity term,
Revised: The regularizer consists of two terms: the selectivity term,

Original: [Equation]
Revised: [Equation]

Original: where l is a convolutional layer, L is number of layers, u is a unit (i.e. feature map), U is the number of units in a given layer, and SI u is the class selectivity index of unit u .
Revised: where l is a convolutional layer, L is number of layers, u is a unit (i.e. feature map), U is the number of units in a given layer, and SI u is the class selectivity index of unit u .

Original: The selectivity term of the regularizer is obtained by computing the selectivity index for each unit in a layer, then computing the mean selectivity index across units within each layer, then computing the mean selectivity index across layers.
Revised: The selectivity term of the regularizer is obtained by computing the selectivity index for each unit in a layer, then computing the mean selectivity index across units within each layer, then computing the mean selectivity index across layers.

Original: Computing the mean within layers before computing the mean across layers (as compared to computing the mean across all units in the network) mitigates the biases induced by the larger numbers of units in deeper layers.
Revised: Computing the mean within layers before computing the mean across layers (as compared to computing the mean across all units in the network) mitigates the biases induced by the larger numbers of units in deeper layers.

Original: The remaining term in the regularizer is α , the regularizer scale.
Revised: The remaining term in the regularizer is α , the regularizer scale.

Original: The sign of α determines whether class selectivity is promoted or discouraged.
Revised: The sign of α determines whether class selectivity is promoted or discouraged.

Original: Negative values of α discourage class selectivity in individual units, while positive values promote it.
Revised: Negative values of α discourage class selectivity in individual units, while positive values promote it.

Original: The magnitude of α controls the contribution of the selectivity term to the overall loss.
Revised: The magnitude of α controls the contribution of the selectivity term to the overall loss.

Original: α thus serves as a single knob with which we can modulate class selectivity across all units in the network in aggregate.
Revised: α thus serves as a single knob with which we can modulate class selectivity across all units in the network in aggregate.

Original: During training, the class selectivity index was computed for each minibatch.
Revised: During training, the class selectivity index was computed for each minibatch.

Original: For the results presented here, the class selectivity index was computed across the entire test set.
Revised: For the results presented here, the class selectivity index was computed across the entire test set.

Original: We also experimented with restricting regularization to the ﬁrst or ﬁnal three layers, which yielded qualitatively similar results (Appendix A.14).
Revised: We also tried restricting regularization to the ﬁrst or ﬁnal three layers (Appendix A.15), and warming up the class selectivity regularization over the initial training epochs (Appendix A.16), all of which yielded qualitatively similar results.

Original:  R ESULTS 4. T EST ACCURACY IS IMPROVED OR UNAFFECTED BY REDUCING CLASS SELECTIVITY
Revised:  R ESULTS 4. T EST ACCURACY IS IMPROVED OR UNAFFECTED BY REDUCING CLASS SELECTIVITY

Original: Prior research has yielded equivocal results regarding the importance of class selectivity in individual units.
Revised: Prior research has yielded equivocal results regarding the importance of class selectivity in individual units.

Original: We sidestepped the limitations of previous approaches by regularizing against selectivity directly in the loss function through the addition of the selectivity term (see Approach 3.3), giving us  a knob with which to causally manipulate class selectivity.
Revised: We sidestepped the limitations of previous approaches by regularizing against selectivity directly in the loss function through the addition of the selectivity term (see Approach 3.3), giving us [Figure] a knob with which to causally manipulate class selectivity.

Original: We ﬁrst veriﬁed that the regularizer works as intended (Figure A1).
Revised: We ﬁrst veriﬁed that the regularizer works as intended (Figure A1).

Original: Indeed, class selectivity across units in a network decreases as α becomes more negative.
Revised: Indeed, class selectivity across units in a network decreases as α becomes more negative.

Original: We also conﬁrmed that our class selectivity regularizer hasa similar effect when measured using a different class selectivity metric  (see Appendix A.13), indicating that our results are not unique to the metric used in our regularizer.
Revised: We also conﬁrmed that our class selectivity regularizer has similar effects when measured using a different class selectivity metric and mutual information (see Appendix A.13) and when regularizing to control d (cid:48) —a measure of class discriminability—in individual units (Appendix A.14). The consistency of our observations across metrics of selectivity indicates that our results are not unique to the metric used in our regularizer.

Original: The regularizer thus allows us to to examine the causal impact of class selectivity on test accuracy.
Revised: The regularizer thus allows us to to examine the causal impact of class selectivity on test accuracy.

Original: [Figure]

Original: Regularizing against class selectivity could yield three possible outcomes: If the previously-reported anti-correlation between selectivity and generalization is causal, then test accuracy should increase.
Revised: Regularizing against class selectivity could yield three possible outcomes: If the previously-reported anti-correlation between selectivity and generalization is causal, then test accuracy should increase.

Original: But if class selectivity is necessary for high-ﬁdelity class representations, then we should observe a decrease in test accuracy.
Revised: But if class selectivity is necessary for high-ﬁdelity class representations, then we should observe a decrease in test accuracy.

Original: Finally, if class selectivity is an emergent phenomenon and/or irrelevant to network performance, test accuracy should remain unchanged.
Revised: Finally, if class selectivity is an emergent phenomenon and/or irrelevant to network performance, test accuracy should remain unchanged.

Original: Surprisingly, we observed that reducing selectivity signiﬁcantly improves test accuracy in ResNet trained on Tiny ImageNet for all examined values of α ∈ [ − 0 .
Revised: Surprisingly, we observed that reducing selectivity signiﬁcantly improves test accuracy in ResNet trained on Tiny ImageNet for all examined values of α ∈ [ − 0 .

Original: 1 , − 2 .
Revised: 1 , − 2 .

Original: (Figure 1; p < 0 .
Revised: (Figure 1; p < 0 .

Original: Bonferroni-corrected t-test).
Revised: Bonferroni-corrected t-test).

Original: Test accuracy increases with the magnitude of α , reaching a maximum at α = − 1 . 

Revised: Test accuracy increases with the magnitude of α , reaching a maximum at α = − 1 .

Original: (test accuracy at α − 1 . 

Revised: (test accuracy at α − 1 .

Original: = 53 .
Revised: = 53 .

Original: 60 ± 0 .
Revised: 60 ± 0 .

Original: 13 , α 0 (i.e. no regularization) = 51 .
Revised: 13 , α 0 (i.e. no regularization) = 51 .

Original: 57 ± 0 . 18 ), at which point there is a 1.6x reduction in class selectivity (mean class selectivity at α − 1 . 

Revised: 57 ± 0 . 18 ), at which point there is a 1.6x reduction in class selectivity (mean class selectivity at α − 1 .

Original: = 0 .
Revised: = 0 .

Original: 22 ± 0 .
Revised: 22 ± 0 .

Original: 0009 , α 0 = 0 .
Revised: 0009 , α 0 = 0 .

Original: 35 ± 0 . 0007 ).
Revised: 35 ± 0 . 0007 ).

Original: Test accuracy then begins to decline; at α − 3 . 0 test accuracy is statistically indistinct from α 0 , despite a 3x decrease in class class selectivity (mean class selectivity at α − 3 .
Revised: Test accuracy then begins to decline; at α − 3 . 0 test accuracy is statistically indistinct from α 0 , despite a 3x decrease in class class selectivity (mean class selectivity at α − 3 .

Original: = 0 .
Revised: = 0 .

Original: 12 ± 0 . 0007 , α 0 = 0 .
Revised: 12 ± 0 . 0007 , α 0 = 0 .

Original: 35 ± 0 . 0007 ).
Revised: 35 ± 0 . 0007 ).

Original: Further reducing class selectivity beyond α = − 3 .(mean
Revised: Further reducing class selectivity beyond α = − 3 .(mean

Original: class selectivity = 0 .
Revised: class selectivity = 0 .

Original: 10 ± 0 . 0007 ) has increasingly detrimental effects on test accuracy.
Revised: 10 ± 0 . 0007 ) has increasingly detrimental effects on test accuracy.

Original: These results show that the amount of class selectivity naturally learned by a network (i.e. the amount learned in the absence of explicit regularization) can actually constrain the network’s performance.
Revised: These results show that the amount of class selectivity naturally learned by a network (i.e. the amount learned in the absence of explicit regularization) can actually constrain the network’s performance.

Original: ResNet20 trained on CIFAR10 also learned superﬂuous class selectivity.
Revised: ResNet20 trained on CIFAR10 also learned superﬂuous class selectivity.

Original: Although reducing class selectivity does not improve performance, it causes minimal detriment, except at extreme regularization scales ( α ≤ − 30 ; Figure A2).
Revised: Although reducing class selectivity does not improve performance, it causes minimal detriment, except at extreme regularization scales ( α ≤ − 30 ; Figure A2).

Original: Increasing the magnitude of α decreases mean class selectivity across the network, with little impact on test accuracy until mean class selectivity reaches 0 .
Revised: Increasing the magnitude of α decreases mean class selectivity across the network, with little impact on test accuracy until mean class selectivity reaches 0 .

Original: 003 ± 0 . 

Revised: 003 ± 0 .

Original: at α − 30 (Figure A1d).
Revised: at α − 30 (Figure A1d).

Original: Reducing class selectivity only begins to have a statistically signiﬁcant effect on performance at α − 1 . 

Revised: Reducing class selectivity only begins to have a statistically signiﬁcant effect on performance at α − 1 .

Original: (Figure A2a), at which point mean class selectivity across the network has decreased from 0 .
Revised: (Figure A2a), at which point mean class selectivity across the network has decreased from 0 .

Original: 22 ± 0 .
Revised: 22 ± 0 .

Original: 002 at α 

Revised: 002 at α

Original: (i.e. no regularization) to 0 .
Revised: (i.e. no regularization) to 0 .

Original: 07 ± 0 .
Revised: 07 ± 0 .

Original: 0013 at α − 1 . 0 —a factor of more than 3 (Figure A2c; p = 0 .
Revised: 0013 at α − 1 . 0 —a factor of more than 3 (Figure A2c; p = 0 .

Original: 03 , Bonferroni-corrected t-test).
Revised: 03 , Bonferroni-corrected t-test).

Original: This implies that ResNet20 learns more than three times the amount of class selectivity required for maximum test accuracy.
Revised: This implies that ResNet20 learns more than three times the amount of class selectivity required for maximum test accuracy.

Original: We observed qualitatively similar results for VGG16 (see Appendix A.12).
Revised: We observed qualitatively similar results for VGG16 (see Appendix A.12).

Original: Although the difference is signiﬁcant at α = − 0 .
Revised: Although the difference is signiﬁcant at α = − 0 .

Original: 1 ( p = 0 .
Revised: 1 ( p = 0 .

Original: 004 , Bonferroni-corrected t-test), it is possible to reduce mean class selectivity by a factor of 5 with only a 0.5% decrease in test accuracy, and by a factor of 10 with only a ∼ 1% drop in test accuracy.
Revised: 004 , Bonferroni-corrected t-test), it is possible to reduce mean class selectivity by a factor of 5 with only a 0.5% decrease in test accuracy, and by a factor of 10 with only a ∼ 1% drop in test accuracy.

Original: These differences may be due to VGG16’s naturally higher levels of class selectivity (see Appendix A.15 for comparisons between VGG16 and ResNet20).
Revised: These differences may be due to VGG16’s naturally higher levels of class selectivity (see Figure A16 for comparisons between VGG16 and ResNet20).

Revised: We also observed qualitatively similar results when using our regularization approach to decrease d (cid:48) , a measure of class discriminability, in ResNet20 trained on CIFAR10 and ResNet18 trained on Tiny ImageNet (Appendix A.14).

Revised: Furthermore, we also ﬁnd that regularizing to reduce class

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original:  Together, these results demonstrate that class selectivity in individual units is largely unnecessary for optimal performance in CNNs trained on image classiﬁcation tasks.
Revised: [Figure] [Figure] selectivity improves test accuracy in ResNet50 trained on Tiny ImageNet (Appendix A.18). Together, these results demonstrate that class selectivity in individual units is largely unnecessary for optimal performance in CNNs trained on image classiﬁcation tasks.

Original: 4. D OES SELECTIVITY SHIFT TO A DIFFERENT BASIS SET ?
Revised: 4. D OES SELECTIVITY SHIFT TO A DIFFERENT BASIS SET ?

Original: We were able to reduce mean class selectivity in all examined networks by a factor of at least three with minimal negative impact on test accuracy ( ∼ 1%, at worst, for VGG16).
Revised: We were able to reduce mean class selectivity in all examined networks by a factor of at least three with minimal negative impact on test accuracy ( ∼ 1%, at worst, for VGG16).

Original: However, one trivial solution for reducing class selectivity is for the network to "hide" it from the regularizer by rotating it off of unit-aligned axes or performing some other linear transformation.
Revised: However, one trivial solution for reducing class selectivity is for the network to "hide" it from the regularizer by rotating it off of unit-aligned axes or performing some other linear transformation.

Original: In this scenario the selectivity in individual units would be reduced, but remain accessible through linear combinations of activity across units.
Revised: In this scenario the selectivity in individual units would be reduced, but remain accessible through linear combinations of activity across units.

Original: In order to test this possibility, we used CCA (see Appendix A.3), which is invariant to

Original: [Figure]

Original: r

Original: [Figure]

Original: c

Original: [Figure]

Original: [Figure]

Original: ** ** b)

Original: [Figure] [Figure] [Figure] rotation and other invertible afﬁne transformations, to compare the representations in regularized (i e. low-selectivity) networks to the representations in unregularized networks.
Revised: In order to test this possibility, we used CCA (see Appendix A.3), which is invariant to rotation and other invertible afﬁne transformations, to compare the representations in regularized (i.e. low-selectivity) networks to the representations in unregularized networks.

Original: We ﬁrst established a meaningful baseline for comparison by computing the CCA distances between each pair of 20 replicate networks for a given value of α (we refer to this set of distances as ρ ( α r , α r ) ).
Revised: We ﬁrst established a meaningful baseline for comparison by computing the CCA distances between each pair of 20 replicate networks for a given value of α (we refer to this set of distances as ρ ( α r , α r ) ).

Original: If regularizing against class selectivity causes the network to move selectivity off-axis, the CCA distances between regularized and unregularized networks —which we term ρ ( α r , α 0 ) —should be similar to ρ ( α r , α r ) .
Revised: If regularizing against class selectivity causes the network to move selectivity off-axis, the CCA distances between regularized and unregularized networks —which we term ρ ( α r , α 0 ) —should be similar to ρ ( α r , α r ) .

Original: Alternatively, if class selectivity is suppressed via some non-afﬁne transformation of the representation, ρ ( α r , α 0 ) should exceed ρ ( α r , α r ) .
Revised: Alternatively, if class selectivity is suppressed via some non-afﬁne transformation of the representation, ρ ( α r , α 0 ) should exceed ρ ( α r , α r ) .

Original: Our analyses conﬁrm the latter hypothesis: we ﬁnd that ρ ( α r , α 0 ) signiﬁcantly exceeds ρ ( α r , α r ) for all values of α except α = − 0 .
Revised: Our analyses conﬁrm the latter hypothesis: we ﬁnd that ρ ( α r , α 0 ) signiﬁcantly exceeds ρ ( α r , α r ) for all values of α except α = − 0 .

Original: 1 in ResNet18 trained on Tiny ImageNet (Figure 2 p < 1 .
Revised: 1 in ResNet18 trained on Tiny ImageNet (Figure 2 p < 1 .

Original: 3 × 10 − 5 , paired t-test).
Revised: 3 × 10 − 5 , paired t-test).

Original: The effect is even more striking in ResNet20 trained on CIFAR10; all tested values of α are signiﬁcant (Figure A3; p < 5 × 10 6 , paired t-test).
Revised: The effect is even more striking in ResNet20 trained on CIFAR10; all tested values of α are signiﬁcant (Figure A3; p < 5 × 10 6 , paired t-test).

Original: Furthermore, the size of the effect is proportional to α in both models; larger α values yield representations that are more dissimilar to unregularized representations.
Revised: Furthermore, the size of the effect is proportional to α in both models; larger α values yield representations that are more dissimilar to unregularized representations.

Original: These results support the conclusion that our regularizer doesn’t just cause class selectivity to be rotated off of unit-aligned axes, but also suppresses it.
Revised: These results support the conclusion that our regularizer doesn’t just cause class selectivity to be rotated off of unit-aligned axes, but also suppresses it.

Original: As an additional control to ensure that our regularizer did not simply shift class selectivity to off-axis directions in activation space, we calculated an upper bound on the amount of class selectivity that could be recovered by ﬁnding the linear projection of unit activations that maximizes class selectivity (see Appendix A.7 for methodological details).
Revised: As an additional control to ensure that our regularizer did not simply shift class selectivity to off-axis directions in activation space, we calculated an upper bound on the amount of class selectivity that could be recovered by ﬁnding the linear projection of unit activations that maximizes class selectivity (see Appendix A.7 for methodological details).

Original: For both ResNet18 trained on Tiny ImageNet (Figure 2c) and ResNet20 trained on CIFAR10 (Figure A4b), the amount of class selectivity in the optimized projection decreases as a function of increasing | α | , indicating that regularizing against class selectivity does not simply rotate the selectivity off-axis.
Revised: For both ResNet18 trained on Tiny ImageNet (Figure 2c) and ResNet20 trained on CIFAR10 (Figure A4b), the amount of class selectivity in the optimized projection decreases as a function of increasing | α | , indicating that regularizing against class selectivity does not simply rotate the selectivity off-axis.

Original: Interestingly, the upper bound on class selectivity is very similar across regularization scales in the ﬁnal two convolutional layers in both models (Figure A4a; A4c), indicating that immediate proximity to the output layer may mitigate the effect of class selectivity regularization.
Revised: Interestingly, the upper bound on class selectivity is very similar across regularization scales in the ﬁnal two convolutional layers in both models (Figure A4a; A4c), indicating that immediate proximity to the logits (output) may mitigate the effect of class selectivity regularization.

Original: While we also found that the amount of class selectivity in the optimized projection is consistently higher than the observed axis-aligned class selectivity, we consider this to be an expected result, as the optimized projection represents an upper bound on the amount class selectivity that could be recovered from the models’ representations.
Revised: While we also found that the amount of class selectivity in the optimized projection is consistently higher than the observed axis-aligned class selectivity, we consider this to be an expected result, as the optimized projection represents an upper bound on the amount class selectivity that could be recovered from the models’ representations.

Original: However, the decreasing upper bound as a function of increasing | α | indicates that our class selectivity regularizer decreases selectivity across all basis sets, and not just along unit-aligned axes. 
Revised: However, the decreasing upper bound as a function of increasing | α | indicates that our class selectivity regularizer decreases selectivity across all basis sets, and not just along unit-aligned axes. [Figure]

Original: 4. I NCREASED CLASS SELECTIVITY CONSIDERED HARMFUL
Revised: 4. I NCREASED CLASS SELECTIVITY CONSIDERED HARMFUL

Original: We have demonstrated that class selectivity can be signiﬁcantly reduced with minimal impact on test accuracy.
Revised: We have demonstrated that class selectivity can be signiﬁcantly reduced with minimal impact on test accuracy.

Original: However, we only examined the effects of reducing selectivity.
Revised: However, we only examined the effects of reducing selectivity.

Original: What are the effects of increasing selectivity?
Revised: What are the effects of increasing selectivity?

Original: We examined this question by regularizing for class selectivity, instead of against it.
Revised: We examined this question by regularizing for class selectivity, instead of against it.

Original: This is achieved quite easily, as it requires only a change in the sign of α .
Revised: This is achieved quite easily, as it requires only a change in the sign of α .

Original: We ﬁrst conﬁrmed that changing the sign of the scale term in the loss function causes the intended effect of increasing class selectivity in individual units (see Appendix A.8).
Revised: We ﬁrst conﬁrmed that changing the sign of the scale term in the loss function causes the intended effect of increasing class selectivity in individual units (see Appendix A.8).

Original: Despite class selectivity not being strictly necessary for high performance, its ubiquity across biological and artiﬁcial neural networks leads us to suspect it may still be sufﬁcient.
Revised: Despite class selectivity not being strictly necessary for high performance, its ubiquity across biological and artiﬁcial neural networks leads us to suspect it may still be sufﬁcient.

Original: We thus expect that increasing it would either improve test accuracy or yield no effect.
Revised: We thus expect that increasing it would either improve test accuracy or yield no effect.

Original: For the same reason, we would consider it unexpected if increasing selectivity impairs test accuracy.
Revised: For the same reason, we would consider it unexpected if increasing selectivity impairs test accuracy.

Original: [Figure] Surprisingly, we observe the latter outcome: increasing class selectivity negatively impacts network performance in ResNet18 trained on Tiny ImageNet (Figure 3a).
Revised: Surprisingly, we observe the latter outcome: increasing class selectivity negatively impacts network performance in ResNet18 trained on Tiny ImageNet (Figure 3a).

Original: Scaling the regularization has an immediate effect: a signiﬁcant decline in test accuracy is present even at the smallest tested value of α ( p ≤ 6 × 10 − 5 for all α , Bonferroni-corrected t-test) and falls catastrophically to ∼ 25% by α = 5 . 0 .
Revised: Scaling the regularization has an immediate effect: a signiﬁcant decline in test accuracy is present even at the smallest tested value of α ( p ≤ 6 × 10 − 5 for all α , Bonferroni-corrected t-test) and falls catastrophically to ∼ 25% by α = 5 . 0 .

Original: The effect proceeds even more dramatically in ResNet20 trained on CIFAR10 (Figure A6a).
Revised: The effect proceeds even more dramatically in ResNet20 trained on CIFAR10 (Figure A6a).

Original: Note that we observed a correlation between the strength of regularization and the presence of dead units in ResNet20, but further analyses ruled this out as an explanation for the decline in test accuracy (see Appendix A.10).
Revised: Note that we observed a correlation between the strength of regularization and the presence of dead units in ResNet20 (but not ResNet18), however further analyses ruled this out as an explanation for the decline in test accuracy (see Appendix A.10).

Original: One solution to generate a very high selectivity index is if a unit is silent for the vast majority of inputs and has low activations for remaining set of inputs.
Revised: One solution to generate a very high selectivity index is if a unit is silent for the vast majority of inputs and has low activations for remaining set of inputs.

Original: If this were the case, we would expect that regularizing to increase selectivity would cause units to be silent for the majority of inputs.
Revised: If this were the case, we would expect that regularizing to increase selectivity would cause units to be silent for the majority of inputs.

Original: However, we found that the majority of units were active for ≥ 80% of inputs even at α = 0 .
Revised: However, we found that the majority of units were active for ≥ 80% of inputs even at α = 0 .

Original: 7 , after signiﬁcant performance deﬁcits have emerged in both ResNet18 and ResNet20 (Appendix A.11).
Revised: 7 , after signiﬁcant performance deﬁcits have emerged in both ResNet18 and ResNet20 (Appendix A.11).

Original: These ﬁndings rule out sparsity as a potential explanation for our results.
Revised: These ﬁndings rule out sparsity as a potential explanation for our results.

Original: The results are qualitatively similar for VGG16 (see Appendix A.12), indicating that increasing class selectivity beyond the levels that are learned naturally (i.e. without regularization, α = 0 ) impairs network performance. 
Revised: It is also possible that regularizing to increase class selectivity could discourage individual neurons from changing their preferred class during training, even if changing their preferred class would improve performance. If regularizing to increase class selectivity did indeed lock units in to their initial preferred class, this could impose a constraint on performance. We tested for this possibility in two ways (Appendix A.16): by examining the statistics of units’ changes in preferred class during training (Figures A28,

Revised: A29), and slowly warming up class selectivity regularization over the initial training epochs (Figures

Revised: A30; A31).

Revised: Approximately 100% of units across all examined models and regularization scales change their preferred class at least once during training (Figure A28), and the relationship between class selectivity regularization and the number of changes in a unit’s preferred class over training is inconsistent (Figure A29).

Revised: Furthermore, warming up the regularization has qualitatively similar effects as using a constant α (Figure A31).

Revised: None of these analyses indicate that an inability to change preferred locations can fully explain the class selectivity-induced test accuracy impairment (see

Revised: Appendix A.16 for additional details).

Revised: The effects of regularizing to increase class selectivity are qualitatively similar for VGG16 (see

Revised: Appendix A.12) and ResNet50 (Appendix A.18); we observed across all models that increasing class selectivity beyond the levels that are learned naturally (i.e. without regularization, α = 0 ) impairs network performance.

Original: Recapitulation We directly compare the effects of increasing vs. decreasing class selectivity in
Revised: Recapitulation We directly compare the effects of increasing vs. decreasing class selectivity in

Original: Figure 4 (and Appendix A.15).
Revised: Figure 4 and Appendix A.17.

Original: The effects diverge immediately at | α | = 0 .
Revised: The effects diverge immediately at | α | = 0 .

Original: 1 , and suppressing class selectivity yields a 6% increase in test accuracy relative to increasing class selectivity by | α | = 2 . 0 .
Revised: 1 , and suppressing class selectivity yields a 6% increase in test accuracy relative to increasing class selectivity by | α | = 2 . 0 .

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We examined the causal role of class selectivity in CNN performance by adding a term to the loss function that allows us to directly manipulate class selectivity across all neurons in the network.
Revised: We examined the causal role of class selectivity in CNN performance by adding a term to the loss function that allows us to directly manipulate class selectivity across all neurons in the network.

Original: We found that class selectivity is not strictly necessary for networks to function, and that reducing it can even improve test accuracy.
Revised: We found that class selectivity is not strictly necessary for networks to function, and that reducing it can even improve test accuracy.

Original: In ResNet18 trained on Tiny Imagenet, reducing class selectivity by 1.6 × improved test accuracy by over 2%.
Revised: In ResNet18 trained on Tiny Imagenet, reducing class selectivity by 1.6 × improved test accuracy by over 2%.

Original: In ResNet20 trained on CIFAR10, we could reduce the mean class selectivity of units in a network by factor of ∼ 2.5 with no impact on test accuracy, and by a factor of ∼ 20—nearly to a mean of 0—with only a 2% change in test accuracy.
Revised: In ResNet20 trained on CIFAR10, we could reduce the mean class selectivity of units in a network by factor of ∼ 2.5 with no impact on test accuracy, and by a factor of ∼ 20—nearly to a mean of 0—with only a 2% change in test accuracy.

Original: We conﬁrmed that our regularizer seems to suppress class selectivity, and not simply cause the network to rotate it off of unit-aligned axes.
Revised: We conﬁrmed that our regularizer seems to suppress class selectivity, and not simply cause the network to rotate it off of unit-aligned axes.

Original: We also found that regularizing a network to increase class selectivity in individual units has negative effects on performance.
Revised: We also found that regularizing a network to increase class selectivity in individual units has negative effects on performance.

Original: These results resolve questions about class selectivity that remained inaccessible to previous approaches: class selectivity in individual units is neither necessary nor sufﬁcient for—and can sometimes even constrain—CNN performance.
Revised: These results resolve questions about class selectivity that remained inaccessible to previous approaches: class selectivity in individual units is neither necessary nor sufﬁcient for—and can sometimes even constrain—CNN performance.

Original: One caveat to our results is that they are limited to CNNs trained to perform image classiﬁcation.
Revised: One caveat to our results is that they are limited to CNNs trained to perform image classiﬁcation.

Original: It’s possible that our ﬁndings are due to idiosyncracies of benchmark datasets, and wouldn’t generalize to more naturalistic datasets and tasks.
Revised: It’s possible that our ﬁndings are due to idiosyncracies of benchmark datasets, and wouldn’t generalize to more naturalistic datasets and tasks.

Original: Given that class selectivity is ubiquitous across DNNs trained on different tasks and datasets, future work should examine how broadly our results generalize, and the viability of class selectivity regularization as a general-purpose tool to improve DNN performance. 
Revised: Given that class selectivity is ubiquitous across DNNs trained on different tasks and datasets, future work should examine how broadly our results generalize, and the viability of class selectivity regularization as a general-purpose tool to improve DNN performance. The presence of non-selective units in a network trained to perform image classiﬁcation could appear puzzling. It may be difﬁcult to envision how non-selective units could shape representations in a manner that is useful for a classiﬁcation task. One possibility is that the class-conditional joint distribution of activations across units facilitates readout.

Revised: Put another way, the correlations between units’ activations can help separate the distributions of activations for different classes.

Original: Our results make a broader point about the potential pitfalls of focusing on the properties of  single units when trying to understand DNNs, emphasizing instead the importance of analyses that focus on distributed representations.
Revised: Indeed, there is evidence that correlated variability between neurons can facilitate information readout in the brain (Zylberberg et al., 2016; Leavitt et al., 2017; Zylberberg, 2018; Nogueira et al., 2020). We know that class selectivity in individual units naturally emerges over the course of learning. The single unit ablation studies show that class selectivity can have an effect on the performance of trained networks. And while it is possible that networks trained with selectivity regularization learn different solutions from networks trained without it, our results show that class selectivity is not strictly necessary for networks to learn representations that result in class-selective units.

Revised: This ﬁnding naturally leads to a compelling question: if class selectivity is unnecessary, why does it emerge?

Revised: Our results also make a broader point about the potential pitfalls of focusing on the properties of single units when trying to understand DNNs, emphasizing instead the importance of analyses that focus on distributed representations.

Original: While we consider it essential to ﬁnd tractable, intuitive approaches for understanding complex systems, it’s critical to empirically verify that these approaches actually reﬂect functionally relevant properties of the system being examined.
Revised: While we consider it essential to ﬁnd tractable, intuitive approaches for understanding complex systems, it’s critical to empirically verify that these approaches actually reﬂect functionally relevant properties of the system being examined.

Original: A A PPENDIX A. M ODELS , TRAINING , DATASETS , AND SOFTWARE
Revised: A A PPENDIX A. M ODELS , TRAINING , DATASETS , AND SOFTWARE

Original: Our experiments were performed on ResNet18  (He et al., 2016) trained on Tiny Imagenet (Fei-Fei et al., 2015), and ResNet20 (He et al.
Revised: Our experiments were performed on ResNet18 and ResNet50 (He et al., 2016) trained on Tiny Imagenet (Fei-Fei et al., 2015), and ResNet20 (He et al.

Original: (2016); code modiﬁed from Idelbayev (2020)) and a VGG16like network (Simonyan and Zisserman, 2015), both trained on CIFAR10 (Krizhevsky, 2009).
Revised: (2016); code modiﬁed from Idelbayev (2020)) and a VGG16-like network (Simonyan and Zisserman, 2015), both trained on CIFAR (Krizhevsky, 2009).

Original: All models were trained using stochastic gradient descent (SGD) with momentum = 0.9 and weight decay = 0.0001.
Revised: All models were trained using stochastic gradient descent (SGD) with momentum = 0.9 and weight decay = 0.0001.

Original: The maxpool layer after the ﬁrst batchnorm layer (see He
Revised: The maxpool layer after the ﬁrst batchnorm layer (see He

Original: et al.
Revised: et al.

Original: (2016)) was removed because of the smaller size of Tiny Imagenet images compared to standard ImageNet images (64x64 vs. 256x256, respectively).
Revised: (2016)) was removed because of the smaller size of Tiny Imagenet images compared to standard ImageNet images (64x64 vs. 256x256, respectively).

Original: ResNet18 were trained for 90 epochs with a minibatch size of 4096 samples with a learning rate of 0.1, multiplied (annealed) by 0.1 at epochs 35, 50, 65, and 80. 
Revised: ResNet18 were trained for 90 epochs with a minibatch size of 4096 samples with a learning rate of 0.1, multiplied (annealed) by 0.1 at epochs 35, 50, 65, and 80. ResNet50 was trained identically, except with a batch size of 1400 samples.

Original: Tiny Imagenet (Fei-Fei et al., 2015) consists of 500 training images and 50 images for each of its 200 classes.
Revised: Tiny Imagenet (Fei-Fei et al., 2015) consists of 500 training images and 50 images for each of its 200 classes.

Original: We used the validation set for testing and created a new validation set by taking 50 images per class from the training set, selected randomly for each training run.
Revised: We used the validation set for testing and created a new validation set by taking 50 images per class from the training set, selected randomly for each training run.

Original: The VGG16-like network is identical to the batch norm VGG16 in Simonyan and Zisserman (2015), except the ﬁnal two fully-connected layers of 4096 units each were replaced with a single 512-unit layer.
Revised: The VGG16-like network is identical to the batch norm VGG16 in Simonyan and Zisserman (2015), except the ﬁnal two fully-connected layers of 4096 units each were replaced with a single 512-unit layer.

Original: ResNet20 and VGG16 were trained for 200 epochs using a minibatch size of 256 samples.
Revised: ResNet20 and VGG16 were trained for 200 epochs using a minibatch size of 256 samples.

Original: ResNet20 were trained with a learning rate of 0.1 and VGG16 with a learning rate of 0.01, both annealed by 10 − 1 at epochs 100 and 150.
Revised: ResNet20 were trained with a learning rate of 0.1 and VGG16 with a learning rate of 0.01, both annealed by 10 − 1 at epochs 100 and 150.

Original: We split the 50k CIFAR10 training samples into a 45k sample training set and a 5k validation set, similar to our approach with Tiny Imagenet.
Revised: We split the 50k CIFAR10 training samples into a 45k sample training set and a 5k validation set, similar to our approach with Tiny Imagenet.

Original: All experimental results were derived from the test set with the parameters from the epoch that achieved the highest validation set accuracy over the training epochs.
Revised: All experimental results were derived from the test set with the parameters from the epoch that achieved the highest validation set accuracy over the training epochs.

Original: 20 replicates with different random seeds were run for each hyperparameter  set.
Revised: 20 replicates with different random seeds were run for each hyperparameter set, except for ResNet50, which only used 5 replicates per hyperparameter set.

Original: Selectivity regularization was not applied to the ﬁnal (output) layer, nor was the ﬁnal layer included any of our analyses.
Revised: Selectivity regularization was not applied to the output (logit) layer, nor was the output layer included any of our analyses.

Original: Experiments were conducted using PyTorch (Paszke et al., 2019), analyzed using the SciPy ecosystem (Virtanen et al., 2019), and visualized using Seaborn (Waskom et al., 2017).
Revised: Experiments were conducted using PyTorch (Paszke et al., 2019), analyzed using the SciPy ecosystem (Virtanen et al., 2019), and visualized using Seaborn (Waskom et al., 2017).

Original: A. E FFECT OF SELECTIVITY REGULARIZER ON TRAINING TIME
Revised: A. E FFECT OF SELECTIVITY REGULARIZER ON TRAINING TIME

Original: We quantiﬁed the number of training epochs required to reach 95% of maximum test accuracy ( t 95 ).
Revised: We quantiﬁed the number of training epochs required to reach 95% of maximum test accuracy ( t 95 ).

Original: The t 95 without selectivity regularization ( t 95 α =0 ) for ResNet20 is 45 ± 15 epochs (median ± IQR).
Revised: The t 95 without selectivity regularization ( t 95 α =0 ) for ResNet20 is 45 ± 15 epochs (median ± IQR).

Original: α in [-2, 0.7] had overlapping IQRs with α = 0 .
Revised: α in [-2, 0.7] had overlapping IQRs with α = 0 .

Original: For ResNet18, t 95 α =0 = 35 ± 1 , while t 95 for α in [-2, 1] was as high as 51 ± 1.5.
Revised: For ResNet18, t 95 α =0 = 35 ± 1 , while t 95 for α in [-2, 1] was as high as 51 ± 1.5.

Original: Beyond these ranges, the t 95 exceeded 1.5 × t 95 α =0 and/or was highly variable.
Revised: Beyond these ranges, the t 95 exceeded 1.5 × t 95 α =0 and/or was highly variable.

Original: A. CCA A.3. A N INTUITION
Revised: A. CCA A.3. A N INTUITION

Original: We used Canonical Correlation Analysis (CCA) to examine the effects of class selectivity regularization on hiden layer representations.
Revised: We used Canonical Correlation Analysis (CCA) to examine the effects of class selectivity regularization on hiden layer representations.

Original: CCA is a statistical method that takes two sets of multidimensional variates and ﬁnds the linear combinations of these variates that have maximum correlation with each other (Hotelling, 1936).
Revised: CCA is a statistical method that takes two sets of multidimensional variates and ﬁnds the linear combinations of these variates that have maximum correlation with each other (Hotelling, 1936).

Original: Critically, CCA is invariant to rotation and other invertible afﬁne transformations.
Revised: Critically, CCA is invariant to rotation and other invertible afﬁne transformations.

Original: CCA has been productively applied to analyze and compare representations in (and between) biological and neural networks (Sussillo et al., 2015; Smith et al., 2015; Raghu et al., 2017;
Revised: CCA has been productively applied to analyze and compare representations in (and between) biological and neural networks (Sussillo et al., 2015; Smith et al., 2015; Raghu et al., 2017;

Original: Morcos et al., 2018a; Gallego et al., 2018).
Revised: Morcos et al., 2018a; Gallego et al., 2018).

Original: We use projection-weighted CCA (PWCCA), a variant of CCA introducted in Morcos et al.
Revised: We use projection-weighted CCA (PWCCA), a variant of CCA introducted in Morcos et al.

Original: (2018a) that has been shown to be more robust to noise than traditional CCA and other CCA variants (though for brevity we just use the term "CCA" in the main text).
Revised: (2018a) that has been shown to be more robust to noise than traditional CCA and other CCA variants (though for brevity we just use the term "CCA" in the main text).

Original: PWCCA generates a scalar value, ρ , that can be thought of as the distance or dissimilarity between the two sets of multidimensional variates,
Revised: PWCCA generates a scalar value, ρ , that can be thought of as the distance or dissimilarity between the two sets of multidimensional variates,

Original: L 1 and L 2 .
Revised: L 1 and L 2 .

Original: For example, if L 2 = L 1 , then ρ L 1 ,L 2 = 0 .
Revised: For example, if L 2 = L 1 , then ρ L 1 ,L 2 = 0 .

Original: Now let R be a rotation matrix.
Revised: Now let R be a rotation matrix.

Original: Because
Revised: Because

Original: CCA is invariant to rotation and other invertible afﬁne transformations, if L 2 = RL 1 (i.e. if Lis a rotation of L 1 ), then ρ L 1 ,L 2 = 0 .
Revised: CCA is invariant to rotation and other invertible afﬁne transformations, if L 2 = RL 1 (i.e. if L is a rotation of L 1 ), then ρ L 1 ,L 2 = 0 .

Original: In contrast, traditional similarity metrics such as Pearson’s
Revised: In contrast, traditional similarity metrics such as Pearson’s

Original: Correlation and cosine similarity would obtain different values if L 2 =
Revised: Correlation and cosine similarity would obtain different values if L 2 =

Original: L 1 compared to L 2 = RL 1 .
Revised: L 1 compared to L 2 = RL 1 .

Original: We use the PWCCA implementation available at https://github.com/google/svcca/, as provided in
Revised: We use the PWCCA implementation available at https://github.com/google/svcca/, as provided in

Original: Morcos et al.
Revised: Morcos et al.

Original: (2018a).
Revised: (2018a).

Original: A.3. O UR APPLICATION
Revised: A.3. O UR APPLICATION

Original: As an example for the analyses in our experiments, L 1 is the activation matrix for a layer in a network that was not regularized against class selectivity (i.e. α = 0 ), and L 2 is the activation matrix for the same layer in a network that was structured and initialized identically, but subject to regularization against class selectivity (i.e. α < 0 ).
Revised: As an example for the analyses in our experiments, L 1 is the activation matrix for a layer in a network that was not regularized against class selectivity (i.e. α = 0 ), and L 2 is the activation matrix for the same layer in a network that was structured and initialized identically, but subject to regularization against class selectivity (i.e. α < 0 ).

Original: If regularizing against class selectivity causes the network’s representations to be rotated (or to undergo to some other invertible afﬁne transformation), then ρ L 1 ,L 2 = 0 .
Revised: If regularizing against class selectivity causes the network’s representations to be rotated (or to undergo to some other invertible afﬁne transformation), then ρ L 1 ,L 2 = 0 .

Original: In practice ρ L 1 ,L 2 > 0 due to differences in random seeds and/or other stochastic factors in the training process, so we can determine a threshold value (cid:15) and say ρ L 1 ,L 2 ≤ (cid:15) .
Revised: In practice ρ L 1 ,L 2 > 0 due to differences in random seeds and/or other stochastic factors in the training process, so we can determine a threshold value (cid:15) and say ρ L 1 ,L 2 ≤ (cid:15) .

Original: If regularizing against class selectivity instead causes a non-afﬁne transformation to the network’s representations, then ρ L 1 ,L 2 > (cid:15) .
Revised: If regularizing against class selectivity instead causes a non-afﬁne transformation to the network’s representations, then ρ L 1 ,L 2 > (cid:15) .

Original: In our experiments we empirically establish a distribution of (cid:15) values by computing the PWCCA distances between ρ L 2 a L 2 b , where L 2 a and L 2 b are two networks from the set of 20 replicates for a given hyperparameter combination that differ only in their initial random seed values (and thus have the same α ).
Revised: In our experiments we empirically establish a distribution of (cid:15) values by computing the PWCCA distances between ρ L 2 a L 2 b , where L 2 a and L 2 b are two networks from the set of 20 replicates for a given hyperparameter combination that differ only in their initial random seed values (and thus have the same α ).

Original: This gives (cid:0) 202 (cid:1) = 190 values of (cid:15) .
Revised: This gives (cid:0) 202 (cid:1) = 190 values of (cid:15) .

Original: We then compute the PWCCA distance between each { L 1 , L 2 } replicate pair, yielding a distribution of 20 × 20 = 400 values of ρ L 1 ,L 2 , which we compare to the distribution of (cid:15) .
Revised: We then compute the PWCCA distance between each { L 1 , L 2 } replicate pair, yielding a distribution of 20 × 20 = 400 values of ρ L 1 ,L 2 , which we compare to the distribution of (cid:15) .

Original: A.3.F ORMALLY 
Revised: A.3.

Revised: F ORMALLY

Original: For the case of our analyses, let us start with a dataset X , which consists of M data samples { x 1 , ...
Revised: For the case of our analyses, let us start with a dataset X , which consists of M data samples { x 1 , ...

Original: x M } .
Revised: x M } .

Original: Using the notation from Raghu et al.
Revised: Using the notation from Raghu et al.

Original: (2017), the scalar output (activation) of a single neuron i on layer ι in response to each data sample collectively form the vector
Revised: (2017), the scalar output (activation) of a single neuron i on layer ι in response to each data sample collectively form the vector

Original: [Equation]
Revised: [Equation]

Original: We then collect the activation vector z li of every neuron in layer ι into a matrix L =
Revised: We then collect the activation vector z li of every neuron in layer ι into a matrix L =

Original: { z ι 1 , ..., z ιM } of size N × M , N is the number of neurons in layer ι , and M is the number of data samples.
Revised: { z ι 1 , ..., z ιM } of size N × M , N is the number of neurons in layer ι , and M is the number of data samples.

Original: Given two such activation matrices L 1 , of size N a × M , and L 2 , of size N b × M , CCA ﬁnds the vectors w (in R N a ) and s (in R N b ), such that the inner product
Revised: Given two such activation matrices L 1 , of size N a × M , and L 2 , of size N b × M , CCA ﬁnds the vectors w (in R N a ) and s (in R N b ), such that the inner product

Original: [Equation]
Revised: [Equation]

Original: is maximized.
Revised: is maximized.

Original: [Figure]
Revised: [Figure]

Original: R EGULARIZING TO DECREASE CLASS SELECTIVITY IN R ES N ET 18 AND R ESNET
Revised: R EGULARIZING TO DECREASE CLASS SELECTIVITY IN R ES N ET 18 AND R ESNET

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: A. D ECREASING CLASS SELECTIVITY WITHOUT DECREASING TEST ACCURACY IN R ES N ET
Revised: A. D ECREASING CLASS SELECTIVITY WITHOUT DECREASING TEST ACCURACY IN R ES N ET

Original: [Figure]

Original: A.

Original: [Figure]
Revised: [Figure]

Original:  CCA R ESULTS FOR R ES N ET
Revised: A. CCA R ESULTS FOR R ES N ET

Original: [Figure]
Revised: [Figure]

Original: CC A D i s t a n c e
Revised: i s t a n c e

Original: R a t i o
Revised: R a t i o

Original: A. C ALCULATING AN UPPER BOUND FOR OFF - AXIS SELECTIVITY
Revised: A. C ALCULATING AN UPPER BOUND FOR OFF - AXIS SELECTIVITY

Original: As an additional control to ensure that our regularizer did not simply shift class selectivity to off-axis directions in activation space, we calculated an upper bound on the amount of class selectivity that could be recovered by ﬁnding the linear projection of unit activations that maximizes class selectivity.
Revised: As an additional control to ensure that our regularizer did not simply shift class selectivity to off-axis directions in activation space, we calculated an upper bound on the amount of class selectivity that could be recovered by ﬁnding the linear projection of unit activations that maximizes class selectivity.

Original: To do so, we ﬁrst collected the validation set activation vector z li of every neuron in layer ι into a matrix A val =
Revised: To do so, we ﬁrst collected the validation set activation vector z li of every neuron in layer ι into a matrix A val =

Original: { z ι 1 , ..., z ιM } of size M × N , where M is the number of data samples in validation set and N is the number of neurons in layer ι .
Revised: { z ι 1 , ..., z ιM } of size M × N , where M is the number of data samples in validation set and N is the number of neurons in layer ι .

Original: We then found the projection matrix W ∈ R N × N that minimizes the loss
Revised: We then found the projection matrix W ∈ R N × N that minimizes the loss

Original: [Equation]
Revised: [Equation]

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: i.e. W is orthonormal, where SI is the selectivity index from Equation 1.
Revised: i.e. W is orthonormal, where SI is the selectivity index from Equation 1.

Revised: We constrained W to be orthonormal because the non-orthonormal solution to maximizing selectivity is degenerate: project all axes onto the single direction in activation space with the highest class selectivity.

Original: We constrained W to be orthonormal using Lezcano-Casado (2019)’s toolbox.
Revised: We used Lezcano-Casado (2019)’s toolbox to constrain W to be orthonormal.

Original: Because SI requires inputs ≥ 0 , we shifted the columns of AW by subtracting the columnwise mininum value before computing SI .
Revised: Because SI requires inputs ≥ 0 , we shifted the columns of AW by subtracting the columnwise mininum value before computing SI .

Original: The optimization was performed using Adam (Kingma and Ba, 2015) with a learning rate of 0.001 for 3500 steps or until the magnitude of the change in loss was less than 10 − 6 for 10 steps.
Revised: The optimization was performed using Adam (Kingma and Ba, 2015) with a learning rate of 0.001 for 3500 steps or until the magnitude of the change in loss was less than 10 − 6 for 10 steps.

Original: W was then used to project the activation matrix for the test set A test , and the selectivity index was calculated for each axis of the new activation space (i.e. each column of A test W ) after shifting the columns of A test W to be ≥ 0 .
Revised: W was then used to project the activation matrix for the test set A test , and the selectivity index was calculated for each axis of the new activation space (i.e. each column of A test W ) after shifting the columns of A test W to be ≥ 0 .

Original: A separate W was obtained for each layer of each model and for each replicate and value of α .
Revised: A separate W was obtained for each layer of each model and for each replicate and value of α .

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: EGULARIZING TO INCREASE CLASS SELECTIVITY IN R ES N ET 18 AND R ES N ET
Revised: EGULARIZING TO INCREASE CLASS SELECTIVITY IN R ES N ET 18 AND R ES N ET

Original: [Figure]
Revised: [Figure]

Original: A. I NCREASED CLASS SELECTIVITY IMPAIRS TEST ACCURACY IN R ES N ET
Revised: A. A DDITIONAL EFFECTS OF CLASS SELECTIVITY REGULARIZATION

Revised: IN R ES N ET

Original: [Figure]
Revised: [Figure]

Original: A. S INGLE UNIT NECROMANCY
Revised: A. S INGLE UNIT NECROMANCY

Original: Lethal ReLUs The inconsistent relationship between α and class selectivity for larger values of α led us to question whether the performance deﬁcits were due to an alternative factor, such as the optimization process, rather than class selectivity per se.
Revised: Lethal ReLUs The inconsistent relationship between α and class selectivity for larger values of α led us to question whether the performance deﬁcits were due to an alternative factor, such as the optimization process, rather than class selectivity per se.

Original: Interestingly, we observed that ResNet regularized to increase selectivity contained signiﬁcantly higher proportions of dead units, and the number of dead units is roughly proportional to alpha (see Figure A7a; also note that this was not a problem in ResNet18).
Revised: Interestingly, we observed that ResNet regularized to increase selectivity contained signiﬁcantly higher proportions of dead units, and the number of dead units is roughly proportional to alpha (see Figure A8a). Regularizing to increase class selectivity did not cause units to die in ResNet18 trained on Tiny ImageNet except at very extreme values of α ( α ≥ 30) , though even at these values the proportion of dead units never exceeded 0.

Original:  Removing the dead units  makes the relationship between regularization and selectivity in ResNet20 more consistent at large regularization scales (see Appendix A7).
Revised: (Figure A7). Removing the dead units in ResNet20 makes the relationship between regularization and selectivity in ResNet20 more consistent at large regularization scales (see Appendix A8).

Original: The presence of dead units is not unexpected, as units with the ReLU activation function are known to suffer from the "dying ReLU problem"(Lu et al., 2019):
Revised: The presence of dead units is not unexpected, as units with the ReLU activation function are known to suffer from the "dying ReLU problem"(Lu et al., 2019):

Original: If, during training, a weight update causes a unit to cease activating in response to all training samples, the unit will be unaffected by subsequent weight updates because the ReLU gradient at x ≤ 0 is zero, and thus the unit’s activation will forever remain zero.
Revised: If, during training, a weight update causes a unit to cease activating in response to all training samples, the unit will be unaffected by subsequent weight updates because the ReLU gradient at x ≤ 0 is zero, and thus the unit’s activation will forever remain zero.

Original: The dead units could explain the decrease in performance from regularizing to increase selectivity as simply a decrease in model capacity.
Revised: The dead units could explain the decrease in performance from regularizing to increase selectivity as simply a decrease in model capacity.

Original: Fruitless resuscitation One solution to the dying ReLU problem is to use a leaky-ReLU activation function (Maas et al., 2013), which has a non-zero slope, b (and thus non-zero gradient) for x ≤ 0 .
Revised: Fruitless resuscitation One solution to the dying ReLU problem is to use a leaky-ReLU activation function (Maas et al., 2013), which has a non-zero slope, b (and thus non-zero gradient) for x ≤ 0 .

Original: Accordingly, we re-ran the previous experiment using units with a leaky-ReLU activation in an attempt to control for the potential confound of dead units.
Revised: Accordingly, we re-ran the previous experiment using units with a leaky-ReLU activation in an attempt to control for the potential confound of dead units.

Original: Note that because the class selectivity index assumes activations ≥ 0 , we shifted activations by subtracting the minimum activation when computing selectivity for leaky-ReLUs.
Revised: Note that because the class selectivity index assumes activations ≥ 0 , we shifted activations by subtracting the minimum activation when computing selectivity for leaky-ReLUs.

Original: If the performance deﬁcits from regularizing for selectivity are simply due to dead units, then using leaky-ReLUs should rescue performance.
Revised: If the performance deﬁcits from regularizing for selectivity are simply due to dead units, then using leaky-ReLUs should rescue performance.

Original: Alternatively, if dead units are not the cause of the performance deﬁcits, then leaky-ReLUs should not have an effect.
Revised: Alternatively, if dead units are not the cause of the performance deﬁcits, then leaky-ReLUs should not have an effect.

Original: We ﬁrst conﬁrmed that using leaky-ReLUs solves the dead unit problem.
Revised: We ﬁrst conﬁrmed that using leaky-ReLUs solves the dead unit problem.

Original: Indeed, the proportion of dead units is reduced to 0 in all networks across all tested values of b .
Revised: Indeed, the proportion of dead units is reduced to 0 in all networks across all tested values of b .

Original: Despite complete recovery of the dead units, however, using leaky-ReLUs does not rescue class selectivity-induced performance deﬁcits (Figure A8).
Revised: Despite complete recovery of the dead units, however, using leaky-ReLUs does not rescue class selectivity-induced performance deﬁcits (Figure A9).

Original: While the largest negative slope value improved test accuracy for larger values of α , the improvement was minor, and increasing α still had catastrophic effects.
Revised: While the largest negative slope value improved test accuracy for larger values of α , the improvement was minor, and increasing α still had catastrophic effects.

Original: These results conﬁrm that dead units cannot explain the rapid and catastrophic effects of increased class selectivity on performance.
Revised: These results conﬁrm that dead units cannot explain the rapid and catastrophic effects of increased class selectivity on performance.

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: A. R ULING OUT A DEGENERATE SOLUTION FOR INCREASING SELECTIVITY
Revised: A. R ULING OUT A DEGENERATE SOLUTION FOR INCREASING SELECTIVITY

Original: One degenerate solution to generate a very high selectivity index is for a unit to be silent for the vast majority of inputs, and have low activations for the small set of remaining inputs.
Revised: One degenerate solution to generate a very high selectivity index is for a unit to be silent for the vast majority of inputs [Figure] and have low activations for the small set of remaining inputs.

Original: We refer to this scenario as "activation minimization".
Revised: We refer to this scenario as "activation minimization".

Original: We veriﬁed that activation minimization does not fully account for our results by examining the proportion of activations which elicit non-zero activations in the units in our models.
Revised: We veriﬁed that activation minimization does not fully account for our results by examining the proportion of activations which elicit non-zero activations in the units in our models.

Original: If our regularizer is indeed using activation minimization to generate high selectivity in individual units, then regularizing to increase class selectivity should cause most units to have non-zero activations for only a very small proportion of samples.
Revised: If our regularizer is indeed using activation minimization to generate high selectivity in individual units, then regularizing to increase class selectivity should cause most units to have non-zero activations for only a very small proportion of samples.

Original: In ResNet18 trained on
Revised: In ResNet18 trained on

Original: Tiny ImageNet, we found that sparse units, deﬁned as units that do not respond to at least half of the data samples, only constitute more than 10% of the total population at extreme positive regularization scales ( α ≥ 10 ; Figures A9a, ?? ) , well after large performance deﬁcits >4% emerge (Figure 3).
Revised: Tiny ImageNet, we found that sparse units, deﬁned as units that do not respond to at least half of the data samples, only constitute more than 10% of the total population at extreme positive regularization scales ( α ≥ 10 ; Figure A10a), well after large performance deﬁcits >4% emerge (Figure 3).

Original: In
Revised: In

Original: ResNet20 trained on CIFAR10, networks regularized to have higher class selectivity (i.e. positive α ) did indeed have more sparse units (Figures A9b, ?? ).
Revised: ResNet20 trained on CIFAR10, networks regularized to have higher class selectivity (i.e. positive α ) did indeed have more sparse units (Figure A10b).

Original: However, this effect does not explain away our ﬁndings: by α = 0 .
Revised: However, this effect does not explain away our ﬁndings: by α = 0 .

Original: 7 , the majority of units respond to over 80% of samples (i.e. they are not sparse), but test accuracy has already decreased by 5% (Figure A6).
Revised: 7 , the majority of units respond to over 80% of samples (i.e. they are not sparse), but test accuracy has already decreased by 5% (Figure A6).

Original: These results indicate that activation minimization does not explain class selectivity-related changes in test accuracy.
Revised: These results indicate that activation minimization does not explain class selectivity-related changes in test accuracy.

Original: [Figure]
Revised: [Figure]

Original: A. R ESULTS FOR VGG 
Revised: A. R ESULTS FOR VGG

Original: Modulating class selectivity in VGG16 yielded results qualitatively similar to those we observed in ResNet20.
Revised: Modulating class selectivity in VGG16 yielded results qualitatively similar to those we observed in ResNet20.

Original: The regularizer reliably decreases class selectivity for negative values of α (Figure
Revised: The regularizer reliably decreases class selectivity for negative values of α (Figure

Original: A10), and class selectivity can be drastically reduced with little impact on test accuracy (Figure A11.
Revised: A11), and class selectivity can be drastically reduced with little impact on test accuracy (Figure A12.

Original: Although test accuracy decreases signiﬁcantly at α = − 0 .
Revised: Although test accuracy decreases signiﬁcantly at α = − 0 .

Original: 1 ( p = 0 .
Revised: 1 ( p = 0 .

Original: 004 , Bonferroni-corrected t-test), the effect is small: it is possible to reduce mean class selectivity by a factor of 5 with only a 0.5% decrease in test accuracy, and by a factor of 10—to 0.03—with only a ∼ 1% drop in test accuracy.
Revised: 004 , Bonferroni-corrected t-test), the effect is small: it is possible to reduce mean class selectivity by a factor of 5 with only a 0.5% decrease in test accuracy, and by a factor of 10—to 0.03—with only a ∼ 1% drop in test accuracy.

Original: Regularizing to increase class selectivity also has similar effects in VGG16 and ResNet20.
Revised: Regularizing to increase class selectivity also has similar effects in VGG16 and ResNet20.

Original: Increasing α causes class selectivity to increase, and the effect becomes less consistent at large values of α (Figure A5).
Revised: Increasing α causes class selectivity to increase, and the effect becomes less consistent at large values of α (Figure A5).

Original: Although the class selectivity-induced collapse in test accuracy does not emerge quite as rapidly in VGG16 as it does in ResNet20, the decrease in test accuracy is still signiﬁcant at the smallest tested value of α ( α = 0 .
Revised: Although the class selectivity-induced collapse in test accuracy does not emerge quite as rapidly in VGG16 as it does in ResNet20, the decrease in test accuracy is still signiﬁcant at the smallest tested value of α ( α = 0 .

Original: 1 , p = 0 .
Revised: 1 , p = 0 .

Original: 02 , Bonferroni-corrected t-test), and the effects on test accuracy of regularizing to promote vs. discourage class selectivity become signiﬁcantly different at α = 0 .
Revised: 02 , Bonferroni-corrected t-test), and the effects on test accuracy of regularizing to promote vs. discourage class selectivity become signiﬁcantly different at α = 0 .

Original: 3 ( p = 10 − 4 , Wilcoxon rank-sum test; Figure A14).
Revised: 3 ( p = 10 − 4 , Wilcoxon rank-sum test; Figure A15).

Original: Our observations that class selectivity is neither necessary nor sufﬁcient for performance in both VGG16 and ResNet20 indicates that this is likely a general property of CNNs.
Revised: Our observations that class selectivity is neither necessary nor sufﬁcient for performance in both VGG16 and ResNet20 indicates that this is likely a general property of CNNs.

Original: It is worth noting that VGG16 exhibits greater class selectivity than ResNet20.
Revised: It is worth noting that VGG16 exhibits greater class selectivity than ResNet20.

Original: In the absence of regularization (i.e. α = 0 ), mean class selectivity in ResNet20 is 0.22, while in VGG16 it is 0.35, a 1.6x increase.
Revised: In the absence of regularization (i.e. α = 0 ), mean class selectivity in ResNet20 is 0.22, while in VGG16 it is 0.35, a 1.6x increase.

Original: This could explain why positive values of α seem to have a stronger effect on class selectivity in VGG16 relative to ResNet20 (compare Figure A5 and Figure A12; also see Figure
Revised: This could explain why positive values of α seem to have a stronger effect on class selectivity in VGG16 relative to ResNet20 (compare Figure A5 and Figure A13; also see Figure

Original: A26b).
Revised: A16b).

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Revised: i v i

Original: [Figure]
Revised: [Figure]

Original: l
Revised: I

Original: [Figure]
Revised: [Figure]

Original: ss

Original: [Figure]
Revised: [Figure]

Original: l e c t i v i

Original: t y I n

Original: [Figure]
Revised: [Figure]

Original: e x
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: C l a ss S e l e c t i v i t y I n d e x
Revised: A. A DDITIONAL MEASURES OF CLASS INFORMATION IN SINGLE UNITS

Original: [Figure]
Revised: In order to conﬁrm that the effect of the regularizer is not unique to our chosen class selectivity metric

Original: [Figure]
Revised: [Figure]

Original: A. D IFFERENT SELECTIVITY METRICS

Original: In order to conﬁrm that the effect of the regularizer is not unique to our chosen class selectivity metric we also examined the effect of our regularizer on  the "precision" metric for class selectivity (Zhou et al., 2015; 2018; Gale et al., 2019).
Revised: we also examined the effect of our regularizer on two different measures of class information in single units: the "precision" metric for class selectivity (Zhou et al., 2015; 2018; Gale et al., 2019), and mutual information (Cover, 1999) [Figure]

Original: The precision metric is calculated by ﬁnding the N images that most strongly activate a given unit, then ﬁnding the image class C i that constitutes the largest proportion of the N images.
Revised: The precision metric is calculated by ﬁnding the N images that most strongly activate a given unit, then ﬁnding the image class C i that constitutes the largest proportion of the N images.

Original: Precision is deﬁned as this proportion.
Revised: Precision is deﬁned as this proportion.

Original: For example, if N = 200 , and the "cats" class, with 74 samples, constitutes the largest proportion of those 200 activations for a unit, then the precision of the unit is 74200 = 0 .
Revised: For example, if N = 200 , and the "cats" class, with 74 samples, constitutes the largest proportion of those 200 activations for a unit, then the precision of the unit is 74200 = 0 .

Original: Note that for a given number of classes C , precision is bounded by [ 1 C , 1] , thus in our experiments the lower bound on precision is 0.1.
Revised: Note that for a given number of classes C , precision is bounded by [ 1 C , 1] , thus in our experiments the lower bound on precision is 0.1.

Original: Zhou et al.
Revised: Zhou et al.

Original: (2015) used N = 60 , while Gale et al.
Revised: (2015) used N = 60 , while Gale et al.

Original: (2019) used N = 100 .
Revised: (2019) used N = 100 .

Original: We chose to use the number of samples per class in the test set data and thus the largest possible sample size.
Revised: We chose to use the number of samples per class in the test set data and thus the largest possible sample size.

Original: This yielded N = 1000 for CIFAR10 and N = 50 for Tiny Imagenet.
Revised: This yielded N = 1000 for CIFAR10 and N = 50 for Tiny Imagenet.

Original: The class selectivity regularizer has similar effects on precision as it does on the class selectivity index.
Revised: The class selectivity regularizer has similar effects on precision as it does on the class selectivity index.

Original: Regularizing against class selectivity has a consistent effect on precision (Figure A15), while regularizing to promote class selectivity has a consistent effect in ResNet18 trained on Tiny ImageNet and for smaller values of α in ResNet20 trained on CIFAR10.
Revised: Regularizing against class selectivity has a consistent effect on precision (Figure A17), while regularizing to promote class selectivity has a consistent effect in ResNet18 trained on Tiny ImageNet and for smaller values of α in ResNet20 trained on CIFAR10.

Original: However, the relationship between precision and the class selectivity index becomes less consistent for larger positive values of α in
Revised: However, the relationship between precision and the class selectivity index becomes less consistent for larger positive values of α in

Original: ResNet20 trained on CIFAR10.
Revised: ResNet20 trained on CIFAR10.

Original: One explanation for this is that activation sparsity is a valid solution for maximizing the class selectivity index but not precision.
Revised: One explanation for this is that activation sparsity is a valid solution for maximizing the class selectivity index but not precision.

Original: For example, a unit that responded only to ten samples from the class "cat" and not at all to the remaining samples would have a class selectivity index of 1, but a precision value of 0.11.
Revised: For example, a unit that responded only to ten samples from the class "cat" and not at all to the remaining samples would have a class selectivity index of 1, but a precision value of 0.11.

Original: This seems likely given the increase in sparsity observed for very large positive values of α (see Appendix A.11).
Revised: This seems likely given the increase in sparsity observed for very large positive values of α (see Appendix A.11).

Revised: The effects of the class selectivity regularizer on mutual information are nearly identical to its effects on precision.

Revised: Regularizing for and against class selectivity has a consistent effect on mutual information in ResNet18 trained on Tiny ImageNet (Figure A18).

Revised: Regularizing to reduce class selectivity also has a consistent effect in ResNet20 trained on CIFAR10, but the relationship between mutual information and the class selectivity index becomes less consistent for larger positive values of α .

Revised: However, given that class selectivity and mutual information are very different quantities, and that Morcos et al.

Revised: (2018b) did not observe a clear relationship between mutual information, class selectivity, and individual unit importance (as measured by impact of ablation), we hesitate to make strong conclusions about the role of mutual information in our selectivity regularizer.

Original: While there are additional class selectivity metrics that we could have used to further assess the effect of our regularizer, many of them are based on relating the activity of a neuron to the accuracy of the network’s
Revised: While there are additional class selectivity metrics that we could have used to further assess the effect of our regularizer, many of them are based on relating the activity of a neuron to the accuracy of the network’s

Original: output(s)
Revised: output(s)

Original: (e.g. top class selectivity Gale et al.
Revised: (e.g. top class selectivity Gale et al.

Original: and
Revised: and

Original: class correlation Li et al.
Revised: class correlation Li et al.

Original: Zhou et al.
Revised: Zhou et al.

Original: (2018)), confounding classiﬁcation accuracy and class selectivity.
Revised: (2018)), confounding classiﬁcation accuracy and class selectivity.

Original: Accordingly, these metrics are unﬁt for use in experiments that examine the relationship between class selectivity and classiﬁcation accuracy, which is exactly what we do here. 
Revised: Accordingly, these metrics are unﬁt for use in experiments that examine the relationship between class selectivity and classiﬁcation accuracy, which is exactly what we do here. [Figure] [Figure] l

Original:  [Figure]
Revised: I [Figure]

Revised: f

Revised: [Figure]

Revised: r

Revised: [Figure]

Revised: t

Original:  A. R ESTRICTING CLASS SELECTIVITY REGULARIZATION TO THE FIRST THREE OR FINAL THREE LAYERS
Revised: i [Figure] A. R EGULARIZING TO CONTROL CLASS DISCRIMINABILITY

Revised: We further conﬁrmed that our results are not unique to our chosen class selectivity metric by examining a related, but distinct metric: class discriminability.

Revised: d (cid:48) is a measure of the discriminability between two distributions (Stanislaw and Todorov, 1999; Macmillan and Creelman, 2004), deﬁned as

Revised: [Equation]

Revised: The numerator of d (cid:48) is identical to the numerator of the selectivity index (Equation 1)—the sum of µ max , the largest class-conditional mean activation, and µ − max , the mean response to the remaining (i.e. non- µ max ) classes—but the denominator differs.

Revised: σ 2 denotes the variance across activations.

Revised: 1 (cid:15) is a small value to prevent division by zero (we used 10 − 7 ).

Revised: Because d (cid:48) is differentiable with respect to the model parameters, we can control the amount of class discriminability learned by individual units using the same approach as when regularizing to control class selectivity.

Revised: Instead of calculating the class selectivity index for each unit, we calculated d (cid:48) .

Revised: This leads to the following loss function:

Revised: [Equation]

Revised: This loss is identical to Equation 2, except the selectivity term, µ SI , is replaced with µ d (cid:48) :

Revised: [Equation]

Revised: As when computing µ SI (Equation 3), l is a convolutional layer, L is the number of layers, u is a unit (i.e. feature map), U is the number of units in a given layer, and u is a unit.

Revised: The procedure for regularizing d (cid:48) is otherwise identical to regularizing class selectivity (Approach 3.3).

Revised: Regularizing to control d (cid:48) has the intended effect in both ResNet18 trained on Tiny ImageNet (Figure A19a) and ResNet20 trained on CIFAR10 (Figure A19c); the amount of class discriminability learned by individual units varies as a function of the sign and scale of regularization ( α ).

Revised: d (cid:48) also correlates strongly with class selectivity across units (Spearman’s ρ = 0.83 for ResNet18 trained on Tiny ImageNet; ρ = 0.90 for ResNet20 trained on CIFAR10; p <

Original: To investigate the layer-speciﬁcity of the effects of class selectivity regularization, we also examined the effects of restricting class selectivity regularization to the ﬁrst three or last three layers of the networks.
Revised: 10 − 5 for both). We observe very similar effects on test accuracy as when manipulating class selectivity: increasing d (cid:48) has rapid negative effects on test accuracy, while decreasing d (cid:48) has more modest effects. Though we do not observe an improvement in test accuracy as we did when regularizing to decrease class selectivity in ResNet18 trained on Tiny ImageNet, the asymmetry in effect between increasing vs. decreasing d (cid:48) is nevertheless consistent, indicating that neither class selectivity nor discriminability are sufﬁcient nor strictly necessary for CNN performance.

Original:  Interestingly, we found that much of the effect of regularizing for or against selectivity on test accuracy was replicated even when the regularization was restricted to the ﬁrst or ﬁnal three layers.
Revised: [Figure] A. R ESTRICTING CLASS SELECTIVITY REGULARIZATION TO THE FIRST THREE OR FINAL THREE LAYERS To investigate the layer-speciﬁcity of the effects of class selectivity regularization, we also examined the effects of restricting class selectivity regularization to the ﬁrst three or last three layers of the networks. Interestingly, we found that much of the effect of regularizing for or against selectivity on test accuracy was replicated even when the regularization was restricted to the ﬁrst or ﬁnal three layers.

Original: For example, reducing class selectivity in the ﬁrst three layers either improves test accuracy—in
Revised: For example, reducing class selectivity in the ﬁrst three layers either improves test accuracy—in

Original: ResNet18 trained on Tiny ImageNet—or has little-to-no effect on test accuracy—in ResNet20 trained on CIFAR10 (Figures A16 and A17).
Revised: ResNet18 trained on Tiny ImageNet—or has little-to-no effect on test accuracy—in ResNet20 trained on CIFAR10 (Figures A20 and A21).

Original: Likewise, regularizing to increase class selectivity in the ﬁrst three layers had an immediate negative impact on test accuracy in both models (Figures A18 and
Revised: Likewise, regularizing to increase class selectivity in the ﬁrst three layers had an immediate negative impact on test accuracy in both models (Figures A22 and

Original: A19).
Revised: A23).

Original: Regularizing against class selectivity in the ﬁnal three layers (Figures A20 and A21) caused a modest increase in test accuracy over a narrow range of α in ResNet18 trained on Tiny ImageNet: less than half a percent gain at most (at α = − 0 . 2 ), and no longer present by α = − 0 .
Revised: Regularizing against class selectivity in the ﬁnal three layers (Figures A24 and A25) caused a modest increase in test accuracy over a narrow range of α in ResNet18 trained on Tiny ImageNet: less than half a percent gain at most (at α = − 0 . 2 ), and no longer present by α = − 0 .

Original: 4 (Figure A21b).
Revised: 4 (Figure A25b). In

Original: In ResNet20, regularizing against class selectivity in the ﬁnal three layers actually causes a decrease in test accuracy (Figures A17c and A17d).
Revised: ResNet20, regularizing against class selectivity in the ﬁnal three layers actually causes a decrease in test accuracy (Figures A21c and A21d).

Original: Given that the output layer of CNNs trained for image classiﬁcation are by deﬁnition class-selective, we thought that regularizing to increase class selectivity in the ﬁnal three layers could improve performance, but surprisingly it causes an immediate drop in test accuracy in both models (Figures A22 and A23).
Revised: Given that the logit (output) layer of CNNs trained for image classiﬁcation are by deﬁnition class-selective, we thought that regularizing to increase class selectivity in the ﬁnal three layers could improve performance, but surprisingly it causes an immediate drop in test accuracy in both models (Figures A26 and A27).

Original: Our observation that regularizing to decrease class selectivity provides greater beneﬁts (in the case of ResNet18) or less impairment (in the case of
Revised: Our observation that regularizing to decrease class selectivity provides greater beneﬁts (in the case of ResNet18) or less impairment (in the case of

Original: ResNet20) in the ﬁrst three layers compared to the ﬁnal three layers leads to the conclusion that class selectivity is less necessary (or more detrimental) in early layers compared to late layers.
Revised: ResNet20) in the ﬁrst three layers compared to the ﬁnal three layers leads to the conclusion that class selectivity is less necessary (or more detrimental) in early layers compared to late layers.

Original: [Figure]
Revised: [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] A. A N INABILITY TO CHANGE PREFERRED CLASSES DURING TRAINING DOES NOT EXPLAIN SELECTIVITY - INDUCED PERFORMANCE DEFICITS

Original: [Figure]
Revised: One potential limitation of our selectivity regularizer is that regularizing to increase class selectivity could discourage individual neurons from changing their preferred class during training; units would be locked in to the class they preferred at initialization. If changing preferred classes is necessary to improve test accuracy, then regularizing to increase class selectivity could impose a constraint on performance. We controlled for this possibility in two ways. First, we analyzed the statistics of preferred class changes that occur as a function of the class selectivity regularization scale.

Original: [Figure]
Revised: We examined the proportion of units that change classes at least once during training when α ≥ 0 . The mean proportion is one, for all examined regularization scales and models (Figure A28), indicating that regularizing to increase selectivity does not lock units into their initial preferred class. We also examined whether regularizing to increase selectivity affects the number of times a unit changes its preferred class during training. In ResNet18 trained on Tiny ImageNet, we found that the mean number of preferred class changes decreases as a function of α for α ≥ 0 .

Original: [Figure]
Revised: 3 (Figure A29a).

Original: [Figure]
Revised: However, we note that the number of preferred class changes is roughly equal for α = { 0 , 0 . 1 , 0 . 2 } , even though test accuracy for α = { 0 . 1 , 0 . 2 } is signiﬁcantly lower than for α = 0 , indicating that preferred class changes cannot fully account for decreased test accuracy. Furthermore, in ResNet20 trained on CIFAR10 there is no clear relationship between α and the number of preferred class changes (Figure A29b).

Original: [Figure]
Revised: These results indicate that the test accuracy impairment caused by regularizing to increase class selectivity cannot be fully explained by the regularizer preventing units from changing their preferred class during training. We also controlled for the possibility of preferred class "lock-in" by warming up selectivity regularization during training, which would allow units’ preferred classes to change during the early period of training, when learning-induced changes are most signiﬁcant (Achille et al., 2018; Gur-Ari et al., 2018; Frankle et al., 2020). We implemented selectivity regularization warmup in an analogous manner to learning rate warmup—by scaling α (see Equation 2) linearly over the interval [0,1] across the epochs ﬁrst ﬁve epochs of training. We observed qualitatively similar results as when we did not warm up selectivity regularization (Figure A30, Figure A31).

Original: A. A DDITIONAL RESULTS
Revised: Regularizing to reduce class selectivity either has minimal negative effects (ResNet20 trained on CIFAR10) or improves test accuracy (ResNet18 trained on Tiny ImageNet), while regularizing to increase class selectivity impairs test accuracy in all examined models (Figure A31). Interestingly, the severity of the test accuracy impairment is slightly reduced when using regularization warmup. This indicates that some of the test accuracy deﬁcit from regularizing to increase class selectivity is attributable to the regularizer forcing the network into suboptimal solutions early in training. Nevertheless, the test accuracy deﬁcit imparted by increased class selectivity remains even when the class selectivity regularizer is warmed up, further supporting the claim that that class selectivity is not sufﬁcient for network performance

Original: [Figure]
Revised: [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Table] [Figure]

Original: [Figure]
Revised: [Figure]

Original: l a
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: S e l e c t i v i
Revised: A. E FFECTS OF CLASS SELECTIVITY REGULARIZATION IN R ES N ET 50 TRAINED ON T INY I MAGE N ET

Original: t y
Revised: We also examined the effect regularizing to control class selectivity in ResNet50 trained on Tiny ImageNet. Our results were qualitatively similar to those observed in ResNet18: class selectivity in trained networks correlates with α (Figure A34a), and regularizing to decrease class selectivity can improve test accuracy, while regularizing to increase class selectivity impairs test accuracy (Figure A34b).

Original: [Figure]
Revised: [Figure]