Original: Much Easier Said Than Done: Falsifying the Causal Relevance of Linear Decoding MethodsAnonymous Author(s)
Revised: Much Easier Said Than Done: Falsifying the Causal Relevance of Linear Decoding Methods

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Linear classifier probes are frequently utilized to better understand how neuralnetworks function.
Revised: Linear classifier probes are frequently utilized to better understand how neural networks function.

Original: Researchers have approached the problem of determining unitimportance in neural networks by probing their learned, internal representations.
Revised: Researchers have approached the problem of determining unit importance in neural networks by probing their learned, internal representations.

Original: Linear classifier probes identify highly selective units as the most important fornetwork function.
Revised: Linear classifier probes identify highly selective units as the most important for network function.

Original: Whether or not a network actually relies on high selectivityunits can be tested by removing them from the network using ablation.
Revised: Whether or not a network actually relies on high selectivity units can be tested by removing them from the network using ablation.

Original: Surprisingly, when highly selective units are ablated they only produce small performancedeficits, and even then only in some cases.
Revised: Surprisingly, when highly selective units are ablated they only produce small performance deficits, and even then only in some cases.

Original: In spite of the absence of ablation effects for selective neurons, linear decoding methods can be effectively used tointerpret network function, leaving their effectiveness a mystery.
Revised: In spite of the absence of ablation effects for selective neurons, linear decoding methods can be effectively used to interpret network function, leaving their effectiveness a mystery.

Original: To falsify theexclusive role of selectivity in network function and resolve this contradiction, wesystematically ablate groups of units in subregions of activation space.
Revised: To falsify the exclusive role of selectivity in network function and resolve this contradiction, we systematically ablate groups of units in subregions of activation space.

Original: Here, wefind a weak relationship between neurons identified by probes and those identified by ablation.
Revised: Here, we find a weak relationship between neurons identified by probes and those identified by ablation.

Original: More specifically, we find that an interaction between selectivityand the average activity of the unit better predicts ablation performance deficitsfor groups of units in Alexnet, VGG16, MobileNetV2, and ResNet101.
Revised: More specifically, we find that an interaction between selectivity and the average activity of the unit better predicts ablation performance deficits for groups of units in AlexNet, VGG16, MobileNetV2, and ResNet101.

Original: Lineardecoders are likely somewhat effective because they overlap with those units thatare causally important for network function.
Revised: Linear decoders are likely somewhat effective because they overlap with those units that are causally important for network function.

Original: Interpretability methods could beimproved by focusing on causally important units.
Revised: Interpretability methods could be improved by focusing on causally important units.

Original:  Introduction
Revised:  Introduction

Original: Neural networks play critical roles in everything from driving to systems of justice, but trusting neural networks in such roles requires understanding how they function.
Revised: Neural networks play critical roles in everything from driving to systems of justice, but trusting neural networks in such roles requires understanding how they function.

Original: Widespread adoption of neuralnetworks has raised concerns about trust and bias [1].
Revised: Widespread adoption of neural networks has raised concerns about trust and bias [1].

Original: For example in medicine, the incorporation ofneural networks requires considering how uncertainty in how a decision was made could be arguedto undermine medical authority [2].
Revised: For example in medicine, the incorporation of neural networks requires considering how uncertainty in how a decision was made could be argued to undermine medical authority [2].

Original: The requirement for transparency is so widely recognized it hasbegun to be incorporated in regulation [3].
Revised: The requirement for transparency is so widely recognized it has begun to be incorporated in regulation [3].

Original: Improving our understanding of neural network functionis in both the interest of the public and AI development communities.
Revised: Improving our understanding of neural network function is in both the interest of the public and AI development communities.

Original: Unfortunately, most methodsused for interpretation have not been strongly linked to neural network function.
Revised: Unfortunately, most methods used for interpretation have not been strongly linked to neural network function.

Original: One approach to understanding neural network function is to examine how they represent information internally [4–8].
Revised: One approach to understanding neural network function is to examine how they represent information internally [4–8].

Original: Internal representations have been shown to be remarkably useful in interpreting neuralnetworks. Representations from networks trained on the same dataset show consistencyacross different initial randomizations [9, 10]. They are also flexible and generalize well enoughto be used as the basis for learning new tasks in stitched networks [11].
Revised: Internal representations have been shown to be remarkably useful in interpreting neural networks [9–11].

Original: More specifically, linearclassifier probes [12] are commonly used  for interpretability [5, 13–15] and to test the efficacy ofrepresentation learning algorithms [16–19].
Revised: Linear classifier probes [12] are commonly used as a tool for interpreting [5, 13–15] and testing internal representations [16–19].

Original: Although linear probes have utility for understanding network representations, studies removing(ablating) groups of neurons identified by linear probes have inconsistent outcomes.
Revised: Although linear probes have utility for understanding network representations, studies removing (ablating) groups of neurons identified by linear probes have inconsistent outcomes.

Original: The best wayto confirm the  importance of groups of neurons is to remove them and see if the network stopsperforming its function [20], hereafter called ablation.
Revised: The best way to confirm the causal importance of groups of neurons is to remove them and see if the network stops performing its function [20], hereafter called ablation.

Original: Performance is not consistently degradedwhen ablating the units most selective for the target image [21, 6, 22].
Revised: Performance is not consistently degraded when ablating the units most selective for the target class (i.e. units that activate more for the target class than other images) [21, 6, 22].

Original: As an example, Donnellyand Roegiest [23] ablate a neuron highly selective for sentiment and observe negligible deficits inperformance.
Revised: As an example, Donnelly and Roegiest [23] ablate a neuron highly selective for sentiment and observe negligible deficits in performance.

Original: This argument can be extended to both training against class selectivity [24] and for theeffect of class selectivity on overall performance [25, 22, 6].
Revised: This argument can be extended to both training against class selectivity [24] and for the effect of class selectivity on overall performance [25, 22, 6].

Original: However, there are examples in whichresearchers observed a weak correlation between class selectivity and ablation deficits [22, 26].
Revised: However, there are examples in which researchers observed a weak correlation between class selectivity and ablation deficits [22, 26].

Original: Thesignificant issue remains that it is difficult to predict the effects caused by ablating the selective unitsidentified by linear probes.
Revised: The significant issue remains that it is difficult to predict the effects caused by ablating the selective units identified by linear probes.

Original: Most ablation studies of performance remove a single unit at a time or are based on single measuresof importance (activation strength, selectivity, decoding weight, information-theoretic measures,etc.)
Revised: Most ablation studies of performance remove a single unit at a time or are based on single measures of importance (activation strength, selectivity, decoding weight, information-theoretic measures, etc.)

Original: [6, 22, 27, 21, 28–30], preventing study of the aggregate effects of groups of units.
Revised: [6, 22, 27, 21, 28–30], preventing study of the aggregate effects of groups of units.

Original: Single unitablation avoids the combinatorially difficult problem of choosing which units to group, but resultsin small or no changes in performance.
Revised: Single unit ablation avoids the combinatorially difficult problem of choosing which units to group, but results in small or no changes in performance.

Original: Larger performance deficits can be achieved by targeting aggregate groups of units based on a single metric (e.g., selectivity).
Revised: Larger performance deficits can be achieved by targeting aggregate groups of units based on a single metric (e.g., selectivity).

Original: However, utilizing single units orsingle dimensions precludes a systematic exploration of how function and representation vary acrossactivation space.
Revised: However, utilizing single units or single dimensions precludes a systematic exploration of how function and representation vary across activation space.

Original: An approach that accounts for these considerations is warranted.
Revised: An approach that accounts for these considerations is warranted.

Original: In this work, wefocus on four heavily studied convolutional neural networks: AlexNet, VGG16, MobileNetV2, and
Revised: In this work, we focus on four heavily studied convolutional neural networks: AlexNet, VGG16, MobileNetV2, and

Original: ResNet101.
Revised: ResNet101.

Original: We efficiently group units by their positions in activation space along two dimensions:average class selectivity and average activation magnitude, to highlight the regions occupied byfunction and representation.
Revised: We efficiently group units by their positions in activation space along two dimensions: average class selectivity and average activation magnitude, to highlight the regions occupied by function and representation.

Original: This paper documents these contributions:
Revised: This paper documents these contributions:

Original: • We demonstrate a reliable and fast method of grouping units in activation space.
Revised: • We demonstrate a reliable and fast method of grouping units in activation space.

Original: This method improves on past clustering work [31] by using a simpler algorithm which clustersunits into groups in activation space.•
Revised: This method improves on past clustering work [31] by using a simpler algorithm which clusters units into groups in activation space.

Original:  We show that many groups of units in activation space produce high linear classificationprobe accuracies with higher accuracies for groups of units that are selective for the targetand highly selective for not-the-target.• We identify a region of activation space in which ablation selectively reduces performance.
Revised: • We show that many groups of units in activation space produce high linear classification probe accuracies with higher accuracies for groups of units that are selective for the target and highly selective for not-the-target. • We identify a region of activation space in which ablation selectively reduces performance.

Original: This subregion of activation space partially overlaps with regions of activation space thatproduce strong linear classification.
Revised: This region partially overlaps with parts of activation space that produce strong linear classification.

Original: This partial overlap falsifies the specificity of linearprobes for isolating function.•
Revised: This partial overlap falsifies the specificity of linear probes for isolating function.

Original:  We show that this region is consistent across four widely studied convolutional neural networks.
Revised: • We show that this region is consistent across four popular convolutional neural networks.

Original:  Methods
Revised:  Methods

Original: For our experiments, we investigated AlexNet [32], VGG16 [33], MobileNetV2 [34], and
Revised: For our experiments, we investigated AlexNet [32], VGG16 [33], MobileNetV2 [34], and

Original: ResNet101 [35] all pre-trained on ImageNet [36].
Revised: ResNet101 [35] all pre-trained on ImageNet [36] 2 .

Original: We modified these pre-trained networks to include lambda masking layers which allowed us to selectively ablate individual units or unit groupsin the network.
Revised: We modified these pre-trained networks to include lambda masking layers which allowed us to selectively ablate individual units or unit groups in the network.

Original: Ablating a unit sets its outgoing activation to zero, preventing information flowfrom that unit to the next layer.
Revised: Ablating a unit sets its outgoing activation to zero, preventing information flow from that unit to the next layer.

Original: Our process of relating linear probe accuracies and causal networkperformance for unit groups consists of two steps: 1) Selection of unit groups, 2) Measuring linearprobe accuracies and causal performance through ablation, outlined in the following sections.
Revised: Our process of relating linear probe accuracies and causal network performance for unit groups consists of two steps: 1) Selection of unit groups, 2) Measuring linear probe accuracies and causal performance through ablation, outlined in the following sections.

Original: 2.1 Selection of unit groups
Revised: 2.1 Selection of unit groups

Original: This section outlines a cheap algorithm for grouping important units in activation space, this allowsfor all experiments to be run on a laptop, minimizing environmental impact.
Revised: This section outlines a cheap algorithm for grouping important units in activation space, this allows for all experiments to be run on a laptop, minimizing environmental impact.

Original: This algorithm groupsunits on a reduced activation space constructed using activation characteristics that were implicatedin performance based on previous ablation studies.
Revised: This algorithm groups units on a reduced activation space constructed using activation characteristics that were implicated in performance based on previous ablation studies.

Original: To find groups of units important for classspecific performance in AlexNet, we measured activations for each unit in the network for a set of2500 images representing 50 classes (e.g., junco bird, brain coral, etc.)
Revised: To find groups of units important for classspecific performance in AlexNet, we measured activations for each unit in the network for a set of 2500 images representing 50 classes (e.g., junco bird, brain coral, etc.)

Original: drawn from the ImageNetvalidation set.
Revised: drawn from the ImageNet validation set.

Original: We extended our findings to three other networks using a subset of 8 image classes.
Revised: We extended our findings to three other networks using a subset of 8 image classes.

Original: We averaged the activity for each unit for our class of interest and across a random subset of other
Revised: We averaged the activity for each unit for our class of interest and across a random subset of other

Original: [Figure]
Revised: [Figure]

Original: classes separately for each layer of thenetwork. We then plotted neurons in this two-dimensionalactivation space with average activation forour class ofinterest along one dimension and averageactivation forother classes across the other dimension.
Revised: the for of for

Original: We then rotated this space to form two new dimensions: class selectivity and activation magnitude(Figure 1a).
Revised: We then rotated this space to form two new dimensions: class selectivity and activation magnitude (Figure 1a).

Original: In this space, class selectivity measures the difference in the average activation ofeach unit between our class of interest and other classes.
Revised: In this space, class selectivity measures the difference in the average activation of each unit between our class of interest and other classes.

Original: Activation magnitude, on the other handmeasures each unit’s average activation in response to all images.
Revised: Activation magnitude, on the other hand measures each unit’s average activation in response to all images.

Original: Over this space of neurons weconstructed a 4x4 grid with rectangular cells.
Revised: Over this space we constructed a 4x4 grid with rectangular cells.

Original: The cells vary in area, but each contains an equalnumber of units.
Revised: The cells vary in area, but each contains an equal number of units.

Revised: These cells were constructed by first partitioning the space into four horizontal strips at intervals along the selectivity dimension, each with the same number of neurons.

Revised: These strips were then individually partitioned into four quadrants each along the magnitude dimension so that each of the sixteen resulting cells contained the same number of neurons.

Original: Controlling for the number of units in each cell allowed for easy comparisonsbetween cells.
Revised: Controlling for the number of units in each cell allowed for easy comparisons between cells.

Original: Figure 1b shows the constructed grid overlaid on the activation space.
Revised: Figure 1b shows the constructed grid overlaid on the activation space.

Original: Cells wereselected one at a time and all the units from that cell were used to measure linear probing accuracyand causal performance deficits, as outlined in the next two subsections.
Revised: Cells were selected one at a time and all the units from that cell were used to measure linear probing accuracy and causal performance deficits, as outlined in the next subsection.

Original: 2.2 Measuring linear probe accuracies and causal performance deficits
Revised: 2.2 Measuring linear probe accuracies and causal performance deficits

Original: To measure linear probe decoding accuracies, we followed the methodology presented in the originalpaper [12], fitting linear probes to each cell’s unit group to predict class identities.
Revised: To measure linear probe decoding accuracies, we followed the methodology presented in the original paper [12], fitting linear probes to each cell’s unit group to predict class identities.

Original: For comparison,the importance of each cell’s unit group for class-specific task performance was measured usingablation.
Revised: For comparison, the importance of each cell’s unit group for class-specific task performance was measured using ablation.

Original: First, before ablation, the network’s baseline performance was recorded.
Revised: First, before ablation, the network’s baseline performance was recorded.

Original: During inferenceon each image, classes were ranked according to the magnitude of their corresponding softmax outputs.
Revised: During inference on each image, classes were ranked according to the magnitude of their corresponding softmax outputs.

Original: For instance, a rank of three for a particular image denotes that the network predicted that thecorrect class was the third most likely class given the image.
Revised: For instance, a rank of three for a particular image denotes that the network predicted that the correct class was the third most likely class given the image.

Original: After recording baseline ranks, everyunit in a given cell was ablated using the lambda mask mentioned previously.
Revised: After recording baseline ranks, every unit in a given cell was ablated using the lambda mask mentioned previously (see Section 2).

Original: A performance impactscore was calculated for each image by measuring how many ranks the correct class fell compared tobaseline.
Revised: A performance impact score was calculated for each image by measuring how many ranks the correct class fell compared to baseline, which we refer to as rank deficits.

Original: Selective ablation rank deficit scores were calculated by averaging the performance impactscores across the class of interest.
Revised: Selective ablation rank deficit scores were calculated by averaging the performance impact scores across the class of interest.

Revised: A network’s top-K classification accuracy depends exclusively on the rank of the correct class, making the rank deficit score a direct measure of the impact of ablation on network performance.

Original:  Results
Revised:  Results

Original: We show grids colored by linear probe accuracies for each image class in our set (rows in Figure2a) and for each layer in AlexNet (columns in Figure 2a).
Revised: We show grids colored by linear probe accuracies for each image class in our set (rows in Figure 2a) and for each layer in AlexNet (columns in Figure 2a).

Original: Correspondingly, we show performancedeficit grids colored by rank deficit for each image class and each layer (Figure 2b).
Revised: Correspondingly, we show performance deficit grids colored by rank deficit for each image class and each layer (Figure 2b).

Original: The last rowdepicts the average grid scores for each layer averaged across all 50 classes.
Revised: The last row depicts the average grid scores for each layer averaged across all 50 classes.

Original: The final column depictsthe average grid scores for each class averaged across all seven layers of AlexNet.
Revised: The final column depicts the average grid scores for each class averaged across all seven layers of AlexNet.

Original: In all cases, thecolorbar displays the range in scores for each class.
Revised: In all cases, the colorbar displays the range in scores for each class.

Original: In addition to AlexNet, we show results forthree other networks: VGG16, MobileNetV2, and ResNet101.
Revised: In addition to AlexNet, we show results for three other networks: VGG16, MobileNetV2, and ResNet101.

Original: Figure 3 depicts the ablation andlinear probe grid scores averaged across all layers for each network we tested.
Revised: Figure 3 depicts the ablation and linear probe grid scores averaged across all layers for each network we tested.

Original: 3.1 Linear probe accuracy
Revised: 3.1 Linear probe accuracy

Original: Linear probe accuracy is somewhat localized in activation space (Figure 2a).
Revised: Linear probe accuracy is somewhat localized in activation space (Figure 2a).

Original: In general, across alleight image classes, unit groups with maximum accuracy scores are located in the most selectiveregions of activation space.
Revised: In general, across all eight image classes, unit groups with maximum accuracy scores are located in the most selective regions of activation space.

Original: High linear-probe scores come from cells that are both highly selectivefor the target class and highly selective on average for all other classes over the target class.
Revised: High linear-probe scores come from cells that are both highly selective for the target class and highly selective on average for all other classes over the target class.

Original: Importantly, deleting these neurons has no effect on the networks ability to identify the class, as shown in
Revised: Importantly, deleting these neurons has no effect on the networks ability to identify the class, as shown in

Original: Figure 2b.
Revised: Figure 2b.

Original: Overall, like ablation impacts, decoding accuracies tend to increase with depth.
Revised: Overall, like ablation impacts, decoding accuracies tend to increase with depth.

Original: 3.2 Rank deficit score
Revised: 3.2 Rank deficit score

Original: Rank deficits are localized in activation space.
Revised: Rank deficits are localized in activation space.

Original: In general, across all image classes, unit groups witha maximum rank deficit are located in the most selective and most active regions of activation space(Figure 2b).
Revised: In general, across all image classes, unit groups with a maximum rank deficit are located in the most selective and most active regions of activation space (Figure 2b).

Original: In general the most selective and active unit group produces rank deficits far greater thanneighboring unit groups.
Revised: In general the most selective and active unit group produces rank deficits far greater than neighboring unit groups.

Original: This pattern shows a combination of selectivity and magnitude contributeto unit importance.
Revised: This pattern shows a combination of selectivity and magnitude contribute to unit importance.

Original: This relationship appears discontinuous rather than graded.
Revised: This relationship appears discontinuous rather than graded.

Original: Even though highimpact cells consistently emerge in the most selective and active regions of activation space, the layerwith the highest impact cell varies across classes.
Revised: Even though high impact cells consistently emerge in the most selective and active regions of activation space, the layer with the highest impact cell varies across classes.

Original: For instance, the junco bird class has the highestimpact cell in the penultimate layer of the network (Dense 2), while the bookcase class shows thehighest rank impacts in Conv 3.
Revised: For instance, the junco bird class has the highest impact cell in the penultimate layer of the network (Dense 2), while the bookcase class shows the highest rank impacts in Conv 3.

Original: Additionally, rank deficit scores vary considerably between layersand between classes.
Revised: Additionally, rank deficit scores vary considerably between layers and between classes.

Original: Ablating the most selective and most active cell in Conv 3 produces a rankdeficit score of nearly 300 for the bookcase class.
Revised: Ablating the most selective and most active cell in Conv 3 produces a rank deficit score of nearly 300 for the bookcase class.

Original: But, junco class rank deficits never exceed 60.3.3 Only a subset of selective unit groups contribute to causal function
Revised: But, junco class rank deficits never exceed 60. 3.3 Only a subset of selective unit groups contribute to causal function

Original: We found the units that are the most active while also being selective are the strongest drivers ofperformance across all four networks.
Revised: We found the units that are the most active while also being selective are the strongest drivers of performance across all four networks.

Original: This pattern is most prominent in AlexNet and VGG16.
Revised: This pattern is most prominent in AlexNet and VGG16.

Original: In
Revised: In

Original: ResNet101 and MobileNetV2, the most selective and active cell continues to serve as the strongestdriver of performance, but unlike VGG16 and ResNet101, other cells become increasingly important.
Revised: ResNet101 and MobileNetV2, the most selective and active cell continues to serve as the strongest driver of performance, but unlike VGG16 and AlexNet, other cells become increasingly important.

Original: Particularly, high magnitude cells are consistently important in MobileNetV2.
Revised: Particularly, high magnitude cells are consistently important in MobileNetV2.

Original: MobileNetV2and VGG16 both utilize batch normalization layers which have been shown to spread out classspecific information across units [22].
Revised: MobileNetV2 and VGG16 both utilize batch normalization layers which have been shown to spread out class-specific information across units [22].

Original: Further investigation is needed to establish explicit linksbetween regularizers, like batch normalization, representation, and performance.
Revised: Further investigation is needed to establish explicit links between regularizers, like batch normalization, representation, and performance.

Original:  Discussion
Revised:  Discussion

Original: In this work, we used a computationally inexpensive method for selecting important groups of unitsin neural networks.
Revised: In this work, we used a computationally inexpensive method for selecting important groups of units in neural networks.

Original: Ablating these groups revealed a consistent pattern whereby the most importantunit groups emerge in highly selective and highly active areas of activation space.
Revised: Ablating these groups revealed a consistent pattern whereby the most important unit groups emerge in highly selective and highly active areas of activation space.

Original: Previous ablationstudies may have failed to identify this pattern for the following reasons.
Revised: Previous ablation studies may have failed to identify this pattern for the following reasons.

Original: Morcos et al.
Revised: Morcos et al.

Original: [6] and
Revised: [6] and

Original: Zhou et al.
Revised: Zhou et al.

Original: [22] ablate single units across the whole network based on class selectivity alone, butnot activation magnitude.
Revised: [22] ablate single units across the whole network based on class selectivity alone, but not activation magnitude.

Original: In Figure 2b, selectivity alone would have a weak effect on performancebecause it averages both strong and weak effects, one potential explanation for the weak relationshipbetween selectivity and performance from [22].
Revised: In Figure 2b, selectivity alone would have a weak effect on performance because it averages both strong and weak effects, one potential explanation for the weak relationship between selectivity and performance from [22].

Original: In contrast, Meyes et al.
Revised: In contrast, Meyes et al.

Original: [21] visualize both classselectivity and class-specific magnitudes, but not per layer.
Revised: [21] visualize both class selectivity and class-specific magnitudes, but not per layer.

Original: Since units from early layers are typicallymore active and selective, clusters in activation space containing units from all layers lump togetherhighly selective and active units from later layers with less selective and low activation units fromearly layers.
Revised: Since units from early layers are typically more active and selective, clusters in activation space containing units from all layers lump together highly selective and active units from later layers with less selective and active units from early layers.

Original: In this way, network-wide groups are too heterogeneous to reveal patterns that existprimarily at theper layer level.
Revised: In this way, network-wide groups are too heterogeneous to reveal patterns that exist primarily at the layer level.

Original: This insight is leveraged by Liu et al.
Revised: This insight is leveraged by Liu et al.

Original: [26] to find highly impactfulneurons usingclass selectivity  for small fully-connected networks.
Revised: [26] to find highly impactful neurons using selectivity (and other promising measures) for small MLPs.

Original: Here, we show that only themost selective and active groups of units contribute to causalclass-specific performance in fourlarge CNNs.
Revised: Here, we show that only the most selective and active groups of units contribute to causal performance in four large CNNs.

Original: Rather than ablating units, most neural network interpretability studies rely on interpreting functionfrom internal representations.
Revised: Rather than ablating units, most neural network interpretability studies rely on interpreting function from internal representations.

Original: Alain et al.
Revised: Alain et al.

Original: [12] proposed using linear classifier probes to extract thesemantic concepts learned by specific units.
Revised: [12] proposed using linear classifier probes to extract the semantic concepts learned by specific units.

Original: We use linearclassifier probes to show that the mostselective units in activation space do indeed contain the most decodable semantic information.
Revised: We use linear probes to show that the most selective [Figure] [Figure] units in activation space do indeed contain the most decodable semantic information.

Original: However, ablating these highly selective units in activation space does not necessarily produce the largestperformance deficits.
Revised: However, ablating these highly selective units in activation space does not necessarily produce the largest performance deficits.

Original: In fact, only a relatively small subset of selective units produce performance[Figure] [Figure] deficits.
Revised: In fact, only a relatively small subset of selective units produce performance deficits.

Original: Across four popular CNNs, high ablation impact neuron groups tend to have both highselectivity and magnitude.
Revised: Across four popular CNNs, high ablation impact neuron groups tend to have both high selectivity and magnitude.

Original: This finding suggests that interpretability methods which rely on linearclassifier probes may be seeking out directions in activation space that onlyhave minimal effects onnetwork performance when removed.
Revised: This finding suggests that interpretability methods which rely on linear probes may be seeking out directions in activation space with only minimal effects on network performance.

Original: We improve on past work by identifying a region of activationspace crucial for connecting interpretation to function.
Revised: We improve on past work by identifying a region of activation space crucial for interpreting function.

Original:  Limitations
Revised:  Limitations

Original: Limitations in this work stem from  assumptions made in our methods for defining selectivity andgenerating groups of units.
Revised: Limitations in this work stem from three assumptions made in our methods.

Original: First, we only investigate class-specific selectivity, while other studieshave emphasized that class-specific selectivity can play a different role in network function fromlow-level feature selectivity [37].
Revised: First, we only investigate class-specific selectivity, while other studies have emphasized that class-specific selectivity can play a different role in network function from low-level feature selectivity [37].

Revised: Additionally, other importance measures (i.e. mutual information [26]) are not studied here, but should be the focus of future study.

Original: Second, our unit grouping method relies on handpicked hyperparamters that determine the number of cells and produces cells that contain different numbers ofunits in each layer, making it difficult to compare scores across layers.
Revised: Second, our unit grouping method relies on handpicked hyperparamters that determine the number of cells and produces cells that contain different numbers of units in each layer, making it difficult to compare scores across layers.

Revised: Lastly, we study the performance changes at the output of the network induced by ablation, while neglecting other causal changes in the rest of the network.

Original:  Conclusion
Revised:  Conclusion

Original: This work contributes directly to the interpretability of neural networks.
Revised: This work contributes directly to the interpretability of neural networks.

Original: We demonstrate a reliableand fast method of grouping units in activation space.
Revised: We demonstrate a reliable and fast method of grouping units in activation space.

Original: We identify a region of activation space inwhich ablation selectively reduces causal performance. This region is consistent across four widelystudied convolutional neural networks.
Revised: We identify a region of activation space in which ablation selectively reduces causal performance, consistent across four popular CNNs.

Original: Surprisingly,across the four popular CNNs, we found thatimportant ablation impact units only occupy a relatively small subset of the activation space coveredby selectively important units identified by linear probes.
Revised: Surprisingly, we found that important ablation impact units only occupy a relatively small subset of the activation space covered by selectively important units identified by linear probes.

Original: As a result, interpretability tools that relyon decoding semantic information likely only weakly relate to performance.
Revised: As a result, interpretability tools that rely on decoding semantic information likely only weakly relate to performance.

Original: Many opportunities exist to augment this research with further investigation into additional networks and tasks to producegeneral theories that govern the relationship betweenaspects of learned representations and functionin neural networks. Checklist For all authors...
Revised: Future investigations would benefit from extending this research to more networks and tasks to uncover theoretical relationships between neural network representation and function.

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes] (b) Did you describe the limitations of your work? [Yes] See Section
Revised: Acknowledgments and Disclosure of Funding

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See potentialbenefits in the introduction and discussion sections.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?[Yes]

Original: Yes, we have read them.

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results? [N/A](b) Did you include complete proofs of all theoretical results? [N/A]3.
Revised: This work was supported by the University of Colorado Boulder Computer Science Department.

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimentalresults (either in the supplemental material or as a URL)?

Original: [Yes] See supplementary materials(b)

Original: Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Section 2(c)

Original: Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [N/A]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?

Original: [Yes] All analyses were conducted on a laptop.

Original: See methodssection.4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a)

Original: If your work uses existing assets, did you cite the creators?

Original: [Yes] We used AlexNet pre-trainedon ImageNet and cited both in Section 2(b)

Original: Did you mention the license of the assets?

Original: [No] We discovered that ImageNet uses a Custom

Original: (non-commercial) license, but were unsure of how to cite this in the paper.

Original: Additionally, we

Original: were unable to find a license for AlexNet.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] Linked

Original: Github in supplementary material.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re us-

Original: ing/curating?

Original: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: infor-

Original: mation or offensive content?

Original: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]

Original: b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)

Original: approvals, if applicable?

Original: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on

Original: participant compensation?

Original: [N/A]