Original: N ATURAL L ANGUAGE D ESCRIPTIONS OF D EEP  F EATURES
Revised: N ATURAL L ANGUAGE D ESCRIPTIONS OF D EEP V ISUAL F EATURES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Some neurons in deep networks specialize in recognizing highly speciﬁc perceptual, structural, or semantic features of inputs.
Revised: Some neurons in deep networks specialize in recognizing highly speciﬁc perceptual, structural, or semantic features of inputs.

Original: In computer vision, techniques exist for identifying neurons that respond to individual concept categories like colors, textures, and object classes.
Revised: In computer vision, techniques exist for identifying neurons that respond to individual concept categories like colors, textures, and object classes.

Original: But these techniques are limited in scope, labeling only a small subset of neurons and behaviors in any network.
Revised: But these techniques are limited in scope, labeling only a small subset of neurons and behaviors in any network.

Original: Is a richer characterization of neuron-level computation possible?
Revised: Is a richer characterization of neuron-level computation possible?

Original: We introduce a procedure (called MILAN , for m utual- i nformation-guided l inguistic a nnotation of n eurons) that automatically labels neurons with open-ended, compositional, natural language descriptions.
Revised: We introduce a procedure (called MILAN , for m utual- i nformation-guided l inguistic a nnotation of n eurons) that automatically labels neurons with open-ended, compositional, natural language descriptions.

Original: Given a neuron, MILAN generates a description by searching for a natural language string that maximizes pointwise mutual information with the image regions in which the neuron is active.
Revised: Given a neuron, MILAN generates a description by searching for a natural language string that maximizes pointwise mutual information with the image regions in which the neuron is active.

Original: MILAN produces ﬁne-grained descriptions that capture categorical, relational, and logical structure in learned features.
Revised: MILAN produces ﬁne-grained descriptions that capture categorical, relational, and logical structure in learned features.

Original: These descriptions obtain high agreement with human-generated feature descriptions across a diverse set of model architectures and tasks, and can aid in understanding and controlling learned models.
Revised: These descriptions obtain high agreement with human-generated feature descriptions across a diverse set of model architectures and tasks, and can aid in understanding and controlling learned models.

Original: We highlight three applications of natural language neuron descriptions.
Revised: We highlight three applications of natural language neuron descriptions.

Original: First, we use MILAN for analysis , characterizing the distribution and importance of neurons selective for attribute, category, and relational information in vision models.
Revised: First, we use MILAN for analysis , characterizing the distribution and importance of neurons selective for attribute, category, and relational information in vision models.

Original: Second, we use MILAN for auditing , surfacing neurons sensitive to protected categories like race and gender in models trained on datasets intended to obscure these features.
Revised: Second, we use MILAN for auditing , surfacing neurons sensitive to protected categories like race and gender in models trained on datasets intended to obscure these features.

Original: Finally, we use MILAN for editing , improving robustness in an image classiﬁer by deleting neurons sensitive to text features spuriously correlated with class labels.
Revised: Finally, we use MILAN for editing , improving robustness in an image classiﬁer by deleting neurons sensitive to text features spuriously correlated with class labels.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A surprising amount can be learned about the behavior of a deep network by understanding the individual neurons that make it up.
Revised: A surprising amount can be learned about the behavior of a deep network by understanding the individual neurons that make it up.

Original: Previous studies aimed at visualizing or automatically categorizing neurons have identiﬁed a range of interpretable functions across models and application domains: low-level convolutional units in image classiﬁers implement color detectors and Gabor ﬁlters (Erhan et al., 2009), while some later units activate for speciﬁc parts and object categories (Zeiler & Fergus, 2014;
Revised: Previous studies aimed at visualizing or automatically categorizing neurons have identiﬁed a range of interpretable functions across models and application domains: low-level convolutional units in image classiﬁers implement color detectors and Gabor ﬁlters (Erhan et al., 2009), while some later units activate for speciﬁc parts and object categories (Zeiler & Fergus, 2014;

Original: Bau et al., 2017).
Revised: Bau et al., 2017).

Original: Outside of computer vision, single neurons have been found to encode sentiment in language data (Radford et al., 2017) and biological function in computational chemistry (Preuer et al., 2019).
Revised: Outside of computer vision, single neurons have been found to encode sentiment in language data (Radford et al., 2017) and biological function in computational chemistry (Preuer et al., 2019).

Original: Given a new model trained to perform a new task, can we automatically ﬁnd, describe, and catalog these behaviors?
Revised: Given a new model trained to perform a new task, can we automatically ﬁnd, describe, and catalog these behaviors?

Original: Techniques for characterizing the behavior of individual neurons are still quite limited.
Revised: Techniques for characterizing the behavior of individual neurons are still quite limited.

Original: Approaches based on visualization (Zeiler & Fergus, 2014; Girshick et al., 2014; Karpathy et al., 2015; Mahendran & Vedaldi, 2015; Olah et al., 2017) leave much of the work of interpretation up to human users, and cannot be used for large-scale analysis.
Revised: Approaches based on visualization (Zeiler & Fergus, 2014; Girshick et al., 2014; Karpathy et al., 2015; Mahendran & Vedaldi, 2015; Olah et al., 2017) leave much of the work of interpretation up to human users, and cannot be used for large-scale analysis.

Original: Existing automated labeling techniques (Bau et al., 2017; 2019; Mu & Andreas, 2020) require researchers to pre-deﬁne a ﬁxed space of candidate neuron labels; they label only a subset of neurons in a given network and cannot be used to surface novel or unexpected behaviors.
Revised: Existing automated labeling techniques (Bau et al., 2017; 2019; Mu & Andreas, 2020) require researchers to pre-deﬁne a ﬁxed space of candidate neuron labels; they label only a subset of neurons in a given network and cannot be used to surface novel or unexpected behaviors.

Original: This paper develops an alternative paradigm for labeling neurons with expressive, compositional, and open-ended annotations in the form of natural language descriptions .
Revised: This paper develops an alternative paradigm for labeling neurons with expressive, compositional, and open-ended annotations in the form of natural language descriptions .

Original: We focus on the visual domain: building on past work on information-theoretic approaches to model interpretability, we formulate neuron labeling as a problem of ﬁnding informative descriptions of a neuron’s pattern
Revised: We focus on the visual domain: building on past work on information-theoretic approaches to model interpretability, we formulate neuron labeling as a problem of ﬁnding informative descriptions of a neuron’s pattern

Original: [Figure]
Revised: [Figure]

Original: of activation on input images.
Revised: of activation on input images.

Original: We describe a procedure (called MILAN , for m utual- i nformationguided l inguistic a nnotation of n eurons) that labels individual neurons with ﬁne-grained natural language descriptions by searching for descriptions that maximize pointwise mutual information with the image regions in which neurons are active.
Revised: We describe a procedure (called MILAN , for m utual- i nformationguided l inguistic a nnotation of n eurons) that labels individual neurons with ﬁne-grained natural language descriptions by searching for descriptions that maximize pointwise mutual information with the image regions in which neurons are active.

Original: To do so, we ﬁrst collect a new dataset of ﬁne-grained image annotations ( MILANNOTATIONS , Figure 1c), then use these to construct learned approximations to the distributions over image regions (Figure 1b) and descriptions.
Revised: To do so, we ﬁrst collect a new dataset of ﬁne-grained image annotations ( MILANNOTATIONS , Figure 1c), then use these to construct learned approximations to the distributions over image regions (Figure 1b) and descriptions.

Original: In some cases,
Revised: In some cases,

Original: MILAN surfaces neuron descriptions that more speciﬁc than the underlying training data (Figure 1d).
Revised: MILAN surfaces neuron descriptions that more speciﬁc than the underlying training data (Figure 1d).

Original: MILAN is largely model-agnostic and can surface descriptions for different classes of neurons, ranging from convolutional units in CNNs to fully connected units in vision transformers, even when the target network is trained on data that differs systematically from MILANNOTATIONS ’ images.
Revised: MILAN is largely model-agnostic and can surface descriptions for different classes of neurons, ranging from convolutional units in CNNs to fully connected units in vision transformers, even when the target network is trained on data that differs systematically from MILANNOTATIONS ’ images.

Original: These descriptions can in turn serve a diverse set of practical goals in model interpretability and dataset design.
Revised: These descriptions can in turn serve a diverse set of practical goals in model interpretability and dataset design.

Original: Our experiments highlight three: using MILAN -generated descriptions to (1) analyze the role and importance of different neuron classes in convolutional image classiﬁers, (2) audit models for sensitive demographic data by comparing comparing their features when trained on anonymized (blurred) and non-anonymized datasets, and (3) identify and mitigate the effects of spurious correlations with text features, improving classiﬁer performance on adversarially distributed test sets.
Revised: Our experiments highlight three: using MILAN -generated descriptions to (1) analyze the role and importance of different neuron classes in convolutional image classiﬁers, (2) audit models for sensitive demographic data by comparing comparing their features when trained on anonymized (blurred) and non-anonymized datasets, and (3) identify and mitigate the effects of spurious correlations with text features, improving classiﬁer performance on adversarially distributed test sets.

Original: Taken together, these results show that ﬁne-grained, automatic annotation of deep network models is both possible and practical: rich explanations produced by automated annotation procedures can surface meaningful and actionable information about model behavior.
Revised: Taken together, these results show that ﬁne-grained, automatic annotation of deep network models is both possible and practical: rich explanations produced by automated annotation procedures can surface meaningful and actionable information about model behavior.

Original:  R ELATED W ORK Interpreting deep networks
Revised:  R ELATED W ORK Interpreting deep networks

Original: MILAN builds on a long line of recent approaches aimed at explaining the behavior of deep networks by characterizing the function of individual neurons, either by visualizing the inputs they select for (Zeiler & Fergus, 2014;
Revised: MILAN builds on a long line of recent approaches aimed at explaining the behavior of deep networks by characterizing the function of individual neurons, either by visualizing the inputs they select for (Zeiler & Fergus, 2014;

Original: Girshick et al., 2014; Karpathy et al., 2015; Mahendran & Vedaldi, 2015; Olah et al., 2017) or by automatically categorizing them according to the concepts they recognize (Bau et al., 2017; 2018; Mu & Andreas, 2020; Morcos et al., 2018; Dalvi et al., 2019).
Revised: Girshick et al., 2014; Karpathy et al., 2015; Mahendran & Vedaldi, 2015; Olah et al., 2017) or by automatically categorizing them according to the concepts they recognize (Bau et al., 2017; 2018; Mu & Andreas, 2020; Morcos et al., 2018; Dalvi et al., 2019).

Original: Past approaches to automatic neuron labeling require ﬁxed, pre-deﬁned label sets; in computer vision, this has limited exploration to pre-selected object classes, parts, materials, and simple logical combinations of these concepts.
Revised: Past approaches to automatic neuron labeling require ﬁxed, pre-deﬁned label sets; in computer vision, this has limited exploration to pre-selected object classes, parts, materials, and simple logical combinations of these concepts.

Original: While manual inspection of neurons has revealed that a wider range of features play an important role in visual recognition (e.g. orientation, illumination, and spatial relations; Cammarata et al. 2021)
Revised: While manual inspection of neurons has revealed that a wider range of features play an important role in visual recognition (e.g. orientation, illumination, and spatial relations; Cammarata et al. 2021)

Original: MILAN is the ﬁrst automated approach that can identify such features at scale.
Revised: MILAN is the ﬁrst automated approach that can identify such features at scale.

Original: Discrete categorization is also possible for directions in representation space (Kim et al., 2018; Andreas et al., 2017; Schwettmann et al., 2021); as MILAN requires only a primitive procedure for generating model inputs maximally associated with the feature or direction of interest, future work might extend it to these settings as well.
Revised: Discrete categorization is also possible for directions in representation space (Kim et al., 2018; Andreas et al., 2017; Schwettmann et al., 2021) and for clusters of images induced by visual representations (Laina et al., 2020); in the latter, an off-the-shelf image captioning model is used to obtain language descriptions of the unifying visual concept for the cluster, although the descriptions miss low-level visual commonalities. As MILAN requires only a primitive procedure for generating model inputs maximally associated with the feature or direction of interest, future work might extend it to these settings as well.

Original: Natural language explanations of decisions Previous work aimed at explaining computer vision classiﬁers using natural language has focused on generating explanations for individual classiﬁcation decisions (e.g., Hendricks et al., 2016; Park et al., 2018; Hendricks et al., 2018; Zellers et al., 2019).
Revised: Natural language explanations of decisions Previous work aimed at explaining computer vision classiﬁers using natural language has focused on generating explanations for individual classiﬁcation decisions (e.g., Hendricks et al., 2016; Park et al., 2018; Hendricks et al., 2018; Zellers et al., 2019).

Original: Outside of computer vision, several recent papers have proposed procedures for generating natural language explanations of decisions in text classiﬁcation models (Zaidan & Eisner, 2008; Camburu et al., 2018; Rajani et al., 2019; Narang et al., 2020) and of representations in more general sequence modeling problems (Andreas & Klein, 2017).
Revised: Outside of computer vision, several recent papers have proposed procedures for generating natural language explanations of decisions in text classiﬁcation models (Zaidan & Eisner, 2008; Camburu et al., 2018; Rajani et al., 2019; Narang et al., 2020) and of representations in more general sequence modeling problems (Andreas & Klein, 2017).

Original: These approaches require task-speciﬁc datasets and often specialized training procedures, and do not assist with interpretability at the model level.
Revised: These approaches require task-speciﬁc datasets and often specialized training procedures, and do not assist with interpretability at the model level.

Original: To the best of our knowledge, MILAN is the ﬁrst approach for generating compositional natural language descriptions for interpretability at the level of individual features rather than input-conditional decisions or representations.
Revised: To the best of our knowledge, MILAN is the ﬁrst approach for generating compositional natural language descriptions for interpretability at the level of individual features rather than input-conditional decisions or representations.

Original: More fundamentally, MILAN can do so independently of the model being described, making it (as shown in Section 4) modular, portable, and to a limited extent task-agnostic.
Revised: More fundamentally, MILAN can do so independently of the model being described, making it (as shown in Section 4) modular, portable, and to a limited extent task-agnostic.

Original:  A PPROACH
Revised:  A PPROACH

Original: Neurons and exemplars Consider the neuron depicted in Figure 1b, located in a convlutional network trained to classify scenes (Zhou et al., 2017).
Revised: Neurons and exemplars Consider the neuron depicted in Figure 1b, located in a convlutional network trained to classify scenes (Zhou et al., 2017).

Original: When the images in Figure 1 are provided as input to the network, the neuron activates in patches of grass near animals , but not in grass without animals nearby.
Revised: When the images in Figure 1 are provided as input to the network, the neuron activates in patches of grass near animals , but not in grass without animals nearby.

Original: How might we automate the process of automatically generating such a description?
Revised: How might we automate the process of automatically generating such a description?

Original: While the image regions depicted in Fig.
Revised: While the image regions depicted in Fig.

Original: 1b do not completely characterize the neuron’s function in the broader network, past work has found that actionable information can be gleaned from such regions alone.
Revised: 1b do not completely characterize the neuron’s function in the broader network, past work has found that actionable information can be gleaned from such regions alone.

Original: Bau et al.
Revised: Bau et al.

Original: (2020; 2019) use them to identify neurons that can trigger class predictions or generative synthesis of speciﬁc objects; Andreas & Klein (2017) use them to predict sequence outputs on novel inputs; Olah et al.
Revised: (2020; 2019) use them to identify neurons that can trigger class predictions or generative synthesis of speciﬁc objects; Andreas & Klein (2017) use them to predict sequence outputs on novel inputs; Olah et al.

Original: and Mu & Andreas (2020) use them to identify adversarial vulnerabilities.
Revised: and Mu & Andreas (2020) use them to identify adversarial vulnerabilities.

Original: Thus, building on this past work, our approach to neuron labeling also begins by representing each neuron via the set of input regions on which its activity exceeds a ﬁxed threshold.
Revised: Thus, building on this past work, our approach to neuron labeling also begins by representing each neuron via the set of input regions on which its activity exceeds a ﬁxed threshold.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: Let f : X → Y be a neural network, and let f i ( x ) denote the activation value of the i th neuron in f given an input x . 

Revised: Let f : X → Y be a neural network, and let f i ( x ) denote the activation value of the i th neuron in f given an input x .

Original: Then, the exemplar representation of the neuron f
Revised: Then, the exemplar representation of the neuron f

Original: i is given by:
Revised: i is given by:

Original: [Equation]
Revised: [Equation]

Original: for some activation threshold η i (discussed in more detail below).
Revised: for some activation threshold η i (discussed in more detail below).

Original: Exemplars and descriptions Given this explicit representation of f i ’s behavior, it remains to construct a description d
Revised: Exemplars and descriptions Given this explicit representation of f i ’s behavior, it remains to construct a description d

Original: i of the neuron.
Revised: i of the neuron.

Original: Past work (Bau et al., 2017; Andreas et al., 2017) begins with a ﬁxed inventory of candidate descriptions (e.g. object categories), deﬁnes an exemplar set E (cid:48)
Revised: Past work (Bau et al., 2017; Andreas et al., 2017) begins with a ﬁxed inventory of candidate descriptions (e.g. object categories), deﬁnes an exemplar set E (cid:48)

Original: d for each such category (e.g. via the output of a semantic segmentation procedure) then labels neurons by optimizing d i :=
Revised: d for each such category (e.g. via the output of a semantic segmentation procedure) then labels neurons by optimizing d i :=

Original: arg min d δ ( E i , E (cid:48) d ) for some measure of set distance (e.g. Jaccard, 1912).
Revised: arg min d δ ( E i , E (cid:48) d ) for some measure of set distance (e.g. Jaccard, 1912).

Original: In this work, we instead adopt a probabilistic approach to neuron labeling.
Revised: In this work, we instead adopt a probabilistic approach to neuron labeling.

Original: In computer vision applications, each E i is a set of image patches.
Revised: In computer vision applications, each E i is a set of image patches.

Original: Humans are adept at describing such patches (Rashtchian et al., 2010) and one straightforward possibility might be to directly optimize d i :=
Revised: Humans are adept at describing such patches (Rashtchian et al., 2010) and one straightforward possibility might be to directly optimize d i :=

Original: arg max d p ( d |
Revised: arg max d p ( d |

Original: E i ) .
Revised: E i ) .

Original: In practice, however, the distribution of human descriptions given images may not be wellaligned with the needs of model users.
Revised: In practice, however, the distribution of human descriptions given images may not be wellaligned with the needs of model users.

Original: Fig.
Revised: Fig.

Original: 2 includes examples of human-generated descriptions for exemplar sets.
Revised: 2 includes examples of human-generated descriptions for exemplar sets.

Original: Many of them (e.g. text for AlexNet conv3-252) are accurate, but generic; in reality, the neuron responds speciﬁcally to text on screens.
Revised: Many of them (e.g. text for AlexNet conv3-252) are accurate, but generic; in reality, the neuron responds speciﬁcally to text on screens.

Original: The generated description of a neuron should capture the speciﬁcity of its function—especially relative to other neurons in the same model .
Revised: The generated description of a neuron should capture the speciﬁcity of its function—especially relative to other neurons in the same model .

Original: We thus adopt an information-theoretic criterion for selecting descriptions: our ﬁnal neuron description procedure optimizes pointwise mutual information between descriptions and exemplar sets:
Revised: We thus adopt an information-theoretic criterion for selecting descriptions: our ﬁnal neuron description procedure optimizes pointwise mutual information between descriptions and exemplar sets:

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: The max-mutual-information description of the neuron f
Revised: The max-mutual-information description of the neuron f

Original: i is given by:
Revised: i is given by:

Original: [Equation]
Revised: [Equation]

Original: This choice of criterion aligns with previous work on probing deep representations (e.g., Pimentel et al., 2020).

Original: To turn Eq.
Revised: To turn Eq.

Original: (2) into a practical procedure for annotating neurons, three additional steps are required: constructing a tractable approximation to the exemplar set E
Revised: (2) into a practical procedure for annotating neurons, three additional steps are required: constructing a tractable approximation to the exemplar set E

Original: i (Section 3.1), using human-generated image descriptions to model p ( d | E ) and p ( d ) (Section 3.2 and Section 3.3), and ﬁnding a high-quality description d in the inﬁnite space of natural language strings (Section 3.4).
Revised: i (Section 3.1), using human-generated image descriptions to model p ( d | E ) and p ( d ) (Section 3.2 and Section 3.3), and ﬁnding a highquality description d in the inﬁnite space of natural language strings (Section 3.4).

Original: 3. A PPROXIMATING THE EXEMPLAR SET
Revised: 3. A PPROXIMATING THE EXEMPLAR SET

Original: As written, the exemplar set in Equation (1) captures a neuron’s behavior on all image patches.
Revised: As written, the exemplar set in Equation (1) captures a neuron’s behavior on all image patches.

Original: This set is large (limited only by the precision used to represent individual pixel values), so we follow past work (Bau et al., 2017) by restricting each E i to the set of images that cause the greatest activation in the neuron f i .
Revised: This set is large (limited only by the precision used to represent individual pixel values), so we follow past work (Bau et al., 2017) by restricting each E i to the set of images that cause the greatest activation in the neuron f i .

Original: For convolutional neurons in image processing tasks, sets E i ultimately comprise k images with activation masks indicating the regions of those images in which f i ﬁred (Fig.
Revised: For convolutional neurons in image processing tasks, sets E i ultimately comprise k images with activation masks indicating the regions of those images in which f i ﬁred (Fig.

Original: 1a; see
Revised: 1a; see

Original: Bau et al.
Revised: Bau et al.

Original: 2017 for details).
Revised: 2017 for details).

Original: Throughout this paper, we use exemplar sets with k = 15 images and choose η i equal to the 0.99 percentile of activations for the neuron f i .
Revised: Throughout this paper, we use exemplar sets with k = 15 images and choose η i equal to the 0.99 percentile of activations for the neuron f i .

Original: 3. M ODELING p ( d | E ) AND p ( d )
Revised: 3. M ODELING p ( d | E ) AND p ( d )

Original: The term pmi( d ; E i ) in Equation (2) can be expressed in terms of two distributions: the probability p ( d | E i ) that a human would describe an image region with d , and the probability p ( d ) that a human would use the description d for any neuron.
Revised: The term pmi( d ; E i ) in Equation (2) can be expressed in terms of two distributions: the probability p ( d | E i ) that a human would describe an image region with d , and the probability p ( d ) that a human would use the description d for any neuron.

Original: p ( d | E i ) is, roughly speaking, a distribution over image captions (Donahue et al., 2015).
Revised: p ( d | E i ) is, roughly speaking, a distribution over image captions (Donahue et al., 2015).

Original: Here, however, the input to the model is not a single image but a set of image regions (the masks in Fig.
Revised: Here, however, the input to the model is not a single image but a set of image regions (the masks in Fig.

Original: 1a)
Revised: 1a)

Original: ; we seek natural language descriptions of the common features of those regions.
Revised: ; we seek natural language descriptions of the common features of those regions.

Original: We approximate p ( d | E i ) with learned model—speciﬁcally the Show-Attend-Tell image description model of Xu et al.
Revised: We approximate p ( d | E i ) with learned model—speciﬁcally the Show-Attend-Tell image description model of Xu et al.

Original: (2015) trained on the MILANNOTATIONS dataset described below, and with several modiﬁcations tailored to our use case.
Revised: (2015) trained on the MILANNOTATIONS dataset described below, and with several modiﬁcations tailored to our use case.

Original: We approximate p ( d ) with a two-layer LSTM language model (Hochreiter & Schmidhuber, 1997) trained on the text of about both models
Revised: We approximate p ( d ) with a two-layer LSTM language model (Hochreiter & Schmidhuber, 1997) trained on the text of about both models

Original: 3. C OLLECTING HUMAN ANNOTATIONS
Revised: 3. C OLLECTING HUMAN ANNOTATIONS

Original: As p ( d | E i ) and p ( d ) are both estimated using learned models, they require training data.
Revised: As p ( d | E i ) and p ( d ) are both estimated using learned models, they require training data.

Original: In particular, modeling p ( d | E i ) requires a dataset of captions that describe regions from multiple different images, such as the ones shown in Fig.
Revised: In particular, modeling p ( d | E i ) requires a dataset of captions that describe regions from multiple different images, such as the ones shown in Fig.

Original: These descriptions must describe not only objects and actions, but all other details that individual neurons select for.
Revised: These descriptions must describe not only objects and actions, but all other details that individual neurons select for.

Original: Existing image captioning datasets, like MSCOCO (Lin et al., 2014) and Conceptual Captions (Sharma et al., 2018), only focus on scene-level details about a single image and do not provide suitable annotations for this task.
Revised: Existing image captioning datasets, like MSCOCO (Lin et al., 2014) and Conceptual Captions (Sharma et al., 2018), only focus on scene-level details about a single image and do not provide suitable annotations for this task.

Original: We therefore collect a novel dataset of captions for image regions to train the models underlying MILAN .
Revised: We therefore collect a novel dataset of captions for image regions to train the models underlying MILAN .

Original: [Table]
Revised: [Table]

Original: First, we must obtain a set of image regions to annotate.
Revised: First, we must obtain a set of image regions to annotate.

Original: To ensure that these regions have a similar distribution to the target neurons themselves, we derive them directly from the exemplar sets of neurons in a set of seed models .
Revised: To ensure that these regions have a similar distribution to the target neurons themselves, we derive them directly from the exemplar sets of neurons in a set of seed models .

Original: We obtain the exemplar sets for a subset of the units in each seed model in Table 1 using the method from Section 3.1.
Revised: We obtain the exemplar sets for a subset of the units in each seed model in Table 1 using the method from Section 3.1.

Original: We then present each set to a human annotator and ask them to describe what is common to the image regions.
Revised: We then present each set to a human annotator and ask them to describe what is common to the image regions.

Original: Table 1 summarizes the dataset, which we call MILANNOTATIONS .
Revised: Table 1 summarizes the dataset, which we call MILANNOTATIONS .

Original: In total, we construct exemplar sets using neurons from seven vision models, totaling 20k neurons.
Revised: In total, we construct exemplar sets using neurons from seven vision models, totaling 20k neurons.

Original: These models include two architectures for supervised image classiﬁcation, AlexNet (Krizhevsky et al., 2012) and ResNet152 (He et al., 2015); one architecture for image generation, BigGAN (Brock et al., 2018); and one for unsupervised representation learning trained with a “Bootsrap Your Own Latent” (BYOL) objective (Chen & He, 2020; Grill et al., 2020), DINO (Caron et al., 2021).
Revised: These models include two architectures for supervised image classiﬁcation, AlexNet (Krizhevsky et al., 2012) and ResNet152 (He et al., 2015); one architecture for image generation, BigGAN (Brock et al., 2018); and one for unsupervised representation learning trained with a “Bootsrap Your Own Latent” (BYOL) objective (Chen & He, 2020; Grill et al., 2020), DINO (Caron et al., 2021).

Original: These models cover two datasets, speciﬁcally ImageNet (Deng et al., 2009) and Places365 (Zhou et al., 2017), as well as two completely different families of models, CNNs and Vision Transformers (ViT) (Dosovitskiy et al., 2021).
Revised: These models cover two datasets, speciﬁcally ImageNet (Deng et al., 2009) and Places365 (Zhou et al., 2017), as well as two completely different families of models, CNNs and Vision Transformers (ViT) (Dosovitskiy et al., 2021).

Original: Each exemplar set is shown to three distinct human participants, resulting 60k total annotations.
Revised: Each exemplar set is shown to three distinct human participants, resulting 60k total annotations.

Original: Examples are provided in Appendix A (Fig. 10).
Revised: Examples are provided in Appendix A (Fig. 10).

Original: We recruit participants from Amazon Mechanical Turk.
Revised: We recruit participants from Amazon Mechanical Turk.

Original: This data collection effort was approved by our institutional review board.
Revised: This data collection effort was approved by our institutional review board.

Original: To control for quality, workers were required to have a HIT acceptance rate of at least 95%, have at least 100 approved HITs, and pass a short qualiﬁcation test.
Revised: To control for quality, workers were required to have a HIT acceptance rate of at least 95%, have at least 100 approved HITs, and pass a short qualiﬁcation test.

Original: Full details about our data collection process and the collected data can be found in Appendix A.
Revised: Full details about our data collection process and the collected data can be found in Appendix A.

Original: 3. S EARCHING IN THE SPACE OF DESCRIPTIONS
Revised: 3. S EARCHING IN THE SPACE OF DESCRIPTIONS

Original: Directly decoding descriptions from pmi( d ; E i ) tends to generate disﬂuent descriptions.
Revised: Directly decoding descriptions from pmi( d ; E i ) tends to generate disﬂuent descriptions.

Original: This is because the p ( d ) term inherently discourages common function words like the from appearing in descriptions.
Revised: This is because the p ( d ) term inherently discourages common function words like the from appearing in descriptions.

Original: Past work language generation (Wang et al., 2020) has found that this can be remedied by ﬁrst introducing a hyperparameter λ to modulate the importance of p ( d ) when computing PMI, giving a new weighted PMI objective:
Revised: Past work language generation (Wang et al., 2020) has found that this can be remedied by ﬁrst introducing a hyperparameter λ to modulate the importance of p ( d ) when computing PMI, giving a new weighted PMI objective:

Original: [Equation]
Revised: [Equation]

Original: Next, search is restricted to a set of captions that are high probability under p ( d  ) , which are reranked according to Eq.
Revised: Next, search is restricted to a set of captions that are high probability under p ( d | E i ) , which are reranked according to Eq.

Revised: Speciﬁcally, we run beam search on p ( d | E i ) , and use the full beam after the ﬁnal search step as a set of candidate descriptions.

Original: For allof our experiments, we set λ
Revised: For all experiments, we set λ

Original: = .
Revised: = .

Original: 2 andobtain an initial description set using beamsearch with a beam of size  50.
Revised: 2 and beam size to 50.

Original:  D OES MILAN GENERALIZE ?
Revised:  D OES MILAN GENERALIZE ?

Original: Because it is trained on a set of human-annotated exemplar sets obtained from a set of seed networks, MILAN is useful as an automated procedure only if it generalizes and correctly describes neurons in trained models with new architectures, new datasets, and new training objectives.
Revised: Because it is trained on a set of human-annotated exemplar sets obtained from a set of seed networks, MILAN is useful as an automated procedure only if it generalizes and correctly describes neurons in trained models with new architectures, new datasets, and new training objectives.

Original: Thus, before describing applications of MILAN to speciﬁc interpretability problems, we perform cross-
Revised: Thus, before describing applications of MILAN to speciﬁc interpretability problems, we perform cross-

Original: [Table]
Revised: [Table]

Original: validation experiments within the MILANNOTATIONS data to validate that MILAN can reliably label new neurons.
Revised: validation experiments within the MILANNOTATIONS data to validate that MILAN can reliably label new neurons.

Original: We additionally verify that MILAN provides beneﬁts over other neuron annotation techniques by comparing its descriptions to three baselines:
Revised: We additionally verify that MILAN provides beneﬁts over other neuron annotation techniques by comparing its descriptions to three baselines:

Original: NetDissect (Bau et al., 2017), which assigns a single concept label to each neuron by comparing the neuron’s exemplars to semantic segmentations of the same images; Compositional
Revised: NetDissect (Bau et al., 2017), which assigns a single concept label to each neuron by comparing the neuron’s exemplars to semantic segmentations of the same images; Compositional

Original: [Table]
Revised: [Table]

Original: Explanations (Mu & Andreas, 2020), which follows a similar procedure to generate logical concept labels; and ordinary image captioning (selecting descriptions using p ( d | E ) instead of pmi( d ; E ) ).
Revised: Explanations (Mu & Andreas, 2020), which follows a similar procedure to generate logical concept labels; and ordinary image captioning (selecting descriptions using p ( d | E ) instead of pmi( d ; E ) ).

Original: Method In each experiment, we train MILAN on a subset of MILANNOTATIONS and evaluate its performance on a held-out subset.
Revised: Method In each experiment, we train MILAN on a subset of MILANNOTATIONS and evaluate its performance on a held-out subset.

Original: To compare MILAN to the baselines, we train on all data except a single held-out network; we obtain the baseline labels by running the publicly available code with the default settings on the held-out network.
Revised: To compare MILAN to the baselines, we train on all data except a single held-out network; we obtain the baseline labels by running the publicly available code with the default settings on the held-out network.

Original: To test generalization within a network, we train on 90% of neurons from each network and test on the remaining 10%.
Revised: To test generalization within a network, we train on 90% of neurons from each network and test on the remaining 10%.

Original: To test generalization across architectures, we train on all AlexNet (ResNet) neurons and test on all ResNet (AlexNet) neurons; we also train on all
Revised: To test generalization across architectures, we train on all AlexNet (ResNet) neurons and test on all ResNet (AlexNet) neurons; we also train on all

Original: CNN neurons and test on ViT neurons.
Revised: CNN neurons and test on ViT neurons.

Original: To test generalization across datasets, we train on all neurons from models trained on ImageNet (Places) and test on neurons from models for the other datasets.
Revised: To test generalization across datasets, we train on all neurons from models trained on ImageNet (Places) and test on neurons from models for the other datasets.

Original: To test generalization across tasks, we train on all classiﬁer neurons (GAN neurons) and test on all GAN neurons (classiﬁer neurons).
Revised: To test generalization across tasks, we train on all classiﬁer neurons (GAN neurons) and test on all GAN neurons (classiﬁer neurons).

Original: We measure performance via BERTScore (Zhang et al., 2020) relative to the human annotations.
Revised: We measure performance via BERTScore (Zhang et al., 2020) relative to the human annotations.

Original: Hyperparameters for each of these experiments are in Appendix C.
Revised: Hyperparameters for each of these experiments are in Appendix C.

Original: Results Table 2 shows results for MILAN and all three baselines applied to four different networks.
Revised: Results Table 2 shows results for MILAN and all three baselines applied to four different networks.

Original: MILAN obtains higher agreement with human annotations on held-out networks than baselines.
Revised: MILAN obtains higher agreement with human annotations on held-out networks than baselines.

Original: It is able to surface highly speciﬁc behaviors in its descriptions, like the splashes of water neuron shown in Figure 2 ( splashes has no clear equivalent in the concept sets used by NetDissect (ND) or Compositional Explanations (CE)).
Revised: It is able to surface highly speciﬁc behaviors in its descriptions, like the splashes of water neuron shown in Figure 2 ( splashes has no clear equivalent in the concept sets used by NetDissect (ND) or Compositional Explanations (CE)).

Original: MILAN also outperforms the ablated p ( d | E ) decoder, justifying the choice of pmi as an objective for obtaining speciﬁc and high-quality descriptions.
Revised: MILAN also outperforms the ablated p ( d | E ) decoder, justifying the choice of pmi as an objective for obtaining speciﬁc and high-quality descriptions.

Original: Table 3 shows that MILAN exhibits different degrees of generalization across models, with generalization to new GAN neurons in the same network easiest and GAN-to-classiﬁer generalization hardest.
Revised: Table 3 shows that MILAN exhibits different degrees of generalization across models, with generalization to new GAN neurons in the same network easiest and GAN-to-classiﬁer generalization hardest.

Original: MILAN can generalize to novel architectures .
Revised: MILAN can generalize to novel architectures .

Original: It correctly labels ViT neurons (in fully connected layers) as often as it correctly labels other convolutional units (e.g., in AlexNet).
Revised: It correctly labels ViT neurons (in fully connected layers) as often as it correctly labels other convolutional units (e.g., in AlexNet).

Original: We observe that transferability across tasks is asymmetric: agreement scores are higher when transferring from classiﬁer neurons to GAN neurons than the reverse.
Revised: We observe that transferability across tasks is asymmetric: agreement scores are higher when transferring from classiﬁer neurons to GAN neurons than the reverse.

Revised: Finally, Figure 3 presents some of MILAN ’s failure cases: when faced with new visual concepts, MILAN sometimes mislabels the concept (e.g., by calling brass instruments noodle dishes ), prefers a vague description (e.g., similar color patterns ), or ignores the highlighted regions and describes the context instead.

Original: We emphasize thatthe results in this section are primarily intended as a sanity check of the learned models underlying MILAN , and not as direct evidence of its usefulness or reliability as a tool for interpretability.
Revised: We emphasize that this section is primarily intended as a sanity check of the learned models underlying MILAN , and not as direct evidence of its usefulness or reliability as a tool for interpretability.

Original: We follow Vaughan & Wallach (2020) in arguing that the ﬁnal test of any such tool must be its ability to produce actionable insights for human users, as in the three applications described below.
Revised: We follow Vaughan & Wallach (2020) in arguing that the ﬁnal test of any such tool must be its ability to produce actionable insights for human users, as in the three applications described below.

Original:  A NALYZING F EATURE I MPORTANCE
Revised:  A NALYZING F EATURE I MPORTANCE

Original: The previous section shows that MILAN can generalize to new architectures, datasets, and tasks.
Revised: The previous section shows that MILAN can generalize to new architectures, datasets, and tasks.

Original: The remainder of this paper focuses on applications that use generated labels to understand how neurons inﬂuence model behavior.
Revised: The remainder of this paper focuses on applications that use generated labels to understand how neurons inﬂuence model behavior.

Original: As a ﬁrst example: descriptions in Figure 2 reveal that neurons have different degrees of speciﬁcity.
Revised: As a ﬁrst example: descriptions in Figure 2 reveal that neurons have different degrees of speciﬁcity.

Original: Some neurons detect objects with spatial constraints ( the area on top of the line ), while others ﬁre for low-level but highly speciﬁc perceptual qualities ( long, thin objects ).
Revised: Some neurons detect objects with spatial constraints ( the area on top of the line ), while others ﬁre for low-level but highly speciﬁc perceptual qualities ( long, thin objects ).

Original: Still others detect perceptually similar but fundamentally different objects ( dog faces and cupcakes ).
Revised: Still others detect perceptually similar but fundamentally different objects ( dog faces and cupcakes ).

Original: How important are these different classes of neurons to model behavior?
Revised: How important are these different classes of neurons to model behavior?

Original: Method We use MILAN trained on all convolutional units in MILANNOTATIONS to annotate every neuron in ResNet18ImageNet.
Revised: Method We use MILAN trained on all convolutional units in MILANNOTATIONS to annotate every neuron in ResNet18ImageNet.

Original: We then score each neuron according to one of seven criteria that capture different syntactic or structural properties of the caption.
Revised: We then score each neuron according to one of seven criteria that capture different syntactic or structural properties of the caption.

Original: Four syntactic criteria each count the number of times that a speciﬁc part of speech appears in a caption: nouns , verbs , prepositions , and adjectives .
Revised: Four syntactic criteria each count the number of times that a speciﬁc part of speech appears in a caption: nouns , verbs , prepositions , and adjectives .

Original: Three structural criteria measure properties of the entire caption: its length , the depth of its parse tree (a rough measure of its compositional complexity, obtained from the spaCy parser of Honnibal et al. 2020), and its maximum word difference (a measure of the semantic coherence of the description, measured as the maximum Euclidean distance between any two caption words, again obtained via spaCy).
Revised: Three structural criteria measure properties of the entire caption: its length , the depth of its parse tree (a rough measure of its compositional complexity, obtained from the spaCy parser of Honnibal et al. 2020), and its maximum word difference (a measure of the semantic coherence of the description, measured as the maximum Euclidean distance between any two caption words, again obtained via spaCy).

Original: Finally, neurons are incrementally ablated in order of their score.
Revised: Finally, neurons are incrementally ablated in order of their score.

Original: The network is tested on the
Revised: The network is tested on the

Original: ImageNet validation set and its accuracy recorded.
Revised: ImageNet validation set and its accuracy recorded.

Original: This procedure is then repeated, deleting 2% of neurons at each step.
Revised: This procedure is then repeated, deleting 2% of neurons at each step.

Original: We also include ﬁve trials in which neurons are ordered randomly .
Revised: We also include ﬁve trials in which neurons are ordered randomly .

Original: Further details and examples of ablated neurons are provided in Appendix D
Revised: Further details and examples of ablated neurons are provided in Appendix D.

Original: [Figure] Results Figure 4 plots accuracy on the ImageNet validation set as a function of the number of ablated neurons.
Revised: Results Figure 4 plots accuracy on the ImageNet validation set as a function of the number of ablated neurons.

Original: Linguistic features of neuron descriptions highlight several important differences between neurons.
Revised: Linguistic features of neuron descriptions highlight several important differences between neurons.

Original: First, neurons captioned with many adjectives or prepositions (that is, neurons that capture attributes and relational features) are relatively important to model behavior.
Revised: First, neurons captioned with many adjectives or prepositions (that is, neurons that capture attributes and relational features) are relatively important to model behavior.

Original: Ablating these neurons causes a rapid decline in performance compared to ablating random neurons or  nouns.
Revised: Ablating these neurons causes a rapid decline in performance compared to ablating random neurons or [Figure] nouns.

Original: Second, neurons that detect dissimilar concepts appear to be less important.
Revised: Second, neurons that detect dissimilar concepts appear to be less important.

Original: When the caption contains highly dissimilar words ( max word diff. ), ablation hurts performance substantially less than ablating random neurons.
Revised: When the caption contains highly dissimilar words ( max word diff. ), ablation hurts performance substantially less than ablating random neurons.

Original: Such neurons sometimes detect non-semantic compositions of concepts like the dog faces and cupcakes neuron shown in Fig.2; past work has found that these
Revised: Such neurons sometimes detect non-semantic compositions of concepts like the dog faces and cupcakes neuron shown in Fig.

Original: [Figure] units contribute to non-robust model behavior Mu & Andreas (2020).
Revised: 2; Mu & Andreas (2020) ﬁnd that these units contribute to non-robust model behavior.

Revised: We reproduce their robustness experiments using these neurons in Section 5 (Figure 14) and reach similar conclusions.

Original: Finally, Figure 4 highlights that neurons satisfying each criterion are not evenly distributed across layers—for example, middle layers contain the largest fraction of relation-selective neurons measured via prepositions.
Revised: Finally, Figure 4 highlights that neurons satisfying each criterion are not evenly distributed across layers—for example, middle layers contain the largest fraction of relation-selective neurons measured via prepositions.

Original:  A UDITING A NONYMIZED M ODELS
Revised:  A UDITING A NONYMIZED M ODELS

Original: One recent line of work in computer vision aims to construct privacy-aware datasets, e.g. by detecting and blurring all faces to avoid leakage of information about speciﬁc individuals into trained models (Yang et al., 2021).
Revised: One recent line of work in computer vision aims to construct privacy-aware datasets, e.g. by detecting and blurring all faces to avoid leakage of information about speciﬁc individuals into trained models (Yang et al., 2021).

Original: But to what extent does this form of anonymization actually reduce
Revised: But to what extent does this form of anonymization actually reduce

Original: [Figure]
Revised: [Figure]

Original: models’ reliance on images of humans?
Revised: models’ reliance on images of humans?

Original: We wish to understand if models trained on blurred data still construct features that can identify speciﬁc individuals, or that select for speciﬁc demographic categories.
Revised: We wish to understand if models trained on blurred data still construct features that can identify speciﬁc individuals, or that select for speciﬁc demographic categories.

Original: A core function of tools for interpretable machine learning is to enable auditing of trained models for such behavior; here, we apply MILAN to investigate the effectiveness of blurring-based dataset privacy.
Revised: A core function of tools for interpretable machine learning is to enable auditing of trained models for such behavior; here, we apply MILAN to investigate the effectiveness of blurring-based dataset privacy.

Original: Method We use MILAN to caption a subset of convolutional units in 12 different models pretrained for image classiﬁcation on the blurred
Revised: Method We use MILAN to caption a subset of convolutional units in 12 different models pretrained for image classiﬁcation on the blurred

Original: ImageNet images ( blurred models).
Revised: ImageNet images ( blurred models).

Original: These models are distributed by the original authors of the blurred ImageNet dataset (Yang et al., 2021).
Revised: These models are distributed by the original authors of the blurred ImageNet dataset (Yang et al., 2021).

Original: We caption the same units in models pretrained on regular ImageNet ( unblurred models) obtained from torchvision (Paszke et al., 2019).
Revised: We caption the same units in models pretrained on regular ImageNet ( unblurred models) obtained from torchvision (Paszke et al., 2019).

Original: We then manually inspect all neurons in the blurred and unblurred models for which MILAN descriptions contain the words face , head , nose , eyes , and mouth (using exemplar sets containing only unblurred images).
Revised: We then manually inspect all neurons in the blurred and unblurred models for which MILAN descriptions contain the words face , head , nose , eyes , and mouth (using exemplar sets containing only unblurred images)

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] [Figure] [Figure]
Revised: BreakingtheoverallaccuracyintoindividualcategoriesinILSVRC

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Revised: Privacybreachescanalsohappenduringtraining/inference.

Original: [Figure]
Revised: [Figure]

Original: (b–c) Exemplar sets and labels for two neu- rons in a blurred model that activate on unblurred faces—and appear to preferentially (but not exclusively) respond to faces in speciﬁc demographic categories.

Original: Results Across models trained on ordinary ImageNet,
Revised: Results Across models trained on ordinary ImageNet,

Original: MILAN identiﬁed 213 neurons selective for human faces.
Revised: MILAN identiﬁed 213 neurons selective for human faces.

Original: Across models trained on blurred ImageNet, MILAN identiﬁed 142 neurons selective for human faces.
Revised: Across models trained on blurred ImageNet, MILAN identiﬁed 142 neurons selective for human faces.

Original: MILAN can distinguish between models trained on blurred and unblurred data (Fig. 5).
Revised: MILAN can distinguish between models trained on blurred and unblurred data (Fig. 5).

Original: However, it also reveals that models trained on blurred data acquire neurons selective for unblurred faces .
Revised: However, it also reveals that models trained on blurred data acquire neurons selective for unblurred faces .

Original: Indeed, it is possible to use
Revised: Indeed, it is possible to use

Original: MILAN ’s labels to extract these face-selective neurons directly.
Revised: MILAN ’s labels to extract these face-selective neurons directly.

Original: Doing so reveals that several of them are not simply face detectors, but select for speciﬁc, protected attributes such as gender and ethnicity (Fig. 6).
Revised: Doing so reveals that several of them are not simply face detectors, but select for speciﬁc, protected attributes such as gender and ethnicity (Fig. 6).

Original: Blurring does not prevent models from extracting highly speciﬁc features for these attributes.
Revised: Blurring does not prevent models from extracting highly speciﬁc features for these attributes.

Original: Our results in this section highlight the use of MILAN for both quantitative and qualitative, human-in-the loop auditing of model behavior.
Revised: Our results in this section highlight the use of MILAN for both quantitative and qualitative, human-in-the loop auditing of model behavior.

Original:  E DITING S PURIOUS F EATURES
Revised:  E DITING S PURIOUS F EATURES

Original: Spurious correlations between features and labels are a persistent problem in machine learning applications, especially in the presence of mismatches between training and testing data (Storkey, 2009).
Revised: Spurious correlations between features and labels are a persistent problem in machine learning applications, especially in the presence of mismatches between training and testing data (Storkey, 2009).

Original: In object recognition, one frequent example is correlation between backgrounds and objects (e.g. cows are more likely to appear with green grass in the background, while ﬁsh are more likely to appear with a blue background; Xiao et al. 2020).
Revised: In object recognition, one frequent example is correlation between backgrounds and objects (e.g. cows are more likely to appear with green grass in the background, while ﬁsh are more likely to appear with a blue background; Xiao et al. 2020).

Original: In a more recent example, models trained on joint text and image data are subject to “text-based adversarial attacks”, in which e.g. an apple with the word iPod written on it is classiﬁed as an iPod (Goh et al., 2021).
Revised: In a more recent example, models trained on joint text and image data are subject to “text-based adversarial attacks”, in which e.g. an apple with the word iPod written on it is classiﬁed as an iPod (Goh et al., 2021).

Original: Our ﬁnal experiment shows that MILAN can be used to reduce models’ sensitivity to these spurious features.
Revised: Our ﬁnal experiment shows that MILAN can be used to reduce models’ sensitivity to these spurious features.

Original: Data We create a controlled dataset imitating Goh et al.
Revised: Data We create a controlled dataset imitating Goh et al.

Original: (2021)’s spurious text features.
Revised: (2021)’s spurious text features.

Original: The dataset consists of 10 ImageNet classes.
Revised: The dataset consists of 10 ImageNet classes.

Original: In the training split, there are 1000 images per class;are annotated with (correct) text labels in the top-left corner.
Revised: In the training split, there are 1000 images per class;are annotated with (correct) text labels in the top-left corner.

Original: The test set contains 100 images per
Revised: The test set contains 100 images per

Original: [Figure]
Revised: [Figure]

Original: class (from the ImageNet validation set) ; in all  images, a random (usually incorrect) text label is 
Revised: class (from the ImageNet validation set); in all these images, a random (usually incorrect) text label is included.

Original: We train and evaluate a fresh ResNet18 model on this dataset, holding out 10% of the training data as a validation dataset for early stopping.
Revised: We train and evaluate a fresh ResNet18 model on this dataset, holding out 10% of the training data as a validation dataset for early stopping.

Original: Training details can be found in Appendix E.
Revised: Training details can be found in Appendix E.

Original: Method We use MILAN to obtain descriptions of every residual neuron in the model as well as the ﬁrst convolutional layer.
Revised: Method We use MILAN to obtain descriptions of every residual neuron in the model as well as the ﬁrst convolutional layer.

Original: We identify all neurons whose description contains text , word , or letter .
Revised: We identify all neurons whose description contains text , word , or letter .

Original: To identify spurious neurons, we ﬁrst assign each text neuron an independent importance score by removing it from the network and measuring the resulting drop in validation accuracy (with nonadversarial images).
Revised: To identify spurious neurons, we ﬁrst assign each text neuron an independent importance score by removing it from the network and measuring the resulting drop in validation accuracy (with nonadversarial images).

Original: We then sort neurons by importance score (with the least important ﬁrst), and successively ablate them from the model.
Revised: We then sort neurons by importance score (with the least important ﬁrst), and successively ablate them from the model.

Original: Results
Revised: Results

Original: The result of this procedure on adversarial test accuracy is shown in Fig.
Revised: The result of this procedure on adversarial test accuracy is shown in Fig.

Original: Training on the spurious data substantially reduces ResNet18’s performance on the adversarial test set: the model achieves 58.8% accuracy, as opposed to 80.8% when trained on non-spurious
Revised: Training on the spurious data substantially reduces ResNet18’s performance on the adversarial test set: the model achieves 58.8% accuracy, as opposed to 80.8% when trained on non-spurious

Original: [Figure]
Revised: [Figure]

Revised: neurons accuracy still improves, much

Original: MILAN identiﬁes 300 text-related convolutional units (out of 1024 examined) in the model, conﬁrming that the model has indeed devoted substantial capacity to identifying text labels in the image.
Revised: MILAN identiﬁes 300 text-related convolutional units (out of 1024 examined) in the model, conﬁrming that the model has indeed devoted substantial capacity to identifying text labels in the image.

Original: Figure 7c shows an example neurons speciﬁcally selective for airline and truck text.
Revised: Figure 7c shows an example neurons speciﬁcally selective for airline and truck text.

Original: By deleting only 13 such neurons, test accuracy is improved by 4.9% (a 12% reduction in error rate).
Revised: By deleting only 13 such neurons, test accuracy is improved by 4.9% (a 12% reduction in error rate).

Original: This increase cannot be explained by the sorting procedure described above: if instead we sort all neurons according to validation accuracy (orange line), accuracy improves by less than 1%.
Revised: This increase cannot be explained by the sorting procedure described above: if instead we sort all neurons according to validation accuracy (orange line), accuracy improves by less than 1%.

Original: Thus, while this experiment does not completely eliminate the model’s reliance on text features, it shows that MILAN ’s predictions enable direct editing of networks to partially mitigate sensitivity to spurious feature correlations.
Revised: Thus, while this experiment does not completely eliminate the model’s reliance on text features, it shows that MILAN ’s predictions enable direct editing of networks to partially mitigate sensitivity to spurious feature correlations.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We have presented MILAN , an approach for automatically labeling neurons with natural language descriptions of their behavior.
Revised: We have presented MILAN , an approach for automatically labeling neurons with natural language descriptions of their behavior.

Original: MILAN selects these descriptions by maximizing pointwise mutual information with image regions in which each neuron is active.
Revised: MILAN selects these descriptions by maximizing pointwise mutual information with image regions in which each neuron is active.

Original: These mutual information estimates are in turn produced by a pair of learned models trained on MILANNOTATIONS , a dataset of ﬁne-grained image annotations released with this paper.
Revised: These mutual information estimates are in turn produced by a pair of learned models trained on MILANNOTATIONS , a dataset of ﬁne-grained image annotations released with this paper.

Original: Descriptions generated by MILAN surface diverse aspects of model behavior, and can serve as a foundation for numerous analysis, auditing, and editing techniques workﬂows for users of deep network models.
Revised: Descriptions generated by MILAN surface diverse aspects of model behavior, and can serve as a foundation for numerous analysis, auditing, and editing techniques workﬂows for users of deep network models.

Original: E THICS STATEMENT
Revised: E THICS STATEMENT

Original: In contrast to most past work on neuron labeling, MILAN generates neuron labels using another black-box learned model trained on human annotations of visual concepts.
Revised: In contrast to most past work on neuron labeling MILAN generates neuron labels using another black-box learned model trained on human annotations of visual concepts.

Original: With this increase in expressive power come a number of potential limitations: exemplar-based explanations have known shortcomings (Bolukbasi et al., 2021), human annotations of exemplar sets may be noisy, and the captioning model may itself behave in unexpected ways far outside the training domain.
Revised: With this increase in expressive power come a number of potential limitations: exemplar-based explanations have known shortcomings (Bolukbasi et al., 2021), human annotations of exemplar sets may be noisy and the captioning model may itself behave in unexpected ways far outside the training domain.

Original: The MILANNOTATIONS dataset was collected with annotator tests to address potential data quality issues, and our evaluation in Section 4 characterizes prediction quality on new networks; we nevertheless emphasize that these descriptions are partial and potentially noisy characterizations of neuron function via their behavior on a ﬁxed-sized set of representative inputs.
Revised: The MILANNOTATIONS dataset was collected with annotator tests to address potential data quality issues, and our evaluation in Section 4 characterizes prediction quality on new networks; we nevertheless emphasize that these descriptions are partial and potentially noisy characterizations of neuron function via their behavior on a ﬁxed-sized set of representative inputs.

Original: MILAN complements, rather than replaces, both formal veriﬁcation (Dathathri et al., 2020) and careful review of predictions and datasets by expert humans (Gebru et al., 2018; Mitchell et al., 2019).
Revised: MILAN complements, rather than replaces, both formal veriﬁcation (Dathathri et al., 2020) and careful review of predictions and datasets by expert humans (Gebru et al., 2018; Mitchell et al., 2019).

Original: A MILANNOTATIONS
Revised: A MILANNOTATIONS

Original: We recruited annotators from Amazon Mechanical Turk to describe one neuron at a time given its top-activating images.
Revised: We recruited annotators from Amazon Mechanical Turk to describe one neuron at a time given its top-activating images.

Original: A screenshot of the template is shown in Figure 9b.
Revised: A screenshot of the template is shown in Figure 9b.

Original: Participants were given the instructions:
Revised: Participants were given the instructions:

Original: Instructions:
Revised: Instructions:

Original: In one sentence, summarize everything shown inside the highlighted regions in the images.
Revised: In one sentence, summarize everything shown inside the highlighted regions in the images.

Original: They might all show the same thing, or they might show several different things.
Revised: They might all show the same thing, or they might show several different things.

Original: In your answer, DO NOT mention that you are describing highlighted regions in images.
Revised: In your answer, DO NOT mention that you are describing highlighted regions in images.

Original: [Table]
Revised: [Table]

Original: Workers were given up to an hour to complete each annotation, but early trials revealed they required about 30 seconds per HIT.
Revised: Workers were given up to an hour to complete each annotation, but early trials revealed they required about 30 seconds per HIT.

Original: We paid workers $0.08 per annotation, which at $9.60 per hour exceeds the United States federal minimum wage.
Revised: We paid workers $0.08 per annotation, which at $9.60 per hour exceeds the United States federal minimum wage.

Original: To control for quality, we required workers to pass a short qualiﬁcation test in which they had to choose the most descriptive caption for two manually chosen neurons from VGG-16 (Simonyan &
Revised: To control for quality, we required workers to pass a short qualiﬁcation test in which they had to choose the most descriptive caption for two manually chosen neurons from VGG-16 (Simonyan &

Original: Zisserman, 2015) trained on ImageNet (not included as part of MILANNOTATIONS ).
Revised: Zisserman, 2015) trained on ImageNet (not included as part of MILANNOTATIONS ).

Original: A screenshot of this test is shown in Figure 9a.
Revised: A screenshot of this test is shown in Figure 9a.

Original: Table 4 shows the inter-annotator agreement of neuron annotations for each model, and Table 5 shows some corpus statistics broken down by model and layer.
Revised: Table 4 shows the inter-annotator agreement of neuron annotations for each model, and Table 5 shows some corpus statistics broken down by model and layer.

Original: Layers closest to the image (early layers in CNNs and later layers in GANs) are generally described with more adjectives than other layers, while annotations for layers far- ther from the image include more nouns, perhaps highlighting the low-level perceptual role of the former and the scene- and object-centric behavior of the latter.
Revised: Layers closest to the image (early layers in CNNs and later layers in GANs) are generally described with more adjectives than other layers, while annotations for layers far- ther from the image include more nouns, perhaps highlighting the low-level perceptual role of the former and the scene- and object-centric behavior of the latter.

Original: Layers farther from the image tend to have longer descriptions (e.g. in BigGAN-ImageNet, AlexNet-ImageNet), but this trend is not consistent across all models (e.g. in models trained on Places365, the middle layers have the longest average caption length).
Revised: Layers farther from the image tend to have longer descriptions (e.g. in BigGAN-ImageNet, AlexNet-ImageNet), but this trend is not consistent across all models (e.g. in models trained on Places365, the middle layers have the longest average caption length).

Original: B MILAN IMPLEMENTATION DETAILS B. I MPLEMENTING p ( d | E )
Revised: B MILAN IMPLEMENTATION DETAILS B. I MPLEMENTING p ( d | E )

Original: We build on the Show, Attend, and Tell (SAT) model for describing images (Xu et al., 2015).
Revised: We build on the Show, Attend, and Tell (SAT) model for describing images (Xu et al., 2015).

Original: SAT is designed for describing the high-level content of a single images, so we must make several modiﬁcations to support our use case, where our goal is to describe sets of regions in images.
Revised: SAT is designed for describing the high-level content of a single images, so we must make several modiﬁcations to support our use case, where our goal is to describe sets of regions in images.

Original: In the original SAT architecture, a single input image x is ﬁrst converted to visual features by passing it through an encoder network g , typically an image classiﬁer pretrained on a large dataset.
Revised: In the original SAT architecture, a single input image x is ﬁrst converted to visual features by passing it through an encoder network g , typically an image classiﬁer pretrained on a large dataset.

Original: The output of the last convolutional layer is extracted as a matrix of visual features:
Revised: The output of the last convolutional layer is extracted as a matrix of visual features:

Original: [Equation]
Revised: [Equation]

Original: These visual features are passed to a decoder LSTM whose hidden state is initialized as a function of the mean of the visual features v = 1 /k (cid:80)
Revised: These visual features are passed to a decoder LSTM whose hidden state is initialized as a function of the mean of the visual features v = 1 /k (cid:80)

Original: i v i .
Revised: i v i .

Original: At each time step, the decoder attends over the features using an additive attention mechanism (Bahdanau et al., 2015), then consumes the attenuated visual features and previous token as input to predict the next token.
Revised: At each time step, the decoder attends over the features using an additive attention mechanism (Bahdanau et al., 2015), then consumes the attenuated visual features and previous token as input to predict the next token.

Original: The SAT architecture makes few assumptions about the structure of the visual features.
Revised: The SAT architecture makes few assumptions about the structure of the visual features.

Original: We will take advantage of this generality and modify how v is constructed to support our task, leaving the decoder architecture intact.
Revised: We will take advantage of this generality and modify how v is constructed to support our task, leaving the decoder architecture intact.

Original: [Table]
Revised: [Table]

Original: layer   5.
Revised: layer   5.

Original: BigGAN-ImageNet layer   3. layer   3. layer   3. layer   3. layer   3. layer   2.
Revised: BigGAN-ImageNet layer   3. layer   3. layer   3. layer   3. layer   3. layer   2.

Original: BigGAN-Places layer   4.
Revised: BigGAN-Places layer   4.

Original: layer   3. layer   4. layer   5. layer   4. layer   4.
Revised: layer   3. layer   4. layer   5. layer   4. layer   4.

Original: DINO-ImageNet layer   4.
Revised: DINO-ImageNet layer   4.

Original: layer   4. layer   3. layer   3. layer   3. layer   3. layer   3.
Revised: layer   4. layer   3. layer   3. layer   3. layer   3. layer   3.

Original: [Table]
Revised: [Table]

Original: Now, instead of a single image x , the model inputs are the k top-activating images x j for a neuron as well as a mask m j for each image that highlights the regions of greatest activation.
Revised: Now, instead of a single image x , the model inputs are the k top-activating images x j for a neuron as well as a mask m j for each image that highlights the regions of greatest activation.

Original: Our task is to describe what the neuron is detecting, based strictly on the highlighted regions of the x j .
Revised: Our task is to describe what the neuron is detecting, based strictly on the highlighted regions of the x j .

Original: In support of this, the visual features must (1) include information about all k images, (2) encode multiple resolutions of the images to capture both low-level perceptual and high-level scene details about the image, and (3) pay most (but not exclusive) attention to the regions of greatest activation in the image.
Revised: In support of this, the visual features must (1) include information about all k images, (2) encode multiple resolutions of the images to capture both low-level perceptual and high-level scene details about the image, and (3) pay most (but not exclusive) attention to the regions of greatest activation in the image.

Original: Describing sets of images The k features in SAT correspond to different spatial localities of a single image.
Revised: Describing sets of images The k features in SAT correspond to different spatial localities of a single image.

Original: In our architecture, each feature v j corresponds to one input image x j .
Revised: In our architecture, each feature v j corresponds to one input image x j .

Original: Encoding multiple resolutions Instead of encoding the image with just the last convolutional layer of g , we use pooled convolutional features from every layer.
Revised: Encoding multiple resolutions Instead of encoding the image with just the last convolutional layer of g , we use pooled convolutional features from every layer.

Original: Formally, let g (cid:96) ( x ) denote the output of layer (cid:96) in the pretrained image encoder with L layers, and let pool denote a pooling function that uses the mask to pool the features (described further below).
Revised: Formally, let g (cid:96) ( x ) denote the output of layer (cid:96) in the pretrained image encoder with L layers, and let pool denote a pooling function that uses the mask to pool the features (described further below).

Original: The feature vector for the j th image x j is:
Revised: The feature vector for the j th image x j is:

Original: [Equation]
Revised: [Equation]

Original: Highlighting regions of greatest activation Each of the top-activating images x j that we hand to our model comes with a mask m j highlighting the image regions of greatest activation.
Revised: Highlighting regions of greatest activation Each of the top-activating images x j that we hand to our model comes with a mask m j highlighting the image regions of greatest activation.

Original: We incorporate these masks into the pooling function pool from above.
Revised: We incorporate these masks into the pooling function pool from above.

Original: Speciﬁcally, we ﬁrst downsample the mask m j to the same spatial shape as g (cid:96) ( x j ) using bilinear interpolation, which we denote upsample ( m j ) .
Revised: Speciﬁcally, we ﬁrst downsample the mask m j to the same spatial shape as g (cid:96) ( x j ) using bilinear interpolation, which we denote upsample ( m j ) .

Original: We then apply the mask to each channel c at layer (cid:96) , written g (cid:96),c ( x j ) , via elementwise multiplication ( (cid:12) ) with upsample ( m j ) .
Revised: We then apply the mask to each channel c at layer (cid:96) , written g (cid:96),c ( x j ) , via elementwise multiplication ( (cid:12) ) with upsample ( m j ) .

Original: Finally, we sum spatially along each channel, resulting in a length c vector.
Revised: Finally, we sum spatially along each channel, resulting in a length c vector.

Original: Formally:
Revised: Formally:

Original: [Equation]
Revised: [Equation]

Original: Each v i is thus a length (cid:80) (cid:96) C (cid:96) vector, where C (cid:96) is the number of channels at layer (cid:96) of g .
Revised: Each v i is thus a length (cid:80) (cid:96) C (cid:96) vector, where C (cid:96) is the number of channels at layer (cid:96) of g .

Original: Throughout our experiments, g is a ResNet101 pretrained for image classiﬁcation on ImageNet, provided by PyTorch Paszke et al.
Revised: Throughout our experiments, g is a ResNet101 pretrained for image classiﬁcation on ImageNet, provided by PyTorch Paszke et al.

Original: We extract visual features from the ﬁrst convolutional layer and all four residual layers.
Revised: We extract visual features from the ﬁrst convolutional layer and all four residual layers.

Original: We do not ﬁne tune any parameters in the encoder.
Revised: We do not ﬁne tune any parameters in the encoder.

Original: The decoder is a single LSTM cell with an input embedding size of 128 and a hidden size of 512.
Revised: The decoder is a single LSTM cell with an input embedding size of 128 and a hidden size of 512.

Original: The attention mechanism linearly maps the current hidden state and all visual feature vectors to size 512 vectors before computing attention weights.
Revised: The attention mechanism linearly maps the current hidden state and all visual feature vectors to size 512 vectors before computing attention weights.

Original: We always decode for a maximum of 15 steps.
Revised: We always decode for a maximum of 15 steps.

Original: The rest of the decoder is exactly the same as in Xu et al.
Revised: The rest of the decoder is exactly the same as in Xu et al.

Original: The model is trained to minimize cross entropy on the training set using the AdamW optimizer
Revised: The model is trained to minimize cross entropy on the training set using the AdamW optimizer

Original: Loshchilov & Hutter (2019) with a learning rate of 1e-3 and minibatches of size 64.
Revised: Loshchilov & Hutter (2019) with a learning rate of 1e-3 and minibatches of size 64.

Original: We include the double stochasticity regularization term used by Xu et al.
Revised: We include the double stochasticity regularization term used by Xu et al.

Original: (2015) with λ = 1 .
Revised: (2015) with λ = 1 .

Original: We also apply dropout ( p = . 5 ) to the hidden state before predicting the next word.
Revised: We also apply dropout ( p = . 5 ) to the hidden state before predicting the next word.

Original: Across conﬁgurations, 10% of
Revised: Across conﬁgurations, 10% of

Original: [Figure]

Original: ResNet18,

Original: [Figure]

Original: [Figure]

Original: The

Original: [Figure]

Original: of

Original: [Figure]

Original: or radial

Original: [Figure]

Original: ResNet

Original: [Figure]

Original: - # nouns

Original: Text on a sign, text on a web page,

Original: [Figure]

Original: menu # adjectives

Original: Purple, red and blue fluorescent blue and purple objects # prepositions ResNet18, layer2Space on the right side of a object

Original: [Figure]
Revised: [Table]

Original: above a bird

Original: [Figure]

Original: conv1ResNet

Original: [Figure] - # verbs [Figure] the training data is held out and used as a validation set, and training stops when the model’s BLEU score (Papineni et al., 2002) does not improve on this set for 4 epochs, up to a maximum ofepochs.
Revised: the training data is held out and used as a validation set, and training stops when the model’s BLEU score (Papineni et al., 2002) does not improve on this set for 4 epochs, up to a maximum ofepochs.

Original: B. I MPLEMENTING p ( d )
Revised: B. I MPLEMENTING p ( d )

Original: We implement p ( d ) using a two-layer LSTM language model (Hochreiter & Schmidhuber, 1997).
Revised: We implement p ( d ) using a two-layer LSTM language model (Hochreiter & Schmidhuber, 1997).

Original: We use an input embedding size of 128 with a hidden state size and cell size of 512.
Revised: We use an input embedding size of 128 with a hidden state size and cell size of 512.

Original: We apply dropout to non-recurrent connections ( p = . 5) during training and hold out 10% of the training dataset as a validation set and following the same early stopping procedure as in Appendix B.1, except we stop on validation loss instead of BLEU.
Revised: We apply dropout to non-recurrent connections ( p = . 5) during training and hold out 10% of the training dataset as a validation set and following the same early stopping procedure as in Appendix B.1, except we stop on validation loss instead of BLEU.

Original: C G ENERALIZATION EXPERIMENT DETAILS
Revised: C G ENERALIZATION EXPERIMENT DETAILS

Original: In each experiment, MILAN is trained with the hyperparameters described in Appendix B and Section 3.4, with the sole exception being the within-network splits—for these, we increase the early stopping criterion to require 10 epochs of no improvement to account for the training instability caused by the small training set size.
Revised: In each experiment, MILAN is trained with the hyperparameters described in Appendix B and Section 3.4, with the sole exception being the within-network splits—for these, we increase the early stopping criterion to require 10 epochs of no improvement to account for the training instability caused by the small training set size.

Original: To obtain NetDissect labels, we obtain image exemplars with the same settings as we do for MILAN , and we obtain segmentations using the full segmentation vocabulary minus the textures.
Revised: To obtain NetDissect labels, we obtain image exemplars with the same settings as we do for MILAN , and we obtain segmentations using the full segmentation vocabulary minus the textures.

Original: To obtain Compositional Explanations labels, we search for up to length 3 formulas (comprised of not , and , and or operators) with a beam size of 5 and no length penalty.
Revised: To obtain Compositional Explanations labels, we search for up to length 3 formulas (comprised of not , and , and or operators) with a beam size of 5 and no length penalty.

Original: Image region exemplars and corresponding segmentations come from the ADE20k dataset (Zhou et al., 2019).
Revised: Image region exemplars and corresponding segmentations come from the ADE20k dataset (Zhou et al., 2019).

Revised: Finally, Table 6 shows statistics for MILAN descriptions generated on the held out sets from each generalization experiment.

Revised: Compared to human annotators (Table 5), MILAN descriptions are on average shorter (2.7 vs. 4.5 tokens), use fewer unique words (1k vs. 4.6k), and contain adjectives twice as often (9.4% vs. 19.8%).

Revised: Figure 12 contains additional examples, chosen at random.

Original: D A NALYZING EXPERIMENT DETAILS
Revised: D A NALYZING EXPERIMENT DETAILS

Original: We obtain the ResNet18 model pretrained on ImageNet from torchvision (Paszke et al., 2019).
Revised: We obtain the ResNet18 model pretrained on ImageNet from torchvision (Paszke et al., 2019).

Original: We obtain neuron descriptions for the same layers that we annotate in ResNet152 (Section 3.3) using the MILAN hyperparameters described in Section 3.2 and Section 3.4.
Revised: We obtain neuron descriptions for the same layers that we annotate in ResNet152 (Section 3.3) using the MILAN hyperparameters described in Section 3.2 and Section 3.4.

Original: We obtain part of speech tags, parse trees, and word vectors for each description from spaCy (Honnibal et al., 2020).
Revised: We obtain part of speech tags, parse trees, and word vectors for each description from spaCy (Honnibal et al., 2020).

Original: Figure 12 shows examples of neurons that scored high under each criterion (and consequently were among the ﬁrst ablated in Fig. 5).
Revised: Figure 13 shows examples of neurons that scored high under each criterion (and consequently were among the ﬁrst ablated in Fig. 5).

Original: Note that these examples include some failure cases of MILAN : for example, in the # verbs example, MILAN incorrectly categorizes all brass instruments as ﬂutes; and in the # adjectives example, the description is disﬂuent.
Revised: Note that these examples include some failure cases of MILAN : for example, in the # verbs example, MILAN incorrectly categorizes all brass instruments as ﬂutes; and in the # adjectives example, the description is disﬂuent.

Original: Nevertheless, these examples conﬁrm our intuitions about the kinds of neurons selected for by each scoring criterion, as described in Section 5.
Revised: Nevertheless, these examples conﬁrm our intuitions about the kinds of neurons selected for by each scoring criterion, as described in Section 5.

Revised: We hypothesized in Section 5 that neurons scoring high on the max-word-diff criterion correspond to non-robust behavior by the model.

Revised: Figure 14 provides some evidence for this hypothesis: we construct cut-and-paste adversarial inputs in the style of Mu & Andreas (2020).

Revised: Speciﬁcally, we look at the example max-word-diff neuron shown in Figure 13, crudely copy and paste one of the objects mentioned in its description (e.g., a vehicle -related object like a half track), and show that this can cause the model to predict one of the other concepts in the description (e.g., an animal -related class like amphibian ).

Original: E E DITING EXPERIMENT DETAILS
Revised: max word difference Animals, vehicles, and vases ResNet18, layer4parse depth / caption length The center part of a flower, or radial spikes that extend from a center

Revised: ResNet18, layer3-

Revised: [Figure]

Revised: Text on a sign, text on a web page, text on a menu # adjectives

Revised: Purple, red and blue fluorescent blue and purple objects # prepositions ResNet18, layer2Space on the right side of a object, space above a bird

Revised: ResNet18, conv

Revised: [Figure]

Revised: ResNet

Revised: [Figure]

Revised: A man holding a flute, a flute, a man playing a flute

Revised: ResNet

Original: [Figure]
Revised: [Figure]

Revised: layer

Revised: [Figure]

Original:  Hyperparameters We train a randomly initialized ResNet18 on the spurious training dataset for a maximum of 100 epochs with a learning rate of 1e-4 and a minibatch size of 128.
Revised: [Figure] E E DITING EXPERIMENT DETAILS Hyperparameters We train a randomly initialized ResNet18 on the spurious training dataset for a maximum of 100 epochs with a learning rate of 1e-4 and a minibatch size of 128.

Original: We annotate the same convolutional and residual units we did for ResNet152 in Section 3.3.
Revised: We annotate the same convolutional and residual units we did for ResNet152 in Section 3.3.

Original: We stop training when validation loss does not improve for 4 epochs.
Revised: We stop training when validation loss does not improve for 4 epochs.

Original: How many neurons should we remove?
Revised: How many neurons should we remove?

Original: In practice, we cannot incrementally test our model on an adversarial set.
Revised: In practice, we cannot incrementally test our model on an adversarial set.

Original: So how do we decide on the number of neurons to zero?
Revised: So how do we decide on the number of neurons to zero?

Original: One option is to look solely at validation accuracy.
Revised: One option is to look solely at validation accuracy.

Original: Figure 13 recreates Figure 8 with accuracy on the held out validation set (which is distributed like the training dataset) instead of accuracy on the adversarial test set.
Revised: Figure 15 recreates Figure 8 with accuracy on the held out validation set (which is distributed like the training dataset) instead of accuracy on the adversarial test set.

Original: The accuracy starts peaks and starts decreasing earlier than in Fig. 8, but if we were to choose the number to be the largest before validation accuracy permanently decreases , we would choose 8 neurons, which would still result in a 3.1% increase in adversarial accuracy. 
Revised: The accuracy starts peaks and starts decreasing earlier than in Fig. 8, but if we were to choose the number to be the largest before validation accuracy permanently decreases , we would chooseneurons, which would still result in a 3.1% increase in adversarial accuracy. [Figure]