Original: W HAT DO V ISION T RANSFORMERS L EARN ? A V ISUAL E XPLORATION
Revised: W HAT DO V ISION T RANSFORMERS L EARN ? A V ISUAL E XPLORATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Vision transformers (ViTs) are quickly becoming the de-facto architecture for computer vision, yet we understand very little about why they work and what they learn.
Revised: Vision transformers (ViTs) are quickly becoming the de-facto architecture for computer vision, yet we understand very little about why they work and what they learn.

Original: While existing studies visually analyze the mechanisms of convolutional neural networks, an analogous exploration of ViTs remains challenging.
Revised: While existing studies visually analyze the mechanisms of convolutional neural networks, an analogous exploration of ViTs remains challenging.

Original: In this paper, we first address the obstacles to performing visualizations on ViTs.
Revised: In this paper, we first address the obstacles to performing visualizations on ViTs.

Original: Assisted by these solutions, we observe that neurons in ViTs trained with language model supervision (e.g., CLIP) are activated by semantic concepts rather than visual features.
Revised: Assisted by these solutions, we observe that neurons in ViTs trained with language model supervision (e.g., CLIP) are activated by semantic concepts rather than visual features.

Original: We also explore the underlying differences between ViTs and CNNs, and we find that transformers detect image background features, just like their convolutional counterparts, but their predictions depend far less on high-frequency information.
Revised: We also explore the underlying differences between ViTs and CNNs, and we find that transformers detect image background features, just like their convolutional counterparts, but their predictions depend far less on high-frequency information.

Original: On the other hand, both architecture types behave similarly in the way features progress from abstract patterns in early layers to concrete objects in late layers.
Revised: On the other hand, both architecture types behave similarly in the way features progress from abstract patterns in early layers to concrete objects in late layers.

Original: In addition, we show that ViTs maintain spatial information in all layers except the final layer.
Revised: In addition, we show that ViTs maintain spatial information in all layers except the final layer.

Original: In contrast to previous works, we show that the last layer most likely discards the spatial information and behaves as a learned global pooling operation.
Revised: In contrast to previous works, we show that the last layer most likely discards the spatial information and behaves as a learned global pooling operation.

Original: Finally, we conduct large-scale visualizations on a wide range of ViT variants, including DeiT, CoaT, ConViT, PiT, Swin, and Twin, to validate the effectiveness of our method.
Revised: Finally, we conduct large-scale visualizations on a wide range of ViT variants, including DeiT, CoaT, ConViT, PiT, Swin, and Twin, to validate the effectiveness of our method.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent years have seen the rapid proliferation of vision transformers (ViTs) across a diverse range of tasks from image classification to semantic segmentation to object detection (Dosovitskiy et al., 2020;
Revised: Recent years have seen the rapid proliferation of vision transformers (ViTs) across a diverse range of tasks from image classification to semantic segmentation to object detection (Dosovitskiy et al., 2020;

Original: He et al., 2021; Dong et al., 2021;
Revised: He et al., 2021; Dong et al., 2021;

Original: Liu et al., 2021; Zhai et al., 2021; Dai et al., 2021).
Revised: Liu et al., 2021; Zhai et al., 2021; Dai et al., 2021).

Original: Despite their enthusiastic adoption and the constant introduction of architectural innovations, little is known about the inductive biases or features they tend to learn.
Revised: Despite their enthusiastic adoption and the constant introduction of architectural innovations, little is known about the inductive biases or features they tend to learn.

Original: While feature visualizations and image reconstructions have provided a looking glass into the workings of CNNs (Olah et al., 2017; Zeiler &
Revised: While feature visualizations and image reconstructions have provided a looking glass into the workings of CNNs (Olah et al., 2017; Zeiler &

Original: Fergus, 2014; Dosovitskiy & Brox, 2016), these methods have shown less success for understanding
Revised: Fergus, 2014; Dosovitskiy & Brox, 2016), these methods have shown less success for understanding

Original: ViT representations, which are difficult to visualize.
Revised: ViT representations, which are difficult to visualize.

Original: In this work we show that, if properly applied to the correct representations, feature visualizations can indeed succeed on VITs.
Revised: In this work we show that, if properly applied to the correct representations, feature visualizations can indeed succeed on VITs.

Original: This insight allows us to visually explore ViTs and the information they glean from images.
Revised: This insight allows us to visually explore ViTs and the information they glean from images.

Original: Edges
Revised: Edges

Original: Textures
Revised: Textures

Original: Patterns
Revised: Patterns

Original: Parts
Revised: Parts

Original: Objects
Revised: Objects

Original: In order to investigate the behaviors of vision transformers, we first establish a visualization framework that incorporates improved techniques for synthesizing images that maximally activate neurons.
Revised: In order to investigate the behaviors of vision transformers, we first establish a visualization framework that incorporates improved techniques for synthesizing images that maximally activate neurons.

Original: Through dissecting and visualizing the internal representations in the transformer architecture, we find that patch tokens preserve spatial information, even in individual channels, throughout all layers except the last attention block.
Revised: Through dissecting and visualizing the internal representations in the transformer architecture, we find that patch tokens preserve spatial information, even in individual channels, throughout all layers except the last attention block.

Original: The last layer of ViTs learns a token-mixing operation akin to average pooling, such that the classification head exhibits comparable accuracy when ingesting a random token instead of the CLS token.
Revised: The last layer of ViTs learns a token-mixing operation akin to average pooling, such that the classification head exhibits comparable accuracy when ingesting a random token instead of the CLS token.

Original: After probing the role of spatial information, we delve into the behavioral differences between ViTs and CNNs.
Revised: After probing the role of spatial information, we delve into the behavioral differences between ViTs and CNNs.

Original: When performing activation maximizing visualizations, we notice that ViTs consistently generate higher quality image backgrounds than CNNs.
Revised: When performing activation maximizing visualizations, we notice that ViTs consistently generate higher quality image backgrounds than CNNs.

Original: Thus, we try masking out image foregrounds during inference, and find that ViTs consistently outperform CNNs when exposed only to image backgrounds.
Revised: Thus, we try masking out image foregrounds during inference, and find that ViTs consistently outperform CNNs when exposed only to image backgrounds.

Original: These findings bolster the observation that transformer models extract information from many sources in an image to exhibit superior performance on out-of-distribution generalization (Paul & Chen, 2021) as well as adversarial robustness (Shao et al., 2021).
Revised: These findings bolster the observation that transformer models extract information from many sources in an image to exhibit superior performance on out-of-distribution generalization (Paul & Chen, 2021) as well as adversarial robustness (Shao et al., 2021).

Original: Additionally, convolutional neural networks are known to rely heavily on high-frequency texture information in images (Geirhos et al., 2018).
Revised: Additionally, convolutional neural networks are known to rely heavily on high-frequency texture information in images (Geirhos et al., 2018).

Original: In contrast, we find that ViTs perform well even when high-frequency content is removed from their inputs.
Revised: In contrast, we find that ViTs perform well even when high-frequency content is removed from their inputs.

Original: We further visualize the effects of language model supervision, i.e. CLIP (Radford et al., 2021), on the features extracted by vision transformers.
Revised: We further visualize the effects of language model supervision, i.e. CLIP (Radford et al., 2021), on the features extracted by vision transformers.

Original: While both ImageNet-trained ViTs and CLIP-trained vision transformers possess neurons that are activated by visual features (e.g. shapes and colors) and distinct classes, the neurons of CLIP-trained vision transformers are also activated by features that do not represent physical objects, such as visual characteristics relating to parts of speech (e.g. epithets, adjectives, and prepositions) or broader concepts such as morbidity.
Revised: While both ImageNet-trained ViTs and CLIP-trained vision transformers possess neurons that are activated by visual features (e.g. shapes and colors) and distinct classes, the neurons of CLIP-trained vision transformers are also activated by features that do not represent physical objects, such as visual characteristics relating to parts of speech (e.g. epithets, adjectives, and prepositions) or broader concepts such as morbidity.

Original: Our contributions are summarized as follows:
Revised: Our contributions are summarized as follows:

Original: I. We observe that uninterpretable and adversarial behavior occurs when applying standard methods of feature visualization to the relatively low-dimensional components of transformer-based models, such as keys
Revised: I. We observe that uninterpretable and adversarial behavior occurs when applying standard methods of feature visualization to the relatively low-dimensional components of transformer-based models, such as keys

Original: , queries, or values.
Revised: , queries, or values.

Original: However, applying these tools to the relatively high-dimensional features of the position-wise feedforward layer results in successful and informative visualizations.
Revised: However, applying these tools to the relatively high-dimensional features of the position-wise feedforward layer results in successful and informative visualizations.

Original: We conduct large-scale visualizations on a wide range of transformer-based vision models, including
Revised: We conduct large-scale visualizations on a wide range of transformer-based vision models, including

Original: ViTs, DeiT, CoaT, ConViT, PiT, Swin, and Twin, to validate the effectiveness of our method.
Revised: ViTs, DeiT, CoaT, ConViT, PiT, Swin, and Twin, to validate the effectiveness of our method.

Original: II.
Revised: II.

Original: We show that patch-wise image activation patterns for ViT features essentially behave like saliency maps, highlighting the regions of the image a given feature attends to.
Revised: We show that patch-wise image activation patterns for ViT features essentially behave like saliency maps, highlighting the regions of the image a given feature attends to.

Original: This behavior persists even for relatively deep layers, showing the model preserves the positional relationship between patches instead of using them as global information stores.
Revised: This behavior persists even for relatively deep layers, showing the model preserves the positional relationship between patches instead of using them as global information stores.

Original: III.
Revised: III.

Original: We compare the behavior of ViTs and CNNs, finding that ViTs make better use of background information and rely less on high-frequency, textural attributes.
Revised: We compare the behavior of ViTs and CNNs, finding that ViTs make better use of background information and rely less on high-frequency, textural attributes.

Original: Both types of networks build progressively more complex representations in deeper layers and eventually contain features responsible for detecting distinct objects.
Revised: Both types of networks build progressively more complex representations in deeper layers and eventually contain features responsible for detecting distinct objects.

Original: IV.
Revised: IV.

Original: We investigate the effect of natural language supervision with CLIP on the types of features extracted by ViTs.
Revised: We investigate the effect of natural language supervision with CLIP on the types of features extracted by ViTs.

Original: We find CLIP-trained models include various features clearly catered to detecting components of images corresponding to caption text, such as prepositions, adjectives, and conceptual categories.
Revised: We find CLIP-trained models include various features clearly catered to detecting components of images corresponding to caption text, such as prepositions, adjectives, and conceptual categories.

Original:  R ELATED W ORK 2. O PTIMIZATION -B ASED V ISUALIZATION
Revised:  R ELATED W ORK 2. O PTIMIZATION -B ASED V ISUALIZATION

Original: One approach to understanding what models learn during training is using gradient descent to produce an image which conveys information about the inner workings of the model.
Revised: One approach to understanding what models learn during training is using gradient descent to produce an image which conveys information about the inner workings of the model.

Original: This has proven to be a fruitful line of work in the case of understanding CNNs specifically.
Revised: This has proven to be a fruitful line of work in the case of understanding CNNs specifically.

Original: The basic strategy underlying this approach is to optimize over input space to find an image which maximizes a particular attribute of the model.
Revised: The basic strategy underlying this approach is to optimize over input space to find an image which maximizes a particular attribute of the model.

Original: For example, Erhan et al.
Revised: For example, Erhan et al.

Original: (2009) use this approach to visualize images which maximally activate specific neurons in early layers of a network, and Olah et al.
Revised: (2009) use this approach to visualize images which maximally activate specific neurons in early layers of a network, and Olah et al.

Original: (2017) extend this to neurons, channels, and layers throughout a network.
Revised: (2017) extend this to neurons, channels, and layers throughout a network.

Original: Simonyan et al.
Revised: Simonyan et al.

Original: Yin et al.
Revised: Yin et al.

Original: (2020) produce images which maximize the score a model assigns to a particular class.
Revised: (2020) produce images which maximize the score a model assigns to a particular class.

Original: Mahendran & Vedaldi (2015) apply a similar method to invert the feature representations of particular image examples.
Revised: Mahendran & Vedaldi (2015) apply a similar method to invert the feature representations of particular image examples.

Original: Recent work Ghiasi et al.
Revised: Recent work Ghiasi et al.

Original: (2021) has studied techniques for extending optimization-based class visualization to ViTs.
Revised: (2021) has studied techniques for extending optimization-based class visualization to ViTs.

Original: We incorporate and adapt some of these proposed techniques into our scheme for feature visualization.
Revised: We incorporate and adapt some of these proposed techniques into our scheme for feature visualization.

Original: 2. O THER V ISUALIZATION A PPROACHES
Revised: 2. O THER V ISUALIZATION A PPROACHES

Original: Aside from optimization-based methods, many other ways to visualize CNNs have been proposed.
Revised: Aside from optimization-based methods, many other ways to visualize CNNs have been proposed.

Original: Dosovitskiy & Brox (2016) train an auxiliary model to invert the feature representations of a CNN.
Revised: Dosovitskiy & Brox (2016) train an auxiliary model to invert the feature representations of a CNN.

Original: Zeiler & Fergus (2014) use ‘deconvnets’ to visualize patches which strongly activate features in various layers.
Revised: Zeiler & Fergus (2014) use ‘deconvnets’ to visualize patches which strongly activate features in various layers.

Original: Simonyan et al.
Revised: Simonyan et al.

Original: (2014) introduce saliency maps, which use gradient information to identify what parts of an image are important to the model’s classification output.
Revised: (2014) introduce saliency maps, which use gradient information to identify what parts of an image are important to the model’s classification output.

Original: Zimmermann et al.
Revised: Zimmermann et al.

Original: (2021) demonstrate that natural image samples which maximally activate a feature in a CNN may be more informative than generated images which optimize that feature.
Revised: (2021) demonstrate that natural image samples which maximally activate a feature in a CNN may be more informative than generated images which optimize that feature.

Original: We draw on some aspects of these approaches and find that they are useful for visualizing ViTs as well.
Revised: We draw on some aspects of these approaches and find that they are useful for visualizing ViTs as well.

Original: 2. U NDERSTANDING V I T S
Revised: 2. U NDERSTANDING V I T S

Original: Given their rapid proliferation, there is naturally great interest in how ViTs work and how they may differ from CNNs.
Revised: Given their rapid proliferation, there is naturally great interest in how ViTs work and how they may differ from CNNs.

Original: Although direct visualization of their features has not previously been explored, there has been recent progress in analyzing the behavior of ViTs.
Revised: Although direct visualization of their features has not previously been explored, there has been recent progress in analyzing the behavior of ViTs.

Original: Paul & Chen (2021); Naseer et al.
Revised: Paul & Chen (2021); Naseer et al.

Original: (2021); Shao et al.
Revised: (2021); Shao et al.

Original: (2021) demonstrate that ViTs are inherently robust to many kinds of adversarial perturbations and corruptions.
Revised: (2021) demonstrate that ViTs are inherently robust to many kinds of adversarial perturbations and corruptions.

Original: Raghu et al.
Revised: Raghu et al.

Original: (2021) compare how the internal representation structure and use of spatial information differs between ViTs and CNNs.
Revised: (2021) compare how the internal representation structure and use of spatial information differs between ViTs and CNNs.

Original: Chefer et al.
Revised: Chefer et al.

Original: (2021) produce ‘image relevance maps’ (which resemble saliency maps) to promote interpretability of ViTs.
Revised: (2021) produce ‘image relevance maps’ (which resemble saliency maps) to promote interpretability of ViTs.

Revised: Park & Kim (2022) demonstrates that multi-head self-attention are low-pass filters, but Convs are high-pass filters.

Revised: They also examined the effect of removing a specific frequency and concluded that ViTs are resistant to high-frequency removal and CNNs are resistant to low-frequency removal.

Original:  V I T F EATURE V ISUALIZATION
Revised:  V I T F EATURE V ISUALIZATION

Original: Like many visualization techniques, we take gradient steps to maximize feature activations starting from random noise (Olah et al., 2017).
Revised: Like many visualization techniques, we take gradient steps to maximize feature activations starting from random noise (Olah et al., 2017).

Original: To improve the quality of our images, we penalize total variation (Mahendran & Vedaldi, 2015), and also employ the Jitter augmentation (Yin et al., 2020), the ColorShift augmentation, and augmentation ensembling (Ghiasi et al., 2021).
Revised: To improve the quality of our images, we penalize total variation (Mahendran & Vedaldi, 2015), and also employ the Jitter augmentation (Yin et al., 2020), the ColorShift augmentation, and augmentation ensembling (Ghiasi et al., 2021).

Original: Finally, we find that Gaussian smoothing facilitates better visualization in our experiments as is common in feature visualization (Smilkov et al., 2017; Cohen et al., 2019).
Revised: Finally, we find that Gaussian smoothing facilitates better visualization in our experiments as is common in feature visualization (Smilkov et al., 2017; Cohen et al., 2019).

Original: Each of the above techniques can be formalized as follows.
Revised: Each of the above techniques can be formalized as follows.

Original: A ViT represents each patch p (of an input x ) at layer l by an array A l,p with d entries.
Revised: A ViT represents each patch p (of an input x ) at layer l by an array A l,p with d entries.

Original: We define a feature vector f to be a stack composed of one entry from each of these arrays.
Revised: We define a feature vector f to be a stack composed of one entry from each of these arrays.

Original: Let f l,i be formed by concatenating the i th entry in A l,p for all patches p .
Revised: Let f l,i be formed by concatenating the i th entry in A l,p for all patches p .

Original: This vector f will have dimension equal to the number of patches.
Revised: This vector f will have dimension equal to the number of patches.

Original: The optimization objective starts by maximizing the sum of the entries of f over inputs x .
Revised: The optimization objective starts by maximizing the sum of the entries of f over inputs x .

Original: The main loss is then
Revised: The main loss is then

Original: [Equation]
Revised: [Equation]

Original: We employ total variation regularization by adding the term λTV ( x ) to the objective.
Revised: We employ total variation regularization by adding the term λTV ( x ) to the objective.

Original: TV represents the total variation, and λ is the hyperparameter controlling the strength of its regularization effect.
Revised: TV represents the total variation, and λ is the hyperparameter controlling the strength of its regularization effect.

Original: We can ensemble augmentations of the input to further improve results.
Revised: We can ensemble augmentations of the input to further improve results.

Original: Let A define a distribution of augmentations to be applied to the input image x , and let a be a sample from A .
Revised: Let A define a distribution of augmentations to be applied to the input image x , and let a be a sample from A .

Original: To create a minibatch of inputs from a single image, we sample several augmentations { a k } from A .
Revised: To create a minibatch of inputs from a single image, we sample several augmentations { a k } from A .

Original: Finally, the optimization problem is:
Revised: Finally, the optimization problem is:

Original: [Equation]
Revised: [Equation]

Original:  We achieve the best visualizations when A is GS ( CS ( Jitter ( x ))) , where GS denotes Gaussian smoothing and CS denotes ColorShift, whose formulas are:[Equation] [Figure]
Revised: [Figure] We achieve the best visualizations when A is GS ( CS ( Jitter ( x ))) , where GS denotes Gaussian smoothing and CS denotes ColorShift, whose formulas are:

Original: [Equation]
Revised: [Equation]

Original: Note that even though ϵ and µ are both additive noise, they act on the input differently since µ is applied per channel (i.e. has dimension three), and ϵ is applied per pixel.
Revised: Note that even though ϵ and µ are both additive noise, they act on the input differently since µ is applied per channel (i.e. has dimension three), and ϵ is applied per pixel.

Original: For more details on hyperparameters, refer to Appendix B.
Revised: For more details on hyperparameters, refer to Appendix B.

Original: To better understand the content of a visualized feature, we pair every visualization with images from the ImageNet validation/train set that most strongly activate the relevant feature.
Revised: To better understand the content of a visualized feature, we pair every visualization with images from the ImageNet validation/train set that most strongly activate the relevant feature.

Original: Moreover, we plot the feature’s activation pattern by passing the most activating images through the network and showing the resulting pattern of feature activations.
Revised: Moreover, we plot the feature’s activation pattern by passing the most activating images through the network and showing the resulting pattern of feature activations.

Original: Figure 2(a) is an example of such a visualization.
Revised: Figure 2(a) is an example of such a visualization.

Original: From the leftmost panel, we hypothesize that this feature corresponds to gravel.
Revised: From the leftmost panel, we hypothesize that this feature corresponds to gravel.

Original: The most activating image from the validation set (middle) contains a lizard on a pebbly gravel road.
Revised: The most activating image from the validation set (middle) contains a lizard on a pebbly gravel road.

Original: Interestingly, the gravel background lights up in the activation pattern (right), while the lizard does not.
Revised: Interestingly, the gravel background lights up in the activation pattern (right), while the lizard does not.

Original: The activation pattern in this example behaves like a saliency map (Simonyan et al., 2014), and we explore this phenomenon across different layers of the network further in Section 4. 
Revised: The activation pattern in this example behaves like a saliency map (Simonyan et al., 2014), and we explore this phenomenon across different layers of the network further in Section 4. [Figure] [Equation] [Figure]

Revised: [Equation]

Revised: [Figure]

Original: The model we adopt for the majority of our demonstrations throughout the paper is ViT-B16, implemented based on the work of Dosovitskiy et al.
Revised: [Equation] [Figure] [Equation] [Figure]

Revised: The visualization both fails to extract interpretable features and to distinguish between early and deep layers.

Revised: High-frequency patterns

Revised: The model we adopt for the majority of our demonstrations throughout the paper is

Revised: ViT-B16, implemented based on the work of Dosovitskiy et al.

Original: In addition, in the Appendix, we conduct large-scale visualizations on a wide range of ViT variants, including DeiT Touvron et al.
Revised: In addition, in the Appendix, we conduct large-scale visualizations on a wide range of ViT variants, including DeiT Touvron et al.

Original: (2021a),
Revised: (2021a),

Original: CoaT Xu et al.
Revised: CoaT Xu et al.

Original: (2021), ConViT d’Ascoli et al.
Revised: (2021), ConViT d’Ascoli et al.

Original: (2021), PiT Heo et al.
Revised: (2021), PiT Heo et al.

Original: , Swin Liu et al.
Revised: Swin Liu et al.

Original: (2021), and Twin Chu et al.
Revised: (2021), and Twin Chu et al.

Original: (2021), 38 models in total, to validate the effectiveness of our method.
Revised: (2021), 38 models in total, to validate the effectiveness of our method.

Original: ViT-B16 is composed of 12 blocks, each consisting of multi-headed attention layers, followed by a projection layer for mixing attention heads, and finally followed by a position-wise-feed-forward layer.
Revised: ViT-B16 is composed of 12 blocks, each consisting of multi-headed attention layers, followed by a projection layer for mixing attention heads, and finally followed by a positionwise-feed-forward layer.

Original: For brevity, we henceforth refer to the position-wise-feed-forward layer simply as the feedforward layer.
Revised: For brevity, we henceforth refer to the position-wise-feedforward layer simply as the feed-forward layer.

Original: In this model, every patch is always represented by a vector of size 768 except in the feed-forward layer which has a size of 3072 ( 4 times larger than other layers).
Revised: In this model, every patch is always represented by a vector of size 768 except in the feed-forward layer which has a size of 3072 ( 4 times larger than other layers).

Original: We first attempt to visualize features of the multi-headed attention layer, including visualization of the keys, queries, and values, by performing activation maximization.
Revised: We first attempt to visualize features of the multi-headed attention layer, including visualization of the keys, queries, and values, by performing activation maximization.

Revised: We find that the visualized feed-forward features are significantly more interpretable than other layers.

Original: We find that the visualized
Revised: We attribute this difficulty

Original: [Figure]
Revised: [Figure]

Original: feed-forward features are significantly more interpretable than other layers. We attribute this difficulty of visualizing other layers to the property that ViTs pack a tremendous amount of information into only 768 features, (e.g. in keys, queries, and values) which then behave similar to multi-modal neurons, as discussed by Goh et al.
Revised: of visualizing other layers to the property that ViTs pack a tremendous amount of information into only 768 features, (e.g. in keys, queries, and values) which then behave similar to multi-modal neurons, as discussed by Goh et al.

Original: (2021), due to many semantic concepts being encoded in a low dimensional space.
Revised: (2021), due to many semantic concepts being encoded in a low dimensional space.

Original: Furthermore, we find that this behaviour is more extreme in deeper layers.
Revised: Furthermore, we find that this behaviour is more extreme in deeper layers.

Original: See
Revised: See

Original: Figure 3 for examples of visualizations of keys, queries and values in both early and deep layers of the ViT. Inspired by these observations, we visualize the features within the feed-forward layer across all 12 blocks of the ViT. We refer to these blocks interchangeably as layers. 
Revised: Figure 3 for examples of visualizations of keys, queries and values in both early and deep layers of the ViT. Inspired by these observations, we visualize the features within the feed-forward layer across all 12 blocks of the ViT. We refer to these blocks interchangeably as layers. [Figure]

Revised: increases the dimension of the feature space, and the second one brings it back to its initial dimension.

Original: The feed-forward layer depicted in Figure 3 takes an input of size d = 768 , projects it into a t = 4 times higher dimensional space, applies the non-linearity GELU, and then projects back to d dimensional space.
Revised: The feed-forward layer depicted in Figure 4 takes an input of size d = 768 , projects it into a t = 4 times higher dimensional space, applies the non-linearity GELU, and then projects back to d dimensional space.

Original: Unless otherwise stated, we always visualize the output of the GELU layers in our experiments.
Revised: Unless otherwise stated, we always visualize the output of the GELU layers in our experiments.

Original: We hypothesize that the network exploits these high-dimensional spaces to store relatively disentangled representations.
Revised: We hypothesize that the network exploits these high-dimensional spaces to store relatively disentangled representations.

Original: On the other hand, compressing the features into a lower dimensional space may result in the jumbling of features, yielding uninterpretable visualizations.
Revised: On the other hand, compressing the features into a lower dimensional space may result in the jumbling of features, yielding uninterpretable visualizations.

Original:  L AST -L AYER T OKEN M IXING
Revised:  L AST -L AYER T OKEN M IXING

Original: In this section, we investigate the preservation of patch-wise spatial information observed in the visualizations of patch-wise feature activation levels which, as noted before, bear some similarity to saliency maps.
Revised: In this section, we investigate the preservation of patch-wise spatial information observed in the visualizations of patch-wise feature activation levels which, as noted before, bear some similarity to saliency maps.

Original: Figure 2(a) demonstrates this phenomenon in layer 5, where the visualized feature is strongly activated for almost all rocky patches but not for patches that include the lizard.
Revised: Figure 2(a) demonstrates this phenomenon in layer 5, where the visualized feature is strongly activated for almost all rocky patches but not for patches that include the lizard.

Original: Additional examples can be seen in Figure 4 and the Appendix, where the activation maps approximately segment the image with respect to some relevant aspect of the image.
Revised: Additional examples can be seen in Figure 5 and the Appendix, where the activation maps approximately segment the image with respect to some relevant aspect of the image.

Original: We find it surprising that even though every patch can influence the representation of every other patch, these representations remain local, even for individual channels in deep layers in the network.
Revised: We find it surprising that even though every patch can influence the representation of every other patch, these representations remain local, even for individual channels in deep layers in the network.

Original: While a similar finding for CNNs, whose neurons may have a limited receptive field, would be unsurprising, even neurons in the first layer of a ViT have a complete receptive field.
Revised: While a similar finding for CNNs, whose neurons may have a limited receptive field, would be unsurprising, even neurons in the first layer of a ViT have a complete receptive field.

Original: In other words, ViTs learn to preserve spatial information, despite lacking the inductive bias of CNNs.
Revised: In other words, ViTs learn to preserve spatial information, despite lacking the inductive bias of CNNs.

Original: Spatial information in patches of deep layers has been explored in Raghu et al.
Revised: Spatial information in patches of deep layers has been explored in Raghu et al.

Original: (2021) through the CKA similarity measure, and
Revised: (2021) through the CKA similarity measure, and

Original: we further show that spatial information is in fact present in individual channels.
Revised: we further show that spatial information is in fact present in individual channels.

Original: The last layer of the network, however, departs from this behavior and instead appears to serve a role similar to average pooling.
Revised: The last layer of the network, however, departs from this behavior and instead appears to serve a role similar to average pooling.

Original: Figure 2(b) shows one example of our visualizations for a feature from the last layer that is activated by shopping carts.
Revised: Figure 2(b) shows one example of our visualizations for a feature from the last layer that is activated by shopping carts.

Original: The activation pattern is fairly uniform across the image.
Revised: The activation pattern is fairly uniform across the image.

Original: For classification purposes, ViTs use a fully connected layer applied only on the class token (the CLS token).
Revised: For classification purposes, ViTs use a fully connected layer applied only on the class token (the CLS token).

Original: It is possible that the network globalizes information in the last layer to ensure that the CLS token has access to the entire image, but because the CLS token is treated the same as every other patch by the transformer, this seems to be achieved by globalizing across all tokens.
Revised: It is possible that the network globalizes information in the last layer to ensure that the CLS token has access to the entire image, but because the CLS token is treated the same as every other patch by the transformer, this seems to be achieved by globalizing across all tokens.

Revised: [Table]

Original: Based on the preservation of spatial information in patches, we hypothesize that the CLS token plays a relatively minor role throughout the network and is not used for globalization until the last layer.
Revised: Based on the preservation of spatial information in patches, we hypothesize that the CLS token plays a relatively minor role throughout the network and is not used for globalization until the last layer.

Original: To demonstrate this, we perform inference on images without using the CLS token in layers 1-11,[Figure] [Table] meaning that in these layers, each patch only attends to other patches and not to the CLS token.
Revised: To demonstrate this, we perform inference on images without using the CLS token in layers 1-11, meaning that in these layers, each patch only attends to other patches and not to the CLS token.

Original: At layer 12, we then insert a value for the CLS token so that other patches can attend to it and vice versa.
Revised: At layer 12, we then insert a value for the CLS token so that other patches can attend to it and vice versa.

Original: This value is obtained by running a forward pass using  the CLS token and no image patches; this value is constant across all input 
Revised: This value is obtained by running a forward pass using only the CLS token and no image patches; this value is constant across all input images.

Original: The resulting hacked network that only has CLS access in the last layer can still successfully classify 78 .
Revised: The resulting hacked network that only has CLS access in the last layer can still successfully classify 78 .

Original: 61% of the ImageNet validation set as shown in Table 1.
Revised: 61% of the ImageNet validation set as shown in Table 1.

Original: From this result, we conclude that the CLS token captures global information mostly at the last layer, rather than building a global representation throughout the network.
Revised: From this result, we conclude that the CLS token captures global information mostly at the last layer, rather than building a global representation throughout the network.

Original: [Figure] [Figure]
Revised: ViT-B       on the validation set when we apply the classification head trained to classify images on the top of the CLS token to the other patches.

Original: We perform a second experiment to show this lastlayer globalization behaviour is not exclusive to the
Revised: We perform a second experiment to show this lastlayer globalization behaviour is not exclusive to the

Original: CLS token, but actually occurs across every patch in the last layer.
Revised: CLS token, but actually occurs across every patch in the last layer.

Original: We take the fully connected layer trained to classify images on top of the CLS token, and without any fine-tuning or adaptation , we apply it to each patch, one at a time.
Revised: We take the fully connected layer trained to classify images on top of the CLS token, and without any fine-tuning or adaptation , we apply it to each patch, one at a time.

Original: This setup still successfully classifies 75 .
Revised: This setup still successfully classifies 75 .

Original: 75% of the validation set, on average across individual patches, and the patch with the maximum performance achieves 80 .
Revised: 75% of the validation set, on average across individual patches, and the patch with the maximum performance achieves 80 .

Original: 16% accuracy (see Table 1), further confirming that the last layer performs a token mixing operation so that all tokens contain roughly identical information.
Revised: 16% accuracy (see Table 1), further confirming that the last layer performs a token mixing operation so that all tokens contain roughly identical information.

Original: Figurecontains a heat-map depicting the performance of this setup across spatial patches.
Revised: Figurecontains a heat-map depicting the performance of this setup across spatial patches.

Original: This observation stands in stark contrast to the suggestions of Raghu et al.
Revised: This observation stands in stark contrast to the suggestions of Raghu et al.

Original: (2021) that ViTs possess strong localization throughout the entire network, and their further hypothesis that the addition of global pooling is required for mixing tokens at the end of the network.
Revised: (2021) that ViTs possess strong localization throughout the entire network, and their further hypothesis that the addition of global pooling is required for mixing tokens at the end of the network.

Original: We conclude by noting that the information structure of a ViT is remarkably similar to a CNN, in the sense that the information is positionally encoded and preserved until the final layer.
Revised: We conclude by noting that the information structure of a ViT is remarkably similar to a CNN, in the sense that the information is positionally encoded and preserved until the final layer.

Original: Furthermore, the final layer in ViTs appears to behave as a learned global pooling operation that aggregates information from all patches, which is similar to its explicit average-pooling counterpart in CNNs.
Revised: Furthermore, the final layer in ViTs appears to behave as a learned global pooling operation that aggregates information from all patches, which is similar to its explicit average-pooling counterpart in CNNs.

Original:  C OMPARISON OF V I T S AND CNN S
Revised:  C OMPARISON OF V I T S AND CNN S

Original: As extensive work has been done to understand the workings of convolutional networks, including similar feature visualization and image reconstruction techniques to those used here, we may be able to learn more about ViT behavior via direct comparison to CNNs.
Revised: As extensive work has been done to understand the workings of convolutional networks, including similar feature visualization and image reconstruction techniques to those used here, we may be able to learn more about ViT behavior via direct comparison to CNNs.

Original: An important observation is thatin CNNs, early layers recognize color, edges, and texture, while deeper layers pick out increasingly complex structures eventually leading to entire objects (Olah et al., 2017). Visualization of features from different layers in a ViT, such as those in Figures 1 and 6, reveal that ViTs exhibit this kind of progressive specialization as well.
Revised: An important observation is that

Original: [Figure]
Revised: [Figure]

Original: Parts
Revised: Parts

Original: Object
Revised: Object

Original: L5 F
Revised: L5 F

Original: L6 F
Revised: L6 F

Original: L7 F L7 F
Revised: L7 F L7 F

Original: L9 F
Revised: L9 F

Original: L10 F
Revised: L10 F

Original: O u r s
Revised: O u r s

Original: V a l
Revised: V a l

Original: T r a i n 
Revised: T r a i n in CNNs, early layers recognize color, edges, and texture, while deeper layers pick out increasingly complex structures eventually leading to entire objects (Olah et al., 2017). Visualization of features from different layers in a ViT, such as those in Figures 1 and 7, reveal that ViTs exhibit this kind of progressive specialization as well.

Original: On the other hand, we observe that there are also important differences between the ways CNNs and
Revised: On the other hand, we observe that there are also important differences between the ways CNNs and

Original: ViTs recognize images.
Revised: ViTs recognize images.

Original: In particular, we examine the reliance of ViTs and CNNs on background and foreground image features using the bounding boxes provided by ImageNet Deng et al.
Revised: In particular, we examine the reliance of ViTs and CNNs on background and foreground image features using the bounding boxes provided by ImageNet Deng et al.

Original: We filter the ImageNet-1k training images and only use those which are accompanied by bounding boxes.
Revised: We filter the ImageNet-1k training images and only use those which are accompanied by bounding boxes.

Original: If several objects are present in an image, we only take the bounding boxes corresponding to the true class label and ignore the additional bounding boxes.
Revised: If several objects are present in an image, we only take the bounding boxes corresponding to the true class label and ignore the additional bounding boxes.

Original: Figure 7(b) shows an example of an image and variants in which the background and foreground, respectively, are masked.
Revised: Figure 8(b) shows an example of an image and variants in which the background and foreground, respectively, are masked.

Original: [Figure]
Revised: [Figure]

Original: Figure 7(a) displays an example of ViTs’ ability to detect background information present in the
Revised: Figure 8(a) displays an example of ViTs’ ability to detect background information present in the

Original: ImageNet dataset.
Revised: ImageNet dataset.

Original: This particular feature appears responsible for recognizing the pairing of grass and snow.
Revised: This particular feature appears responsible for recognizing the pairing of grass and snow.

Original: The rightmost panel indicates that this feature is solely activated by the background, and not at all by the patches of the image containing parts of the wolf.
Revised: The rightmost panel indicates that this feature is solely activated by the background, and not at all by the patches of the image containing parts of the wolf.

Original: To quantitatively assess each architecture’s dependence on different parts of the image on the dataset level, we mask out the foreground or background on a set of evaluation images using the aforementioned ImageNet bounding boxes, and we measure the resulting change in top-5 accuracy.
Revised: To quantitatively assess each architecture’s dependence on different parts of the image on the dataset level, we mask out the foreground or background on a set of evaluation images using the aforementioned ImageNet bounding boxes, and we measure the resulting change in top-5 accuracy.

Original: These tests are performed across a number of pretrained ViT models, and we compared to a set of common CNNs in Table 2.
Revised: These tests are performed across a number of pretrained ViT models, and we compared to a set of common CNNs in Table 2.

Original: Further results can be found in Table 3.
Revised: Further results can be found in Table 3.

Original: We observe that ViTs are significantly better than CNNs at using the background information in an image to identify the correct class.
Revised: We observe that ViTs are significantly better than CNNs at using the background information in an image to identify the correct class.

Original: At the same time, ViTs also suffer noticeably less from the removal of the background, and thus seem to depend less on the background information to make their classification.
Revised: At the same time, ViTs also suffer noticeably less from the removal of the background, and thus seem to depend less on the background information to make their classification.

Original: A possible, and likely, confounding variable here is the imperfect separation of the background from the foreground in the ImageNet bounding box data set.
Revised: A possible, and likely, confounding variable here is the imperfect separation of the background from the foreground in the ImageNet bounding box data set.

Original: A rectangle containing the wolf in Figure 7(a), for example, would also contain a small amount of the grass and snow at the wolf’s feet.
Revised: A rectangle containing the wolf in Figure 8(a), for example, would also contain a small amount of the grass and snow at the wolf’s feet.

Original: However, the foreground is typically contained entirely in a bounding box, so masking out the bounding box interiors is highly effective at removing the foreground.
Revised: However, the foreground is typically contained entirely in a bounding box, so masking out the bounding box interiors is highly effective at removing the foreground.

Original: Because ViTs are better equipped to make sense of background information, the leaked background may be useful for maintaining superior performance.
Revised: Because ViTs are better equipped to make sense of background information, the leaked background may be useful for maintaining superior performance.

Original: Nonetheless, these results suggest that ViTs consistently outperform CNNs when information, either foreground or background, is missing.Next
Revised: Nonetheless, these results suggest that ViTs consistently outperform CNNs when information, either foreground or background, is missing.

Original: [Table]
Revised: [Table]

Original:  we study the role of texture in ViT predictions.
Revised: Next, we study the role of texture in ViT predictions.

Original: To this end, we filter out high-frequency components from ImageNet test images via low-pass filtering.
Revised: To this end, we filter out high-frequency components from ImageNet test images via lowpass filtering.

Original: While the predictions of ResNets suffer greatly when high-frequency texture information is removed from their inputs, ViTs are seemingly resilient.
Revised: While the predictions of ResNets suffer greatly when highfrequency texture information is removed from their inputs, ViTs are seemingly resilient.

Original: See Figure 14 for the decay in accuracy of ViT and ResNet models as textural information is removed.[Table]
Revised: See Figurefor the decay in accuracy of ViT and ResNet models as textural information is removed.

Original:  V I T S WITH L ANGUAGE M ODEL S UPERVISION
Revised:  V I T S WITH L ANGUAGE M ODEL S UPERVISION

Original: Recently, ViTs have been used as a backbone to develop image classifiers trained with natural language supervision and contrastive learning techniques (Radford et al., 2021).
Revised: Recently, ViTs have been used as a backbone to develop image classifiers trained with natural language supervision and contrastive learning techniques (Radford et al., 2021).

Original: These CLIP models are state-of-the-art in transfer learning to unseen datasets.
Revised: These CLIP models are state-of-the-art in transfer learning to unseen datasets.

Original: The zero-shot ImageNet accuracy of these models is even competitive with traditionally trained ResNet-50 competitors.
Revised: The zero-shot ImageNet accuracy of these models is even competitive with traditionally trained ResNet-50 competitors.

Original: We compare the feature visualizations for ViT models with and without CLIP training to study the effect of natural language supervision on the behavior of the transformer-based backbone.
Revised: We compare the feature visualizations for ViT models with and without CLIP training to study the effect of natural language supervision on the behavior of the transformer-based backbone.

Original: The training objective for CLIP models consists of matching the correct caption from a list of options with an input image (in feature space).
Revised: The training objective for CLIP models consists of matching the correct caption from a list of options with an input image (in feature space).

Original: Intuitively, this procedure would require the network to extract features not only suitable for detecting nouns (e.g. simple class labels like ‘bird’), but also modifying phrases like prepositions and epithets.
Revised: Intuitively, this procedure would require the network to extract features not only suitable for detecting nouns (e.g. simple class labels like ‘bird’), but also modifying phrases like prepositions and epithets.

Original: Indeed, we observe several such features that are not present in ViTs trained solely as image classifiers.
Revised: Indeed, we observe several such features that are not present in ViTs trained solely as image classifiers.

Original: [Figure]
Revised: [Figure]

Original: Figure 8(a) shows the image optimized to maximally activate a feature in the fifth layer of a ViT
Revised: Figure 9(a) shows the image optimized to maximally activate a feature in the fifth layer of a ViT

Original: CLIP model alongside its two highest activating examples from the ImageNet dataset.
Revised: CLIP model alongside its two highest activating examples from the ImageNet dataset.

Original: The fact that all three images share sharp boundaries indicates this feature might be responsible for detecting caption texts relating to a progression of images.
Revised: The fact that all three images share sharp boundaries indicates this feature might be responsible for detecting caption texts relating to a progression of images.

Original: Examples could include “before and after," as in the airport images or the adjective “step-by-step" for the iPod teardown.
Revised: Examples could include “before and after," as in the airport images or the adjective “step-by-step" for the iPod teardown.

Original: Similarly, Figure 8(b) and 8(c)
Revised: Similarly, Figure 9(b) and 9(c)

Original: [Figure]
Revised: [Figure]

Original: The presence of features that represent conceptual categories is another consequence of CLIP training.
Revised: The presence of features that represent conceptual categories is another consequence of CLIP training.

Original: Unlike ViTs trained as classifiers, in which features detect single objects or common background information, CLIP-trained ViTs produce features in deeper layers activated by objects in clearly discernible conceptual categories.
Revised: Unlike ViTs trained as classifiers, in which features detect single objects or common background information, CLIP-trained ViTs produce features in deeper layers activated by objects in clearly discernible conceptual categories.

Original: For example, the top left panel of Figure 9(a) shows a feature activated by what resembles skulls alongside tombstones.
Revised: For example, the top left panel of Figure 10(a) shows a feature activated by what resembles skulls alongside tombstones.

Original: The corresponding seven highly activating images from the dataset include other distinct objects such as bloody weapons, zombies, and skeletons.
Revised: The corresponding seven highly activating images from the dataset include other distinct objects such as bloody weapons, zombies, and skeletons.

Original: From a strictly visual point of view, these classes have very dissimilar attributes, indicating this feature might be responsible for detecting components of an image relating broadly to morbidity.
Revised: From a strictly visual point of view, these classes have very dissimilar attributes, indicating this feature might be responsible for detecting components of an image relating broadly to morbidity.

Original: In Figure 9(b), we see that the top leftmost panel shows a disco ball, and the corresponding images from the dataset contain boomboxes, speakers, a record player, audio recording equipment, and a performer.
Revised: In Figure 10(b), we see that the top leftmost panel shows a disco ball, and the corresponding images from the dataset contain boomboxes, speakers, a record player, audio recording equipment, and a performer.

Original: Again, these are visually distinct classes, yet they are all united by the concept of music.
Revised: Again, these are visually distinct classes, yet they are all united by the concept of music.

Original: Given that the space of possible captions for images is substantially larger than the mere one thousand classes in the ImageNet dataset, high performing CLIP models understandably require higher level organization for the objects they recognize.
Revised: Given that the space of possible captions for images is substantially larger than the mere one thousand classes in the ImageNet dataset, high performing CLIP models understandably require higher level organization for the objects they recognize.

Original: Moreover, the CLIP dataset is scraped from the internet, where captions are often more descriptive than simple class labels.
Revised: Moreover, the CLIP dataset is scraped from the internet, where captions are often more descriptive than simple class labels.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In order to dissect the inner workings of vision transformers, we introduce a framework for optimization-based feature visualization.
Revised: In order to dissect the inner workings of vision transformers, we introduce a framework for optimization-based feature visualization.

Original: We then identify which components of a ViT are most amenable to producing interpretable images, finding that the high-dimensional inner projection of the feed-forward layer is suitable while the key, query, and value features of self-attention are not.
Revised: We then identify which components of a ViT are most amenable to producing interpretable images, finding that the high-dimensional inner projection of the feed-forward layer is suitable while the key, query, and value features of self-attention are not.

Original: Applying this framework to said features, we observe that ViTs preserve spatial information of the patches even for individual channels across all layers with the exception of the last layer, indicating that the networks learn spatial relationships from scratch.
Revised: Applying this framework to said features, we observe that ViTs preserve spatial information of the patches even for individual channels across all layers with the exception of the last layer, indicating that the networks learn spatial relationships from scratch.

Original: We further show that the sudden disappearance of localization information in the last attention layer results from a learned token mixing behavior that resembles average pooling.
Revised: We further show that the sudden disappearance of localization information in the last attention layer results from a learned token mixing behavior that resembles average pooling.

Original: In comparing CNNs and ViTs, we find that ViTs make better use of background information and are able to make vastly superior predictions relative to CNNs when exposed only to image backgrounds despite the seemingly counter-intuitive property that ViTs are not as sensitive as CNNs to the loss of high-frequency information, which one might expect to be critical for making effective use of background.
Revised: In comparing CNNs and ViTs, we find that ViTs make better use of background information and are able to make vastly superior predictions relative to CNNs when exposed only to image backgrounds despite the seemingly counter-intuitive property that ViTs are not as sensitive as CNNs to the loss of high-frequency information, which one might expect to be critical for making effective use of background.

Original: We also conclude that the two architectures share a common property whereby earlier layers learn textural attributes, whereas deeper layers learn high level object features or abstract concepts.
Revised: We also conclude that the two architectures share a common property whereby earlier layers learn textural attributes, whereas deeper layers learn high level object features or abstract concepts.

Original: Finally, we show that ViTs trained with language model supervision learn more semantic and conceptual features, rather than object-specific visual features as is typical of classifiers.
Revised: Finally, we show that ViTs trained with language model supervision learn more semantic and conceptual features, rather than object-specific visual features as is typical of classifiers.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: We make our code repository available at: https://github.com/anonymous2023iclr/
Revised: We make our code repository available at: https://github.com/anonymous2023iclr/

Original: ViTVis
Revised: ViTVis

Original: A F AILED E XAMPLES
Revised: A F AILED E XAMPLES

Original: Figure 10 shows few examples of our visualization method failing when applied on low dimensional spaces.
Revised: Figure 11 shows few examples of our visualization method failing when applied on low dimensional spaces.

Original: We attribute this to entanglement of more than 768 features when represented by vectors of size 768 .
Revised: We attribute this to entanglement of more than 768 features when represented by vectors of size 768 .

Original: We note that, due to skip connections, activation in previous layers can cause activation in the next layer for the same feature, consequently, the visualizations of the same features in different layers can share visual similarities.Feature
Revised: We note that, due to skip connections, activation in previous layers can cause activation in the next layer for the same feature, consequently, the visualizations of the same features in different layers can share visual similarities.

Original: Feature
Revised: Feature

Original: [Table]
Revised: [Table]

Original: Feature
Revised: Feature

Original: [Table]
Revised: [Table]

Original: Layer Layer
Revised: Layer Layer

Original: Layer Layer
Revised: Layer Layer

Original: Layer Layer
Revised: Layer Layer

Original: Layer Layer
Revised: Layer Layer

Original: Layer Layer
Revised: Layer Layer

Original: Layer Layer
Revised: Layer Layer

Original: We believe thatroughly interpretable visualizations in layers, since there are not many different features to be embedded.
Revised: We believe that layers, since there are not many different features to be embedded.

Original: However, in deeper layers the features are entangled, so it is more difficult to visualize them.
Revised: However, in deeper layers the features are entangled, so it is more difficult to visualize them.

Original: For example,the
Revised: example,

Original: B E XPERIMENTAL S ETUP AND H YPERPARAMETERS
Revised: B E XPERIMENTAL S ETUP AND H YPERPARAMETERS

Original: As mentioned before, we ensemble augmentations to the input.
Revised: As mentioned before, we ensemble augmentations to the input.

Original: More specifically, we use A is
Revised: More specifically, we use A is

Original: GS ( CS ( Jitter ( x ))) as our augmentation.
Revised: GS ( CS ( Jitter ( x ))) as our augmentation.

Original: The bound for Jitter is ( − 32 , 32) for both directions vertical and horizontal.
Revised: The bound for Jitter is ( − 32 , 32) for both directions vertical and horizontal.

Original: The hyper parameters for CS are always mean = 1 . 0 and std = 1 . 

Revised: The hyper parameters for CS are always mean = 1 . 0 and std = 1 .

Original: in all of the experiments.
Revised: in all of the experiments.

Original: For GS , the mean is always 0 , however, for the std we have a linear scheduling, where at the beginning of the optimization the std = 0 .
Revised: For GS , the mean is always 0 , however, for the std we have a linear scheduling, where at the beginning of the optimization the std = 0 .

Original: 5 and at the end of the optimization std = 0 . 0 .
Revised: 5 and at the end of the optimization std = 0 . 0 .

Original: We use a batch-size n = 8 for all of our experiments.
Revised: We use a batch-size n = 8 for all of our experiments.

Original: We use ADAM as our choice of optimizer with β = (0 . 5 , 0 . 99) .
Revised: We use ADAM as our choice of optimizer with β = (0 . 5 , 0 . 99) .

Original: Optimization is done in 400 steps and at every step, we re-sample the augmentations
Revised: Optimization is done in 400 steps and at every step, we re-sample the augmentations

Original: Jitter , GS and CS .
Revised: Jitter , GS and CS .

Original: We also use a CosineAnealing learning rate scheduling, starting from lr = 0 .at
Revised: We also use a CosineAnealing learning rate scheduling, starting from lr = 0 .at

Original: the beginning and l = 0 the end.
Revised: the beginning and l = 0 the end.

Original: The hyper-parameter used for total variation λ tv = 0 . 00005 .
Revised: The hyper-parameter used for total variation λ tv = 0 . 00005 .

Original: For all ViT experiments, we use the pretrained models from https://github.com/lukemelas/PyTorchPretrained-ViT. For clip models, we use pretrained models by Wightman (2019).
Revised: For all ViT experiments, we use the pretrained models from https://github.com/lukemelas/PyTorchPretrained-ViT. For clip models, we use pretrained models by Wightman (2019).

Original: The rest of models we use are from https://github.com/pytorch/vision.
Revised: The rest of models we use are from https://github.com/pytorch/vision.

Original: For all of our experiments, we use GeForce RTX 2080 Ti GPUs with 12GB of memory.
Revised: For all of our experiments, we use GeForce RTX 2080 Ti GPUs with 12GB of memory.

Original: All inferences on ImageNet are done under 20 minutes on validation set and under 1 hour on training set using only 1 GPU.
Revised: All inferences on ImageNet are done under 20 minutes on validation set and under 1 hour on training set using only 1 GPU.

Original: All visualization experiments take at most 90 seconds to complete.
Revised: All visualization experiments take at most 90 seconds to complete.

Original: C M ODELS
Revised: C M ODELS

Original: In our experiments, we use publicly available pre-trained models from various sources.
Revised: In our experiments, we use publicly available pre-trained models from various sources.

Original: The following tables list the models used from each source, along with references to where they are introduced in the literature.
Revised: The following tables list the models used from each source, along with references to where they are introduced in the literature.

Original: [Table]
Revised: [Table]

Original: Dosovitskiy et al.
Revised: Dosovitskiy et al.

Original: [Table]
Revised: [Table]

Original: Dosovitskiy et al.
Revised: Dosovitskiy et al.

Original: [Figure]
Revised: [Figure]

Original: Paper
Revised: Paper

Original: [Figure]
Revised: [Figure]

Original: Touvron et al.
Revised: Touvron et al.

Original: (2021b)
Revised: (2021b)

Original: [Figure]
Revised: [Figure]

Original: patch16_
Revised: patch16_

Original: Touvron et al.
Revised: Touvron et al.

Original: (2021b) 
Revised: (2021b) deit

Revised: [Figure]

Original: [-deit_base_patch
-]
Revised: [-deit_base_patch
-]base_patch16_

Original: Touvron et al.
Revised: Touvron et al.

Original: (2021b)
Revised: (2021b)

Original: deit_tiny_distilled_patch16_
Revised: deit_tiny_distilled_patch16_

Original: Touvron et al.
Revised: Touvron et al.

Original: (2021b)
Revised: (2021b)

Original: deit_small_distilled_patch16_
Revised: deit_small_distilled_patch16_

Original: Touvron et al.
Revised: Touvron et al.

Original: (2021b)
Revised: (2021b)

Original: deit_base_distilled_patch16_
Revised: deit_base_distilled_patch16_

Original: Touvron et al.
Revised: Touvron et al.

Original: (2021b)
Revised: (2021b)

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: D E FFECT OF LOW - PASS FILTERING
Revised: D E FFECT OF LOW - PASS FILTERING

Original: [Figure]
Revised: [Figure]

Original: E A DDITIONAL VISUALIZATIONS
Revised: E A DDITIONAL VISUALIZATIONS

Original: L0 F
Revised: L0 F

Original: L0 F L0 F
Revised: L0 F L0 F

Original: L1 F L1 F
Revised: L1 F L1 F

Original: L2 F L2 F
Revised: L2 F L2 F

Original: O u r s
Revised: O u r s

Original: V a l
Revised: V a l

Original: L2 F
Revised: L2 F

Original: L3 F L3 F
Revised: L3 F L3 F

Original: L3 F
Revised: L3 F

Original: L4 F L4 F
Revised: L4 F L4 F

Original: L4 F
Revised: L4 F

Original: O u r s
Revised: O u r s

Original: V a l
Revised: V a l

Original: L5 F
Revised: L5 F

Original: L5 F L5 F
Revised: L5 F L5 F

Original: L6 F L6 F
Revised: L6 F L6 F

Original: L6 F L6 F
Revised: L6 F L6 F

Original: O u r s
Revised: O u r s

Original: V a l 
Revised: V a l L6 F L6 F L7 F

Original: L6 F L6 F
Revised: L7 F L7 F

Original: L7 F L7 F
Revised: L7 F L7 F

Original: L7 F L7 F L7 F O u r s
Revised: O u r s

Original: V a l
Revised: V a l

Original: L7 F L7 F
Revised: L7 F L7 F

Original: L7 F L7 F
Revised: L7 F L7 F

Original: L8 F
Revised: L8 F

Original: L8 F L8 F
Revised: L8 F L8 F

Original: [Figure] 
Revised: [Figure] [Figure] [Figure]

Revised: L1 F

Original: [Figure]
Revised: [Figure]

Revised: O

Original: [Figure]
Revised: [Figure]

Original: s
Revised: s

Original: V a l
Revised: V a l

Original: T r a i n
Revised: T r a i n

Original: L2 F
Revised: L2 F

Original: L
Revised: L

Original: [Figure]
Revised: [Figure]

Original: L2 F
Revised: L2 F

Original: L2 F L
Revised: L2 F L2 F

Original: [Figure] L [Figure]
Revised: L3 F

Original: L3 F
Revised: L3 F

Original: O u r s
Revised: O u r s

Original: V a l
Revised: V a l

Original: T r a i n
Revised: T r a i n

Original: [Figure]
Revised: [Figure]

Original: L
Revised: L

Original: [Figure]
Revised: [Figure]

Revised: L4 F

Original: s
Revised: L

Original: V a l
Revised: [Figure] L4 F L4 F L4 F

Revised: [Figure]

Original: T
Revised: u

Original: [Figure]
Revised: [Figure]

Original:  a i n
Revised: s V a l T r a i n

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: L6 F L6 F
Revised: L6 F L6 F

Original: L7 F L7 F
Revised: L7 F L7 F

Original: L7 F L7 F
Revised: L7 F L7 F

Original: L7 F
Revised: L7 F

Original: O u r s
Revised: O u r s

Original: V a l
Revised: V a l

Original: T r a i n
Revised: T r a i n

Original: L7 F L7 F
Revised: L7 F L7 F

Original: L7 F L7 F
Revised: L7 F L7 F

Original: L8 F L8 F
Revised: L8 F L8 F

Original: L8 F
Revised: L8 F

Original: O u r s
Revised: O u r s

Original: V a l
Revised: V a l

Original: T r a i n
Revised: T r a i n

Original: L8 F L8 F
Revised: L8 F L8 F

Original: L8 F L8 F
Revised: L8 F L8 F

Original: L8 F L8 F
Revised: L8 F L8 F

Original: L8 F
Revised: L8 F

Original: O u r s
Revised: O u r s

Original: V a l
Revised: V a l

Original: T r a i n
Revised: T r a i n

Original: L8 F L8 F
Revised: L8 F L8 F

Original: L8 F L8 F 
Revised: L8 F L8 F L8 F

Original: L8 F L9 F
Revised: L9 F L9 F

Revised: [Figure]

Original: L9 F
Revised: [Figure] [Figure] V a l T r a i n

Original: [Figure]
Revised: [Figure]

Revised: L

Original: [Figure]
Revised: [Figure]

Revised: L

Original: [Figure]
Revised: [Figure]

Original: s
Revised: u

Original: V a l T r a i n L [Figure]
Revised: [Figure]

Original: L [Figure] L6 F L
Revised: s

Original: [Figure]

Original: L7 F

Original: L7 F

Original: O u r s

Original: V a l
Revised: V a l

Original: T r a i n
Revised: T r a i n

Original: [Figure]

Original: F

Original: L7 F

Original: L

Original: [Figure]

Original: F

Original: [Figure]
Revised: [Figure]

Original: s
Revised: s

Original: V a l
Revised: V a l

Original: T r a i n
Revised: T r a i n

Original: [Figure]
Revised: [Figure]

Original: L8 F
Revised: L8 F

Original: L8 F L8 F
Revised: L8 F L8 F

Original: L8 F L8 F
Revised: L8 F L8 F

Original: L9 F L9 F
Revised: L9 F L9 F

Original: O
Revised: O

Original: [Figure]
Revised: [Figure]

Original: V a l
Revised: V a l

Original: T r a i n
Revised: T r a i n

Original: L9 F L9 F
Revised: L9 F L9 F

Original: L9 F L9 F
Revised: L9 F L9 F

Original: L9 F L9 F
Revised: L9 F L9 F

Original: L9 F
Revised: L9 F

Original: O u r s
Revised: O u r s

Original: V a
Revised: V a

Original: [Figure]
Revised: [Figure]

Original: T r a i n
Revised: T r a i n

Original: L9 F
Revised: L9 F

Original: L9 F L9 F
Revised: L9 F L9 F

Original: L9 F L9 F
Revised: L9 F L9 F

Original: L10 F L10 F
Revised: L10 F L10 F

Original: [Figure]
Revised: [Figure]

Original: T
Revised: T

Original: [Figure]
Revised: [Figure]

Original: i n
Revised: i n

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: V a
Revised: V a

Original: [Figure]
Revised: [Figure]

Original: T r a in L
Revised: T r a i

Original: [Figure]
Revised: [Figure]

Original: L
Revised: L

Original: [Figure]
Revised: [Figure]

Original: L
Revised: L

Original: [Figure]

Original: L

Original: [Figure]

Original: L

Original: [Figure]

Original: L

Original: [Figure]

Original: u

Original: [Figure]
Revised: [Figure]

Original: s

Original: V a
Revised: V a

Original: [Figure]
Revised: [Figure]

Original: T r a i

Original: [Figure]

Original: L
Revised: T

Original: [Figure]
Revised: [Figure]

Original: V
Revised: T

Original: [Figure]
Revised: [Figure]

Original: l T
Revised: i n

Original: [Figure]
Revised: [Figure]

Original: a i n
Revised: [Figure] [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: a i 
Revised: i n

Original: [Figure]

Original: V
Revised: V

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] [Figure] [Figure]
Revised: l

Original: [Figure]
Revised: [Figure]

Original: V

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: a i n
Revised: a i n

Original: V a l
Revised: V a l

Original: O u r s
Revised: O u r s

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Table]

Original: [Figure]
Revised: In the following experiments, we find the most activating images for each feature. Then, we forward these images to the network. We call a patch active if the activation for this patch is higher than 0.5.

Original: [Figure]
Revised: First we mask out all the inactive patches meaning that we replace them with black patches. Then, we mask out x percent of the active patches in the current image. Then, we forward this new image to the network.

Original: [Figure]
Revised: Finally, we plot the number/sum of active patches of the modified image divided by the number/sum of the active patches in the initial image for different percentages. If the spatial information is present in a layer, we expect this number to have a linear trend. As we see in figures 62, and 63, all the layers except for the last one, have a linear trend, indicating that the loss of spatial information in individual channels is mostly made in the last layer.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: R a t i o = P A c t i v a t i o n s ( B e f o r e M a s k i n g ) P A c t i v a t i o n s ( A f t e r

Revised: M a s k i n g )

Original: [Figure]
Revised: [Figure]

Original: [Table]