W HAT D O S ELF -S UPERVISED V ISION T RANSFORMERS L EARN ?
A BSTRACT
We present a comparative study on how and why contrastive learning (CL) and masked image modeling (MIM) differ in their representations and in their performance of downstream tasks.
In particular, we demonstrate that self-supervised
Vision Transformers (ViTs) have the following properties: (1) CL trains selfattentions to capture longer-range global patterns than MIM, such as the shape of an object, especially in the later layers of the ViT architecture.
This CL property helps ViTs linearly separate images in their representation spaces.
However, it also makes the self-attentions collapse into homogeneity for all query tokens and heads.
Such homogeneity of self-attention reduces the diversity of representations, worsening scalability and dense prediction performance.
(2) CL utilizes the lowfrequency signals of the representations, but MIM utilizes high-frequencies.
Since low- and high-frequency information respectively represent shapes and textures,
CL is more shape-oriented and MIM more texture-oriented.
(3) CL plays a crucial role in the later layers, while MIM mainly focuses on the early layers.
Upon these analyses, we ﬁnd that CL and MIM can complement each other and observe that even the simplest harmonization can help leverage the advantages of both methods.
I NTRODUCTION
Contrastive Learning (CL)
(He et al., 2020; Chen et al., 2020a;b; 2021) has been the most popular self-supervised learning methods until recently.
It aims to learn the invariant semantics of two random views (Tian et al., 2020a;b) by making global projections of representations similar for positive samples and dissimilar for negative samples.
Since CL exploits the globally projected representations to contrast each other, it can be deemed as an “image-level” self-supervised learning approach.
Deviating from CL, masked image modeling (MIM) (Bao et al., 2022;
Xie et al., 2022b;
He et al., 2022) has risen as a strong competitor of CL in the era of Vision Transformers (ViTs) (Dosovitskiy et al., 2021) with its impressive performances of downstream tasks.
MIM trains ViTs by reconstructing the correct semantics of masked input patches.
Unlike CL, it learns the semantics of patch tokens and this can be deemed as a “token-level” self-supervised learning approach.
Since MIM outperforms CL in ﬁne-tuning accuracy, it may appear prima facie as a more effective pre-training method than CL.
However, a different trend is observed for linear probing accuracy with CL outperforming MIM (See
Figure 1).
For further exposition on CL and MIM, we refer the reader to Appendix B.
Then, which method—CL or MIM—should we use for the self-supervised learning of ViTs?
Although both methods are widely used, little is known about what they learn.
This paper sheds light on their nature by showing that ViTs trained through CL and MIM learn opposite knowledge.
In particular, we raise questions to better understand self-supervised learning, and then ﬁnd the answers that can potentially affect future improvements.
The questions posed can be divided into the following properties of Vision Transformers: the behavior of self-attentions, the transformation of the representations, and the position of lead role components.
Our key questions and ﬁndings are elaborated below.
How do self-attentions behave?
(Section 2)
We ﬁnd that CL primarily captures global relationships, while MIM captures local relationships.
This implies that the representations of CL contain more global patterns, such as object shapes, than those of MIM.
On the one hand, this property helps
Linear Probing 82.
F i n e - t u n i n g
MAE
BEiT
SimMIM
MoCo
DINO
Ti
S
B
L H
Model Size 65.
F i n e - t u n i n g
DINO
MoCo MoCo
MAE BEiT
SimMIM 48.
COCO 82.
I m a g e N e t
Supervised
MoCo
MAE
BEiT
CL recognize objects and distinguish images.
On the other hand, however, it also suggests that CL struggles to preserve local information.
In particular, we observe that self-attentions of CL in the later layers for all query tokens and heads collapse into homogeneous attention maps.
In such cases, most self-attention maps focus on object boundaries, meaning that they can capture object shapes but may lose interaction diversity between tokens.
Consequently, CL and MIM each have advantages over different tasks: CL works well for linear probing and classiﬁcation tasks with smaller models, whereas MIM outperforms CL in ﬁne-tuning and dense prediction tasks with larger models.
How are representations transformed?
(Section 3) CL transforms representations mainly based on image-level information, and its self-attentions collect information on object shape over entire tokens.
This process makes tokens similar rather than diversifying them.
As a result, CL distinguishes images well but has difﬁculty distinguishing tokens.
On the contrary, MIM preserves and ampliﬁes token-level information.
Thus, the self-attentions for each token are substantially different and prohibit each token from including redundant information.
We observe the consistent property from our Fourier analysis: CL primarily utilizes the low-frequency signals, but MIM utilizes high-frequencies.
This observation suggests that CL is shape-biased and MIM is texture-biased.
In sum, self-supervised models trained with CL and MIM learn the representations in different levels of detail.
Which components play an important role?
(Section 4) Analyses of the importance of each CL and MIM layer demonstrate that the later layers in CL and early layers in MIM play a key role.
We interpret this as a consistent observation since early layers are usually known to capture low-level features—e.g., local patterns, high-frequency signals, and texture information—and later layers capture global patterns, low-frequency signals, and shape information (Dosovitskiy et al., 2021;
Raghu et al., 2021; d’Ascoli et al., 2021; Graham et al., 2021;
Dai et al., 2021; Park & Kim, 2022b).
From the above analyses and insights, we ﬁnd that CL and MIM can complement each other and show in Section 5 that even the simplest implementation, such as a linear combination of CL and
MIM objectives, can take advantage of both methods.
Surprisingly, the hybrid models outperform those pre-trained with either CL or MIM both in terms of ﬁne-tuning and linear probing accuracy.
H OW D O S ELF -A TTENTIONS B EHAVE ?
We point out that CL and MIM may not be silver bullets for all tasks, as shown in Figure 1.
CL generally outperforms MIM in linear probing, while MIM dominates CL in the ﬁne-tuning scheme.
However, when we dissect the size of the model, CL outperforms MIM after ﬁne-tuning for small models (cf.
(Wang et al., 2022)), while MIM performs better on large models.
Also, MIM yields effective representations for dense prediction tasks, such as object detection, but CL falls short on those tasks.
This section explains these phenomena by investigating the behavior of self-attentions.
[Figure]
Our analyses mainly compare ViT-B/16 pre-trained on ImageNet-1K (Russakovsky et al., 2015) with
MoCo v3 (Chen et al., 2021) and SimMIM (Xie et al., 2022b).
We use the ImageNet validation images for our experiments.
We observe that other methods, e
[Figure]
g., DINO (Caron et al., 2021), BEiT (Bao et al., 2022), and MAE (He et al., 2022), have consistent properties (See Figure C.1)
[Figure] [Figure]
CL mainly captures global relationships.
We measure the ranges of self-attentions via attention distance (Dosovitskiy et al., 2021).
Attention distance is deﬁned as the average distance between the query tokens and key tokens considering their self-attention weights.
Therefore, it conceptually corresponds to the size of the receptive ﬁelds in CNNs.
Figure 3 shows that the attention distance of CL (MoCo) is signiﬁcantly higher than that of MIM (SimMIM), especially in the later layers.
As seen in Figure 2, the qualitative visualization, this implies that the representations of CL contain global patterns and shape information, so CL can help ViTs distinguish between objects of images.
Conversely, the self-attentions of
MIM mainly capture local relationships; i.e., MIM may have difﬁculty recognizing whole objects and their shapes.
Sectionalso discuss this claim from a representational perspective.
Self-attentions of CL collapse into homogeneity.
We observe an interesting behavior of CL in Figure 2, which shows the attention maps for query tokens from two different spatial locations.
The self-attentions of CL surprisingly indicate almost identical object shapes for the two query tokens, compared to that of MIM.
We describe this phenomenon as an attention collapse into homogeneity .
This collapsing trend in the selfattentions of CL is observed across all the heads and query tokens.
In contrast, the self-attentions of MIM are more faithful to the two query tokens, as expected.
We use normalized mutual information (NMI) (Strehl & Ghosh, 2002) to measure the attention collapse.
Let p ( q ) be a distribution of query tokens, and assume that these query tokens are uniformly distributed since a single query token is given for each spatial coordinate, i.e., p ( q )
= 1 / N where N is the number of the tokens.
Then the joint distribution of query and key tokens is p ( q, k ) =
π ( k | q ) p ( q ) where
[Figure]
π ( k | q ) is the softmax-normalized self-attention matrix.
Thus, the normalized mutual information is I ( q,k ) √ H ( q ) H ( k ) where I ( · , · ) is the mutual information and H ( · ) is the marginal entropy.
Low mutual information values show that attention maps are less dependent on the query tokens, implying an attention collapse into homogeneity.
Conversely, high mutual information means that the attention maps strongly depend on the query tokens.
Figure 4 shows the degree of attention collapse in terms of the normalized mutual information (NMI).
Results show that the mutual information of CL is signiﬁcantly lower than that of MIM in the later layers, suggesting that the self-attentions of CL tend to collapse into homogeneous distributions.
Attention collapse reduces representational diversity.
We conjecture that the self-attention collapse into homogeneity eventually leads to homogeneous token representations.
To support this argument, we measure representational cosine similarities.
In particular, we design three similarities: between different self-attention heads (heads), between the before and after self-attention layers (depths), and between different tokens (tokens).
Figure 5 shows the results, reporting the representation similarities for heads, depths, and tokens.
As expected, the similarities of CL are notably higher than those of MIM in the later layers, indicating that the representations of CL have signiﬁcant homogeneity.
Even increasing the model size does not solve the problem CL has and may rather worsen it.
Increasing the number of heads (ViT-S to ViT-B;
Figure 5a) improves the representational diversity of MIM, but hardly improves the diversity of CL.
Increasing the depth of CL (ViT-B to ViT-L; Figure 5b) only adds redundant modules.
Implications of the behaviors we observed.
In conclusion, the self-attention of CL captures global patterns and shapes of objects.
However, CL suffers from the problem of attention collapse into homogeneity, which reduces the diversity of token representations.
On the other hand, MIM primarily captures local patterns and thus does not suffer from the attention collapse problem.
The behaviors mentioned above can explain the phenomena we observed in Figure 1:
• CL outperforms MIM in linear probing tasks because it captures shapes, which helps recognize objects and distinguish images.
Although MIM preserves the texture and diversity of representations, their correlation with objects or content may not be as strong as shapes do.
• The attention collapse prohibits CL from fully exploiting heads, depths, and tokens of ViTs.
Since homogeneous representations are not very helpful in improving token representations, ViTs trained with CL waste a large part of network capability.
Therefore, the ﬁne-tuning accuracy of MIM is signiﬁcantly higher than CL in large models.
• CL is not suitable for dense prediction since the token features are homogeneous with respect to their spatial
coordinates.
[Figure]
We further investigate the self-attention’s behavior with restricted receptive ﬁelds in Figure D.1.
As shown in the experiment, locally restricted self-attentions lead to lower linear probing but higher ﬁne-tuning accuracy, which is consistent with our observations.
H OW A RE R EPRESENTATIONS T RANSFORMED ?
In this section, we analyze the token representations of ViTs pre-trained with CL and MIM to demonstrate how the properties of self-attentions we observed in Section 2 affect the representations differently.
We use the same pre-trained ViT-B/16 models by default default just as we did in Section 2.
CL transforms all tokens in unison, while MIM does so individually.
To show how CL and
MIM transform token representations, we visualize them in representation space.
Figure 6 shows 196 (14 × 14 patches) tokens before and after self-attention modules from a single image sample of the ImageNet validation set.
We use the three large singular vectors obtained via singular value decomposition (SVD) as the bases of the space.
To better visualize this, we display the representation of MoCo and SimMIM in their crucial layers—the last layer and the ﬁrst layer, respectively.
Figure 6a visualizes the changes that occur in the tokens of CL when transformed by self-attention module; it indicates that the self-attentions of CL translate all tokens in unison.
This phenomenon occurs because the self-attention maps of CL are homogeneous, i.e., self-attention is almost independent of the spatial coordinates and query tokens.
Therefore, the modules add near-constant to all the token representations.
As a result, the inter-representation distance and the volume of representations do not increase, which implies that CL cares less about individual tokens.
Nevertheless, self-attentions are essential for the discriminative power of CL.
As shown in Figure 6b, they help distinguish images by moving “the centers of the representation distribution” away from each other.
In short, this ﬁgure suggests that CL makes the image linearly separable even though it loses the ability to distinguish tokens.
In contrast, MIM applies a different transformation to individual tokens, as shown in Figure 6c, because different self-attentions are assigned to the individual spatial tokens.
Thus, MIM alters the distance between tokens of a single image as well as the volume of the representation distribution.
We ﬁnd consistent results in quantitative analysis.
Inspired by Jing et al.
, Figure 7 visualizes singular value spectra for tokens and images.
A singular value spectrum provides singular values of a representation distribution obtained by SVD, so we can use it to represent the effective volume of distributions in a representation space.
The higher the singular value in a spectrum, the larger the
MoCo
SimMIM
D e p t h  11 Depth -4.
L o g s i n g u l a r v a l u e
MoCo SimMIM
[Figure]
[Figure] [Figure]
from a image (token-level)
MoCo
SimMIM
D e p t h  11 Depth -10.
L o g s i n g u l a r v a l u e
MoCo SimMIM
[Figure]
volume of a representation distribution.
To calibrate the scale, we use the relative log singular value ( ∆ Log singular value), the difference with the (second) largest singular value for a depth.
Figure 7a shows singular value spectra of tokens from a single image.
We calculate them for each image in the ImageNet validation set and report averaged singular values over the dataset.
In this ﬁgure, the CL layers hardly increase or even decrease the singular value; consistent with the explanation above, this implies that CL hardly distinguishes tokens.
In contrast, MIM increases the singular value, meaning that it changes the volume of tokens and can distinguish tokens.
Another interesting observation is that a few later layers of MIM decrease the volume, even though they capture local patterns as shown in Figures 3 and 4.
This is because they behave like decoders.
Section 4 discusses this in detail.
Figure 7b shows the singular value spectra of images.
We average all tokens in an image to build an image-level representation vector and conduct a singular value spectrum over the collection of representations in the validation set.
As opposed to the previous case, the representational volume of
CL is larger than that of MIM, which implies that CL makes the image-level representation separable.
CL exploits low-frequencies, and MIM exploits high-frequencies.
We hypothesize that CL captures low-frequency and MIM captures high-frequency information in spatial dimensions since CL provides image-level self-supervision to capture global patterns, while MIM provides token-level self-supervision to exploit local patterns.
To support this argument from a frequency perspective, we conduct a Fourier analysis of the representations as following Park & Kim (2022b).
In particular, we report the relative log amplitude of Fourier-transformed representations by calculating the amplitude difference between the highest and lowest frequencies of representations.
Figure 9 visualizes the relative amplitudes of CL and MIM.
It shows that the high-frequency amplitude of CL is signiﬁcantly smaller than that of MIM, suggesting that CL mainly utilizes low-frequency spatial information such as global structures and shapes.
On the contrary, MIM usually uses highfrequency spatial information such as narrow structures and ﬁne textures.
[Figure] [Figure]
Moreover, a few last layers of CL reduce high-frequency by capturing global patterns.
MIM also reduces it even though they capture local patterns, because the later layers behave like decoders.
See also Figure 11.
Another interesting ﬁnding is that the last few layers of
MIM reduce the high frequencies even though they only focus on local areas (See Figure 3).
We conjecture that MIM implicitly divides ViTs into the encoder-decoder structure and allows intermediate layers to have linearly separable information.
In contrast, CL allows the last layer to have such information.
This is further elaborated in Figure 11.
CL is shape-biased, but MIM is texture-biased.
Based on the results of the Fourier analysis, we assume that CL and
MIM each have a bias toward shapes and textures, respectively.
To demonstrate this claim, we use Stylized ImageNet (Geirhos et al., 2019), a texture-altered dataset, by using
AdaIN (Huang & Belongie, 2017).
Figure 8a reports the linear probing results on Stylized ImageNet to evaluate the shape and texture biases of pre-trained models.
Compared to the model pre-trained with supervised learning, CL depends more on the shape and MIM depends on texture of images to classify images.
In other words, CL is robust to texture changes, and MIM is vulnerable to them.
Figure 8b shows the consistent results.
In this experiment, we follow Park & Kim (2022a;b) and measure the decrease in accuracy on the ImageNet dataset with frequency-based random noise.
The results suggest that CL is robust to high-frequency noises, but MIM is signiﬁcantly more vulnerable to them.
Since high-frequency noises harm the ﬁne details of images, we arrive at the same conclusion that CL is more shape-biased and MIM is texture-biased.
This can explain the robustness of CL against adversarial perturbations (Bordes et al., 2022).
W HICH C OMPONENTS P LAY AN I MPORTANT R OLE ?
The previous sections consistently show through various perspectives that CL exploits image-level global patterns while MIM captures token-level local patterns.
This section analyzes pre-trained ViTs from an architectural perspective and shows that the key components in CL and MIM are different.
11 Depth 0.
N o r m a l
i z e d M I
MAE MoCo
SimMIM  19 Depth 0.
(a) Self-attention  10 Depth -3.
L o g a m p l i t u d e
MAE SimMIM  18 Depth -3.
(b) Fourier analysis
Later layers of CL and early layers of MIM are important.
According to studies on ViT (Graham et al., 2021; Dai et al., 2021; Park & Kim, 2022b), the later layers use high-level information, and the early layers exploit low-level information.
Since CL and MIM each exploit global and local patterns, we expect that the later layers of CL and early layers of MIM play a key role.
[Figure]
We report linear probing accuracies by using the representations of the intermediate layers.
CL outperforms MIM in later layers, and MIM outperforms
CL in early layers.
To evaluate the importance of each layer, we measure the linear probing accuracy using intermediate representations with the conﬁguration of Table A.1.
In Figure 11, we observe the following properties: First, the linear probing accuracy of MIM is higher than that of CL at the beginning.
Conversely, CL outperforms MIM at the end of the model.
Such result indicates that the later layers of CL and early layers of MIM play an important role in making linearly separable representations.
Second, the accuracy of CL increases with increasing depth as expected, but the accuracy of MIM surprisingly decreases at the end of the model, i.e., the later layers of MIM are not very helpful in separating representations.
We explain this observation as a phenomenon in which
MIM methods with shallow prediction heads, e.g., SimMIM, use later layers of the backbone as a decoder.
Therefore,
MIM with a deep self-attention decoder, e.g., MAE (He et al., 2022), can be useful for linear probing performance.
Moreover, it also explains why SimMIM’s high-frequency component and representational volumes drop in the later layers as shown in Figures 7 and 9.
Third, even the highest linear probing accuracy of MIM is lower than that of CL.
The explicit decoder helps ViTs further leverage the advantages of MIM.
Several previous observations ﬁnd that the implicit decoder of MIM with a shallow prediction head, such as SimMIM, can impair performance.
MAE (He et al., 2022) addresses this problem by introducing deep explicit
ViT decoders and reconstructing masked tokens only in the separate decoders.
In Figure 10, we analyze MAE to understand the properties of decoders more deeply.
Figure 10a shows the self-attention behaviors.
The results indicate that the mutual information of MAE is lower than that of SimMIM in the later layers of the encoder but higher in the decoder, implying that the decoder reconstructs masked tokens based on its neighborhood tokens.
Figure 10b shows the results of the Fourier analysis.
As explained in Figure 9, the last four layers of
SimMIM reduce the high-frequency components.
In contrast, the later layers (excluding the last layer) of MAE do not reduce them.
Instead, the decoder of MAE prioritizes low-frequency information compared with the encoder, allowing the backbone to efﬁciently utilize high-frequency information.
60 Linear Probe 81.
F i n et u n i n g =0.
=0.
=0. =0.
=1.
(a) Performance  11 Depth 0.
N o r m a l
i z e d M I
MoCo SimMIM
SimMIM+MoCo (b) Self-attention  10 Depth -3.
L o g a m p l i t u d e
MoCo SimMIM
SimMIM+MoCo (c) Fourier analysis
A RE THE T WO M ETHODS C OMPLEMENTARY TO E ACH O THER ?
We present comparative analyses on CL and MIM from three perspectives: self-attentions, representation transforms, and the position of important layers.
All of our results indicate that CL and
MIM train ViTs differently.
These differences naturally imply that combining CL and MIM to train a backbone may help leverage the advantages of both methods.
To show that CL and MIM are complementary, we introduce the simplest way to harmonize CL and MIM by linearly combining two losses, i.e., L = (1 − λ )
L MIM + λ L CL where L MIM and L CL each indicate the losses of MIM and CL, and λ is the importance weight of CL.
We ﬁnd that this simple hybrid model trained with combined losses efﬁciently exploits the strengths of both methods.
Figure 12a shows linear probing and ﬁne-tuning accuracy on ImageNet with varying λ .
Surprisingly, the hybrid models outperform MIM ( λ = 0 ) and CL ( λ = 1 ) in both aspects.
Figure 12b and
Figure 12c can provide insights on how hybrid models behave by analyzing the model with λ = 0 .in terms of self-attentions in Section 2 and Fourier analysis in Section 3, respectively; both results show that the hybrid model exploits MIM properties in the early layers and CL properties in the later layers.
In particular, Figure 12b indicates that the self-attentions of the early layers are changed according to the query token but those of the later layers are not.
Likewise, Figure 12c shows that the early layers exploit high-frequency, while the later layers try to exploit low-frequency.
C ONCLUSION
We conducted a comparative study highlighting various facets of two widely used self-supervised learning methods for vision transformers: contrastive learning (CL) and masked image modeling (MIM).
The study demonstrated many opposing properties of the two methods: image information (image-level vs .
token-level; as in Section 2), feature representations (low-frequency vs .
highfrequency; as in Section 3), and lead role components (later layers vs .
early layers; as in Section 4).
Furthermore, we suggested a possible application that exploits only the beneﬁts from both methods and showed how a combined model can outperform individual methods.
Future directions.
Various future directions can be explored based on our study.
We believe that there are better ways than a simple linear combination of CL and MIM objectives.
For example, a novel self-supervised learning approach, in which CL is applied in the later layers and MIM in the early layers, can be considered.
Moreover, we may extend our ﬁndings on self-supervision for multi-stage ViTs, such as PiT (Heo et al., 2021) and Swin (Liu et al., 2021).
Another interesting direction is to enhance the individual properties of CL and MIM.
Techniques that help CL or MIM learn shapes or textures, respectively, may also improve performance.
A S ETUP
We build the conﬁgurations based on Xie et al.
(2022b) for ﬁne-tuning and Caron et al.
(2021) for linear probing.
Table A.1 summarizes the conﬁgurations.
Most analyzes of ViTs use the ofﬁcial ViT-B pre-trained models, and some analyzes use ViT-{S, L} pre-trained with ofﬁcial conﬁgurations but epochs of 100.
Due to memory limitations, ViT-L is pre-trained with a quarter batch size of the other models.
The hybrid model introduced in Section 5 uses the ViT backbone architecture of Xie et al.
(2022b) and employes a conﬁgureation based on their work for pre-training as shown in Table A.1.
For data augmentation and regularization, we adopt widely used settings, e.g., Randaugment (Cubuk et al., 2020), label smoothing (Szegedy et al., 2016), mixup (Zhang et al., 2018), cutmix (Yun et al., 2019), stochastic depth (Huang et al., 2016).
Layer decay (Bao et al., 2022) is also used for ﬁne-tuning.
Neural network models are implemented in PyTorch (Paszke et al., 2019).
The code for analysis is available at https://github.com/naver-ai/cl-vs-mim .
All experiments use {1, 4, 8} NVIDIA
A100 Tensor Core GPU.
NSML (Kim et al., 2018) has been used for experiments.
[Table]
B R ELATED W ORK
CL is a method based on comparing the global projection of two different random views.
However, this approach usually suffers from the collapsing problem, where all representations collapse into constant solutions.
To solve this problem, various methods such as negative samples and InfoNCE (Oord et al., 2018) have been proposed.
Negative samples is an effective technique to avoid the collapsing problems, but they cause dimensional collapse (Jing et al., 2022) and require extra large batches (Chen et al., 2020a) or memory queues (He et al., 2020; Chen et al., 2020b) to retrieve them.
We mainly analyze MoCo v3 (Chen et al., 2021), since the method includes these de facto standard components—global projection, random views, and negative samples.
Some SSL methods, e.g. Grill et al.
; Caron et al.
(2021), do not use negative samples and use the projections of their Siamese representations as the positives.
Such self-distillation has been explored theoretically and empirically (Chen & He, 2021; Tian et al., 2021) to prevent the collapsing problem, but we do not discuss the distillation scheme in this paper.
Wei et al.
(2022b) shows that feature distillation improves the ﬁne-tuning performance of CL by diversifying attention ranges; this observation is consistent with our ﬁndings.
While they focus on distillation to improve CL, we reveal the fundamental nature of self-supervised learning by rigorously comparing CL and MIM.
[Figure]
[Figure] [Figure]
Compared with CL, MIM has been rarely explored in vision tasks.
Various methods, such as histograms of oriented gradients (Wei et al., 2022a) and tokenization (Bao et al., 2022), have been proposed as part of porting masked language models to the image domain with ViTs.
Among them,
SimMIM (Xie et al., 2022b) and MAE (He et al., 2022) are simple yet effective methods to reconstruct masked tokens without complicated pretext tasks.
Because of its simplicity and superior performance in downstream operations, MIM is attracting attention as a promising technique in image processing.
Nevertheless, we ﬁnd hints suggesting that CL and MIM utilize different aspects of the data, making them complementary.
For example, Zhou et al.
; Wang et al.
Yu et al.
(2022) achieve high predictive performance by harmonizing the image-level and the token-level self-supervised learning.
Xie et al.
(2022a) also observe that, unlike supervised pre-trained models or CL, selfattentions in SimMIM focus locally; this is a consistent result with our ﬁndings.
C O UR I NSIGHTS A RE G ENERALIZABLE TO V ARIOUS M ODELS
In the main text, we analyze ViT-B pre-trained using MoCo and SimMIM.
Furthermore, we observe consistent characteristics across various sizes of ViTs that have been pre-trained using other selfsupervised learning methods.
To support this claim, we delve into the properties of self-attentions.
[Figure]
Figure C.1 visualizes the self-attention behaviors of different self-supervised learning methods in terms of attention distance and normalized mutual information.
As depicted in the ﬁgure, all CLs and
MIMs exhibit consistent properties.
Similarly, Figure C.2 demonstrates that various sizes of models also demonstrate consistent properties.
D L OCALITY I NDUCTIVE B IAS I MPROVES F INE -T UNING A CCURACY OF CL
[Figure]
harms linear probing accuracy but improves ﬁne-tuning accuracy.
We report the linear probing and ﬁnetuning accuracy of ViTs with restricted self-attentions via an attention mask.
In Section 2, we demonstrate that the homogeneity of selfattention map, i.e., attention collapse of CL, helps ViT distinguish images but harms ﬁne-tuning accuracy.
As a result, we anticipate that incorporating a locality inductive bias into
CL will improve ﬁne-tuning accuracy but degrade linear probing accuracy.
One simple method to inject locality into self-attentions is to limit the receptive ﬁeld of self-attention by using attention masks.
Figure D.1 shows the predictive performance of ViT with restricted local self-attentions.
As expected, the results are similar to the performance of MIM; As the kernel size decreases, the linear probing accuracy decreases but the ﬁnetuning accuracy increases.
These results are consistent with our ﬁndings.
E A C LOSER L OOK AT THE R OLE OF S ELF - SUPERVISED V I T L AYERS
The main text provides the key characteristics of CL and MIM.
This section delves deeper into the details not covered in the main text to provide a more comprehensive understanding of the subjects.
The role of the early modules.
Figures 3 and 4 suggest that most layers the CL model capture global patterns and have only a weak correlation with query tokens.
However, one or two of CL layers exhibit unusual behavior.
For example, the 3 rd layer of CL focuses on local areas and its self-attention map is dependent on the query.
We explore this property in more detail.
[Figure]
Figure E.1a provides the variance of normalized mutual information with respect of heads.
As the results show, the variance of SimMIM is consistent across all depths whereas that of CL is not.
In particular, the 3 rd layer of CL has high variance even though other layers do not.
This suggests that, while most of CL’s self-attention heads capture global patterns and have weak correlation with query tokens, some heads deviate from this behavior and exhibit a different pattern.
Figure E.1b shows the distribution of normalized mutual information among heads in the 3 rd layer to analyse this phenomenon.
In this ﬁgure, we use kernel density estimation with Gaussian kernel to visualize the distribution.
As expected, the results reveal several outlier heads in CL with mutual information close to 1.0.
A comprehensive view through visualization of tokens from multiple images.
Figure 6 visualizes how self-attention layers transform tokens from one or two images in representation space.
The ﬁgure demonstrates that MoCo transforms all tokens in union while SimMIM transforms them individually.
As a result, CL separates the representations at the image-level and SimMIM separates them at the token-level.
The t-SNE visualization (Van der Maaten & Hinton, 2008) in Figure E.2 provides consistent results and offers even a more comprehensive perspective.
In this ﬁgure, we visualize the last representations of 3528 tokens from 18 images that belong to three different classes.
The visualization demonstrates that MoCo separates the representations into distinct classes and even images, while maintaining the tokens close together in compact image clusters.
On the other hand, SimMIM separates tokens from images, resulting in a wide representation space for each image, but the images or even classes may be challenging to linearly distinguish.
The ﬁrst layer of MoCo aggregates tokens into compact clusters.
Figures 6 and 7 shows that all modules, except the ﬁrst module, in MoCo behave consistently.
However, we observe that MoCo’s ﬁrst module behaves differently and unusually than the others.
We elaborate the behaviour of the ﬁrst layer of module.
Figure E.3a shows the qualitative visualization of tokens for a sample image, similar to Figure 6.
This visualization shows that the ﬁrst MoCo layer aggregates tokens into compact clusters.
Although this ﬁgure only uses a single image, the layer aggregates all images into a small representation space as well.
In terms of singular values, we observe consistent results.
Similar to Figure 7, Figures E.3b and E.3c report the second largest log singular value, instead of the relative log singular value, to investigate the absolute volume of the representations.
As expected, most layers in both CL and
SimMIM increase the singular value, but surprisingly, the ﬁrst layer of CL reduces the singular value, meaning that the volumes of representations are decreased at both the token-level and image-level.
[Figure] [Figure]
, the mutual information of MoCo’s self-attention maps is generally lower compared to that of SimMIM.
However, it is noteworthy that the mutual information of the later self-attention maps in SimMIM decreases signiﬁcantly.
This is because the later layers of a model trained with supervision or ﬁne-tuning tend to capture global information
Similarly,
SimMIM utilizes higher frequency information
MoCo.
Based on these observations, we conjecture that the ﬁrst module of MoCo behaves like an embedding component.
F F INE -T UNED M ODELS I NHERIT THE P ROPERTIES OF P RE - TRAINED M ODELS
The main text focuses on highlighting the key properties of pre-trained models.
This section demonstrates that these properties are also utilized by ﬁne-tuned models.
As a result, we can safely apply the insights gained from the main text to various situations.
Consistent results in self-attention and Fourier analysis.
Figures 3 and 4 demonstrate that MoCo captures global areas and that its self-attentions are less related to the query tokens, compared with
[Figure]
S h a p e c a t e g
o
r
i e s (a) Stylized ImageNet 0.
Noise frequency -4.
A cc u r a c y ( % )
MoCo SimMIM
[Figure]
SimMIM.
In addition, Figure 9 shows that MoCo captures low-frequency information as opposite to
SimMIM.
These results are consistent in the ﬁne-tuning scheme.
Figure F.1a reveals the self-attention behaviours of ﬁne-tuned MoCo and SimMIM in terms of normalized mutual information.
Similar to pre-trained models, the ﬁne-tuned self-attention maps of
MoCo have generally lower mutual information compared to those of SimMIM.
The only signiﬁcant difference is that the mutual information of the later self-attention maps in ﬁne-tuned SimMIM decreases signiﬁcantly, as later layers in models trained with supervision or ﬁne-tuning tend to capture more global information.
As a result, the gap between the two methods is reduced.
This is also reﬂected in the consistent results of Fourier analysis as shown in Figure F.1b.
In this analysis,
SimMIM captures higher-frequency information compared to MoCo in ﬁne-tuning scheme as well.
However, the later layers of SimMIM attempt to capture low-frequency information.
Therefore, the gap of ﬁne-tuned models is smaller than that of pre-trained models.
CL is shape-biased and MIM is texture-biased in ﬁne-tuning scheme.
In Figure 8, we demonstrate that linear probing model with CL is more shape-biased and that with MIM is texture-
[Figure]
the ﬁne-tuning scheme.
To show this, we initialize a few blocks and measure the decrease in the tuning accuracy of models biased, compared with each other.
As in the experiment, we calculate the classiﬁcation results of ImageNet ﬁnetuned CL and MIM on Stylized-ImageNet, and measure the decrease in accuracy against frequency-based random noise.
As we would expected, Figure F.2 shows that the property also extends to the ﬁne-tuned model.
Even though we still observe the difference between
CL and MIM, the performance gap between CL and
MIM is quite reduced compared to the gap between the linear probing models.
Later layers of CL and early layers of MIM are important in ﬁnd-tuning phases.
As shown in Figure 11, the later layers of the CL and the early layers of the MIM are linearly separable.
This ﬁnding suggests that these layers are signiﬁcant, however, it does not provide direct evidence that such properties are preserved during ﬁne-tuning phases.
We demonstrate that these layers play a crucial role in ﬁne-tuning phases as well.
To support this claim, we conduct a study to measure the accuracy drop of ﬁne-tuned models using pre-trained models with a few blocks initialized.
As shown in Figure F.3a, the results indicate that the initializing a few early blocks in the pre-training models of MIM signiﬁcantly harms the ﬁne-tuning accuracy, compared to CL.
These observations suggest that early layers of MIM play an important role in ﬁne-tuning.
Conversely, Figure F.3b shows that initializing later blocks in the pre-training models of MIM does not signiﬁcantly harms the ﬁne-tuning accuracy, suggesting that they are not important in ﬁne-tuning compared with CL.
One limitation of this experiment is the evaluation of the accuracy drop in a single run.
Since the accuracy drop of MoCo is marginally higher than that of SimMIM at the ﬁrst initialization depth in
Figure F.3b, additional experiments may improve the results.
In this experiment, we utilized the same ﬁne-tuning settings for both MoCo and SimMIM; but experiments with ﬁne-tuning settings tailored to each method may provide further insight.
G H YBRID M ODELS O UTPERFORM CL AND MIM IN D OWNSTREAM T ASKS
[Table]
The claim that CL and MIM are complementary is demonstrated only on ImageNet in Section 5.
To validate this claim in tasks beyond
ImageNet, we evaluated the pre-trained models of the hybrid method introduced in Sectionfor another classiﬁcation task and a semantic segmentation task.
In particluar, we measured the accuracy on iNaturalist 2018 (Van Horn et al., 2018) and the mIoU on ADE20K
(Zhou et al., 2019).
As shown in Table G.1, the hybrid model of SimMIM and MoCo outperforms both SimMIM and MoCo in various downstream tasks.
Therefore, we conclude that the effectiveness of this claim extends beyond ImageNet.