W HAT D O
S ELF -S UPERVISED V ISION T RANSFORMERS L EARN ?
Anonymous authors
Paper under double-blind review
A BSTRACT
We present a comparative study on how and why contrastive learning (CL) andmasked image modeling (MIM) differ in their representations and in their performance of downstream tasks.
In particular, we demonstrate that self-supervised Vision Transformers (ViTs) have the following properties: (1) CL trains selfattentions to capture longer-range global patterns than MIM, such as the shape ofan object, in the later layers of the ViT architecture.
This CL property helps ViTslinearly separate images in their representation spaces.
However, it also makesthe self-attentions collapse into homogeneity for all query tokens and heads.
Suchhomogeneity of self-attention reduces the diversity of representations, worseningscalability and dense prediction performance. (2) CL utilizes the low-frequencysignals of the representations, but MIM utilizes high-frequencies.
Since low- andhigh-frequency information respectively represent shapes and textures, CL is moreshape-oriented and MIM more texture-oriented.
(3) CL plays a crucial role in thelater layers, while MIM mainly focuses on the early layers.
Upon these analyses,we ﬁnd that CL and MIM can complement each other and observe that even thesimplest harmonization can help leverage the advantages of both methods.
I NTRODUCTION
Contrastive Learning (CL)
(He et al., 2020; Chen et al., 2020a;b; 2021) has been the most popularself-supervised learning methods until recently.
It aims to learn the invariant semantics of two randomviews (Tian et al., 2020a;b) by making global projections of representations similar for positivesamples and dissimilar for negative samples.
Since CL exploits the globally projected representationsto contrast each other, it can be deemed as an “image-level” self-supervised learning approach.
Deviating from CL, masked image modeling (MIM) (Bao et al., 2022; Xie et al., 2022; He et al.
, 2022)has risen as a strong competitor of CL in the era of Vision Transformers (ViTs) (Dosovitskiy et al.,2021) with its impressive performances of downstream tasks.
MIM trains ViTs by reconstructing thecorrect semantics of masked input patches.
Unlike CL, it learns the semantics of patch tokens andthis can be deemed as a “token-level” self-supervised learning approach.
Since MIM outperforms CLin ﬁne-tuning accuracy, it may appear prima facie as a more effective pre-training method than CL.
However, a different trend is observed for linear probing accuracy with CL outperforming MIM (see Figure 1).
For further exposition on CL and MIM, we refer the reader to Appendix B.
Then, which method—CL or MIM—should we use for the self-supervised learning of ViTs?
Althoughboth methods are widely used, little is known about what they learn.
This paper sheds light on theirnature by showing that ViTs trained through CL and MIM learn opposite knowledge.
In particular,we raise questions to better understand self-supervised learning, and then ﬁnd the answers that canpotentially affect future improvements.
The questions posed can be divided into the following properties of Vision Transformers: the behavior of self-attentions, the transformation of the representations,and the position of lead role components.
Our key questions and ﬁndings are elaborated below.
How do self-attentions behave?
(Section 2)
We ﬁnd that CL primarily captures global relationships,while MIM captures local relationships.
This implies that the representations of CL contain moreglobal patterns, such as object shapes, than those of MIM.
On the one hand, this property helps CL recognize objects and distinguish images.
On the other hand, however, it also suggests that CLstruggles to preserve local information that ViTs trained with CL lack such information.
In particular,we observe that self-attentions of CL in the later layers for all query tokens and heads collapse into
Linear Probing 82.
F i n e - t u n i n g
MAE
BEiT
SimMIM
MoCo
DINO
Ti
S
B
L H
Model Size 65.
F i n e - t u n i n g
DINO
MoCo MoCo
MAE BEiT
SimMIM 48.
COCO 82.
I m a g e N e t
Supervised
MoCo
MAE
BEiT homogeneous attention maps.
In such cases, most self-attention maps focus on object boundaries,meaning that they can capture object shapes but may lose interaction diversity of self-attention.
Consequently, CL and MIM each have advantages over different tasks: CL works well for linearprobing and classiﬁcation tasks with smaller models, whereas MIM outperforms CL in ﬁne-tuningand dense prediction tasks with larger models.
How are representations transformed?
(Section 3) CL transforms representations mainly based onimage-level information, and its self-attentions collect information on object shape over entire tokens.
This process makes tokens similar rather than diversifying them.
As a result, CL distinguishes imageswell but has difﬁculty distinguishing tokens.
On the contrary, MIM preserves and ampliﬁes token-levelinformation.
Thus, the self-attentions for each token are substantially different and prohibit eachtoken from including redundant information.
We observe the consistent property from our Fourieranalysis: CL primarily utilizes the low-frequency signals, but MIM utilizes high-frequencies.
Thisobservation suggests that CL is shape-biased and MIM is texture-biased.
In sum, self-supervisedmodels trained with CL and MIM learn the representations in different levels of detail.
Which components play an important role?
(Section 4) Analyses of the importance of each CLand MIM layer demonstrate that the later layers in CL and early layers in MIM play a key role.
Weinterpret this as a consistent observation since early layers are usually known to capture low-levelfeatures—e.g., local patterns, high-frequency signals, and texture information—and later layerscapture global patterns, low-frequency signals, and shape information (Dosovitskiy et al., 2021; Raghu et al., 2021; d’Ascoli et al., 2021; Graham et al., 2021; Dai et al., 2021; Park & Kim, 2022b).
From the above analyses and insights, we ﬁnd that CL and MIM can complement each other andshow in Section 5 that even the simplest implementation, such as a linear combination of CL and MIM objectives, can take advantage of both methods.
Surprisingly, the hybrid models outperformthose pre-trained with either CL or MIM both in terms of ﬁne-tuning and linear probing accuracy.
H OW D O S ELF -A TTENTIONS B EHAVE ?
We point out that CL and MIM may not be silver bullets for all tasks by visualizing their performancefrom three perspectives.
First, Figure 1 shows that CL generally outperforms MIM in linear probing,while MIM dominates CL in the ﬁne-tuning scheme.
However, when we dissect the size of themodel, CL outperforms MIM after ﬁne-tuning for small models (cf. (Wang et al., 2022)), while MIMperforms better on large models.
Also, MIM yields effective representations for dense prediction tasks,such as object detection, but CL falls short on those tasks.
This section explains these phenomena byinvestigating the behavior of self-attentions.
[Figure] Our analyses mainly compare ViT-B/16 pre-trained on ImageNet-1K (Russakovsky et al., 2015) with
MoCo v3 (Chen et al., 2021) and SimMIM (Xie et al., 2022).
We use the ImageNet validation imagesfor the experiments.
We observe that other methods, e.g., DINO (Caron et al., 2021), BEiT (Baoet al., 2022), and MAE (He et al., 2022), have consistent properties (See Figure B.1). [Figure]
[Figure]
CL mainly captures global relations, but MIM does not.
We measure the ranges of self-attentions via attention distance(Dosovitskiy et al., 2021). Attention distance is deﬁned as theaverage distance between the query tokens and key tokensconsidering their self-attention weights, and it conceptuallycorresponds to the receptive ﬁelds in CNNs.
Figure 3 shows that the attention distance of CL (MoCo) issigniﬁcantly higher than that of MIM (SimMIM), especially inthe later layers.
Considering the qualitative visualization, Figure 2, this implies that the representations of CL contain globalpatterns and shape information, so it can help ViT distinguishbetween objects or images.
Conversely, the self-attentions of MIM mainly capture local relationships, i.e., MIM may havedifﬁculty recognizing whole objects and their shapes.
Section 3also discuss this claim from a representational perspective.
Self-attentions of CL collapse into homogeneity.
We observe an interesting behavior of CL in Figure 2, which showsattention maps for query tokens from two different spatial locations.
Surprisingly, the later self-attentions of CL indicatealmost identical object shapes for the two query tokens, compared to that of MIM.
We describe this phenomenon as attentioncollapse into homogeneity .
This attention collapse trend in selfattentions of CL is observed across all the heads and querytokens.
In contrast, MIM shows more faithful self-attentionsfor two query tokens, as expected.
We use normalized mutual information (NMI) (Strehl & Ghosh,2002) to measure the attention collapse.
Let p ( q ) be a distribution of query tokens, and assume they are uniformly distributedsince a single query token is given for each spatial coordinate,i.e., p ( q )
= 1 / N where N is the number of the tokens.
Thenthe joint distribution of query and key tokens is p ( q, k ) =
π ( k | q ) p ( q ) where π ( k | q ) is softmax- [Figure] normalized self-attention matrix.
Thus, the normalized mutual information is
I ( q,k ) √
H ( q ) H ( k ) where
I ( · , · ) is the mutual information and H ( · ) is the marginal entropy.
The low mutual information valuerepresents the attention map is less dependent on the query tokens implying the attention collapseinto homogeneity.
Conversely, the high the mutual information means the attention maps stronglydepend on the query token.
Figure 4 shows the degree of attention collapse in terms of the normalized mutual information (NMI).
The results show that the mutual information of CL is signiﬁcantly lower than that of MIM in thelater layers, suggesting that self-attentions of CL tend to collapse into homogeneous distributions.
The attention collapse reduces representation diversity.
We conjecture that the self-attentioncollapse into homogeneity eventually leads to homogeneous token representations.
To support thisargument, we measure representational cosine similarities.
In particular, we design three similarities:between different self-attention heads (heads), between the before and after self-attention layers(depths), and between different tokens (tokens).
Figure 5 shows the result reporting the representation cosine similarities for heads, depths, and tokens.
As expected, the similarities of CL are notably higher than those of MIM in the later layers, indicatingthat the representations of CL have signiﬁcant homogeneity. Moreover, increasing the model size doesnot solve or even exacerbates this problem of CL. Increasing the number of heads (ViT-S to ViT-B; Figure 5a) improves the diversity of MIM due to its self-attention diversity, but hardly improves CL.
Increasing the depth of CL (ViT-B to ViT-L; Figures 5b and 5c) only adds redundant modules.
Implications of the behaviors we observed.
In conclusion, the self-attention of CL captures globalpatterns and shapes of objects.
However, CL suffers from the problem of attention collapse intohomogeneity, which reduces the diversity of token representations.
On the contrary, MIM does notsuffer from the attention collapse problem by primarily capturing local patterns.
The behaviors mentioned above can explain the phenomena we observed in Figure 1:
• CL outperforms MIM in linear probing tasks because capturing shapes by CL helps recognize ob-
jects and distinguish images.
Although MIM preserves texture and the diversity of representation,
these may not strongly correlate with objects or content, compared to shapes.
• The attention collapse prohibits CL from fully exploiting heads, depths, and tokens of ViTs.
Since homogeneous representations is not very helpful for improving token representations, ViTstrained with CL wastes a large part of network capability.
Therefore, the ﬁne-tuning accuracy of MIM with large models is signiﬁcantly higher than that of CL.
• CL is not suitable for dense prediction since the token features are homogeneous with respect to
their spatial coordinates.
[Figure]
We further investigate the self-attention’s behavior with restricted receptive ﬁelds in Figure A.1.
Asshown in the experiment, locally restricted self-attentions lead to lower linear probing but higherﬁne-tuning accuracy, which is consistent with our observations.
H OW A RE R EPRESENTATIONS T RANSFORMED ?
In this section, we analyze the token representations of ViTs pre-trained with CL and MIM todemonstrate how the properties of self-attentions we observed in Section 2 affect the representationsdifferently.
We use the same pre-trained ViT-B/16 models by default as well as we did in Section 2.
CL transforms all tokens in unison, while MIM’s tokens are transformed individually.
Toshow how CL and MIM transform token representations, we visualize them in representation space.
Figure 6 represents 196 (14 × 14 patches) tokens before and after the ﬁrst self-attention module froma single image of the ImageNet validation set.
We use the top three singular vectors obtained viasingular value decomposition (SVD) as the bases of the representation space.
Figure 6a visualizes the changes in the tokens of CL transformed by the self-attention module; itindicates that the self-attentions of CL translate all tokens in unison.
This phenomenon occurs sincethe self-attention maps of CL are homogeneous, i.e., the self-attention is almost independent of thespatial coordinates and the query tokens.
Therefore, the module adds near-constant to all the tokenrepresentations.
As a result, the inter-representation distance and the representations’ volume remainalmost invariant, which implies that CL cares less about individual tokens.
Nevertheless, self-attentions are essential for the discriminative power of CL.
As shown in Figure 6b,they distinguish images by moving “the centers of the representation distribution” away from eachother.
In short, this ﬁgure suggests that CL makes the image linearly separable even though it losesthe ability to distinguish tokens.
In contrast, MIM applies a different transform to individual tokens, as shown in Figure 6c, because thedifferent self-attentions are assigned to the individual spatial tokens.
Thus, MIM alters the distancebetween the tokens of a single image and the volume of the representation distribution.
We ﬁnd consistent results in quantitative analysis.
Inspired by Jing et al.
(2022), Figure 7 visualizesingular value spectra for tokens and images.
The singular value spectrum provides singular valuesof representation distribution obtained by SVD, so we can use it to represent the effective volume
[Figure]
s i n g u l a r v a l u e
[Figure]
[Figure]
D e p t h [Figure]
L o g s i n g u l a r v a l u e
[Figure] SimMIM
[Figure]
shows the singular value spectra of tokens for a single image.
We calculate it for eachimage in ImageNet validation set and report averaged singular values over the dataset.
In this ﬁgure,the layers of CL hardly increase or even decrease the singular value; consistent with the explanationabove, it implies that CL hardly distinguishes tokens.
In contrast, MIM increases the singular value,so it changes the volume of tokens and can distinguish tokens.
One interesting observation is that afew later layers of MIM decrease the volume, even though they captures local patterns as shown in
Figures 3 and 4.
This is because they behave like decoders, and Section 4 discuss this in detail.
Figure 7b shows the singular value spectra of images.
We averaged all tokens in an image to buildan image-level representation vector and conduct the singular value spectra over the collection ofrepresentations in the validation set.
As opposed to the previous case, the representational volume of
CL is larger than that of MIM, which implies that CL makes the image-level representation separable.
CL exploits low-frequencies, and MIM exploits high-frequencies.
We hypothesize that CL captures low-frequency and MIM captures high-frequency information in spatial dimensions since CLprovides image-level self-supervision to capture global patterns, while MIM provides token-levelself-supervision to exploit local patterns.
To support this argument from a frequency perspective, weconduct Fourier analysis on the representations as following Park & Kim (2022b).
In particular, wereport the relative log amplitude of Fourier transformed representation by calculating the amplitudedifference between the representations’ highest and lowest frequencies.
Figure 9 visualizes the relative amplitudes of CL and MIM.
It shows that the high-frequency amplitudeof CL is signiﬁcantly smaller than that of MIM, suggesting that CL mainly utilizes low-frequencyspatial information such as global structures and shapes.
On the contrary, MIM usually uses highfrequency spatial information such as narrow structures and ﬁne textures.
[Figure]
Fraction of
[Figure] shape [Figure] decisions
[Figure]
S h a p e c a t e g
o
r
i e s
[Figure]
[Figure] Moreover, a few last layers of CL reduce high-frequency by capturing global patterns. MIM also reduces it even though they capture local patterns, because the later layers behave like decoders.
See also Figure 11.
Another interesting ﬁnding is that the last few layers of MIMreduce the high frequencies even though they only focus onthe local areas (See Figure 3).
We conjecture that MIM implicitly divides ViTs into the encoder-decoder structure andlets the intermediate layers have linearly separable information.
In contrast, CL lets the last layer have it.
This is furtherelaborated in Figure 11.
CL is shape-biased, but MIM is texture-biased.
Basedon the results of Fourier analysis, we assume that CL and MIM have a bias toward shape and texture, respectively.
Todemonstrate this claim, we use Stylized-ImageNet (Geirhoset al., 2019), a texture-altered dataset, by using AdaIN(Huang & Belongie, 2017).
Figure 8a report the linear probing results on Stylized-ImageNet to evaluate the shape andtexture biases of pre-trained models.
Compared to the modelpre-trained with supervised learning, CL depends more onthe shape of images to classify images, and MIM dependson texture.
In other words, CL is robust to texture changes,and MIM is vulnerable to them.
Figure 8b shows the consistent results.
In this experiment, we follow Park & Kim (2022a;b) andmeasure the decrease in accuracy on the ImageNet dataset with frequency-based random noise.
Theresults suggest that CL is robust to high-frequency noises, but MIM is signiﬁcantly more vulnerableto them.
Since high-frequency noises harm the ﬁne details of images, we arrive at the same conclusionthat CL is more shape-biased and MIM is texture-biased.
This could serve as an explanation for therobustness of CL against adversarial perturbations (Bordes et al., 2021).
W HICH C OMPONENTS P LAY AN I MPORTANT R OLE ?
In the previous sections, various perspectives consistently show that CL exploits image-level globalpatterns and MIM captures token-level local patterns.
This section analyzes pre-trained ViTs fromthe architecture perspective layer-wisely and shows key components in CL and MIM are different.
10 Depth 0.
N o r m a l i z e d M I
MAE MoCo
SimMIM  18 Depth 0.
MAE (a) Self-attention  10 Depth -3. L o g a m p l i t u d e
MAE SimMIM 15 17 20 Depth -3.
[Figure]
Later layers of CL and early layers of MIM are important.
According to studies on ViT architectures (Graham et al., 2021; Dai et al., 2021; Park & Kim, 2022b), the later layers use high-levelinformation, and the early layers exploit low-level information.
Since CL and MIM exploit globaland local patterns, we expect that the later layers of CL and early layers of MIM play key roles.
[Figure]
We report linear probing accuracies by using representations of the intermediate layer.
CL outperforms MIM in later layers, and MIM outperforms CL in early layers.
To evaluate the importance of each layer, we measure thelinear probing accuracy using the intermediate representations with the conﬁguration of Table A.1.
In Figure 11, weobserve the following properties: First, the linear probingaccuracy of MIM is higher than that of CL at the beginning.
Conversely, CL outperforms MIM at the end of the model.
This result indicates that later layers of CL and early layersof MIM play an important role in making linearly separablerepresentations.
Second, the accuracy of CL increases withincreasing depth as expected, but surprisingly, the accuracyof MIM decreases at the end of the model, i.e., later layersof MIM are not very helpful in separating representations.
We explain this as MIM methods with a shallow predictionhead, e.g., SimMIM, use later layers of the backbone as adecoder.
Therefore, MIM with a deep self-attention decoder,e.g., MAE (He et al., 2022), can be useful for linear probingperformance.
Moreover, it also explains SimMIM’s highfrequency component and representational volumes drop atlater layers as shown in Figures 7 and 9.
Third, the highestlinear probing accuracy of MIM is lower than that of CL.
The explicit decoder help further leverage the advantages of MIM. Several previous observations ﬁnd that implicit decoders in MIMs with shallow prediction heads, such as SimMIM, canimpair performance. MAE (He et al., 2022) addresses this problem by introducing deep explicit ViTdecoders and using masked tokens only for the decoders.
In Figure 10, we analyze the token representations of MAE to understand the properties of decodersmore deeply. Figure 10a represents the self-attention behavior.
This result indicates that the mutualinformation of MAE is lower than that of SimMIM in the later encoder layers but higher in thedecoder, implying that decoders and self-attentions are correlated.
Figure 10b shows the result of Fourier analysis.
The later layers (except the last layer) of MAE do notreduce high frequency component, whereas the last four layers of SimMIM reduce it.
The decoderof MAE utilizes low-frequency information compared with the encoder. As a result, the backboneof MAE does not have an unnecessary decoder part to learn effective token representations, and itmakes efﬁcient use of the limited capacity (cf. (Wang et al., 2022)).
60 Linear Probe 81.
F i n e - t u n i n g =0.
=0.
=0. =0.
=1.
(a) Performance  10 Depth 0.
N o r m a l
i z e d M I
MoCo SimMIM
SimMIM+MoCo (b) Self-attention  10 Depth -3.
L o g a m p l i t u d e
MoCo SimMIM
SimMIM+MoCo (c) Fourier analysis  C AN W E B ENEFIT FROM B OTH M ETHODS ? We present comparative analyses on CL and MIM from three perspectives: self-attentions, representation transforms, and the position of important layers. All of our results indicate that CL and
MIM train ViTs differently.
These differences naturally imply that combining CL and MIM may helpleverage the advantages of both methods.
To show that CL and MIM are complementary, we introduce the simplest way to harmonize CLand MIM by linearly combining two losses, i.e., L = (1 − λ ) L MIM + λ L CL where L MIM and L CLeach indicate the losses of MIM and CL, and λ is the importance weight of CL. We ﬁnd that thissimple hybrid model trained with combined losses efﬁciently exploits the strengths of both methods. Figure 12a shows linear probing and ﬁne-tuning accuracy on ImageNet with varying λ .
Surprisingly,the hybrid models outperform MIM ( λ = 0 ) and CL ( λ = 1 ) in both aspects.
Figure 12b and Figure 12c can provide insights on how hybrid models behave by analyzing the model with λ = 0 . 2in terms of self-attentions in Section 2 and Fourier analysis in Section 3, respectively; both resultsshow that the hybrid model exploits MIM properties in the early layers and CL’s properties in thelater layers.
In particular, Figure 12b indicates that the self-attentions of the early layers are changedaccording to the query token but those of the later layers are not.
Likewise, Figure 12c shows that theearly layers exploit high-frequency, while the later layers try to exploit low-frequency.
C ONCLUSION
We have conducted comparative studies highlighting various facets of two widely used self-supervisedlearning methods for vision transformers: contrastive learning and masked image modeling.
Thestudies demonstrated many opposing properties of two methods: image information (image-levelvs .
token-level; as in Section 2), feature representations (low-frequency vs . high-frequency; as in
Section 3), and different lead role components (later layers vs . early layers; as in Section 4).
Wefurther imagined the possible application exploiting only the beneﬁts from both ends and showedhow their mixture outperforms them.
Future directions.
Based on our study, diverse future directions can be explored.
We believe thereare better ways beyond a simple linear combination of CL and MIM objectives.
For example, novelself-supervised learning can be considered, which applies CL for the later layers and MIM for theearly layers.
Moreover, we may extend our ﬁndings on self-supervision for multi-stage ViTs, suchas PiT (Heo et al., 2021) and Swin (Liu et al., 2021).
Another interesting direction is to enhancethe properties of CL or MIM.
Techniques that help CL or MIM learn shapes or textures also mayimprove performance.
R EPRODUCIBILITY S TATEMENT
We will provide the code and experimental details for reproducing the results.
The code will bereleased as open source.
See Appendix A for detailed conﬁgurations.
A S ETUP
We build the conﬁgurations based on Xie et al.
for ﬁne-tuning and pre-training, and Caron et al.(2021) for linear probing.
For data augmentation and regularization, we adopt widely used settings,e.g., Randaugment (Cubuk et al., 2020), label smoothing (Szegedy et al., 2016), mixup (Zhang et al.,2018), cutmix (Yun et al., 2019), stochastic depth (Huang et al., 2016).
Layer decay (Bao et al.
2022)is also used.
Table A.1 summarize the conﬁgurations.
We use {1, 4, 8} NVIDIA A100 Tensor Core
GPU for all experiments.
Neural network models are implemented in PyTorch(Paszke et al., 2019);
The code will be released publicly.
C ONFIGURATION
[Table] [Table]
gradient
[Table] [Table]
B R ELATED W ORK
[Figure]
harms linear probing accuracy but improves ﬁne-tuning accuracy.
We report the ﬁne-tuning accuracy and linear probing accuracy of ViTs with restricted self-attention via attention mask.
CL is a method based on comparing the global projection oftwo different random views.
However, this approach usuallysuffers from the collapsing problem, where all representations collapse into constant solutions.
To solve this problem,various methods such as negative samples and InfoNCE(Oord et al., 2018) have been proposed.
Negative samplesis an effective technique to avoid the collapsing problems,but they cause dimensional collapse (Jing et al., 2022) andrequire extra large batches (Chen et al., 2020a) or memoryqueues (He et al., 2020; Chen et al., 2020b) to retrieve them.
Some SSL methods, e.g. Grill et al.
(2020); Caron et al.(2021), do not use negative samples and use the projectionsof their Siamese representations as the positives.
Such selfdistillation has been explored theoretically and empirically(Chen & He, 2021; Tian et al., 2021) to prevent the collapsing problem, but we do not discuss the distillation scheme inthis paper.
Wei et al.
(2022b) shows that feature distillationimproves the ﬁne-tuning performance of CL by diversifying attention ranges; this observation is consistent with our
[Figure]
[Figure] [Figure]
ﬁndings.
While they focus on distillation to improve CL, we reveal the fundamental nature ofself-supervised learning by rigorously comparing CL and MIM.
Compared with CL, MIM has been rarely explored.
Various methods, such as histograms of orientedgradients (Wei et al., 2022a) and tokenization (Bao et al., 2022), have been proposed as part ofporting masked language models to the image domain with ViTs.
Among them, SimMIM (Xie et al.,2022) and MAE (He et al., 2022) are simple yet effective methods to reconstruct masked tokenswithout complicated pretext tasks.
Because of its simplicity and superior performance in downstreamoperations, MIM is attracting attention as a promising technique in image processing.
Nevertheless, we ﬁnd hints suggesting that CL and MIM utilize different aspects of the data, makingthem complementary.
For example, Zhou et al.
; Wang et al.
Yu et al.
(2022) achievehigh predictive performance by harmonizing the image-level and the token-level self-supervisedlearning.
C R ESTRICTED S ELF -A TTENTION
In Section 2, we demonstrate that the homogeneity of self-attention map, due to the attention collapse,helps ViT distinguish images but harms ﬁne-tuning accuracy.
One simple method to inject localityinto self-attentions is to limit the receptive ﬁeld of self-attention by using attention masks.
{+Figure A.
+}
[Figure]
Fraction of "shape" decisions
[Figure]
of
[Figure]
decisions
S h a p e c a t e g
o
r
i e s
MoCo
[Figure]
SimMIM
[Figure]
shows the predictive performance of ViT with restricted local self-attentions.
As expected, the resultsare similar to the performance of MIM; As the kernel size decreases, the linear probing accuracydecreases but the ﬁne-tuning accuracy increases.
These results are consistent with our ﬁndings.
D F INE -T UNED M ODELS I NHERIT THE P ROPERTIES OF P RE - TRAINEDM ODELS
CL is shape-biased and MIM is texture biased in ﬁne-tuning scheme. In Figure 8, we demonstrate that linear probing model with CL is more shape-biased and that with MIM is texture-biased,compared with each other. As in the experiment, we calculate the classiﬁcation results of ImageNet ﬁne-tuned CL and MIM on Stylized-ImageNet, and measure the decrease in accuracy againstfrequency-based random noise.
As we would expected, Figure C.1 shows that the property alsoextends to the ﬁne-tuned model.
Even though we still observe the difference between CL and MIM,the performance gap between CL and MIM is quite reduced compared to the gap between the linearprobing models.
[Figure] [Figure]
cc [Figure] [Figure] Later layers of CL and early layers of MIM are important in ﬁnd-tuning phases.
Figure 11 shows thatlater layers of CL and early layers of MIM are linearlyseparable.
Even though this result implies that laterlayers of CL and early layers of MIM are important,it does not directly support that these properties aremaintained in ﬁne-tuning phases.
To demonstrate this claim in ﬁne-tuning phases, wemeasure the accuracy drop of ﬁne-tuned models usingpre-trained models with a few blocks initialized.
Asshown in Figure D.1a, initializing a few early blocks inthe pre-training models of MIM signiﬁcantly harms theﬁne-tuning accuracy, compared with CL.
This suggeststhat early layers of MIM play an important role in ﬁnetuning.
Conversely, Figure D.1b shows that initializinglater blocks in the pre-training models of MIM does notharms the ﬁne-tuning accuracy signiﬁcantly, suggestingthat they are not important in ﬁne-tuning compared with CL.
A limitation of this experiment is that the accuracy drop is evaluated in single run.
Since the accuracydrop of MoCo is higher than that of SimMIM within a margin of error at the ﬁrst few initializationdepth in Figure D.1b, additional experiments may help improve the result.
In this experiment, weuse the same ﬁne-tuning settings for MoCo and SimMIM, but experiments with tailored ﬁne-tuningsetting for each method may also be helpful.
E CL T RAINS S ELF -A TTENTIONS AND MIM T RAINS MLP S
MLP. We measure (cid:96) 2 distance between initial weight and optimized weight module-wisely.
The larger the distance, the more the method trained the module.
M ETHOD
Attn
MLP
MoCo
SimMIM
CL captures low-frequency information and MIM captureshigh-frequency information.
Moreover, self-attentions capture low-frequency information and MLPs capture highfrequency information (Park & Kim, 2022b).
Therefore,we conjecture that CL mainly trains self-attentions and MIM trains MLPs.
We measure (cid:96) 2 distance between initial weight and optimized weight module-wisely.
Table E.1 provides the result,indicating that the weight of self-attentions changes significantly in CL learning, and in MIM learning, the weightof MLP changes dominantly.
F H YBRID M ODELS O UTPERFORM CL AND MIM IN D OWNSTREAM T ASKS
[Table]
Section 5 demonstrates the claim—CL and
MIM are complementary—only on ImageNet.
To extend the effectiveness of this claim beyond ImageNet, we evaluate the pre-trainedmodels of the hybrid models in Section 5 inanother classiﬁcation task and a semantic segmentation task.
In particular, we measure accuracy on iNaturalist 2018 (Van Horn et al.
, 2018)and mIoU on ADE20K
(Zhou et al., 2019).
Asin Table F.1, the result shows that the hybridmodel of SimMIM and MoCo outperforms both SimMIM and MoCo in the downstream tasks.