S PARSE AND H IERARCHICAL M ASKED M ODELING FOR C ONVOLUTIONAL R EPRESENTATION L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
We identify and overcome two key obstacles in extending the success of BERT-style pre-training, or the masked image modeling, to convolutional networks (convnets): (i) convolution operation cannot handle irregular, random-masked input images; (ii) the single-scale nature of BERT pre-training is inconsistent with convnet’s hierarchical structure.
For (i), we treat unmasked pixels as flatten 3D point clouds and use sparse convolution to encode.
This is the first usage of sparse convolution for 2D masked image modeling.
For the latter, we develop a hierarchical encoderdecoder architecture to reconstruct the image from multi-scale encoded features.
This fully exploits the advantage of hierarchy.
The resulted framework, called
Spar se mas K ed modeling or SparK , releases convnet’s untapped potential: on three downstream tasks, SparK surpasses both state-of-the-art contrastive learning and transformer-based masked modeling by similarly large margins (around +1.0%).
Improvements on object detection and instance segmentation are more substantial (up to 3.5%), verifying the strong transferability of features learned by SparK. We also show its favorable scaling behavior by observing more gains on larger models.
In sum, SparK suggests a promising future of generative pre-training on convnets.
I NTRODUCTION
The pretrain-finetune paradigm in natural language processing (NLP), popularized by BERT (Devlin et al., 2018; Clark et al., 2020; Dong et al., 2019) and GPT (Radford et al., 2019; Brown et al., 2020), is remarkably effective and thus long envied by our vision community.
It is the emerging masked image modeling (Bao et al., 2021;
He et al., 2021; Xie et al., 2021; Chen et al., 2022) initially extends the success of BERT from language transformers to vision transformers (ViTs).
A bold move that increases the mask ratio to a staggering level (40~75%) is largely credited with this success (Bao et al., 2021;
He et al., 2021).
And the field of visual self-supervised learning on ViTs (Dosovitskiy et al., 2020; Liu et al., 2021) has now shifted from contrastive learning (Chen et al., 2021;
Grill et al., 2020; Caron et al., 2021) to BERT-style masked modeling or a fusion of them (Zhou et al., 2021).
Nonetheless, extending the success of BERT pre-training from transformers to convolutional networks (convnets) is a wonderful, but unrealized vision.
The pioneering work (Pathak et al., 2016; Zhang et al., 2017) precedes BERT but only achieves far worse results than supervised pre-training.
There have been efforts over the past year trying to port BERT to convnets, yet eventually compromise on proposing a non-convolutional model (Gao et al., 2022) or non-masked modeling (Fang et al., 2022).
One may therefore ask: what exactly is preventing the application of BERT to convnets?
We conclude that in essence, the difficulty is rooted in the gap between language and vision in terms of data processing (Bateman, 2014;
He et al., 2021; Yang et al., 2019).
Models in NLP (recurrent networks or transformers) process text as a variable-length sequence of words (well-defined semantic units), while convnets have to extract semantics from objects of different sizes (like “units” at different scales).
This huge disparity rises two challenges: (i) Removing information of the masked “words” is difficult for convnets.
For ViTs, an input image is divided to several non-overlapping patches.
Simply dropping masked patches or replacing them with mask tokens can remove the information.
This ease relies on the transformer being able to handle irregular (variable-length) and non-overlapping patches, thus cannot be achieved on convnets as they not only operate on regular grids, but also perform sliding window with overlapping .
One may zero-out all masked pixels and
[Figure]
feed the image into convnet.
This, however, leads to a severe data distribution shift (shown in figure 1) and more another issues (discussed in section 3.1 or figure 3), thus cannot be an ideal solution.
(ii) Single-scale algorithm cannot learn multi-scale (hierarchical) features.
Multi-scale structures have long been a gold-standard in computer vision, which allows visual processing systems like SIFT descriptors (Lowe, 1999; Bay et al., 2006) and pyramid networks
He et al., 2015; Lin et al., 2017) to cope with changes in object scale.
In contrast, masked modeling from NLP originally works in a single-scale manner.
Applying it directly on convnets will miss the advantage of hierarchy.
In this paper, we clear the hurdles above and make BERT suitable for convnet by proposing Spar se mas K ed modeling with hierarchy (SparK).
We first randomly mask images in a patch-wise manner.
Observing the sparse nature of point clouds coincides with these unmasked pixels, we treat them as a flatten 3D point cloud and use sparse convolution for encoding.
This strategy accurately eliminates the information of masked parts whilst allowing convnets to easily handle irregular masked images.
For decoding, we fill in empty positions on multi-scale encoded features with mask embeddings, and feed them into a multi-scale decoder.
This way, we can embrace the advantage of convnet’s hierarchy.
Evaluation is performed on standard downstream tasks including classification, object detection, and instance segmentation.
We observe that both classical (He et al., 2016) and modern (Liu et al., 2022) convnets benefit a lot (by around +1.0%) from SparK, and with more performance gains on larger models that demonstrates scalability.
On all downstream tasks, convnet-based SparK outperforms both (i) state-of-the-art contrastive learning and (ii) transformer-based masked pre-training by similarly large margins (around +1.0%), which shows great promise in generative pre-training for convnets.
The improvements on COCO tasks are more significant than those on ImageNet (up to 3.5%), indicating the representations learned by SparK are highly transferable.
To summarize, SparK provides:
• A first BERT-style pre-training method for convnets, overcoming their inability to handle irregular masked inputs and fully exploiting their advantage of multi-scale structure.
• A leap in convnet’s performance across downstream tasks, showing superior performance over state-of-the-art contrastive learning and transformer-based masked modeling.
• An insight into designing generative pre-training for convnets, and a promise of extending the success of transformer’s pretrain-finetune paradigm to convnets.
The computer vision community has recently paid more attention to research on vision transformers, while convnets no longer appear in the spotlight (Liu et al., 2021;
Bao et al., 2021; He et al., 2021).
Nonetheless, convolutional neural networks, which condense the essence (scale- and translationequivariant; locality; weight-sharing; hardware-friendly) of many classical vision processing systems (Lowe, 1999; Csurka et al., 2004; Jégou et al., 2010; Farabet et al., 2010), still play an irreplaceable role in tackling a wide range of real-world visual tasks that are more challenging or structural than classification (Liu et al., 2017; Jaderberg et al., 2015; Liu et al., 2022).
We hope SparK’s excellent performance compared to transformer-based pre-training will prompt us to revisit convnets as generic backbones for computer vision, and inspire more future research in exploiting their full potential.
R ELATED W ORK 2. H IERARCHICAL VISUAL PROCESSING SYSTEMS
Hierarchical structure is acknowledged as a gold-standard for visual representation systems.
Many fundamental handcrafted feature descriptors (Lowe, 1999; Bay et al., 2006; Rublee et al., 2011) extract multi-scale visual representations via scale-space extremum on feature pyramid (say, octave).
The crux behind this hierarchical design is to extract scale-invariant (or equivariant) features, thus, allows the system to cope with varying object sizes (scales).
Widely used in visual tasks (Yang et al., 2009; Felzenszwalb et al., 2008), these descriptors also motivate the design principles of convolutional networks (He et al., 2016; Tan & Le, 2019; Liu et al., 2022).
Some recent arts also elaborately design hierarchical modules that allow the information aggregation at different granularities to better tackle detection and segmentation tasks using convnets (Long et al., 2015; Lin et al., 2017; Liu et al., 2016).
2. R ECENT PROGRESS ON VISUAL SELF - SUPERVISED LEARNING
Recently, the contrastive learning formulates self-supervise learning as instance classification (Van den Oord et al., 2018;
He et al., 2020; Chen et al., 2020a).
Efforts have been made (Chen et al., 2020b; Grill et al., 2020; Caron et al., 2020; Chen & He, 2021) to overcome the core issue of mode collapse.
More advanced methods are developed since then (Tian et al., 2020; Zbontar et al., 2021;
Chen et al., 2021; Caron et al., 2021), and this line of work dominated the area of visual unsupervised learning until masked generative pre-training along with the vision transformer came into view.
Masked image modeling, inspired by the recent success of masked language modeling in natural language processing (NLP) (Devlin et al., 2018; Liu et al., 2019), has attracted growing interest for visual pre-training.
The pioneering work (Bao et al.
pre-trains vision transformers by learning to predict token indices of masked patches.
He et al.
(2021) ingeniously takes advantage of transformer’s ability to handle variable-length inputs and implements an efficient and scalable method.
Both He et al.
(2021) and Xie et al.
(2021) regress raw RGBs to simplify the pre-training, while Wei et al.
(2022) selects HoG (Dalal & Triggs, 2005) as targets due to their rich semantics.
Gao et al.
(2022) designs a transformer with a heavier patchifier to perform masked modeling.
So far, enormous studies have successfully verified the efficacy of these algorithms on vision transformers (Zhou et al., 2021; Chen et al., 2022; Fang et al., 2022).
However, on the other hand, their methodology is almost the same as that in NLP (Devlin et al., 2018; Liu et al., 2019), and is therefore difficult to be used for hierarchical convolutional models – on convnets, contrastive learning still remains state-of-the-art.
2. S PARSE CONVOLUTION FOR VISUAL REPRESENTATION
Convolution is widely used in 2D computer vision (Dalal & Triggs, 2005;
He et al., 2016), which typically performs sliding window on regular grids (pixels).
When facing with 3D point clouds, this operator quickly becomes unaffordable due to the cubic increasing number of grids (voxels).
Considering point clouds are highly sparse and irregular, one can skip all empty voxels for speed.
This motivates the sparse convolution (sparseconv) (Liu et al., 2015), which is heavily used in modern convnets for 3D visual tasks (Zhou & Tuzel, 2018; Sindagi et al., 2019).
Minkowski Engine (Choy et al., 2019) is one of the most common sparseconv frameworks.
Some prior arts (Verelst &
Tuytelaars, 2020) also tried to introduce sparseconv for faster 2D visual understanding.
And in this work, we have observed the similarity between 3D point clouds and 2D masked images in BERT-style pre-training.
We thus use sparseconv, for the first time, with the purpose of “facilitating the adaptation of convnet to BERT masked modeling”, rather than of “speeding up the computation of convolution”.
A PPROACH
Illustrated in figure 2, our SparK framework aims to pre-train a convolutional network encoder via hierarchical masked image modeling – masking a portion of image and learning to recover it.
We are going to detail SparK by introducing a sparse masking strategy (section 3.1), a hierarchical encoder-decoder architecture (section 3.2), and a target of pre-training task (section 3.3).
3. S PARSELY G ATHERING U NMASKED P ATCHES
We use patch-wise masking following the convention of masked image modeling.
An image x is divided into non-overlapping square patches, each of which will be masked independently with a given probability called mask ratio.
The key to masked image modeling is how to eliminate the information from masked parts.
Previous transformer-based work either removes masked patches, or replaces them with a mask token to eliminate information.
These strategies are only applicable to vision transformer since the transformer is able to handle irregular (variable-length) input, and it operates on non-overlapping patches.
So, simply corrupting a patch (by removing or replacing) can avoid any information leakage from masked pixels.
But this cannot be used directly on convnets which operate on regular grids with overlapping.
A straightforward way is to set all masked pixels to zero to erase information.
Although applicable to convnets, this solution has three evident shortcomings: (i) computing convolution on masked regions is redundant; (ii) it disturbs the distribution of pixels, as illustrated in figure 1; (iii) patterns on mask maps will vanish after several convolutional layers.
We examine and explain this problem in figure 3, similar to the case in Graham & van der Maaten (2017), where we compare this zero-outing with our sparse masking strategy (introduced next).
The problem is particularly acute when using modern deep convnets because of the large number of successive convolutional blocks.
To overcome the problems, we propose to sparsely gather the unmasked patches into a sparse image, and then use sparse convolutions 1 to encode it.
This strategy: (i) ensures no information is leaked; (ii) is well adapted to convnet architectures; (iii) is efficient as sparse convolution computes only at visible places; (iv) solves the issues of pixel distribution shift as well as the “mask pattern vanishing”.
As shown in figure 3, sparse convolution will skip all masked positions on sparse feature maps, and only computes at unmasked points.
In other words, all empty positions remain empty after sparse convolution, and vice versa.
This helps to prevent the shape of the mask pattern from changing with convolution, thus ensures a consistent masking effect and ratio throughout all convolution layers.
3. H IERARCHICAL E NCODING AND D ECODING
By “hierarchical” encoding, we mean the encoder will generate a set of feature maps with different resolutions, namely different scales .
Taking a ResNet-style model for example, it typically containsstages each with a series of convolutional blocks and a downsampling module.
The feature resolution is downsampled by a factor of 2 after every stage.
For an image shaped as H × W , a sparse ResNetproduces feature maps at 4 scales with resolutions of H 4 × W 4 , H 8 × W 8 , H 16 × W 16 , and H 32 × W 32 .
Let
S 1 , S 2 , S 3 , and S 4 be the sparse features, respectively.
They will be used to decode.
[Figure]
Overall, the decoder follows the design of UNet Ronneberger et al.
We use a relatively light decoder that contains three successive blocks {B 3 , B 2 , B 1 } with upsampling.
To reconstruct a dense image, we need to fill in the empty positions on sparse feature maps before feeding them to the decoder.
This is called “ densifying ”.
Taking the smallest sparse feature S 4 as example, all empty positions (inactive sites) on S 4 are filled with a mask embedding [M 4 ] to get a dense feature S ′ 4 .
A projection layer ϕ 4 is applied then, in case encoder and decoder have different network widths:
[Equation]
So D 4 is the input of decoder’s first block B 3 .
It has the same resolution of S 4 with H 32 × W 32 .
Similarly, we can get D 3 , D 2 , and D 1 (with shapes of H 16 × W 16 , H 8 × W 8 , H 4 × W 4 ) via:
[Equation]
Note that four different mask embeddings [M 4 ~ 1 ] and projection layers ϕ 4 ~ 1 are required: they belong to each of the scales.
The final output of decoder is D 1 , with a shape of H 4 × W 4 .
O PTIMIZATION T ARGET AND T RANSFERRING TO D OWNSTREAM
To reconstruct an image from D 1 , a head module h is needed, which should include two more upsampling layers to reach the original resolution of input H × W .
As for the reconstruction target, we choose per-patch normalized pixels as targets with an L 2 -loss, and calculate errors only on masked positions.
These designs have been proven to facilitate models to learn more informative features in
He et al.
and are also verified by ablation study later in section 4.5.
After pre-training, we discard the decoder and only use the encoder for downstream tasks.
When fine-tuning, the pre-trained sparse encoder can be directly generalized to dense images without any tuning, due to the fact that dense input is a special case of the sparse, where every position is active.
E MPIRICAL R ESULTS 4. S IMPLE I MPLEMENTATION OF P RE - TRAINING
Components.
SparK can use any convolutional network as the encoder by converting all its layers (Conv2d, BatchNorm, MaxPooling, etc. ) to sparse form.
After encoding, mask embeddings [M 4 ~ 1 ] are required to fill in sparse feature maps with different resolutions.
These embeddings are implemented as random-initialized learnable feature vectors.
For decoding, we use a lightweight
UNet decoder (Ronneberger et al., 2015) with three blocks.
See Appendix A for more detailed structure.
Positional embeddings are not used since convnet already encodes the spatial information.
Hyperparameters.
For simplicity, SparK uses the same pre-training hyperparameters for all model architectures (ResNets, ConvNeXts, etc. ) and model sizes, even though tuning each would improve more at face value.
We pre-train models with 1.28 million unlabeled images from ImageNet (Deng et al., 2009) training set for 1600 epochs by default.
Only minimal data augmentation is required (a random horizontal flipping and a random resize cropping to 224 × 224 ).
We use the same masking patch size (32) and ratio (60%) as in SimMIM (Xie et al., 2021).
We train with the LAMB optimizer (You et al., 2019), a batch size of 4096, and a cosine-annealing learning rate lr = 0 .
× batchsize/ 256 with linear warmup.
Following the advanced work (Caron et al., 2021;
Zhou et al., 2021), the weight decay also follows a cosine schedule from 0.04 to 0.2.
4. I MAGE N ET E VALUATION
Performance comparison with self-supervised transformers.
We first validate SparK framework on ImageNet (Deng et al., 2009).
We experiment SparK with the pure convolutional model, ConvNeXt (Liu et al., 2022), and make comparisons between it and self-supervised pre-trained transformers of
Dosovitskiy et al.
; Liu et al.
(2021) in table 1.
Smaller models ({ViT,ConvNeXt}-S, Swin-S) and the bigger ones ({ViT, Swin, ConvNeXt}-B) are compared separately.
By comparing the results vertically in table 1, one can find the convolutional models, with SparK, overwhelmingly outperform all transformer-based pre-training methods by large margins ( +0 . 8 ~ 2 . 7 ), though SparK neither employs external models (DALL-E dVAE (Ramesh et al., 2021)), nor profits from advanced (MIM+CL) pre-training.
This is indeed a surprising result since the transformers are well-known data-hungry models with much less inductive bias than convolutional nets, and therefore are considered to benefit more from large-scale self-supervised training.
The result here conveys a new message: convnets may have much more potential than expected, and their capability in visual representation may not be inferior to that of transformers.
The key depends on how to use powerful pre-training algorithms ( e.g. , SparK – masked modeling) to turn this potential into capability .
Efficiency.
Similar to MAE (He et al., 2021), SparK has an advantage on encoding efficiency, especially compared to contrastive learning that processes two or more images in a forward pass.
For instance, DINO and iBOT by default (Caron et al., 2021; Zhou et al., 2021) use multi-crop withglobal crops of 224 × 224 and 10 locals of 96 × 96 , leading to 2 + 10 (96 / 224) 2 ≈ 3 .
8 times the cost of single image encoding.
In contrast, SparK requires only 40% of the theoretical overhead thanks to the sparsity of masked input: 60% of patches are masked, and sparse convolution only processes the rest.
In practice, we found a sparse ResNet-50 can save ~23% memory footprint (26.4 GB vs. 34.
GB for single batch size of 128).
This allows one can train it on a 32GB Tesla V100, which otherwise is impossible for non-sparse pre-training.
The efficiency helps SparK to scale-up more easily.
[Table]
[Table]
T RANSFERRING TO D OWNSTREAM T ASKS
Previous results on ImageNet classification have exposed the potential of SparK pre-training.
In this part we further evaluate the representation quality on fundamental downstream tasks, including object detection and instance segmentation on COCO (Lin et al., 2014).
These tasks are challenging, serving as professional feature evaluators because they place higher demands than classification: models need to predict not only what , but also where the objects (instances) are.
Here, we consider two different settings: comparison with self-supervised vision transformers, and then with convolutional networks.
Performance vs. self-supervised transformers.
In table 2, we comprehensively compare the fine-tuning results on three downstream tasks: image classification (Cls.), object detection (Det.), and instance segmentation (Seg.).
SparK, as the only convolutional pre-training method, perform consistently well in all three downstream tasks, surpassing all vision-transformer-based counterparts by substantial margins.
Even when compared to the strongest SimMIM (Xie et al., 2021) with swintransformer, SparK still yields superior results by +0 . 8 , +0 . 8 , +0 .
7 on three tasks
respectively.
With the base-sized backbone, all transformer-based masked modeling methods perform closely, while
SparK presents a leap in performance.
These observations are consistent with those in section 4.
and once again validate the effectiveness of generative pre-training on hierarchical convnets.
Performance vs. self-supervised convnets.
We then compare SparK to state-of-the-art convolutional contrastive learning methods.
In table 3, all contrastive methods are comparable or slightly worse than supervised pre-training.
While SparK, the first generative pre-training method for hierarchical 2 “Effective epoch” account the total amount of pre-training.
For instance, a
, so the literal
[Table]
[Table]
[Table]
Cls.
[Table]
Det.
(Box)
Seg.
(Mask) 1 × [Table] × 1 × 2 ×
Supervised − − 79.
SimSiam (Chen & He, 2021)
Contrastive 79.
− − − −
MoCo (He et al., 2020)
Contrastive − 38.
MoCov2 (Chen
[Table]
., 2020b)
[Table]
SimCLR (Chen et al., 2020a)
Contrastive 80.
− − − −
InfoMin (Tian et al., 2020)
Contrastive − 40.
BYOL (Grill et al., 2020)
Contrastive 80.
SwAV (Caron et al., 2020)
Contrastive 80.
− 42.
− 38.
SparK (ours)
[Table]
convnets, performs significantly better than them across all downstream tasks by +0 .
5 ~ 1 . 2 points.
In particular, SparK does not rely on sophisticated augmentations which have proven to be essential for contrastive learning (Chen et al., 2020a; Tian et al., 2020).
We attribute these superior results above to the fact that generative pre-training (SparK) inherently provides more supervisory signals than discriminative methods.
It optimizes a reconstruction loss, a form of regression loss, which is considered to be more dense and localized than contrastive loss.
Feature transferability.
Another intriguing phenomenon in table 3 is that the improvements from
SparK on COCO are more significant than those on ImageNet.
Notice that all methods pre-train on
ImageNet with 224 resolution before transferring to COCO, and there are several key differences between the two datasets: (i) image resolution of COCO is much higher; (ii) most images in ImageNet are object-centric, while COCO images usually contain multiple disorganized objects.
This domain gap poses a challenge for transfer learning, and the performance gap between SparK and contrastive learning suggests that models can learn more transferable features through SparK. Taken all results above together, a promising future of generative pre-training on convnets has been initially shown.
4. S CALING - UP S PAR K
In this experiment, we gradually scale-up the model size and test SparK’s fine-tuning performance.
Results are reported in table 4, where we quote the accuracy from Wightman et al.
(the strongest ResNet baselines) and Liu et al.
(2022) (a modern convolutional architecture ConvNeXt) as “Baseline Acc.”.
We refer to the hyperparameters in Fang et al.
(2022) and Liu et al.
for fine-tuning.
Comparing the results vertically in table 4, one can observe that with SparK pre-training, four models with smaller sizes (ResNet-101, ResNet-152, ConvNeXt-S, ConvNeXt-B) achieve performance on par with their non-pretrained versions of larger sizes.
Such a qualitative leap indicates that SparK can push the convnet to the “next level” in terms of representation capability.
Comparing horizontally, SparK greatly improves all supervised baselines by +0.8~1.5, verifying the superiority of self-supervised learned representation.
Overall, the results demonstrate a favorable scaling ability of SparK as larger models benefit more.
These steady gains across classical and modern architectures make us believe SparK can boost many other state-of-the-art convnets, like VAN (Guo et al., 2022).
[Table] [Table]
A BLATION S TUDY
In this study
[Table]
we gradually ablate the components in SparK framework and evaluate the corresponding performance respectively.
ImageNet fine-tuning results of each SparK’s variants are listed in table 5.
Core designs.
We first remove the two most important designs in SparK: sparse masking strategy and the hierarchical architecture.
By replacing our sparse strategy with the zero-outing discussed in section 3.1, we observe a noticeable performance degradation in row 3 of table 5 that almost reaches the supervised baseline.
This suggests that the issues raised by zero-outing (data distribution shift in figure 1 and mask pattern vanish in figure 3) can lead to ineffective pre-training.
We then remove the hierarchical design (row 4), which results in a single-scale masked modeling that is commonly used for transformers (Devlin et al., 2018; Bao et al., 2021;
He et al., 2021; Xie et al., 2021).
It only uses
[Table]
the features at the end of encoder to reconstruct.
This modification is shown to impair the fine-tuning performance as well.
In sum, both sparse strategy and hierarchy play key roles in SparK pre-training.
Other components.
In addition, we find adding absolute positional embeddings (row 5) is practically useless for learning convolutional representations.
We also observe calculating loss values only on masked patches gives higher accuracy (row 6), which is consistent with He et al.
Finally and reasonably [Table] our SparK benefits from longer pre-training as verified in row
[Table]
4. V ISUALIZATION
We visualize some reconstruction results to check how the model performs in pre-training.
From figure 4 we can see that the model is able to make different but plausible predictions on masked regions ( e.g. , in the 2-nd column).
In the 4-th and 6-th columns, the model can almost recover the original appearance of red fruits from the very small portion of the exposed edges.
The clear texture in the 3-rd column also shows the model can capture the visual signals with medium or high frequencies.
C ONCLUSION
The NLP community has witnessed the rise and proliferation of masked modeling on transformers, followed by recent efforts to validate this paradigm on vision transformers.
However, generalizing this method to convolutional networks (convnets) is a wonderful, but unrealized vision.
Observing the sparse nature of point clouds coincides with those masked patches, we propose to sparsely gather unmasked patches into a sparse image, and use sparse convolution to encode them.
We also employ a hierarchical decoder to make full use of the multi-scale encoded features.
The resulting framework, called SparK, makes masked modeling well suited for convnets, and shows superior downstream performance compared with state-of-the-art contrastive learning and transformer-based pre-training.
Through these results, a promising future of generative pre-training on convnets has been initially shown.
We hope our exploration and discovery will inspire more future work to exploit the potential of convnets (maybe by generative pre-training), and prompt us to refocus attention back to them.
A D ETAILS ON D ECODER A RCHITECTURE
The same lightweight decoder of LightDecoder(768, 32) is used equally for all pre-trained convnets, including all ResNets and ConvNeXts.
Its pytorch implementation is as follows.
import math import torch.nn as nn class UNetBlock2x(nn.
Module): def __init__(self, cin, cout): super().__init__()self.b=nn.
Sequential( nn.Conv2d(cin,cin,kernel_size=3, stride=1, padding=1, bias=False), nn.BatchNorm2d(cin),
nn.ReLU6(inplace=
True), nn.
Conv2d(cin, cout, kernel_size=3, stride=1, padding=1, bias=False), nn.BatchNorm2d(cout), ) def forward(self, x): return self.b(x) class DecoderConv(nn.
Module): def __init__(self, cin, cout): super().__init__()self.up=nn.
ConvTranspose2d(cin, cin, kernel_size=4, stride=2, padding=1, bias=
True) self.conv = UNetBlock2x(cin, cout) def forward(self, x): x = self.up(x) return self.conv(x) class LightDecoder(nn.
Module): def __init__(self, decoder_fea_dim, upsample_ratio): super().__init__()self.fea_dim=decoder_fea_dim n = round(math.log2(upsample_ratio)) channels =
[self.fea_dim // 2 ** i for i in range(n + 1)]
self.dec = nn
B F INE - TUNING R ECIPES
We report the detailed fine-tuning recipes on ImageNet and COCO below.
[Table]
[Table]
C L INEAR E VALUATION R ESULTS
We report the small-sized models’ linear evaluation performance in table 8.
Numbers are directly quoted from Chen et al.
MoCoV3 Chen et al.
(2021) is the only contrastive learning method in this comparison, which inherently learns a global representation and is therefore more suitable than non-contrastive methods for tasks such as linear probing.
SparK shows its good performance relative to other non-contrastive ones.
[Table]