S PARSE AND H IERARCHICAL M ASKED M ODELING FOR C ONVOLUTIONAL R EPRESENTATION L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper presents a simple yet powerful framework to pre-train convolutional network (convnet) with Spar se mas K ed modeling.
SparK addresses key challenges in applying transformer-specialized masked modeling to convolutional models: (i) convolution operation cannot handle irregular, random-masked input; (ii) the single-scale nature of existing masked modeling is inconsistent with convnet’s hierarchical structure.
For (i), we sparsely gather all unmasked pixels to a sparse image and use sparse convolution for feature encoding.
For the latter, we develop a hierarchical encoder-decoder to reconstruct from multi-scale encoded features to fully exploit the advantage of hierarchy.
As the first hierarchical masked modeling method designed for convnets, SparK exploits their untapped potential: on three downstream tasks, SparK surpasses both state-of-the-art contrastive learning and transformer-based masked modeling by similarly large margins (around +1.0%).
Improvements on object detection and instance segmentation are more substantial (>1.0%), verifying the strong transferability of features learned by SparK. We also demonstrate its favorable scaling behavior by observing more gains on larger models.
Taken all results together, SparK suggests a promising future of generative pre-training on convnets.
Codes and models will be made publicly available soon.
I NTRODUCTION
Learning expressive representations from limited supervision is an enduring research topic.
In computer vision, many well-established paradigms have been developed to create supervision without labels, which basically fall into two categories (Chen et al., 2020a): generative and discriminative.
The discriminative ones, represented by contrastive learning (Chen et al., 2020a; He et al., 2020;
Grill et al., 2020; Chen & He, 2021; Caron et al., 2020; Zbontar et al., 2021), used to lie at the center of visual self-supervised learning due to their well-proven representation properties ( e.g. , good linear separability) and compelling downstream performance.
It is the introduction of masked modeling (Devlin et al., 2018; Clark et al., 2020) together with transformer architecture (Vaswani et al., 2017) from natural language processing (NLP) that rekindled public interest in generative visual pre-training.
These approaches (Bao et al., 2021;
He et al., 2021; Xie et al., 2021; Fang et al., 2022) outperform their contrastive and supervised counterparts on vision transformers (Dosovitskiy et al., 2020; Touvron et al., 2021; Liu et al., 2021), thereby showing great promise.
Curiously, the success has not yet spread to convolutional networks , especially since some advanced contrastive methods still remain state-of-the-art (Caron et al., 2020; 2021;
Grill et al., 2020; Tian et al., 2020;
Chen et al., 2021).
We therefore ask: what makes it difficult to apply masked modeling to convnets?
In essence, the difficulty is rooted in the gap between NLP and computer vision in terms of data processing (He et al., 2021;
Devlin et al., 2018; Yang et al., 2019): transformer processes text as a variable-length sequence of words (well-defined semantic units), while convnet has to extract semantics from objects of different sizes (like “units” at different scales).
This big difference gives rise to the following challenges: (i) Removing information of masked parts is difficult for convnet.
In vision transformer, an input image is divided to non-overlapping patches.
Simply dropping masked patches or replacing them with mask tokens removes the information.
This ease relies on the transformer being able to handle irregular (variable-length) and non-overlapping patches, thus cannot be achieved on convnets as they not only operate on regular grids, but also perform sliding window with overlapping .
One may zero-out all masked pixels and feed the image into convnet.
This, however, leads to a severe data distribution shift (shown in figure 1) and cannot be an ideal solution.
[Figure]
(ii) Single-scale algorithm cannot learn multi-scale (hierarchical) convolutional features.
Multiscale structure has long been a gold-standard in computer vision.
It allows visual processing systems like SIFT/SURF descriptors (Lowe, 1999; Bay et al., 2006) and pyramid networks (He et al., 2015;
Lin et al., 2017; Liu et al., 2016) to cope with changes in object scale.
In contrast, masked modeling from NLP typically works in a single-scale manner, since the decoding process is based only on the features at the end of encoder.
Applying it directly on convnets will miss the advantage of hierarchy.
In this paper, we develop a methodology called Spar se mas K ed modeling with hierarchy (SparK) to clear all the hurdles above.
We first mask images in a patch-wise manner.
Observing the sparse nature of point clouds coincides with those masked patches, we then sparsely gather unmasked patches to a sparse image, and use sparse convolutions for encoding.
This strategy accurately eliminates the information from masked parts whilst allowing convnets to easily handle irregular masked images.
For decoding, we fill in empty positions on the multi-scale encoded features with mask embeddings, and feed them into a multi-scale decoder.
This way, we can embrace the advantage of feature hierarchy.
Taken together, SparK makes masked modeling well suited for any convolutional models.
Evaluation is performed on standard downstream tasks including classification, object detection, and instance segmentation.
We observe that both classical (He et al., 2016) and modern (Liu et al., 2022) convnets benefit a lot (by around +1.0%) from SparK, and with more performance gains on larger models that demonstrates scalability.
On all downstream tasks, convnet-based SparK outperforms both (i) state-of-the-art contrastive learning and (ii) transformer-based masked pre-training by similarly large margins (around +1.0%), which shows great promise in generative pre-training for convnets.
The improvements on COCO tasks are more significant than those on ImageNet, indicating the representations learned by SparK are highly transferable.
To summarize, SparK provides:
• A first hierarchical masked modeling method for convnets, overcoming their inability to handle irregular masked inputs and fully exploiting their advantage of multi-scale structure.
• A leap in convnet’s performance across downstream tasks, enabling convnets to surpass self-supervised vision transformers with similar model capacity.
• An insight into designing generative pre-training for convnets, and a promise of extending the success of transformer’s pretrain-finetune paradigm to convnets.
The computer vision community has recently paid more attention to research on vision transformers, while convnets no longer appear in the spotlight (Liu et al., 2021;
Bao et al., 2021; He et al., 2021).
Nonetheless, convolutional neural networks, which condense the essence (scale- and translationequivariant; locality; weight-sharing; hardware-friendly) of many classical vision processing systems (Lowe, 1999; Csurka et al., 2004; Jégou et al., 2010; Farabet et al., 2010), still play an irreplaceable role in tackling a wide range of real-world visual tasks that are more challenging or structural than classification (Liu et al., 2017; Jaderberg et al., 2015; Liu et al., 2022).
We hope SparK’s excellent performance compared to transformer-based pre-training will prompt us to revisit convnets as generic backbones for computer vision, and inspire more future research in exploiting their full potential.
R ELATED W ORK 2. H IERARCHICAL VISUAL PROCESSING SYSTEMS
Hierarchical structure is acknowledged as a gold-standard for visual representation systems.
Many fundamental handcrafted feature descriptors (Lowe, 1999; Bay et al., 2006; Rublee et al., 2011) extract multi-scale visual representations via scale-space extremum on feature pyramid (say, octave).
The crux behind this hierarchical design is to extract scale-invariant (or equivariant) features, thus, allows the system to cope with varying object sizes (scales).
Widely used in visual tasks (Yang et al., 2009; Felzenszwalb et al., 2008), these descriptors also motivate the design principles of convolutional networks (He et al., 2016; Tan & Le, 2019; Liu et al., 2022).
Some recent arts also elaborately design hierarchical modules that allow the information aggregation at different granularities to better tackle detection and segmentation tasks using convnets (Long et al., 2015; Lin et al., 2017; Liu et al., 2016).
2. R ECENT PROGRESS ON VISUAL SELF - SUPERVISED LEARNING
Recently, the contrastive learning formulates self-supervise learning as instance classification (Van den Oord et al., 2018;
He et al., 2020; Chen et al., 2020a).
Efforts have been made (Grill et al., 2020; Caron et al., 2020; Chen & He, 2021) to overcome the core issue of mode collapse.
Moco (He et al., 2020; Chen et al., 2020b) employs a momentum-updated offline bank to store sufficient negative samples.
BYOL (Grill et al., 2020) and SimSiam (Chen & He, 2021) explore stop gradient and asymmetric heads, where they find these techniques helpful to get rid of the reliance on negative examples.
More advanced methods are developed since then (Tian et al., 2020; Zbontar et al., 2021;
Chen et al., 2021; Caron et al., 2021), and this line of work dominated the area of visual unsupervised learning until masked generative pre-training along with the vision transformer came into view.
Masked image modeling, inspired by the recent success of masked language modeling in natural language processing (NLP) (Devlin et al., 2018; Liu et al., 2019), has attracted growing interest for visual pre-training.
The pioneering work Bao et al.
(2021) pre-trains vision transformers by learning to predict token indices of masked patches.
He et al.
(2021) ingeniously takes advantage of transformer’s ability to handle variable-length inputs and implements an efficient and scalable method.
Both He et al.
(2021) and Xie et al.
(2021) regress raw RGBs to simplify the pre-training, while Wei et al.
(2022) selects HoG (Dalal & Triggs, 2005) as targets due to their rich semantics.
Gao et al.
(2022) designs a transformer with a heavier patchifier to perform masked modeling.
So far, enormous studies have successfully verified the efficacy of these algorithms on vision transformers (Zhou et al., 2021; Chen et al., 2022; Fang et al., 2022).
However, on the other hand, their methodology is almost the same as that in NLP (Devlin et al., 2018; Liu et al., 2019), and is therefore difficult to be used for hierarchical convolutional models – on convnets, contrastive learning still remains state-of-the-art.
2. S PARSE CONVOLUTION FOR VISUAL REPRESENTATION
Convolution is widely used in 2D computer vision (Dalal & Triggs, 2005;
He et al., 2016), which typically performs sliding window on regular grids (pixels).
When facing with 3D point clouds, this operator quickly becomes unaffordable due to the cubic increasing number of grids (voxels).
Considering point clouds are highly sparse and irregular, one can skip all empty voxels for speed.
This motivates the sparse convolution (sparseconv) (Liu et al., 2015), which is heavily used in modern convnets for 3D visual tasks (Zhou & Tuzel, 2018; Sindagi et al., 2019).
In practice, there are two different sparseconvs.
One is vanilla sparseconv that has the equivalent effect to the ordinary “dense” convolution but with a faster speed.
The other is submanifold sparseconv, which has a different effect from the ordinary: it only computes when the convolution kernel covers a non-empty position, thus keeps the sparsity during convolution and is faster than the former (Graham & van der Maaten, 2017).
A PPROACH
Illustrated in figure 2, our SparK framework aims to pre-train a convolutional network encoder via hierarchical masked image modeling – masking a portion of image and learning to recover it.
We are going to detail SparK by introducing a sparse masking strategy (section 3.1), a hierarchical encoder-decoder architecture (section 3.2), and a target of pre-training task (section 3.3).
3. S PARSELY G ATHERING U NMASKED P ATCHES
We use patch-wise masking following the convention of masked image modeling.
An image x is divided into non-overlapping square patches, each of which will be masked independently with a given probability called mask ratio.
The key to masked image modeling is how to eliminate the information from masked parts.
Previous transformer-based work either removes masked patches, or replaces them with a mask token to eliminate information.
These strategies are only applicable to vision transformer since the transformer is able to handle irregular (variable-length) input, and it operates on non-overlapping patches.
So, simply corrupting a patch (by removing or replacing) can avoid any information leakage from masked pixels.
But this cannot be used directly on convnets which operate on regular grids with overlapping.
A straightforward way is to set all masked pixels to zero to erase information.
Although applicable to convnets, this solution has three evident shortcomings: (i) computing convolution on masked regions is redundant; (ii) it disturbs the distribution of pixels, as illustrated in figure 1; (iii) patterns on mask maps will vanish after several convolutional layers.
We examine and explain this problem in figure 3, similar to the case in Graham & van der Maaten (2017), where we compare this zero-outing with our sparse masking strategy (introduced next).
The problem is particularly acute when using modern deep convnets because of the large number of successive convolutional blocks.
To overcome the problems, we propose to sparsely gather the unmasked patches into a sparse image, and then use sparse convolutions 1 to encode it.
This strategy: (i) ensures no information is leaked; (ii) is well adapted to convnet architectures; (iii) is efficient as sparse convolution computes only at visible places; (iv) solves the issues of pixel distribution shift as well as the “mask pattern vanishing”.
As shown in figure 3, sparse convolution will skip all masked positions on sparse feature maps, and only computes at unmasked points.
In other words, all empty positions remain empty after sparse convolution, and vice versa.
This helps to prevent the shape of the mask pattern from changing with convolution, thus ensures a consistent masking effect and ratio throughout all convolution layers.
3. H IERARCHICAL E NCODING AND D ECODING
By “hierarchical” encoding, we mean the encoder will generate a set of feature maps with different resolutions, namely different scales .
Taking a ResNet-style model for example, it typically containsstages each with a series of convolutional blocks and a downsampling module.
The feature resolution is downsampled by a factor of 2 after every stage.
For an image shaped as H × W , a sparse ResNetproduces feature maps at 4 scales with resolutions of H 4 × W 4 , H 8 × W 8 , H 16 × W 16 , and H 32 × W 32 .
Let
S 1 , S 2 , S 3 , and S 4 be the sparse features, respectively.
They will be used to decode.
[Figure]
Overall, the decoder follows the design of UNet Ronneberger et al.
We use a relatively light decoder that contains three successive blocks {B 3 , B 2 , B 1 } with upsampling.
To reconstruct a dense image, we need to fill in the empty positions on sparse feature maps before feeding them to the decoder.
This is called “ densifying ”.
Taking the smallest sparse feature S 4 as example, all empty positions (inactive sites) on S 4 are filled with a mask embedding [M 4 ] to get a dense feature S ′ 4 .
A projection layer ϕ 4 is applied then, in case encoder and decoder have different network widths:
[Equation]
So D 4 is the input of decoder’s first block B 3 .
It has the same resolution of S 4 with H 32 × W 32 .
Similarly, we can get D 3 , D 2 , and D 1 (with shapes of H 16 × W 16 , H 8 × W 8 , H 4 × W 4 ) via:
[Equation]
Note that four different mask embeddings [M 4 ~ 1 ] and projection layers ϕ 4 ~ 1 are required: they belong to each of the scales.
The final output of decoder is D 1 , with a shape of H 4 × W 4 .
3. O PTIMIZATION T ARGET AND T RANSFERRING TO D OWNSTREAM
To reconstruct an image from D 1 , a head module h is needed, which should include two more upsampling layers to reach the original resolution of input H × W .
As for the reconstruction target, we choose per-patch normalized pixels as targets with an L 2 -loss, and calculate errors only on masked positions.
These designs have been proven to facilitate models to learn more informative features in
He et al.
and are also verified by ablation study later in section 4.5.
After pre-training, we discard the decoder and only use the encoder for downstream tasks.
When fine-tuning, the pre-trained sparse encoder can be directly generalized to dense images without any tuning, due to the fact that dense input is a special case of the sparse, where every position is active.
E MPIRICAL R ESULTS 4. S IMPLE I MPLEMENTATION OF P RE - TRAINING
Components
SparK can use any convolutional network as the encoder by converting all its layers (Conv2d, BatchNorm, MaxPooling, etc. ) to sparse form.
After encoding, mask embeddings [M 4 ~ 1 ] are required to fill in sparse feature maps with different resolutions.
These embeddings are implemented as random-initialized learnable feature vectors.
For decoding, we use a lightweight
UNet decoder (Ronneberger et al., 2015) with three blocks.
See Appendix A for more detailed structure.
Positional embeddings are not used since convnet already encodes the spatial information.
Hyperparameters.
For simplicity, SparK uses the same pre-training hyperparameters for all model architectures (ResNets, ConvNeXts, etc. ) and model sizes, even though tuning each would improve more at face value.
We pre-train models with 1.28 million unlabeled images from ImageNet (Deng et al., 2009) training set for 1600 epochs by default.
Only minimal data augmentation is required (a random horizontal flipping and a random resize cropping to 224 × 224 ).
We use the same masking patch size (32) and ratio (60%) as in SimMIM (Xie et al., 2021).
We train with the LAMB optimizer (You et al., 2019), a batch size of 4096, and a cosine-annealing learning rate lr = 0 .
× batchsize/ 256 with linear warmup.
Following the advanced work (Caron et al., 2021;
Zhou et al., 2021), the weight decay also follows a cosine schedule from 0.04 to 0.2.
4. I MAGE N ET E VALUATION
Performance comparison with self-supervised transformers.
We first validate SparK framework on ImageNet (Deng et al., 2009).
We experiment SparK with the pure convolutional model, ConvNeXt (Liu et al., 2022), and make comparisons between it and self-supervised pre-trained transformers of
Dosovitskiy et al.
; Liu et al.
(2021) in table 1.
Smaller models ({ViT,ConvNeXt}-S, Swin-T) and the bigger ones ({ViT, Swin, ConvNeXt}-B) are compared separately.
By comparing the results vertically in table 1, one can find the convolutional models, with SparK, overwhelmingly outperform all transformer-based pre-training methods by large margins ( +0 . 8 ~ 2 . 7 ), though SparK neither employs external models (DALL-E dVAE (Ramesh et al., 2021)), nor profits from advanced (MIM+CL) pre-training.
This is indeed a surprising result since the transformers are well-known data-hungry models with much less inductive bias than convolutional nets, and therefore are considered to benefit more from large-scale self-supervised training.
The result here conveys a new message: convnets may have much more potential than expected, and their capability in visual representation may not be inferior to that of transformers.
The key depends on how to use powerful pre-training algorithms ( e.g. , SparK – masked modeling) to turn this potential into capability .
Efficiency.
Similar to MAE (He et al., 2021), SparK has high encoding efficiency among all methods in table 1, especially compared to contrastive learning that processes two or more images in a forward pass.
For instance, DINO and iBOT by default (Caron et al., 2021; Zhou et al., 2021) use multi-crop with 2 global crops of 224 × 224 and 10 locals of 96 × 96 , leading to 2 + 10 (96 / 224) 2 ≈ 3 .
8 times the cost of single image encoding.
In contrast, SparK requires only 40% of the overhead thanks to the sparsity of masked input: 60% of patches are masked, and sparse convolution only needs to process the rest.
This efficiency allows SparK to scale-up as easily as MAE, demonstrated in section 4.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
4. T RANSFERRING TO D OWNSTREAM T ASKS
Previous results on ImageNet classification have exposed the potential of SparK pre-training.
In this part we further evaluate the representation quality on fundamental downstream tasks, including object detection and instance segmentation on COCO (Lin et al., 2014).
These tasks are challenging, serving as professional feature evaluators because they place higher demands than classification: models need to predict not only what , but also where the objects (instances) are.
Here, we consider two different settings: comparison with self-supervised vision transformers, and then with convolutional networks.
Performance vs. self-supervised transformers.
In table 2, we comprehensively compare the fine-tuning results on three downstream tasks: image classification (Cls.), object detection (Det.), and instance segmentation (Seg.).
SparK, as the only convolutional pre-training method, perform consistently well in all three downstream tasks, surpassing all vision-transformer-based counterparts by substantial margins.
Even when compared to the strongest SimMIM (Xie et al., 2021) with swintransformer, SparK still yields superior results by +0 . 8 , +0 . 8 , +0 .
7 on three tasks
respectively.
With the base-sized backbone, all transformer-based masked modeling methods perform closely, while
SparK presents a leap in performance.
These observations are consistent with those in section 4.
and once again validate the effectiveness of generative pre-training on hierarchical convnets.
Performance vs. self-supervised convnets.
We then compare SparK to state-of-the-art convolutional contrastive learning methods.
In table 3, all contrastive methods are comparable or slightly worse than supervised pre-training.
While SparK, the first generative pre-training method for hierarchical
Pre-train task
Eff.
epoch
Cls.
(Acc.)
Det.
(Box)
Seg.
(Mask)
1 × 2 × 1 × 2 ×
Supervised − − 79.
SimSiam (Chen & He, 2021)
Contrastive 79.
− − − −
MoCo (He et al., 2020)
Contrastive − 38.
MoCov2 (Chen et al., 2020b)
Contrastive 79.
SimCLR (Chen et al., 2020a)
Contrastive 80.
− − − −
InfoMin (Tian et al., 2020)
Contrastive − 40.
BYOL (Grill et al., 2020)
Contrastive 80.
SwAV (Caron et al., 2020)
Contrastive 80.
− 42.
− 38.
Spark (ours)
Generative 80.
convnets, performs significantly better than them across all downstream tasks by +0 .
5 ~ 1 . 2 points.
In particular, SparK does not rely on sophisticated augmentations which have proven to be essential for contrastive learning (Chen et al., 2020a; Tian et al., 2020).
We attribute these superior results above to the fact that generative pre-training (SparK) inherently provides more supervisory signals than discriminative methods.
It optimizes a reconstruction loss, a form of regression loss, which is considered to be more dense and localized than contrastive loss.
Feature transferability.
Another intriguing phenomenon in table 3 is that the improvements from
SparK on COCO are more significant than those on ImageNet.
Notice that all methods pre-train on
ImageNet with 224 resolution before transferring to COCO, and there are several key differences between the two datasets: (i) image resolution of COCO is much higher; (ii) most images in ImageNet are object-centric, while COCO images usually contain multiple disorganized objects.
This domain gap poses a challenge for transfer learning, and the performance gap between SparK and contrastive learning suggests that models can learn more transferable features through SparK. Taken all results above together, a promising future of generative pre-training on convnets has been initially shown.
4. S CALING - UP S PAR K
In this experiment, we gradually scale-up the model size and test SparK’s fine-tuning performance.
Results are reported in table 4, where we quote the accuracy from Wightman et al.
(the strongest ResNet baselines) and Liu et al.
(2022) (a modern convolutional architecture ConvNeXt) as “Baseline Acc.”.
We refer to the hyperparameters in Fang et al.
(2022) and Liu et al.
for fine-tuning.
Comparing the results vertically in table 4, one can observe that with SparK pre-training, four models with smaller sizes (ResNet-101, ResNet-152, ConvNeXt-S, ConvNeXt-B) achieve performance on par with their non-pretrained versions of larger sizes.
Such a qualitative leap indicates that SparK can push the convnet to the “next level” in terms of representation capability.
Comparing horizontally, SparK greatly improves all supervised baselines by +0.8~1.5, verifying the superiority of self-supervised learned representation.
Overall, the results demonstrate a favorable scaling ability of SparK as larger models benefit more.
These steady gains across classical and modern architectures make us believe SparK can boost many other state-of-the-art convnets, like VAN (Guo et al., 2022).
[Table] [Table]
A BLATION S TUDY
In this study
[Table]
we gradually ablate the components in SparK framework and evaluate the corresponding performance respectively.
ImageNet fine-tuning results of each SparK’s variants are listed in table 5.
Core designs.
We first remove the two most important designs in SparK: sparse masking strategy and the hierarchical architecture.
By replacing our sparse strategy with the zero-outing discussed in section 3.1, we observe a noticeable performance degradation in row 3 of table 5 that almost reaches the supervised baseline.
This suggests that the issues raised by zero-outing (data distribution shift in figure 1 and mask pattern vanish in figure 3) can lead to ineffective pre-training.
We then remove the hierarchical design (row 4), which results in a single-scale masked modeling that is commonly used for transformers (Devlin et al., 2018; Bao et al., 2021;
He et al., 2021; Xie et al., 2021).
It only uses
[Table]
the features at the end of encoder to reconstruct.
This modification is shown to impair the fine-tuning performance as well.
In sum, both sparse strategy and hierarchy play key roles in SparK pre-training.
Other components.
In addition, we find adding absolute positional embeddings (row 5) is practically useless for learning convolutional representations.
We also observe calculating loss values only on masked patches gives higher accuracy (row 6), which is consistent with He et al.
Finally and reasonably, our SparK benefits from longer pre-training as verified in row
[Table]
4. V ISUALIZATION
We visualize some reconstruction results to check how the model performs in pre-training.
From figure 4 we can see that the model is able to make different but plausible predictions on masked regions ( e.g. , in the 2-nd column).
In the 4-th and 6-th columns, the model can almost recover the original appearance of red fruits from the very small portion of the exposed edges.
The clear texture in the 3-rd column also shows the model can capture the visual signals with medium or high frequencies.
C ONCLUSION
The NLP community has witnessed the rise and proliferation of masked modeling on transformers, followed by recent efforts to validate this paradigm on vision transformers.
However, generalizing this method to convolutional networks (convnets) is a wonderful, but unrealized vision.
Observing the sparse nature of point clouds coincides with those masked patches, we propose to sparsely gather unmasked patches into a sparse image, and use sparse convolution to encode them.
We also employ a hierarchical decoder to make full use of the multi-scale encoded features.
The resulting framework, called SparK, makes masked modeling well suited for convnets, and shows superior downstream performance compared with state-of-the-art contrastive learning and transformer-based pre-training.
Through these results, a promising future of generative pre-training on convnets has been initially shown.
We hope our exploration and discovery will inspire more future work to exploit the potential of convnets (maybe by generative pre-training), and prompt us to refocus attention back to them.
A D ETAILS ON D ECODER A RCHITECTURE
The same lightweight decoder of LightDecoder(768, 32) is used equally for all pre-trained convnets, including all ResNets and ConvNeXts.
Its pytorch implementation is as follows.
import math import torch.nn as nn class UNetBlock2x(nn.
Module): def __init__(self, cin, cout): super().__init__()self.b=nn.
Sequential( nn.Conv2d(cin,cin,kernel_size=3, stride=1, padding=1, bias=False), nn.BatchNorm2d(cin),
nn.ReLU6(inplace=
True), nn.
Conv2d(cin, cout, kernel_size=3, stride=1, padding=1, bias=False), nn.BatchNorm2d(cout), ) def forward(self, x): return self.b(x) class DecoderConv(nn.
Module): def __init__(self, cin, cout): super().__init__()self.up=nn.
ConvTranspose2d(cin, cin, kernel_size=4, stride=2, padding=1, bias=
True) self.conv = UNetBlock2x(cin, cout) def forward(self, x): x = self.up(x) return self.conv(x) class LightDecoder(nn.
Module): def __init__(self, decoder_fea_dim, upsample_ratio): super().__init__()self.fea_dim=decoder_fea_dim n = round(math.log2(upsample_ratio)) channels =
[self.fea_dim // 2 ** i for i in range(n + 1)]
self.dec = nn