

=-1 Transformers have quickly shined in the computer vision world since the emergence of Vision Transformers (ViTs). The dominant role of convolutional neural networks (CNNs) seems to be challenged by increasingly effective transformer-based models. Very recently, a couple of advanced convolutional models strike back with large kernels motivated by the local but large attention mechanism, showing appealing performance and efficiency. While one of them, i.e. RepLKNet, impressively manages to scale the kernel size to 31$\times$31 with improved performance, the performance starts to saturate as the kernel size continues growing, compared to the scaling trend of advanced ViTs such as Swin Transformer. In this paper, we explore the possibility of training extreme convolutions larger than 31$\times$31 and test whether the performance gap can be eliminated by strategically enlarging convolutions. This study ends up with a recipe for applying extremely large kernels from the perspective of sparsity, which can smoothly scale up kernels to 61$\times$61 with better performance. Built on this recipe, we propose *Sparse Large Kernel Network* (**SLaK**), a pure CNN architecture equipped with 51$\times$51 kernels that can perform on par with or better than state-of-the-art hierarchical Transformers and modern ConvNet architectures like ConvNeXt and RepLKNet, on ImageNet classification as well as typical downstream tasks.

<span>.8</span>\[.5,0\](0.53, -0.623)

Introduction
============

=-1 Since invented @fukushima1982neocognitron [@lecun1989handwritten; @lecun1998gradient], convolutional neural networks (CNNs) @Alexnet [@vgg; @resnet; @densenet; @howard2017mobilenets; @xie2017aggregated; @efficientnet] have been one of the most indispensable architectures of machine learning in the last decades. However, the dominance of CNNs has been significantly challenged by Transformer @vaswani2017attention over the past few years. Stemming from natural language processing, Vision Transformers (ViTs) @dosovitskiy2021an [@d2021convit; @touvron2021training; @wang2021pyramid; @liu2021swin; @vaswani2021scaling] have demonstrated strong results in various computer vision tasks including image classification @dosovitskiy2021an [@yuan2021volo], object detection @dai2021dynamic [@liu2021swin], and segmentation @xie2021segformer [@wang2021max; @wang2021end; @cheng2021per]. Meanwhile, works on understanding of ViTs have blossomed. Plausible reasons behind the success of ViTs are fewer inductive bias @dosovitskiy2021an, long-range dependence @vaswani2017attention, advanced architecture @yu2021metaformer, and more human-like representations @tuli2021convolutional, etc.

Recently, there is a rising trend that attributes the supreme performance of ViTs to the ability to capture a large receptive field. In contrast to CNNs which perform convolution in a small sliding window (e.g., 3$\times$3 and 5$\times$5) with shared weights, global attention or local attention with larger window sizes in ViTs @liu2021swin directly enables each layer to capture large receptive field. Inspired by this trend, some recent works on CNNs @liu2022convnet [@ding2022scaling] strike back by designing advanced pure CNN architecture and plugging large kernels into them. For instance, RepLKNet @ding2022scaling successfully scales the kernel size to 31$\times$31, while achieving comparable results to Swin Transformer @liu2021swin. However, large kernels are notoriously difficult to train. Even under the assistance of a parallel branch with small kernels, the performance of RepLKNet starts to saturate as the kernel size continues increasing, compared to the scaling trend of advanced ViTs such as Swin Transformer. It remains mysterious whether we can exceed the Transformer-based models by further scaling the kernel size beyond 31$\times$31.

\[t\]

In this paper, we attempt to answer this research question by exploring the *sparsity* commonly observed in the human visual system. Sparsity has been seen as one of the most important principles in the primary visual cortex (V1) @tong2003primary, where the incoming stimuli have been hypothesized to be sparsely coded and selected @desimone1995neural [@olshausen1997sparse; @vinje2000sparse]. We extensively study the trainability of large kernels and unveil three main observations: **(i)** existing methods that either naively applying larger kernels @liu2022convnet or assisting with structural re-parameterization @ding2022scaling are unable to scale kernel sizes beyond 31$\times$31; **(ii)** replacing one large M$\times$M kernel with two rectangular, parallel kernels (M$\times$N and N$\times$M, where N $<$ M) can smoothly scale the kernel size up to 61$\times$61 with improved performance; **(iii)** constructing with sparse groups while expanding width significantly boosts the performance.

Built upon these observations, we propose SLaK – Sparse Large Kernel Network – a new pure CNNs architecture equipped with an unprecedented kernel size of 51$\times$51. Evaluated across a variety of tasks including ImageNet classification @deng2009imagenet, semantic segmentation on ADE20K @zhou2019semantic, and object detection on PASCAL VOC 2007 @pascal-voc-2007, SLaK achieves higher accuracy than CNN pioneers such as ResNe(X)t @resnet [@xie2017aggregated] and ConvNeXt @liu2022convnet as well as attention-based models e.g., Swin Transformer @liu2021swin on ImageNet. Our analysis of effective receptive field (ERF) shows that when plugged in the recently proposed ConvNeXt, our method is able to cover a large region of ERF than the existing larger kernel paradigms, while introducing the human-like peripheral inductive biases @lettvin1976seeing [@min2022peripheral].

Related Work
============

Large Kernel in Attention
-------------------------

Originally introduced for Natural Language Processing @vaswani2017attention and extended in Computer Vision by @dosovitskiy2021an, self-attention can be viewed as a global depth-wise kernel that enables each layer to have a global receptive field. Swin Transformer @liu2021swin is a ViTs variant that adopts local attention with a shifted window manner. Compared with global attention, local attention @ramachandran2019stand [@vaswani2021scaling; @chu2021twins; @liu2021swin2; @dong2021cswin] can greatly improve the memory and computation efficiency with appealing performance. Since the size of attention windows is at least 7, it can be seen as an alternative class of large kernel. A recent work @guo2022visual proposes a novel large kernel attention module that uses stacked depthwise, small convolution, dilated convolution as well as pointwise convolution to capture both local and global structure.

Large Kernel in Convolution
---------------------------

Large kernels in convolution date back to the 2010s @krizhevsky2012imagenet [@szegedy2015going; @szegedy2017inception], if not earlier, where large kernel sizes such as 7$\times$7 and 11$\times$11 are applied. Global Convolutional Network (GCNs) @peng2017large enlarges the kernel size to 15 by employing a combination of 1$\times$M + M$\times$1 and M$\times$1 + 1$\times$M convolutions. However, the proposed method leads to performance degradation on ImageNet. The family of Inceptions @szegedy2016rethinking [@szegedy2017inception] allows for the utilization of varying convolutional kernel sizes to learn spatial patterns at different scales. With the popularity of VGG @simonyan2014very, it has been common over the past decade to use a stack of small kernels (1$\times$1 or 3$\times$3) to obtain a large receptive field @resnet [@howard2017mobilenets; @xie2017aggregated; @huang2017densely]. Until very recently, some works start to revive the usage of large kernels in CNNs. Han et al. @han2021connection find that dynamic depth-wise convolution (7$\times$7) performs on par with the local attention mechanism if we substitute the latter with the former in Swin Transformer. Liu et al. @liu2022convnet imitate the design elements of Swin Transformer @liu2021swin and design ConvNeXt employed with 7$\times$7 kernels, surpassing the performance of the former. RepLKNet @ding2022scaling for the first time scales the kernel size to 31$\times$31 by constructing a small kernel (e.g., 3$\times$3 or 5$\times$5) parallel to it and achieves comparable performance to the Swin Transformer. Recently,  @chen2022scaling reveals large kernels to be feasible and beneficial for 3D networks too.

Prior works have explored the idea of paralleling @peng2017large or stacking @szegedy2017inception two complementary M$\times$1 and 1$\times$M kernels. However, they limit the short edge to 1 and do not scale the kernel size beyond 51$\times$51. Different from those prior arts, we decompose a large kernel into two complementary non-square kernels (M$\times$N and N$\times$M), improving the scalability of convolutions with large kernels.

\[t\]

Dynamic Sparsity
----------------

=-1 Dynamic sparsity @mocanu2018scalable [@liu2021sparse; @evci2020rigging; @mostafa2019parameter; @jayakumar2020top; @chen2021chasing] is a recently-emerged research topic that attempt to train intrinsically sparse neural networks from scratch using only a small proportion of parameters and FLOPs (as illustrated in Figure \[fig:DST\]). A desirable property of dynamic sparsity is that the model is sparse from the beginning, hence the training and inference FLOPs and memory requirements are only a small fraction of the dense models. Different from post-training pruning @mozer1989using [@han2015deep; @frankle2018lottery], models built with dynamic sparsity can be trained from scratch to match their dense counterparts without involving any pre-training or dense training. Dynamic sparsity stems from Sparse Evolutionary Training (SET) @mocanu2018scalable [@liu2021sparse] which randomly initializes the sparse connectivity between layers with *Erds-R<span>é</span>nyi* topology @24gotErdos1959 and dynamically adjusts the sparse connectivity via a parameter prune-and-grow scheme during the course of training. The parameter prune-and-grow scheme allows the model's sparse structure to gradually evolve, achieving better performance than naively training a static sparse network @liu2021we. Although there exist numerous pruning criteria in the literature, simple magnitude pruning typically performs well in the field of dynamic sparsity. On the other hand, the criteria used to grow weights back differs from method to method, including randomness @mocanu2018scalable [@mostafa2019parameter], momentum @dettmers2019sparse, and gradient @evci2020rigging [@liu2021neuro]. In this paper, our target is not to find sparse networks that can match the corresponding dense networks. Motivated by the principle of ResNeXt @xie2017aggregated [@liu2022convnet] – "use more groups, expand width", we instead attempt to leverage dynamic sparsity to scale neural architectures with extreme kernels.

Recipe for Applying Extremely Large Kernels beyond 31$\times$31 {#sec:recipe}
===============================================================

We first study the performance with extreme kernel sizes larger than 31$\times$31 and share our three main observations here. We take the recently-developed CNN architecture ConvNeXt @liu2022convnet on ImageNet-1K as our benchmark to conduct this study.

=-1 We follow recent works @liu2022convnet [@bao2021beit; @liu2021swin; @ding2022scaling; @touvron2021training] using Mixup @zhang2017mixup, Cutmix @yun2019cutmix, RandAugment @cubuk2020randaugment, and Random Erasing @zhong2020random as data augmentations. Stochastic Depth @huang2016deep and Label Smoothing @szegedy2016rethinking are applied as regularization with the same hyper-parameteres as used in ConvNeXt @liu2022convnet. We train models with AdamW @loshchilov2018decoupled. All models are trained for *a reduced length of 120 epochs* in this section, to just observe the scaling trend of large kernel sizes. Later in Section \[sec:Ima\_Evaluation\], we will adopt the full training recipe and train our models for 300 epochs, to enable fair comparisons with state-of-the-art models. Please refer to Appendix \[app:training\_config\] for more details.

**Observation \#1: existing techniques fail to scale convolutions beyond 31$\times$31.** Very recently, RepLKNet @ding2022scaling successfully scales convolutions up to 31$\times$31 with structural re-parameterization @ding2019acnet [@ding2021repmlpnet]. We further increase the kernel size to 51$\times$51 and 61$\times$61 and see whether larger kernels can bring more gains. Following the design in @ding2022scaling, we successively set the kernel size of each stage as \[51, 49, 47, 13\] and \[61, 59, 57, 13\]. The test accuracy is reported in Table \[tab:51\_51\]. As expected, naively increasing kernel size from 7$\times$7 to 31$\times$31 decreases the performance significantly, whereas RepLKNet can overcome this problem, improving the accuracy by 0.5%. However, this trend does not hold with larger kernels as increasing kernel size to 51$\times$51 starts to hurt the performance.

One plausible explanation is that although the receptive field may be enlarged by using extremely large kernels e.g., 51$\times$51 and 61$\times$61, it might fail to maintain certain desirable properties, e.g., locality. Since the stem cell in standard ResNet @resnet and ConvNeXt results in a 4$\times$ downsampling of the input images, extreme kernels with 51$\times$51 are already roughly equal to the global convolution for the typical $224\times224$ ImageNet. Therefore, this observation makes sense as local attention @dosovitskiy2021an [@chu2021twins; @dong2021cswin; @liu2021swin2] usually outperforms global attention @dosovitskiy2021an in a similar mechanism of ViTs @han2021connection. Motivated by this, we see the opportunity to address this problem by introducing the locality while preserving the ability to capture global relations.

**Observation \#2: decomposing a square large kernel into two rectangular, parallel kernels smoothly scales the kernel size up to 61.** Although using convolutions with medium sizes (e.g., 31$\times$31) seemingly can directly avoid this problem, we want to investigate to see if we can further push the performance of CNNs by using (global) extreme convolutions. Our recipe here is to approximate the large M$\times$M kernel with a combination of two parallel and rectangular convolutions whose kernel size is M$\times$N and N$\times$M (where N $<$ M), respectively, as shown in Figure \[fig:SLaK\]. Following @ding2022scaling, we keep a 5$\times$5 layer parallel to the large kernels and summed up their outputs after a batch norm layer.

This decomposition not only inherits the ability of large kernels to capture long-range dependencies but also can extract local context features with a shorter edge. What is more, existing techniques for large kernel training @liu2022convnet [@ding2022scaling] suffer from quadratic computational and memory overhead as the size of depth-wise kernels increases. In stark contrast, the overhead of this method increases linearly with the kernel size (Figure \[fig:scaling\_efficiency\]). The performance of kernel decomposition with N $=5$ is reported as the "Decomposed" group in Table \[tab:decomposed\_and\_sparse\]. As decomposition reduces the FLOPs, the network is expected to sacrifice a bit of accuracy compared to structural re-parameterization (RepLKNet) with medium kernels i.e. 31$\times$31. However, as the convolution size increases to global convolution, it can surprisingly scale kernel size up to 61 with improved performance.

=-1 **Observation \#3: "use *sparse* groups, expand more" significantly boosts the model capacity.** Recently proposed ConvNeXt @liu2022convnet revisits the principle introduced in ResNeXt @xie2017aggregated that splits convolutional filters into small but more groups. Instead of using the standard group convolution, ConvNeXt simply employs depthwise convolutions with an increased width to achieve the goal of "use more groups, expand width". In this paper, we attempt to extend this principle from an alternative perspective – "use *sparse* groups, expand more".

To be specific, we first replace the dense convolutions with sparse convolutions, where the sparse kernels are randomly constructed based on the layer-wise sparsity ratio of SNIP @lee2018snip due to its strong performance on large-scale models @liu2022the [^2]. After construction, we train the sparse model with dynamic sparsity @mocanu2018scalable [@liu2021sparse], where the sparse weights are dynamically adapted during training by pruning the weights with the lowest magnitude and growing the same number of weights randomly @mocanu2018scalable. Doing so enables dynamic adaptation of sparse weights, leading to better local features. As kernels are sparse throughout training, the corresponding parameter count and training/inference FLOPs are only proportional to the dense models. See Appendix \[app:config\_DS\] for the configurations for dynamic sparsity. To evaluate, we sparsify the decomposed kernels with 40% sparsity and report the performance as the "Sparse Decomposed" group. We can observe in the middle column of Table \[tab:decomposed\_and\_sparse\] that dynamic sparsity notably reduces the FLOPs by more than 2.0G, resulting in a temporary performance degradation.

=-1 We next show that the above high efficiency of dynamic sparsity can be effectively transferred to model scalability. Dynamic sparsity allows us to computation-friendly scale the model size up. For example, using the same sparsity (40%), we can expand the model width by 1.3$\times$ while keeping the parameter count and FLOPs roughly the same as the dense model. This brings us significant performance gains, increasing the performance from 81.3% to 81.6% with extreme 51$\times$51 kernels. Impressively, equipped with 61$\times$61 kernels, our method outperforms the previous state of the arts @liu2022convnet [@ding2022scaling] while saving 55% FLOPs.

Sparse Large Kernel Network - SLaK
==================================

So far, we have discovered our recipe which can successfully scale up kernel size to 61 without backfiring performance. It comprises two sparsity-inspired designs. On the macro level, we construct an intrinsically sparse network and further expand the network to improve the network capacity while preserving a similar model size. On the micro level, we decompose a dense large kernel into two complementary kernels with dynamic sparsity to improve the scalability of large kernels. Different from the conventional post-training pruning @han2015deep [@frankle2018lottery], we directly train our network from scratch without involving any pre-training or fine-tuning. Built on this recipe, we next propose Sparse Large Kernel Network (SLaK), a pure CNN architecture employed with extreme 51$\times$51 kernels.

SLaK is built based on the architecture of ConvNeXt. The design of the stage compute ratio and the stem cell are inherited from ConvNeXt. The number of blocks in each stage is \[3, 3, 9, 3\] for SLaK-T and \[3, 3, 27, 3\] for SLaK-S/B. The stem cell is simply a convolution layer with 4$\times$4 kernels and 4 strides. We increase the kernel size of ConvNeXt's stages to \[51, 49, 47, 13\], respectively, and replace each M$\times$M kernel with a combination of M$\times$5 and 5$\times$M[^3] kernels as illustrated in Figure \[fig:SLaK\]. We find that adding a BatchNorm layer directly after each decomposed kernel is crucial before summing the output up. Following the guideline of "use *sparse* groups, expand more", we further sparsify the whole network and expand the width of stages by 1.3$\times$, ending up with SLaK-T/S/B. Even though we are aware that there is a large space to improve the performance of SLaK by tuning the trade-off between model width and sparsity, we keep the width of all models as 1.3$\times$ for simplicity. The sparsity is set as 40% for all models. While our model is equipped with extreme 51$\times$51 kernels, the overall parameter count and FLOPs do not increase too much and it is quite efficient in practice thanks to the excellent implementation provided by @ding2022scaling.

Evaluation of SLaK {#sec:Ima_Evaluation}
==================

To comprehensively verify the effectiveness of SLaK, we compare it with various state-of-the-art baselines on ImageNet-1K classification @deng2009imagenet, semantic segmentation on ADE20K @zhou2019semantic, and object detection on PASCAL VOC 2007 @pascal-voc-2007.

Evaluation on ImageNet-1K {#sec:eva_ima}
-------------------------

ImageNet-1K is the most highly-used image classification dataset, containing 1,281,167 training images, 50,000 validation images. We use exactly the same training configurations in Section \[sec:recipe\] and train SLaK on ImageNet with AdamW @loshchilov2018decoupled for 300 epochs. Exponential Moving Average (EMA) @polyak1992acceleration is adopted to improve the performance. We observe that models with BatchNorm layers lead to poor performance of EMA when trained with 300 epochs (also pointed by @liu2022convnet). Fortunately, we resolve this by running one additional pass over the training data, as used in @garipov2018loss [@izmailov2018averaging]. Please refer to Appendix \[app:training\_config\] for more details about the training configurations.

We compare the performance of SLaK on ImageNet-1K with various the state-of-the-arts in Table \[tab:imagenet-1k\]. With similar model sizes and FLOPs, SLaK outperforms the existing convolutional models such as ResNe(X)t @resnet [@xie2017aggregated], RepLKNet @ding2022scaling, and ConvNeXt @liu2022convnet. Without using any complex attention modules and patch embedding, SLaK is able to achieve an appealing higher accuracy than the state-of-the-art transformers, e.g., Swin Transformer @liu2021swin and Pyramid Vision Transformer @wang2021pyramid [@wang2022pvt]. Perhaps more interestingly, directly replacing the 7$\times$7 of ConvNeXt-S to 51$\times$51 is able to improve the accuracy over the latter by 0.7%.

<span>l|ccccc</span> Model & Image Size & \#Param. & FLOPs & Top-1 Accuracy (%)\
ResNet-50 @resnet & 224$\times$224 & 26M & 4.1G & 76.5\
ResNeXt-50-32$\times$4d @xie2017aggregated & 224$\times$224 & 25M & 4.3G & 77.6\
ResMLP-24 @touvron2021resmlp & 224$\times$224 & 30M & 6.0G & 79.4\
gMLP-S @liu2021pay & 224$\times$224 & 20M & 4.5G & 79.6\
DeiT-S @touvron2021training & 224$\times$224 & 22M & 4.6G & 79.8\
PVT-Small @wang2021pyramid & 224$\times$224 & 25M & 3.8G & 79.8\
Swin-T @liu2021swin & 224$\times$224 & 28M & 4.5G & 81.3\
TNT-S @han2021transformer & 224$\times$224 & 24M & 5.2G & 81.3\
T2T-ViT$_t$-14 @yuan2021tokens & 224$\times$224 & 22M & 6.1G & 81.7\
ConvNeXt-T @liu2022convnet & 224$\times$224 & 29M & 4.5G & 82.1\

**SLaK-T** & 224$\times$224 & 30M & 5.0G & **82.5**\
Mixer-B/16 @tolstikhin2021mlp & 224$\times$224 & 59M & 11.6G & 76.4\
ResNet-101 @resnet & 224$\times$224 & 45M & 7.9G & 77.4\
ResNeXt101-32x4d @xie2017aggregated & 224$\times$224 & 44M & 8.0G & 78.8\
PVT-Large @wang2021pyramid & 224$\times$224 & 61M & 9.8G & 81.7\
T2T-ViT$_t$-19 @yuan2021tokens & 224$\times$224 & 39M & 9.8G & 82.4\
Swin-S @liu2021swin & 224$\times$224 & 50M & 8.7G & 83.0\
ConvNeXt-S @liu2022convnet & 224$\times$224 & 50M & 8.7G & 83.1\
**SLaK-S** & 224$\times$224 & 55M & 9.8G & **83.8**\
ViT-Base/16 @dosovitskiy2021an & 224$\times$224 & 87M & 17.6G & 77.9\
DeiT-Base/16 @touvron2021training & 224$\times$224 & 87M & 17.6G & 81.8\
RepLKNet-31B @ding2022scaling & 224$\times$224 & 79M & 15.3G & 83.5\
Swin-B @liu2021swin & 224$\times$224 & 88M & 15.4G& 83.5\
ConvNeXt-B @liu2022convnet & 224$\times$224 & 89M & 15.4G & 83.8\
**SLaK-B** & 224$\times$224 & 95M & 17.1G & **84.0**\

\[tab:imagenet-1k\]

Evaluation on ADE20K
--------------------

=-1 For semantic segmentation, we choose the widely-used dataset ADE20K. ADE20K is a large-scale dataset for semantic segmentation which contains 20K images of 150 categories for training and 2K for validation. The backbones are pre-trained on ImageNet-1K with 224$\times$224 input for 120 epochs and then are finetuned with UperNet @xiao2018unified for 80K iterations. We report the mean Intersection over Union (mIoU) with single-scale testing in Table \[tab:ADE20K\] for comparison. Overall, we can see a very clear trend that the performance increases as the kernel size. Specifically, RepLKNet scales the kernel size of ConvNeXt-T from 7$\times$7 to 31$\times$31 and achieves 1.6% higher mIoU. Notably, SLaK-T with larger kernels (51$\times$51) further brings 0.9% mIoU improvement over ConvNeXt-T (RepLKNet) with 31$\times$31 kernels while requiring even fewer FLOPs.

\[tab:ADE20K\]

Evaluation on PASCAL VOC 2007
-----------------------------

=-1 PASCAL VOC 2007 has been widely used as a benchmark for object detection, semantic segmentation. In this section, we evaluate our model on object detection. Models are on pre-trained on ImageNet-1K for 300 epochs and fine-tuned with Faster-RCNN. Table \[tab:PASCAL\] shows the results comparing SLaK-T, ConvNeXt-T, ConvNet-T (RepLKNet), and traditional convolutional networks, i.e., ResNet. Again, large kernels lead to better performance. Specifically, ConvNeXt-T with 31$\times$31 kernels achieves 0.7% higher mean Average Precision (mAP) than the 7$\times$7 kernels and SLaK-T with 51 kernel sizes further brings 1.4% mAP improvement, highlighting the crucial role of extremely large kernels on downstream vision tasks.

\[tab:PASCAL\]

Analysis of SLaK
================

Effective Receptive Field (ERF)
-------------------------------

\[h\]

The concept of receptive field @luo2016understanding [@araujo2019computing] is important for deep CNNs work, as anywhere in an input image outside the receptive field of a unit does not affect the value of that unit. For instance, recent work @ding2022scaling successfully scales kernels up to 31$\times$31, showing enlarged ERF and also higher accuracy over small-kernel models @resnet. As shown in the ERF theory @luo2016understanding, ERF is proportion to $\mathcal{O}(k\sqrt{n})$, where $k$ and $n$ refers to the kernel size and the network depth, respectively. Therefore, the hypothesis behind the kernel decomposition in SLaK is that the two decomposed M$\times$N and N$\times$M kernels can well maintain the ability of large kernels in terms of capturing large ERF, while can also focus on fine-grained local features with the shorter edge (N). To evaluate this hypothesis, we make a comparison between the ERF captured by SLaK and RepLKNet.

We apply RepLKNet (full kernel with structural re-parameterization) to ConvNeXt-T and compare it with SLaK-T in terms of ERF. Following @kim2021dead [@ding2022scaling], we choose and resize 50 images from validation set to 1024$\times$1024 and measure the contribution of the pixel on the input image to the central point of the feature map generated in the last layer and sum them up to a 1024$\times$1024 matrix. The matrix is visualized in Figure \[fig:ERF\]. From the left sub-figure, we can see that although the original ConvNeXt already improves the kernel size to 7$\times$7, the high-contribution pixels only emerge in the center of the input. It is surprising to see that even the 31$\times$31 kernels used by RepLKNet are not sufficient for ConvNeXt to cover the whole input. In contrast, high-contribution pixels of SLaK are distributed in a larger range of input, indicating a larger ERF. Perhaps more interestingly, there is a square of pixels emerging with brighter colors on top of the whole ERF region of SLaK. This observation is perfectly in line with our hypothesis that SLaK is not able to capture just long-range dependences but also the local context features. Moreover, it seems that our method discovers a special type of peripheral vision @lettvin1976seeing [@min2022peripheral] in the human vision system: the entire visual field is partitioned into multiple regions where the center of our gaze is processed with high contribution for discovering low-level details, and the contribution of the far peripheral regions is decreased only for high-level contexts.

                           Kernel Size   $t=20\%$   $t=30\%$    $t=50\%$    $t=99\%$
  ----------------------- ------------- ---------- ----------- ----------- -----------
  ResNet-101                 3-3-3-3       0.9%       1.5%        3.2%        22.4%
  ResNet-152                 3-3-3-3       1.1%       1.8%        3.9%        34.4%
  ConvNeXt-T                 7-7-7-7       2.0%       3.6%        7.7%        55.5%
  ConvNeXt-T (RepLKNet)    31-29-27-13     4.0%       9.1%        19.1%       97.5%
  SLaK-T                   51-49-47-13   **6.9%**   **11.5%**   **23.4%**   **97.5%**

  : **Quantitative analysis on the ERF with the high-contribution area ratio $r$**. A larger value suggests a smoother distribution of high-contribution pixels, hence larger ERF.<span data-label="tab:ERF_ratio"></span>

We further quantify ERF of various models by reporting the high-contribution area ratio $r$ in Table \[tab:ERF\_ratio\] following @ding2022scaling. $r$ refers to the ratio of a minimum rectangle to the overall input area that can cover the contribution scores over a given threshold $t$. Assuming an area of A$\times$A at the center can cover $t=30\%$ contribution scores of an 1024$\times$1024 input, then the area ratio of $t=30\%$ is $r=(A/1024)^2$. Larger $r$ suggests a smoother distribution of high-contribution pixels. We can see that with global kernels, SLaK naturally considers a larger range of pixels to make decisions than ConvNeXt and RepLKNet.

Kernel Scaling Efficiency
-------------------------

\[h\]

As we mentioned in Section \[sec:recipe\], the two components of SLaK, kernel decomposition and dynamic sparsity, substantially improve the scaling efficiency of kernel sizes. To support this, we report the overhead required by various large kernel training methods in Figure \[fig:scaling\_efficiency\]. We simply replace all the kernels in stages of ConvNeXt-T with a set of kernel sizes from 7 to 151 and report the required GFLOPs and the number of parameters. With medium kernel sizes (smaller than 31$\times$31), the difference between full-kernel scaling (e.g., ConvNeXt and RepLKNet) and our methods is negligible. However, we can see a big gap between full-kernel scaling and kernel decomposition as the kernel size increases to sizes beyond 51$\times$51. Even using ultra-large 151$\times$151 kernels, our methods require fewer FLOPs and parameters than full-kernel scaling with 51$\times$51 kernels. Such significant reduction in extreme large kernels can be very important for training with high resolution input images @he2017mask [@Tan2019efficientnet; @yu2022coca; @lin2017feature], where the image resolution can be as large as 600$\times$600 @he2017mask. Growing evidence has demonstrated that training with high resolution is an effective performance booster for classification @Tan2019efficientnet [@wortsman2022model] and object detection @he2017mask [@lin2017feature]. In this scenario, it is likely that we need extremely large kernels to enable the global receptive field.

Trade-off between Sparsity and Width
------------------------------------

  Model     (Sparsity, Width)    \#Param   FLOPs   Top-1 Accuracy (%)
  ------- --------------------- --------- ------- --------------------
           (0.20, 1.1$\times$)     29M     5.0G           81.3
           (0.40, 1.3$\times$)     30M     5.0G           81.6
           (0.55, 1.5$\times$)     30M     4.9G           81.7
           (0.70, 1.9$\times$)     32M     5.0G           81.6
           (0.82, 2.5$\times$)     32M     5.1G           81.2

  : **Trade-off between sparsity and width.** The experiments are conducted on SLaK-T trained for 120 epoch on ImageNet-1K.

\[tab:tradeoff\]

The principle of "use *sparse* groups, expand more" in Observation \#3 essentially sacrifices network sparsity for network width. Therefore, there is a trade-off between model sparsity and width. To have a better understanding of this trade-off, we choose 5 combinations of (Sparsity, Width factor), i.e., (0.20, 1.1$\times$), (0.40, 1.3$\times$), (0.55, 1.5$\times$), (0.70, 1.9$\times$) and (0.82, 2.5$\times$). All the settings have roughly 5.0M FLOPs but with different network widths. The experiments are conducted on SLaK-T. As we expected, the model's performance keeps increasing as model width increases until the width factor reaches 1.5$\times$, after which increasing width further starts to hurt the performance apparently due to the training difficulties associated with highly sparse neural networks.

Conclusion and Discussion of Broader Impact {#sec:conclusion}
===========================================

Recent works on modern ConvNets defend the essential roles of convolution in computer vision by designing advanced architectures and plugging large kernels. However, the largest kernel size is limited to 31$\times$31 and the performance starts to saturate as the kernel size continues growing. In this paper, we investigate the training of ConvNets with extremely large kernels that are beyond 31$\times$31 and consequently provide a recipe for applying extremely large kernels inspired by sparsity. Based on this recipe, we build a pure ConvNet model that smoothly scales up the kernel size beyond 51$\times$51, while achieving better performance than Swin Transformer and ConvNeXt. Our strong results suggest that sparsity, as the "old friend" of deep learning, can make a promising tool to boost neural network scaling. This work is scientific in nature and we do not expect any negative societal impacts.

Configurations of Dynamic Sparsity {#app:config_DS}
==================================

Two key components are crucial for training models with dynamic sparsity: (1) sparse model initialization and (2) sparse weight adaptation. Sparse model initialization determines how to distribute the limited number of parameters across layers, i.e., layer-wise sparsity ratio and weight adaptation controls how we adapt the sparse weights for better sparse structures during training.

Liu et al. @liu2022the discover that the layer-wise sparsity ratio learned by SNIP @lee2018snip outperforms the Erds-R<span>é</span>nyi (ER) based sparsity ratios such as ER @mocanu2018scalable, ERK @evci2020rigging, and ERK+ @liu2022the, with large scale models on ImageNet. We confirm that SNIP ratio achieves the better trade-off between accuracy and FLOPs with SLaK. Following @liu2021we, we specifically tune two factors for SLaK-T that control the strength of weight adaptation, adaptation frequency $f$ and adaptation rate $p$. Adaptation frequency determines after how many training iterations we adjust the sparse weights, and the latter controls the ratio of the weight that we adjust at each adaptation. We share the results in Figure \[Fig:hyper\]. We empirically find that $f=2000$ and $p=0.5$ works best for SLak-T. For SLak-S/B, we directly choose $f=100$ and $p=0.3$ without careful tuning.

\[h\]

![image](./images/Hyperparameters.pdf){width="95.00000%"}

Experimental Settings {#app:training_config}
=====================

ImageNet-1K
-----------

We share the (pre-)training settings of SLaK on ImageNet-1K in this section. All the models are trained on 4/8 A100 GPUs. We train SLaK for 300 epochs (Section \[sec:eva\_ima\]) and 120 epochs (Section \[sec:recipe\]) using AdamW @loshchilov2018decoupled with a batch size of 4096, and a weight decay of 0.05. The only differnce between models training for 300 epochs and 120 epochs is the training time. The learning rate is 4e-3 with a 20-epoch linear warmup followed by a cosine decaying schedule. For data augmentation, we use the default setting of RandAugment @cubuk2020randaugment in Timm @rw2019timm – "rand-m9-mstd0.5-inc1", Label Smoothing @szegedy2016rethinking coefficient of 0.1, Mixup @zhang2017mixup with $\alpha = 0.8$, Cutmix @yun2019cutmix with $\alpha = 1.0$, Random Erasing @zhong2020random with $p=0.25$, Stochastic Depth with drop rate of 0.1 for SLaK-T, 0.4 for SLaK-S, and 0.5 for SLaK-B, Layer Scale @touvron2021going of initial value of 1e-6, and EMA with a decay factor of 0.9999. We train SLaK-T with NVIDIA A100 GPUs and the rest of models are trained with NVIDIA V100.

Semantic Segmentation on ADE20K
-------------------------------

We follow the training setting used in @ding2022scaling using the UperNet @xiao2018unified implemented by MMSegmentation @mmseg2020 with the 80K-iteration training schedule. The backbones are pre-trained on ImageNet-1K with 224$\times$224 input for 120 epochs.

Object Detection on PASCAL VOC 2007
-----------------------------------

We follow @liu2021swin and fine-tune Faster-RCNN on PASCAL VOC dataset with SLaK-T as the backbone. We use multi-scale setting @carion2020end [@sun2021sparse] which leads to the length of the shorter side between 480 and 800 and the ones of the longer side at most 1333. The model is trained with AdamW for 36 epochs with a learning rate of 0.0001, a weight decay of 0.05, and a batch size of 16.

Learning Curve
==============

We here share the learning curve of different architectures on ImageNet-1K classification in Figure \[fig:learning\_curve\]. Again, we can observe that large kernels provide significant training loss gains (yellow lines and blue lines). Increasing kernel size from 31$\times$31 to 51$\times$51 further decreases the training loss with improved test accuracy. It is worth noting that even though SLaK's kernel size is as large as 51$\times$51, it enjoys a very promising convergence speed compared to the models with smaller kernel sizes.

\[h\]

Limitations {#sec:limit}
===========

The main limitation of this work is that the sparse architecture is implemented with binary masks due to the limited support of sparse neural networks by the commonly used hardware such as GPU and TPU. Therefore, the inference FLOPs reported in the main paper are the theoretical values. Traditional works on structured sparsity @han2015deep [@frankle2018lottery; @gale2020sparse; @liu2021neuro] mainly focus on finding sparse subnetworks that can match the performance of their dense counterparts. The promising results in our paper go one step further and demonstrate the large potential of unstructured sparsity for scaling modern neural architectures. Once this great potential is supported in the future, it can have a significant positive impact on our planet by saving a huge amount of energy and reducing overall total carbon emissions. Furthermore, an increasing number of companies and researchers start considering including unstructured sparsity support in their hardware [@liu2021sparse; @gale2020sparse; @nvidia2020; @zhou2021learning; @hubara2021accelerated]. We hope our work can provide more motivation for such advances.

[^1]: Equal contribution.

[^2]: SNIP ratio is obtained by globally selecting the important weights across layers with the highest connection sensitivity score $|g \odot w|$, where $w$ and $g$ is the network weight and gradient, respectively.

[^3]: We empirically find that $N=5$ provides us the best results.

{'source': 'arXiv_out/2207'}