C AN CNN S B E M ORE R OBUST T HAN T RANSFORMERS ?
A BSTRACT
The recent success of Vision Transformers is shaking the long dominance of Convolutional Neural Networks (CNNs) in image recognition for a decade.
Specifically, in terms of robustness on out-of-distribution samples, recent research finds that Transformers are inherently more robust than CNNs, regardless of different training setups.
Moreover, it is believed that such superiority of Transformers should largely be credited to their self-attention-like architectures per se .
In this paper, we question that belief by closely examining the design of Transformers.
Our findings lead to three highly effective architecture designs for boosting robustness, yet simple enough to be implemented in several lines of code, namely a) patchifying input images, b) enlarging kernel size, and c) reducing activation layers and normalization layers.
Bringing these components together, we are able to build pure CNN architectures without any attention-like operations that are as robust as, or even more robust than, Transformers.
We hope this work can help the community better understand the design of robust neural architectures.
The code is publicly available at https://github.com/UCSC-VLAA/RobustCNN.
I NTRODUCTION
The success of deep learning in computer vision is largely driven by Convolutional Neural Networks (CNNs).
Starting from the milestone work AlexNet (Krizhevsky et al., 2012), CNNs keep pushing the frontier of computer vision (Simonyan & Zisserman, 2015; He et al., 2016; Tan & Le, 2019).
Interestingly, the recently emerged Vision Transformer (ViT) (Dosovitskiy et al., 2020) challenges the leading position of CNNs.
ViT offers a completely different roadmap—by applying the pure self-attention-based architecture to sequences of image patches, ViTs are able to attain competitive performance on a wide range of visual benchmarks compared to CNNs.
The recent studies on out-of-distribution robustness (Bai et al., 2021; Zhang et al., 2022; Paul &
Chen, 2022) further heat up the debate between CNNs and Transformers.
Unlike the standard visual benchmarks where both models are closely matched, Transformers are much more robust than CNNs when testing out of the box.
Moreover, Bai et al.
(2021) argue that, rather than being benefited by the advanced training recipe provided in (Touvron et al., 2021a), such strong out-of-distribution robustness comes inherently with the Transformer’s self-attention-like architecture.
For example, simply “upgrading” a pure CNN to a hybrid architecture ( i.e ., with both CNN blocks and Transformer blocks) can effectively improve out-of-distribution robustness (Bai et al., 2021).
Though it is generally believed that the architecture difference is the key factor that leads to the robustness gap between Transformers and CNNs, existing works fail to answer which architectural elements in Transformer should be attributed to such stronger robustness.
The most relevant analysis is provided in (Bai et al., 2021; Shao et al., 2021)—both works point out that Transformer blocks, where self-attention operation is the pivot unit, are critical for robustness.
Nonetheless, given a) the Transformer block itself is already a compound design, and b) Transformer also contains many other layers ( e.g ., patch embedding layer), the relationship between robustness and Transformer’s architectural elements remains confounding.
In this work, we take a closer look at the architecture design of Transformers.
More importantly, we aim to explore, with the help of the architectural elements from Transformers, whether CNNs can be robust learners as well .
Our diagnose delivers three key messages for improving out-of-distribution robustness, from the perspective of neural architecture design.
Firstly, patchifying images into non-overlapped patches can
[Figure]
substantially contribute to out-of-distribution robustness; more interestingly, regarding the choice of patch size, we find the larger the better.
Secondly, despite applying small convolutional kernels is a popular design recipe, we observe adopting a much larger convolutional kernel size ( e.g ., from 3 × 3 to 7 × 7 , or even to 11 × 11 ) is necessary for securing model robustness on out-of-distribution samples.
Lastly, as inspired by the recent work (Liu et al., 2022), we note aggressively reducing the number of normalization layers and activation functions is beneficial for out-of-distribution robustness; meanwhile, as a byproduct, the training speed could be accelerated by up to ∼ 23%, due to fewer normalization layers being used (Gitman & Ginsburg, 2017; Brock et al., 2021).
Our experiments verify that all these three architectural elements consistently and effectively improve out-of-distribution robustness on a set of CNN architectures.
The largest improvement is reported by integrating all of them into CNNs’ architecture design—as shown in Fig. 1, without applying any self-attention-like components, our enhanced ResNet (dubbed Robust-ResNet) is able to outperform a similar-scale Transformer, DeiT-S, by 2.4% on Stylized-ImageNet (16.2% vs .
18.6%), 0.5% on ImageNet-C (42.8% vs .
42.3%), 4.0% on ImageNet-R (41.9% vs .
45.9%) and 3.9% on ImageNet-Sketch (29.1% vs .
We hope this work can help the community better understand the underlying principle of designing robust neural architectures.
R ELATED W ORKS
Vision Transformers.
Transformers (Vaswani et al., 2017), which apply self-attention to enable global interactions between input elements, underpins the success of building foundation models in natural language processing (Devlin et al., 2019;
Yang et al., 2019; Dai et al., 2019; Radford et al., 2018; 2019; Brown et al., 2020; Bommasani et al., 2021).
Recently, Dosovitskiy et al .
(Dosovitskiy et al., 2020) show Transformer attains competitive performance on the challenging ImageNet classification task.
Later works keep pushing the potential of Transformer on a variety of visual tasks, in both supervised learning (Touvron et al., 2021a; Yuan et al., 2021; Liu et al., 2021;
Wang et al., 2021; Yuan et al., 2021; Zhai et al., 2021; Touvron et al., 2021b;
Xue et al., 2021) and self-supervised learning (Caron et al., 2021; Chen et al., 2021; Bao et al., 2022; Zhou et al., 2022; Xie et al., 2021;
He et al., 2022), showing a seemly inevitable trend on replacing CNNs in computer vision.
CNNs striking back.
There is a recent surge in works that aim at retaking the position of CNNs as the favored architecture.
Wightman et al .
(Wightman et al., 2021) demonstrate that, with the advanced training setup, the canonical ResNet-50 is able to boost its performance by 4% on ImageNet.
In addition, prior works show modernizing CNNs’ architecture design is essential—by either simply adopting Transformer’s patchify layer (Trockman & Kolter, 2022) or aggressively bringing in every possible component from Transformer (Liu et al., 2022), CNNs are able to attain competitive performance w.r.t.
Transformers across a range of visual benchmarks.
Our work is closely related to ConvNeXt (Liu et al., 2022), while shifting the study focus from standard accuracy to robustness.
Moreover, rather than specifically offering a unique neural architecture as in (Liu et al., 2022), this paper aims to provide a set of useful architectural elements that allows
CNNs to be able to match, or even outperform, Transformers when measuring robustness.
Out-of-distribution robustness.
Dealing with data from shifted distributions is a commonly encountered problem when deploying models in the real world.
To simulate such challenges, several out-of-distribution benchmarks have been established, including measuring model performance
[Table]
on images with common corruptions (Hendrycks & Dietterich, 2018) or with various renditions (Hendrycks et al., 2021).
Recently
[Table]
a set of works (Bai et al., 2021; Zhang et al., 2022; Paul & Chen, 2022) find that Transformers are inherently much more robust than CNNs on out-of-distribution samples.
Our work is a direct follow-up of Bai’s study et al .
(Bai et al., 2021) on Vision Transformers versus CNNs
[Table]
whereas we stand on the opposite side— we show CNNs can in turn outperform Transformers in out-of-distribution robustness
[Table]
S ETTINGS
Following the previous work (Bai et al., 2021)
[Table] in this paper [Table] the robustness comparison is fairly conducted between CNNs and Transformers based on ResNet (He et al., 2016) and ViT (Dosovitskiy et al., 2020), two of the most classic milestone architectures in the deep learning history.
CNN Block Instantiations.
We consider four different block instantiations, as shown in Fig.
The first block is Depth-wise ResNet Bottleneck block (Xie et al., 2017), in which the 3 × 3 vanilla convolution layer is replaced with a 3 × 3 depth-wise convolution layer.
The second block is the
Inverted Depth-wise ResNet Bottleneck block, in which the hidden dimension is four times the input dimension (Sandler et al., 2018).
The third block is instantiated based on the second block by moving up the position of the depthwise convolution layer as in ConvNeXT(Liu et al., 2022).
Note that a ConvNeXT block comes with other modifications such as changes of activation and normalization layers (Liu et al., 2022).
Similarly, based on the second block, the fourth block moves down the position of the depthwise convolution layer.
We replace the bottleneck building block in the original ResNet architecture with these four blocks, and term the resulting models as ResNetDW, ResNet-Inverted-DW, ResNet-Up-Inverted-DW, and ResNet-Down-Inverted-DW, respectively.
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
Computational Cost.
In this work, we consider measuring the model size in terms of FLOPs, i.e.,
FLoating-point Operations Per Second.
Here we note that directly replacing the ResNet bottleneck block with the above four instantiated blocks will significantly reduce the total FLOPs of the model, due to the usage of the depth-wise convolution.
To mitigate the computational cost loss, and to increase model performance following the spirit of ResNeXT (Xie et al., 2017), we change the channel setting of each stage from (64, 128, 256, 512) to (96, 192, 384, 768).
We then tune the block number in stage 3 to keep the total FLOPs of baseline models roughly the same as DeiT-S.
The final FLOPs of our baseline models are shown in Tab. 1.
Unless otherwise stated, all models considered in this work are of a similar scale as DeiT-S.
Robustness Benchmarks.
In this work, we extensively evaluate the model performance on out-ofdistribution robustness using the following benchmarks: 1) Stylized-ImageNet(Geirhos et al., 2018), which contains synthesized images with shape-texture conflicting cues; 2) ImageNet-C(Hendrycks & Dietterich, 2018), with various common image corruptions; 3) ImageNet-R (Hendrycks et al., 2021), which contains natural renditions of ImageNet object classes with different textures and local image statistics; 4) ImageNet-Sketch (Wang et al., 2019), which includes sketch images of the same
ImageNet classes collected online.
Training Recipe.
CNNs may achieve better robustness by simply tuning the training recipe (Bai et al., 2021; Wightman et al., 2021).
Therefore, in this work, we deliberately apply the standard 300epoch DeiT training recipe (Touvron et al., 2021a) for all models unless otherwise mentioned, so that the performance difference between models can only be attributed to the architecture difference.
Baseline Results.
The results are shown in Tab. 1.
For simplicity, we use “IN”, “S-IN”, “INC”, “IN-R”, and “IN-SK” as abbreviations for “ImageNet”, “Stylized-ImageNet”, “ImageNet-C”, “ImageNet-R”, and “ImageNet-Sketch”.
We already know that DeiT-S shows better robustness generalization than ResNet50 from (Bai et al., 2021).
Interestingly, even when equipped with stronger depth-wise convolution layers, the ResNet architecture can only attain similar clean image accuracy while still being non-trivially less robust compared with the DeiT counterpart, suggesting the hidden magic lies in the architecture design of Vision Transformer.
C OMPONENT D IAGNOSIS
In this section, we introduce three effective architectural designs inspired by Transformers, which work surprisingly well on CNNs as well.
Concretely, they are 1) patchifying input images (Section 4.1), b) enlarging the kernel size (Section 4.2), and finally, 3) reducing the number of activation layers and normalization layers (Section 4.3).
Specifically, the first design has a non-negligible effect on network FLOPs, which we mitigate by adding back blocks in the third stage.
4. P ATCHIFY S TEM
A CNN or a Transformer usually down-samples the input image to a proper feature map size at the beginning of the network.
A standard ResNet employs a 7 × 7 convolution layer with stride 2, followed by a 3 × 3 max pooling with stride 2, resulting in a 4 × resolution reduction.
ViT, however, adopts a much more aggressive downsampling strategy.
It partitions an input image into p ×
p nonoverlapping patches and projects each patch with a linear layer.
In practice, this is implemented by a convolution layer with kernel size p and stride p , where p is typically set to 16.
Here we refer to the layers before the typical ResNet block in ResNet, or the layer before the self-attention block in ViT, as stem .
Some works argue for the importance of stem setup in CNN (Trockman & Kolter, 2022) or Transformers (Xiao et al., 2021), but none of them investigate this module from the perspective of robustness.
We replace the ResNet-style stem in the baseline models with the ViT-style patchify stem, a convolution layer with kernel size p and stride p , where p varies from 4 to 16.
We keep the total stride of the model fixed (so a 224 × 224 input image will always lead to a 7 × 7 feature map before the final global pooling layer), which is achieved by setting stride=1 in the first block of some stages.
Specifically, the original ResNet sets stride=2 for the first block in stages 2, 3, and 4.
We set stride=1 for the first block in stage 2 when employing 8 × 8 patchify stem, and set stride=1 for the first block in stages 2 and 3 when employing 16 × 16 patchify stem.
To ensure a fair comparison, we add some blocks in stage 3 to keep the total FLOPs roughly the same as before.
In Tab. 1, we observe that increasing the patch size of the ViT-style patchify stem leads to increased performance on robustness benchmarks, though at the potential cost of clean accuracy.
Specifically, for ResNet-DW, when patch size=8, the performance on all robustness benchmarks is boosted by
[Table]
at least 0.6%; when patch size=16, the performance on all robustness benchmarks is boosted by at least 1.2%, and the biggest improvement is 6.6% on Stylized-ImageNet.
Surprisingly, this simple operation demonstrates significant potential for closing the robustness gap between CNNs and Transformers, suggesting a large portion of the robustness of ViT comes from the patchify stem.
We have also experimented with more advanced patchify stems, and surprisingly found them not so helpful on out-of-distribution robustness.
See Appendix A.2 for details.
It suggests that clean accuracy and out-of-distribution robustness are not always positively correlated.
And useful design for improving clean accuracy may not necessarily be attributed to better robustness, which again, shows the importance of searching for methods that can boost robustness in addition to solely clean accuracy.
4. L ARGE K ERNEL S IZE
One of the most critical properties of self-attention operation that distinguish it from the classic convolution operation, is that it operates on the whole input image or feature map and thus has a global receptive field.
Even before the rise of Vision Transformer, the importance of capturing longrange dependency has been demonstrated for CNNs as well.
One typical example is the Non-local
Neural Network which has been shown to be highly effective for both static and sequential image recognition even when equipped with only one non-local block (Wang et al., 2018).
Still, the most popular design is to stack many 3 × 3 convolution layers to gradually increase the network receptive field as it goes deeper.
In this section, we try to mimic the behavior of the self-attention block, by increasing the kernel size of the depth-wise convolution layer.
We experiment with various kernel sizes: 5, 7, 9, 11, and 13.
The results on different robustness benchmarks are shown in Fig.
From Fig.
3, we observe that larger kernel sizes generally come with better clean accuracy and stronger robustness.
We can also see that the performance gain gradually saturates when the kernel size becomes too large.
Since using convolutions of larger kernels will incur notable extra computation, here we would like to highlight that this particular component design would be impossible without the help of depth-wise convolution layers, which significantly reduces the computational cost of large kernel [Table] convolution layers in all four blocks.
For example, if directly changing the kernel size in ResNet from 3 to 5, the total FLOPs of the resulting model would be 7.4G, which is way larger than its
Transformer counterpart.
Thanks to the depth-wise convolution layer, increasing kernel size fromto 13 typically only increase the FLOPs by 0.3G, which is relatively small compared to the FLOPs of DeiT-S (4.6G), except for ResNet-Inverted-DW.
Because of the large channel dimension in the Inverted Bottleneck design, increasing kernel size from 3 to 13 brings a total of 1.4G FLOPs increase, rendering an unfair comparison to some extent.
Nevertheless, the trend for all four architectures is the same.
And the extra computational cost brought by large kernel size can be mitigated by employing a patchify stem with a large patch size so that our final models will still be on the same scale as DeiT-S (see Sec.
5 for the FLOPs of models with multiple proposed designs).
Note that the results are obtained with the best choice of only one activation layer in a block based on the results of Tab. 3.
We use ✓ to denote keeping a normalization layer and ✗ to denote removing a normalization layer.
It is observed that the highlighted architectural design choices consistently lead to improved robustness upon the baselines.
[Table]
32.4 ✗ ✗ ✓ 80. ResNet-Down-Inverted-DW 4.5G ✓ ✓ ✓ 79. 29.5 ✓ ✗ ✗ 80. 33.3 ✗ ✓ ✗ 80.
31.7 ✗ ✗ ✓ 80.
[Figure]
, 96->
[Figure]
, 384-> 1x1, 384->
[Figure]
BN, ReLU
[Figure]
[Figure]
>
[Figure]
, 96->
[Figure]
[Figure]
->
BN
ReLU 1x1,
96-> 1x1, 384-> d3x3, 96->
BN, ReLU d3x3, 96-> 1x1,
[Figure]
-> 1x1, 384->
BN
ReLU
[Figure]
4. R EDUCING A CTIVATION AND N ORMALIZATION L AYERS
A typical Vision Transformer block usually has fewer activation and normalization layers than a
ResNet block (Dosovitskiy et al., 2020; Liu et al., 2021), an architecture design found to be very effective in ConvNeXT (Liu et al., 2022).
In this work, we directly borrow the idea of reducing activation layers and normalization layers in ConvNeXT for all four block instantiations, and find it to be beneficial to clean accuracy as well as out-of-distribution robustness.
Specifically, in a typical
ResNet block, there is one normalization layer and one activation layer following each convolution layer, resulting in a total of three normalization layers and activation layers in one block.
In our implementation, for each block, we remove two normalization layers and activation layers, and leave only one normalization layer and activation layer.
The results of the models with differently removed activation layers are shown in Tab. 3.
In addition, the results of the models with differently removed normalization and activation layers are shown in Tab. 4.
Note that we only ablate the position of removing normalization layers with the optimal position of the activation layer shown in Tab. 3, since we find that only removing normalization layers does not result in performance improvement in the preliminary experiments. [Table]
We empirically find that among all experimental settings, leaving only one activation layer after the convolution layer where the channel dimension expands ( i.e ., the number of output channels is larger than that of the input channels), and leaving one normalization layer after the first convolution layer yields the best result.
We also visualize the best choice in Fig.
For instance, we observe 1.4% improvement on Stylized-ImageNet (12.9% vs .
14.3%), 2.9% improvement on ImageNet-C (47.9% vs .
45.0%), 4.3% improvement on ImageNet-R (42.9% vs .
47.2%), and 2.3% improvement on ImageNet-Sketch (30.8% vs .
33.1%) for ResNet-Up-Inverted-DW.
Additionally, fewer normalizations lead to less GPU memory cost and accelerated training (e.g., up to 23% speed up simply by removing a few normalization layers)
(Gitman & Ginsburg, 2017; Brock et al., 2021).
C OMPONENTS C OMBINATION
In this section, we investigate whether all components combined will lead to better results.
We choose a 16 × 16 patchify stem, 11 × 11 kernel size, and the corresponding optimal position for placing normalization and activation layer for all architectures (except for ResNet-Inverted-DW, we use 7 × 7 kernel size as we empirically find using too large kernel size (e.g., 11 × 11) can lead to unstable training).
As shown in Tab. 5, and Tab. 9, Tab. 10, Tab. 11 in Appendix A.3, we can see these simple designs not only work well when they are individually employed with ResNet, but even work better when they are simultaneously employed .
Also, when all three designs are used, we see now ResNet beats DeiT on all four out-of-distribution benchmarks.
These results again verify the effectiveness of our proposed architecture design, and also implies that a pure CNN without any self-attention-like block achieves just as good robustness as ViT. From now on, for simplicity, we will term ResNet-DW with our choice of ViT-style patchify stem, large kernel size, and fewer activation and normalization layers as Robust-ResNet-DW (and similar for others).
K NOWLEDGE D ISTILLATION
Knowledge Distillation (Hinton et al., 2015) is a training heuristic aiming at transferring the knowledge of a stronger teacher model to a weaker student model with lower capacity.
Usually, the student model can achieve better or at least similar performance than the teacher model in knowledge distillation.
Yet, directly employing knowledge distillation to let ResNet-50 (student model) learn from
DeiT-S (teacher model) has been found to be less effective: ResNet50 still underperforms DeiT-S by a significant margin; While surprisingly, when the model roles are switched, the student model
DeiT-S is still able to remarkably outperform the teacher model ResNet-50 (Bai et al., 2021).
It is therefore concluded that the key to achieving good robustness of DeiT lies in its architecture that cannot be transferred to ResNet via knowledge distillation.
We run these experiments again with [Table] our training recipe and models with all three architectural designs combined.
We can clearly conclude from Tab. 6 that with the help of our proposed architectural elements from ViT, a ResNet does perform better than DeiT on out-of-distribution samples.
In contrast, the baseline models, though attaining good performance on clean ImageNet, are not as robust as the teacher model DeiT-S.
L ARGER M ODELS
So far, we have only demonstrated the effectiveness of our proposed models on a relatively small model scale.
To show that they are also beneficial to larger models, we adjust our models to match the total FLOPs of DeiT-Base.
Specifically, we change the base channel dimensions to (128, 256, 512, and 1024) [Table] and add over 20 blocks in stage 3 of the network.
We then train it with the ConvNeXT-B recipe (Liu et al., 2022) and compare its performance with DeiT-Base, as shown in Tab. 7.
We have also tried directly applying the DeiT recipe and found that it brings little improvement.
As can be seen from Tab. 7, our proposed Robust-ResNet-Base models still perform favorably against DeiT-B, indicating the great potential of our method when scaling up models [Table]
[Table]
C ONCLUSION
While the recent work (Bai et al., 2021) concludes that Transformers are more robust than CNNs on out-of-distribution samples, it also suggests that Transformer’s self-attention-like architectures are the key.
In this paper [Table] we take a closer look at the Transformer architecture, and find several useful designs other than the self-attention block.
We find that combining these designs and introducing them into ResNet can lead to a pure CNN architecture that can be as robust as, sometimes even more robust than, a Vision Transformer model of the same scale.
We hope our results can encourage the community to rethink the robustness comparison between Transformers and CNNs, and, meanwhile, help the community re-examine and better understand robust architecture design. A CKNOWLEDGEMENT This work is supported by a gift from Open Philanthropy, TPU Research Cloud (TRC) program, and Google Cloud Research Credits program
A A PPENDIX A. T RAINING R ECIPE D ETAILS
We follow the standard 300-epoch DeiT training recipe (Touvron et al., 2021a) in this work.
Specifically, we train all models using AdamW optimizer (Loshchilov & Hutter, 2019); We set the initial base learning rate to 5e-4, and apply the cosine learning rate scheduler to decrease it; Besides weight decay, we additionally adopt six data augmentation & regularization strategies (i.e., RandAug (Cubuk et al., 2020), MixUp (Zhang et al., 2018), CutMix (Yun et al., 2019), Random Erasing (Zhong et al., 2020), Stochastic Depth (Huang et al., 2016), and Repeated Augmentation (Hoffer et al., 2020)) to regularize training.
A. M ORE S TEM E XPERIMENTS
Recent findings also suggest replacing the ViT-style patchify stem with a small number of stacked 2-stride 3 × 3 convolution layers can greatly ease optimization and thus boost the clean accuracy (Graham et al., 2021; Xiao et al., 2021).
To verify its effectiveness on robustness benchmarks, we also implemented the convolution stem for ViT-S in (Graham et al., 2021), with a stack of 4 3 ×convolution layers with stride 2.
The results are shown as follows in Tab. 8.
Surprisingly, though the use of convolutional stem does attain higher clean accuracy, it appears to be not as helpful as
ViT-style patchify stem on out-of-distribution robustness. [Table]
A. M ORE C OMPONENTS C OMBINATION E XPERIMENTS
Here we show more results of combining different components proposed in this paper in Tab. 9,
Tab. [Table] and Tab. 11.
A. M ORE COMPARISON WITH O THER M ODELS
Besides DeiT, here we also evaluate our proposed Robust-ResNet against two state-of-the-art models, ConvNeXt (Liu et al., 2022) and Swin-Transformer (Liu et al., 2021), in terms of out-ofdistribution robustness.
As shown in Tab. 12, all four of our models perform similarly to, or better than ConvNeXt or Swin-Transformer in all out-of-distribution tests.
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
A. R EPEATING E XPERIMENTS
In the above experiments
[Table]
we report the result of a single run with a fixed random seed. In order to demonstrate the statistical significance of the robustness improvements brought by our proposed components, we conduct three runs with different random seeds and report the mean and standard deviation in Tab. 13. We observe only small variations across the three runs, which confirms the consistent and reliable performance gains achieved by our proposed models. A. I MAGE N ET -A E VALUATION
The ImageNet-A dataset comprises a set of natural adversarial samples that considerably degrade the performance of machine learning models.
In Tab. 14, we compare the performance of our RobustResNet models and DeiT on the ImageNet-A dataset.
Notably, while the Robust-ResNet models do not perform as well as DeiT with an input resolution of 224, increasing the input resolution (e.g., to 320) significantly narrows the gap between Robust-ResNet and DeiT on ImageNet-A.
We conjecture this is because objects of interest in ImageNet-A are smaller than in standard ImageNet.
[Figure]
[Table]
A. S TRUCTURAL R E - PARAMETERIZATION
A line of recent works promotes the idea of a training-time multi-branch but inference-time plain model architecture via structural re-parameterization (Ding et al., 2021; 2022a;b).
RepLKNet
(Ding et al., 2022b), in particular, has shown that re-parameterization using small kernels can alleviate the optimization issue associated with large-kernel convolution layers, without incurring additional inference costs.
Given the Robust-ResNet models also adopt large kernels, we also try the idea of structural re-parameterization and leverage a multi-branch block architecture at training to further enhance model performance. The block architecture details are shown in Fig. The results of two different model scales, shown as in Tab. 15 and Tab. 16, demonstrate generally improved performance with this re-parameterization approach. One exception may be Robust-ResNet-Up-InvertedDW, which sometimes exhibits slightly worse robustness with re-parameterization.
Notably, with the re-parameterization technique, we are able to train the Robust-ResNet-Inverted-DW model using a kernel size of 11.
[Table]