T RANSFORMED CNN S : RECASTING PRE - TRAINED CONVOLUTIONAL LAYERS WITH SELF - ATTENTION
Anonymous authors
Paper under double-blind review
A BSTRACT
Vision Transformers (ViT) have recently emerged as a powerful alternative to convolutional networks (CNNs).
Although hybrid models attempt to bridge the gap between these two architectures, the self-attention layers they rely on induce a strong computational bottleneck, especially at large spatial resolutions.
In this work, we explore the idea of reducing the time spent training these layers by initializing them from pre-trained convolutional layers.
This enables us to transition smoothly from any pre-trained CNN to its functionally identical hybrid model, called Transformed CNN (T-CNN).
With only 50 epochs of ﬁne-tuning, the resulting T-CNNs demonstrate signiﬁcant performance gains over the CNN as well as substantially improved robustness.
We analyze the representations learnt by the
T-CNN, providing deeper insights into the fruitful interplay between convolutions and self-attention.
I NTRODUCTION
Since the success of AlexNet in 2012 (Krizhevsky et al., 2017), the ﬁeld of Computer Vision has been dominated by Convolutional Neural Networks (CNNs) (LeCun et al., 1998; 1989).
Their local receptive ﬁelds give them a strong inductive bias to exploit the spatial structure of natural images (Scherer et al., 2010; Schmidhuber, 2015; Goodfellow et al., 2016), while allowing them to scale to large resolutions seamlessly.
Yet, this inductive bias limits their ability to capture long-range interactions.
In this regard, self-attention (SA) layers, originally introduced in language models (Bahdanau et al., 2014; Vaswani et al., 2017; Devlin et al., 2018), have gained interest as a building block for vision Ramachandran et al.
; Zhao et al.
Recently, they gave rise to a plethora of Vision
Transformer (ViT) models, able to compete with state-of-the-art CNNs in various tasks Dosovitskiy et al.
(2020); Touvron et al.
; Wu et al.
(2020); Touvron et al.
Liu et al.
(2021); Heo et al.
(2021) while demonstrating better robustness (Bhojanapalli et al., 2021; Mao et al., 2021).
However, capturing long-range dependencies necessarily comes at the cost of quadratic complexity in input size, a computational burden which many recent directions have tried to alleviate (Bello, 2021; Wang et al., 2020; Choromanski et al., 2020; Katharopoulos et al., 2020).
Additionally, ViTs are generally harder to train (Zhang et al., 2019; Liu et al., 2020), and require vast amounts of pre-training (Dosovitskiy et al., 2020) or distillation from a convolutional teacher (Hinton et al., 2015;
Jiang et al., 2021; Graham et al., 2021) to match the performance of CNNs.
Faced with the dilemma between efﬁcient CNNs and powerful ViTs, several approaches have aimed to bridge the gap between these architectures.
On one side, hybrid models append SA layers onto convolutional backbones (Chen et al., 2018; Bello et al., 2019; Graham et al., 2021; Chen et al., 2021;
Srinivas et al., 2021), and have already fueled successful results in a variety of tasks (Carion et al., 2020;
Hu et al., 2018; Chen et al., 2020; Locatello et al., 2020; Sun et al., 2019).
Conversely, a line of research has studied the beneﬁt of introducing convolutional biases in Transformer architectures to ease learning (d’Ascoli et al., 2021; Wu et al., 2021; Yuan et al., 2021).
Despite these interesting compromises, modelling long-range dependencies at low computational cost remains a challenge for practitioners.
Contributions At a time when pre-training on vast datasets has become common practice, we ask the following question: does one need to train the SA layers during the whole learning process? Could [Figure]
one instead learn cheap components such as convolutions ﬁrst, leaving the SA layers to be learnt at the end?
In this paper, we take a step in this direction by presenting a method to fully reparameterize any pre-trained convolutional layer as a Gated Positional Self-Attention (GPSA) layer (d’Ascoli et al., 2021).
The latter is initialized to reproduce the mapping of the convolutional layer, but is then encouraged to learn more general mappings which are not accessible to the CNN by adjusting positional gating parameters.
We leverage this method to reparametrize pre-trained CNNs as functionally equivalent hybrid models.
After only 50 epochs of ﬁne-tuning, the resulting Transformed CNNs (T-CNNs) boast signiﬁcant performance and robustness improvements as shown in Fig.
1, demonstrating the practical relevance of our method.
We analyze the inner workings of the T-CNNs, and show that they learn more robust representations by combining convolutional heads and attentional heads in a complementary way.
Related work Our work mainly builds on two pillars.
First, the idea that SA layers can express any convolution, introduced by Cordonnier et al.
This idea was recently leveraged by d’Ascoli et al.
(2021), which initialize the SA layers of end-to-end Transformers as random convolutions to imbue them with a local inductive bias and improve their sample efﬁciency.
Our approach leverages the opposite idea: giving an end-to-end CNN the freedom to escape locality by learning self-attention at late times.
Second, we exploit the following learning paradigm: train a simple and fast model, then reparameterize it as a more complex model for the ﬁnal stages of learning.
This approach was studied from a scientiﬁc point of view in d’Ascoli et al.
(2019), which shows that reparameterizing a CNN as a fully-connected network (FCN) halfway through training can lead the FCN to outperform the CNN.
Yet, the practical relevance of this method is limited by the vast increase in number of parameters required by the FCN to functionally represent the CNN.
In contrast, our reparameterization hardly increases the parameter count of the CNN, making it easily applicable to any state-of-the-art CNN.
Note that these reparameterization methods can be viewed an informed version of dynamic architecture growing algorithms such as AutoGrow (Wen et al., 2020).
In the context of hybrid models, various works have studied the performance gains obtained by introducing MHSA layers in ResNets with minimal architectural changes (Srinivas et al., 2021;
Graham et al., 2021; Chen et al., 2021).
However, the MHSA layers used in these works are initialized randomly and need to be trained from scratch.
Our approach is different, as it makes use of GPSA layers, which can be initialized to represent the same function as the convolutional layer it replaces.
We emphasize that the novelty in our work is not in the architectures used, but in the unusual way they are blended together.
B ACKGROUND
Multi-head self-attention
The SA mechanism is based on a trainable associative memory with (key, query) vector pairs.
To extract the semantic interpendencies between the L elements of a sequence X ∈ R L × D in , a sequence of “query” embeddings Q = W qry X ∈ R L × D h is matched against another sequence of “key” embeddings K = W key X ∈ R L × D h using inner products.
The result is an attention matrix whose entry ( ij ) quantiﬁes how semantically relevant Q i is to K j :
[Equation]
Multi-head SA layers use several SA heads in parallel to allow the learning of different kinds of dependencies:
[Equation]
where W h val ∈ R D in × D v and W h out ∈ R D v × D out are two learnable projections.
To incorporate positional information, ViTs usually add absolute position information to the input at embedding time, before propagating it through the SA layers.
Another possibility is to replace the vanilla SA with positional SA (PSA), by including a position-dependent term in the softmax (Ramachandran et al., 2019; Shaw et al., 2018).
Although there are several ways to parametrize the positional attention, we use encodings r ij of the relative position of pixels i and j as in (Cordonnier et al., 2019):
[Equation]
Each attention head learns an embedding v hpos ∈ R D pos , and the relative positional encodings r ij ∈ R D pos only depend on the distance between pixels i and j , denoted denoted as a twodimensional vector δ ij .
Self-attention as a generalized convolution Cordonnier et al.
(2019) shows that a multi-head PSA layer (Eq. 3) with N h heads and dimension D pos ≥ 3 can express any convolutional layer of ﬁlter size √ N h , with D in input channels and min( D v , D out ) output channels, by setting the following:
[Equation]
In the above, the center of attention ∆ h ∈ R 2 is the position to which head h pays most attention to, relative to the query pixel, whereas the locality strength α h > 0 determines how focused the attention is around its center ∆ h .
When α h is large, the attention is focused only on the pixel located at ∆ h ; when α h is small, the attention is spread out into a larger area.
Thus, the PSA layer can achieve a convolutional attention map by setting the centers of attention ∆ h to each of the possible positional offsets of a √ N h × √ N h convolutional kernel, and sending the locality strengths α h to some large value.
A PPROACH
In this section, we introduce our method for mapping a convolutional layer to a functionally equivalent
PSA layer with minimal increase in parameter count.
To do this, we leverage the GPSA layers introduced in d’Ascoli et al.
Loading the ﬁlters We want each head h of the PSA layer to functionally mimic the pixel h of
a convolutional ﬁlter W ﬁlter ∈ R N h × D in × D out , where we typically have D out ≥ D in .
Rewriting the action of the MHSA operator in a more explicit form, we have
[Equation]
In the convolutional conﬁguration of Eq. 4, A h X selects pixel h of X .
Hence, we need to set
W h
= W h ﬁlter .
However, as a product of matrices, the rank of W h is bottlenecked by D v .
To avoid this being a limitation, we need D v ≥ D in (since D out ≥ D in ).
To achieve this with a minimal number of parameters, we choose D v = D in , and simply set the following initialization:
[Equation]
Note that this differs from the usual choice made in SA layers, where D v = (cid:98) D in /N h (cid:99) .
However, to keep the parameter count the same, we share the same W hval across different heads h , since it plays a symmetric role at initialization.
Note that this reparameterization introduces three additional matrices compared to the convolutional ﬁlter: W qry , W key , W val , each containing D in × D in parameters.
However, since the convolutional ﬁlter contains N h × D in × D out parameters, where we typically have N h
= 9 and D out ∈ { D in , 2 D in } , these additional matrices are much smaller than the ﬁlters and hardly increase the parameter count.
This can be seen from the model sizes in Tab. 3.
Gated Positional self-attention
Recent work (d’Ascoli et al., 2021) has highlighted an issue with standard PSA: the fact that the content and positional terms in Eq.
3 are potentially of very different magnitudes, in which case the softmax ignores the smallest of the two.
This can typically lead the
PSA to adopt a greedy attitude: choosing the form of attention (content or positional) which is easiest at a given time then sticking to it.
To avoid this, the authors suggest to sum the content and positional terms after the softmax, with their relative importances governed by a learnable gating parameter λ h (one for each attention head).
The resulting Gated Positional Self-Attention (GPSA) layers are parametrized as follows:
[Equation]
where σ : x (cid:55)→ 1 / ( 1+ e − x ) is the sigmoid function.
In the positional part, the encodings r ij are ﬁxed rather than learnt (see Eq. 4), which makes changing input resolution straightforward (see SM.
C) and leaves only 3 learnable parameters per head: ∆ 1 , ∆ 2 and α 1 .
How convolutional should the initialization be?
The convolutional initialization of GPSA layers involves two parameters, determining how strictly convolutional the behavior is: the initial value of the locality strength α , which determines how focused each attention head is on its dedicated pixel, and the initial value of the gating parameters λ , which determines the importance of the positional information versus content.
If λ h (cid:29) 0 and α (cid:29) 1 , the T-CNN will perfectly reproduce the input-output function of the CNN, but may want to greedily stay in the convolutional conﬁguration.
Conversely, if λ h (cid:28) 0 and α (cid:28) 1 , the T-CNN will forget about the input-output function of the
CNN.
Hence, we choose α = 1 and λ = 1 to lie in between these two extremes, encouraging the
T-CNNs to escape locality throughout training.
Architectural details To make our setup as canonical as possible, we focus on ResNet architectures (He et al., 2016), which contain 5 stages, with spatial resolution halfed and number of channels doubled at each stage.
Our method involves reparameterizing 3 × 3 convolutions as GPSA layers with 9 attention heads.
However, global SA is too costly in the ﬁrst layers, where the spatial resolution is large.
We therefore only reparameterize the last stage of the architecture, while replacing the ﬁrst stride-2 convolution by a stride-1 convolution, exactly as in (Srinivas et al., 2021).
We also add explicit padding layers to account for the padding of the original convolutions. [Figure]
P ERFORMANCE OF THE T RANSFORMED CNN S
In this section, we apply our reparametrization to state-of-the-art CNNs, then ﬁne-tune the resulting
T-CNNs to learn better representations.
This method allows to fully disentangle the training of the SA layers from that of the convolutional backbone, which is of practical interest for two reasons.
First, it minimizes the time spent training the SA layers, which typically have a slower throughput.
Second, it separates the algorithmic choices of the CNN backbone from those of the SA layers, which are typically different; for example, CNNs are typically trained with SGD whereas SA layers perform much better with adaptive optimizers such as Adam (Zhang et al., 2019), an incompatibility which may limit the performance of usual hybrid models.
Note that our reparametrization can also be applied halfway through training: this scenario is investigated in Sec.
Results suggest that reparametrizing at intermediate times is optimal in terms of speed-performance trade-offs.
However, for easier reproducibility, we focus on reparametrizing the fully pre-trained CNNs available in the timm package (Wightman, 2019): this avoids having to retrain the various models from scratch.
Training details We applied our method to pre-trained ResNet-RS (Bello et al., 2021) models, using the weights provided by the timm package (Wightman, 2019).
These models are derived from the original ResNet (He et al., 2016), but use improved architectural features and training strategies, enabling them to reach better speed-accuracy trade-offs than EfﬁcientNets.
To minimize computational cost, we restrict our ﬁne-tuning to 50 epochs 2 .
Following (Zhang et al., 2019), we use the AdamW optimizer, with a batch size of 1024 3 .
The learning rate is warmed up to 10 − 4 then annealed using a cosine schedule.
To encourage the T-CNN to escape the convolutional conﬁguration and learn content-based attention, we use a larger learning rate of 0.1 for the gating parameters of Eq.
7 (one could equivalently decrease the temperature of the sigmoid function).
We use the same data augmentation scheme as the DeiT (Touvron et al., 2020), as well as rather large stochastic depth coefﬁcients d r reported in Tab. 1.
Hoping that our method could be used as an alternative to the commonly used practice of ﬁne-tuning models at higher resolution, we also increase the resolution during ﬁne-tuning (Touvron et al., 2019).
In this setting, a ResNet50 requires only 6 hours of ﬁne-tuning on 16 V100 GPUs, compared to 33 hours for the original training.
For our largest model (ResNet350-RS), the ﬁne-tuning lasts 50 hours.
Performance improvements Results are presented in Tab. 1, where we also report the baseline improvement of ﬁne-tuning in the same setting but without SA.
In all cases, our ﬁne-tuning improves top-1 accuracy, with a signiﬁcant gap over the baseline.
To demonstrate the wide applicability of our method, we report similar improvements for ResNet-D architectures in SM.
E.
Despite the extra ﬁne-tuning epochs and their slower throughput, the resulting T-CNNs also outperforming the original CNNs on the ImageNet validation set at equal training budget, as shown [Table] [Figure] h
[Figure]
n e
[Figure] e l a s t i c t r a n s f o r m p i x e l a t e j p e g c o m p r
e
ss i
o n g l a ss b l u r m o t
i o n b l u r z
o o m
b l u r d e f o c
u s b l u r f o g s h
o t n
o
i s e g a u
ss i a n n
o
i s e f r o s t s n o w i m p
u l s e n o i s e c o n t r a s t R e l .
i m p
r
o v e
m e n t noise blur digital weather in the leftmost panel of Fig. However, the major beneﬁt of the reparametrization is in terms of robustness, as shown in Fig. 2(b) and explained below. Robustness improvements Recent work has shown that Transformer-based architectures offer better robustness and out-of-domain generalization than convolutional architectures (Bhojanapalli et al., 2021; Mao et al., 2021;
Mahmood et al., 2021; Shao et al., 2021).
To investigate whether our ﬁne-tuning procedure is enough to imbue CNNs with these advantages, we evaluate our T-CNNs on various benchmarks:
• Common corruptions : we use ImageNet-C (Hendrycks & Dietterich, 2019), a dataset containing 15 sets of randomly generated corruptions, grouped into 4 categories: ‘noise’, ‘blur’, ‘weather’, and ‘digital’.
Each corruption type has ﬁve levels of severity, resulting in 75 distinct corruptions.
Note that to avoid distorting the corruptions, which are often pixel-based, we keep a resolution of 224 at inference, which disadvantages the large models trained at higher resolutions.
• Adversarial robustness : following (Mao et al., 2021), we evaluate the accuracy of our models under two white -bow attacks 5 : (i) single-step FGSM (Goodfellow et al., 2014) and (ii) multi-step L ∞ -PGD (Madry et al., 2017) with t = 5 steps of size α = 0 .
Both attackers perturb the input image with max magnitude (cid:15) = 1 .
We also evaluate our models on ImageNet-A (Hendrycks et al., 2021), a dataset containing naturally “adversarial” examples from ImageNet.
Note however that since this dataset is built from the ﬂaws of a ResNet, it is potential unfair to CNNs.
• Distribution shifts : we use ImageNet-R (Hendrycks et al., 2020), a dataset with various stylized “renditions” of ImageNet images ranging from paintings to embroidery, which strongly modify the local image statistics.
The full table of results is presented in Tab. 3 of SM.
A, and illustrated in Figs.
1 and 2.
The
T-ResNet-RS substantially outperforms the ResNet-RS on all robustness benchmarks.
For example, our T-ResNet101-RS, which is 50% faster than ResNet200-RS, reaches similar or better results all robustness tasks, despite its lower top-1 accuracy on ImageNet-1k.
This demonstrates that SA improves robustness more than it improves classiﬁcation accuracy.
The most striking improvement is in terms of adversarial robustness, where the smallest T-ResNet-RS is on par with the largest
ResNet-RS despite requiring 5 times less compute.
To better understand where the beneﬁts come from, we decompose the improvement of the TResNet50-RS over the various corruption severeties and categories of ImageNet-C in Fig.
We observe that improvement increases almost linearly with corruption severity.
Although performance is higher in all corruption categories, there is a strong variability: the T-CNN shines particularly in tasks where the objects in the image are less sharp due to lack of contrast, bad weather or blurriness.
We attribute this to the ability of SA to distinguish shapes in the image, as investigated in Sec 4.
D ISSECTING THE T RANSFORMED CNN S
In this section, we analyze various observables to understand how the representations of a TResNet270-RS evolve from those of the ResNet270-RS throughout training.
[Figure]
T o pa cc u r a c y
A
Test
Train  40 Epochs
A tt e n t i o n s p a n B Layer Layer
Layer
Layer  40 Epochs 0.
P o s i t i o n a l i m p o r t a n c e
C Unlearn to better relearn In Fig. 4A, we display the train and test accuracy throughout training 6 .
The dynamics decompose into two distinct phases: accuracy dips down during the learning rate warmup phase (ﬁrst 5 epochs of training), then increases back up as the learning rate is decayed.
As shown in SM.
B, the depth of the dip depends on the learning rate.
For too small learning rates, the dip is small, but the test accuracy increases too slowly after the dip; for too large learning rates, the test accuracy increases rapidly after the dip, but the dip is too deep to be compensated for.
This suggests that the T-CNN needs to “unlearn” to some extent, a phenomenon reminiscent of the “catapult” mechanism of Lewkowycz et al.
(2020) which propels models out of sharp minima to land in wider minima.
[Figure] Escaping the convolutional representation In Fig.
4B, we show the evolution of the “attention span” 1 /α h (see Eq. 4), which reﬂects the size of the receptive ﬁeld of attention head h .
On average (thick lines), this quantity increases in the ﬁrst three layers, showing that the attention span widens, but variability exists among different attention heads (thin lines): some broaden their receptive ﬁeld, whereas others contract it.
In Fig.
4C, we show the evolution of the gating parameters λ h of Eq.
7, which reﬂect how much attention head h pays to position versus content.
Interestingly, the ﬁrst layer stays strongly convolutional on average, as E h σ ( λ h ) rapidly becomes close to one (thick blue line).
The other layers strongly escape locality, with most attention heads focusing on content information at the end of ﬁne-tuning.
In Fig.
5, we display the attention maps after ﬁne-tuning.
A clear divide appears between the “convolutional” attention heads, which remain close to their initialization, and the “content-based” attention heads, which learn more complex dependencies.
Notice that the attention head initially focusing on the query pixel (head 5) stays convolutional in all layers.
Throughout the layers, the edges of the central object is more and more clearly visible, as observed in (Caron et al., 2021).
This supports the hypothesis that robustness gains obtained for blurry corruptions (see Fig. 3) are partly due to the ability of the SA layers to isolate objects from the background.  W HEN SHOULD ONE START LEARNING THE SELF - ATTENTION LAYERS ?
We have demonstrated the beneﬁts of initializing T-CNNs from pre-trained CNNs, a very compelling procedure given the wide availability of pretrained models.
But one may ask: how does this compare to training a hybrid model from scratch?
More generally, given a computational budget, how long should the SA layers be trained compared to the convolutional backbone?
Transformed CNN versus hybrid models To answer the ﬁrst question, we consider a ResNettrained on ImageNet for 400 epochs.
We use SGD with momentum 0.9 and a batch size of 1024, warming up the learning rate for 5 epochs before a cosine decay. To achieve a strong baseline, we use the same augmentation scheme as in Touvron et al. (2020) for the DeiT. Results are reported in Tab. 2.
In this modern training setting, the vanilla ResNet50 reaches a solid performance of 79.04% on ImageNet, well above the 77% usually reported in litterature.
The T-CNN obtained by ﬁne-tuning the ResNet for 50 epochs at same resolution obtains a topaccuracy of 79.88%, with a 15% increase in training time, and 80.84 as resolution 320, with a 35% increase in training time.
In comparison, the hybrid model trained for 400 epochs in the same setting only reaches 79.95%, in spite of a 40% increase in training time.
Hence, ﬁne-tuning yields better results than training the hybrid model from scratch.
What is the best time to reparametrize?
We now study a scenario between the two extreme cases: what happens if we reparametrize halfway through training?
To investigate this question in
[Table]
a systematic way, we train the ResNet50 for t 1 epochs, then reparametrize and resume training for another t 2 epochs, ensuring that t 1 + t 2 = 400 in all cases.
Hence, t 1 = 400 , amounts to the vanilla
ResNet50, whereas t 1 = 0 corresponds to the hybrid model trained from scratch.
To study how ﬁnal performance depends on t 1 in a fair setting, we keep the same optimizer and learning rate after the reparametrization, in contrast with the ﬁne-tuning procedure which uses fresh optimizer.
Results are presented in Tab. 2.
Interestingly, the ﬁnal performance evolves non-monotonically, reaching a maximum of 80 . 44 for t 1 = 100 , then decreasing back down as the SA layers have less and less time to learn.
This non-monotonicity is remarkably similar to that observed in d’Ascoli et al.
(2019), where reparameterizing a CNN as a FCN in the early stages of training enables the FCN to outperform the CNN.
Crucially, this result suggests that reparametrizing during training not only saves time, but also helps the T-CNN ﬁnd better solutions.
D ISCUSSION
In this work, we showed that complex building blocks such as self-attention layers need not be trained from start.
Instead, one can save in compute time while gaining in performance and robustness by initializing them from pre-trained convolutional layers.
At a time where energy savings and robustness are key stakes, we believe this ﬁnding is important.
On the practical side, our ﬁne-tuning method offers an interesting new direction for practitioners.
One limitation of our method is the prohibitive cost of reparametrizing the early stages of CNNs.
This cost could however be alleviated by using linear attention methods (Wang et al., 2020), an important direction for future work.
Note also that while our T-CNNs signiﬁcantly improve the robustness of
CNNs, they do not always reach the performance of end-to-end Transformers such as the DeiT (for example on ImageNet-C, see Fig. 1).
Bridging this gap is an important next step for hybrid models. A P ERFORMANCE TABLE
In Tab.
[Table]
we display the characteristics and the performance of our T-ResNet-RS models and compare them to the original ResNet-RS models as well as several other strong baselines reported in Bhojanapalli et al.
; Mao et al. (2021).
B C HANGING THE LEARNING RATE
We have shown that the learning dynamics decompose into two phases: the learning rate warmup phase, where the test loss drops, then the learning rate decay phase, where the test loss increases again.
This could lead one to think that the maximal learning rate is too high
[Table]
and the dip could be avoided by choosing a lower learning rate.
Yet this is not the case, as shown in Fig.
Reducing the maximal learning rate indeed reduces the dip, but it also slows down the increase in the second phase of learning.
This conﬁrms that the model needs to “unlearn” the right amount to ﬁnd better solutions.
[Figure]
[Figure]
C C HANGING THE TEST RESOLUTION
One advantage of the GPSA layers introduced by d’Ascoli et al.
(2021) is how easily they adapt to different image resolutions.
Indeed, the positional embeddings they use are ﬁxed rather than learnt.
They simply consist in 3 values for each pair of pixels: their euclidean distance (cid:107) δ (cid:107) , as well as their coordinate distance δ 1 , δ 2 (see Eq. 4).
Our implementation automatically adjusts these embeddings to the input image, allowing us to change the test resolution seamlessly.
In Fig.
[Figure]
we show how the top-1 accuracies of our T-ResNet-RS models compares to those of the ResNet-RS models ﬁnetuned at same resolution but without SA.
At test resolution 416, our TResNetRS-350 reaches an impressive top-1 accuracy of 84.9%, beyond those of the best EfﬁcientNets and BotNets Srinivas et al.
D C HANGING THE NUMBER OF EPOCHS
In Tab. 4, we show how the top-1 accuracy of the T-ResNet-RS model changes with the number of ﬁne-tuning epochs.
As expected, performance increases signiﬁcantly as we ﬁne-tune for longer, yet we chose to set a maximum of 50 ﬁne-tuning epochs to keep the computational cost of ﬁne-tuning well below that of the original training.
[Table]
E C HANGING THE ARCHITECTURE
Our framework, which builds on the timm package, makes changing the original CNN architecture very easy.
We applied our ﬁne-tuning procedure to the ResNet-D models He et al.
with the exact same hyperparameters, and observed substantial performance gains, similar to the ones obtained for ResNet-RS, see Tab. 5.
This suggests the wide applicability of our method
[Table]