O MNI -D IMENSIONAL D YNAMIC C ONVOLUTION
A BSTRACT
Learning a single static convolutional kernel 1 in each convolutional layer is the common training paradigm of modern Convolutional Neural Networks (CNNs).
Instead, recent research in dynamic convolution shows that learning a linear combination of n convolutional kernels weighted with their input-dependent attentions can signiﬁcantly improve the accuracy of light-weight CNNs, while maintaining efﬁcient inference.
However, we observe that existing works endow convolutional kernels with the dynamic property through one dimension (regarding the convolutional kernel number) of the kernel space, but the other three dimensions (regarding the spatial size, the input channel number and the output channel number for each convolutional kernel) are overlooked.
Inspired by this, we present
Omni-dimensional Dynamic Convolution (ODConv), a more generalized yet elegant dynamic convolution design, to advance this line of research.
ODConv leverages a novel multi-dimensional attention mechanism with a parallel strategy to learn complementary attentions for convolutional kernels along all four dimensions of the kernel space at any convolutional layer.
As a drop-in replacement of regular convolutions, ODConv can be plugged into many CNN architectures.
Extensive experiments on the ImageNet and MS-COCO datasets show that ODConv brings solid accuracy boosts for various prevailing CNN backbones including both light-weight and large ones, e.g., 3.77% ∼ 5.71% | 1.86% ∼ 3.72% absolute top-1 improvements to MobivleNetV2 | ResNet family on the ImageNet dataset
Intriguingly, thanks to its improved feature learning ability, ODConv with even one single kernel can compete with or outperform existing dynamic convolution counterparts with multiple kernels, substantially reducing extra parameters.
Furthermore, ODConv is also superior to other attention modules for modulating the output features or the convolutional weights.
Code and models will be available at https://github.com/OSVAI/ODConv .
I NTRODUCTION
In the past decade, we have witnessed the tremendous success of deep Convolutional Neural Networks (CNNs) in many computer vision applications (Krizhevsky et al., 2012; Girshick et al., 2014;
Long et al., 2015;
He et al., 2017).
The most common way of constructing a deep CNN is to stack a number of convolutional layers as well as other basic layers organized with the predeﬁned feature connection topology.
Along with great advances in CNN architecture design by manual engineering (Krizhevsky et al., 2012;
He et al., 2016; Howard et al., 2017) and automatic searching (Zoph & Le, 2017; Pham et al., 2018; Howard et al., 2019), lots of prevailing classiﬁcation backbones have been presented.
Recent works (Wang et al., 2017; Hu et al., 2018b; Park et al., 2018;
Woo et al., 2018; Yang et al., 2019; Chen et al., 2020) show that incorporating attention mechanisms into convolutional blocks can further push the performance boundaries of modern CNNs, and thus it has attracted great research interest in the deep learning community.
The well-known SENet (Hu et al., 2018b) uses an attention module consisting of squeeze and excitation operations to adaptively recalibrate the output features of convolutional layers, strengthening the representation power of a CNN via encouraging informative feature channels while suppressing less important ones.
Numerous attentive feature recalibration variants (Woo et al., 2018; Park et al., 2018; Hu et al., 2018a) have been proposed since then.
In Lin et al.
(2020) and Quader et al.
, two attention extensions to modulate the convolutional weights instead of the output features are also presented.
Unlike the aforementioned methods in which the number of convolutional parameters of a target network is ﬁxed, dynamic convolution, which applies the attention mechanism over n additive convolutional kernels to increase the size and the capacity of a network while maintaining efﬁcient inference, has recently become popular in optimizing efﬁcient CNNs.
This line of research is pioneered by Conditionally Parameterized Convolutions (CondConv) (Yang et al., 2019) and Dynamic Convolution (DyConv) (Chen et al., 2020) whose basic ideas are the same.
Generally, unlike a regular convolutional layer which applies the same (i.e., static) convolutional kernel to all input samples, a dynamic convolutional layer learns a linear combination of n convolutional kernels weighted with their attentions conditioned on the input features.
Despite signiﬁcant accuracy improvements for light-weight CNNs, dynamic convolution designed in this way has two limitations.
Firstly, the main limitation lies in the attention mechanism design.
The dynamic property of CondConv and
DyConv comes from computing convolutional kernels as a function of the input features.
However, we observe that they endow the dynamic property to convolutional kernels through one dimension (regarding the convolutional kernel number) of the kernel space while the other three dimensions (regarding the spatial size, the input channel number and the output channel number for each convolutional kernel) are overlooked.
As a result, the weights of each convolutional kernel share the same attention scalar for a given input, limiting their abilities to capture rich contextual cues.
That is, the potentials of dynamic convolutional property have not been fully explored by existing works, and thus they leave considerable room for improving the model performance.
Secondly, at a convolutional layer, replacing regular convolution by dynamic convolution increases the number of convolutional parameters by n times.
When applying dynamic convolution to a lot of convolutional layers, it will heavily increase the model size.
To handle this limitation, Li et al.
(2021) proposes a dynamic convolution decomposition method which can get more compact yet competitive models.
Instead, in this paper we address both of the above limitations in a new perspective: formulating a more diverse and effective attention mechanism and inserting it into the convolutional kernel space .
Our core contribution is a more generalized yet elegant dynamic convolution design called Omnidimensional Dynamic Convolution (ODConv).
Unlike existing works discussed above, at any convolutional layer, ODConv leverages a novel multi-dimensional attention mechanism to learn four types of attentions for convolutional kernels along all four dimensions of the kernel space in a parallel manner.
We show that these four types of attentions learnt by our ODConv are complementary to each other, and progressively applying them to the corresponding convolutional kernels can substantially strengthen the feature extraction ability of basic convolution operations of a CNN.
Consequently, ODConv with even one single kernel can compete with or outperform existing dynamic convolution counterparts with multiple kernels, substantially reducing extra parameters.
As a drop-in design, ODConv can be used to replace regular convolutions in many CNN architectures.
It strikes a better tradeoff between model accuracy and efﬁciency compared to existing dynamic convolution designs, as validated by extensive experiments on the large-scale ImageNet classiﬁcation dataset (Russakovsky et al., 2015) with various prevailing CNN backbones.
ODConv also shows better recognition performance under similar model complexities when compared to other state-of-the-art attention methods for output feature recalibration (Woo et al., 2018;
Hu et al., 2018b; Wang et al., 2020; Lin et al., 2020) or for convolutional weight modiﬁcation (Ma et al., 2020; Lin et al., 2020; Quader et al., 2020).
Furthermore, the performance improvements by ODConv for the pre-trained classiﬁcation models can transfer well to downstream tasks such as object detection on the MS-COCO dataset (Lin et al., 2014), validating its promising generalization ability.
R ELATED W ORK
Deep CNN Architectures.
AlexNet (Krizhevsky et al., 2012) ignited the surge of deep CNNs by winning the ImageNet classiﬁcation challenge 2012.
Since then, lots of well-known CNN architectures such as VGGNet (Simonyan & Zisserman, 2015), InceptionNet (Szegedy et al., 2015),
ResNet (He et al., 2016), DenseNet (Huang et al., 2017) and ResNeXt
(Xie et al., 2017) have been proposed, which are designed to be much deeper and have more sophisticated connection topologies compared with AlexNet.
To ease the deployment of inference models on resource-limited platforms,
MobileNets (Howard et al., 2017; Sandler et al., 2018) and ShufﬂeNet (Zhang et al., 2018b; Ma et al., 2018) are presented.
All the aforementioned CNNs are manually designed.
Recently, researchers have also made great efforts (Zoph & Le, 2017; Pham et al., 2018;
Howard et al., 2019) to automate the network design process.
Our ODConv could be potentially used to boost their performance.
Attentive Feature Recalibration.
Designing attentive feature recalibration modules to improve the performance of a CNN has been widely studied in recent years.
Wang et al.
(2017) proposes a specialized attention module consisting of a trunk branch and a mask branch, and inserts it into the intermediate stages of deep residual networks.
SENet (Hu et al., 2018b) uses a seminal channel attention module termed Squeeze-and-Excitation (SE) to exploit the interdependencies between the channels of convolutional features.
Many subsequent works improve SE from different aspects, following its two-stage design (i.e., feature aggregation and feature recalibration).
BAM (Park et al., 2018) and CBAM (Woo et al., 2018) combine the channel attention module with the spatial attention module.
Misra et al.
(2021) presents an attention module having three branches conditioned on the features rotated along three different dimensions.
GE (Hu et al., 2018a) introduces a gather operator to extract better global context from a large spatial extent.
To enhance the feature aggregation capability, SRM (Lee et al., 2019) replaces the global average by the channel-wise mean and standard deviation.
SKNets (Li et al., 2019) add an attention design over two branches with different sized convolutions to fuse multi-scale feature outputs.
ECA (Wang et al., 2020) provides a more efﬁcient channel attention design using cheaper 1D convolutions to replace the ﬁrst fully connected layer of SE.
Instead of recalibrating the output convolutional features by attention modules, dynamic convolution methods apply attention mechanisms to a linear combination of n convolutional kernels.
Dynamic Weight Networks.
Making the weights of a neural network to be sample-adaptive via dynamic mechanisms has shown great potentials for boosting model capacity and generalization.
Hypernetworks (Ha et al., 2017) use a small network called hypernetwork to generate the weights for a larger recurrent network called main network.
MetaNet (Munkhdalai & Yu, 2017) adopts a meta learning model to parameterize the task-adaptive network for rapid generalization across a sequence of tasks.
Jaderberg et al.
(2015) proposes a Spatial Transformer module conditioned on the learnt features to predict the parametric transformation, and applies it to align the distorted input image.
Dynamic Filter Network (Jia et al., 2016) uses a ﬁlter generation network to produce ﬁlters conditioned on an input, and processes another input with the generated ﬁlters.
DynamoNet (Diba et al., 2019) uses dynamically generated motion ﬁlters to handle the action recognition problem.
Kernel Prediction Networks (Bako et al., 2017; Mildenhall et al., 2018) leverage a CNN architecture to predict spatially varying kernels used for video denoising.
WeightNet (Ma et al., 2020) appends a grouped fully connected layer to the attention feature vector of an SE block, generating the weights of a CNN used for image recognition.
Lin et al.
(2020) modiﬁes the weights of convolutional layers with a gated module under the guidance of global context, while Quader et al.
(2020) directly uses either an SE block or a simple activation function conditioned on the magnitudes of convolutional weights to modify the weights themselves.
Our ODConv aims to address the limitations of recently proposed dynamic convolution (Yang et al., 2019; Chen et al., 2020) which differs from these methods both in focus and formulation, see the Introduction and Method sections for details.
M ETHOD
In this section, we ﬁrst make a review of dynamic convolution via a general formulation.
Then, we describe the formulation of our ODConv, clarify its properties and detail its implementation.
3. R EVIEW OF D YNAMIC C ONVOLUTION
Basic concept.
A regular convolutional layer has a single static convolutional kernel which is applied to all input samples.
For a dynamic convolutional layer, it uses a linear combination of n convolutional kernels weighted dynamically with an attention mechanism, making convolution operations be input-dependent.
Mathematically, dynamic convolution operations can be deﬁned as
[Equation]
where x ∈ R h × w × c in and y ∈ R
h × w × c out denote the input features and the output features (having c in / c out channels with the height h and the width w ), respectively; W i denotes the i th convolutional kernel consisting of c out ﬁlters
W m i ∈
R k × k × c in , m = 1 , ..., c out ; α wi ∈ R is the attention scalar
+ * * 𝒙 GAP
FC
ReLU 𝑾𝑾… 𝑾 𝒏 𝜶 𝒘 𝜶 𝒘 … 𝜶 𝒘𝒏 .
FC Softmax 𝒚 + 𝒙 𝒚
GAP
FC
ReLU 𝑾𝑾… 𝑾 𝒏 𝜶 𝒔 𝜶 𝒔 … 𝜶 𝒔𝒏 𝜶 𝒄 𝜶 𝒄 … 𝜶 𝒄𝒏 .
FC Sigmoid
FC Sigmoid FC Sigmoid
FC Softmax 𝜶 𝒇 𝜶 𝒇 … 𝜶 𝒇𝒏 .
𝜶 𝒘 𝜶 𝒘 … 𝜶 𝒘𝒏 .
(a) (b) for weighting W is by an wi
CondConv vs. DyConv.
Although the concept of dynamic convolution deﬁned in Eq.
1 is proposed separately in CondConv (Yang et al., 2019) and DyConv (Chen et al., 2020), their implementations are different, mainly in the structure of π wi ( x ) to compute α wi , the model training strategy, and the layer locations to apply dynamic convolutions.
Speciﬁcally, both methods choose the modiﬁed SE structure for π wi ( x ) , and CondConv uses a Sigmoid function while DyConv uses a Softmax function as the activation function to compute α wi .
DyConv adopts a temperature annealing strategy in the training process to suppress the near one-hot output of the Softmax function.
For all their tested CNN architectures, CondConv replaces the convolutional layers in the ﬁnal several blocks (e.g., 6 for the
MobileNetV2 backbones and 3 for the ResNet backbones) and the last fully connected layer, while
DyConv replaces all convolutional layers except the ﬁrst layer.
These implementation differences lead to different results in model accuracy, size and efﬁciency for CondConv and DyConv.
Limitation Discussions.
According to Eq. 1, dynamic convolution has two basic components: the convolutional kernels { W 1 , ...W n } , and the attention function π wi ( x ) to compute their attention scalars { α w 1 , ...
α wn } .
Given n convolutional kernels, the corresponding kernel space has four dimensions regarding the spatial kernel size k × k , the input channel number c in and the output channel number c out for each convolutional kernel, and the convolutional kernel number n .
However, for
CondConv and DyConv, we can observe that π wi ( x ) allocates a single attention scalar α wi to the convolutional kernel W i , meaning that all its c out ﬁlters W mi ∈ R k × k × c in , m = 1 , ..., c out have the same attention value for the input x .
In other words, the spatial dimension, the input channel dimension and the output channel dimension for the convolutional kernel W i are ignored by CondConv and DyConv.
This leads to a coarse exploitation of the kernel space when they design their attention mechanisms for endowing n convolutional kernels with the dynamic property.
This may also be one of the reasons why CondConv and DyConv show much lower performance gains to relatively larger CNNs compared to efﬁcient ones.
Besides, compared to a regular convolutional layer, a dynamic convolutional layer increases the number of convolutional parameters by n times (although the increase of Multiply-Adds (MAdds) is marginal due to the additive property of n convolutional kernels).
Typically, CondConv sets n = 8 and DyConv sets n = 4 .
Therefore, it will heavily increase the model size when applying dynamic convolution to a lot of convolutional layers.
However, we empirically ﬁnd that removing the attention mechanism from CondConv | DyConv (i.e., setting α wi = 1 ) almost diminishes the accuracy boosts for prevailing CNN backbones on the ImageNet dataset close to zero.
For instance, on ResNet18, the top-1 gain averaged over 3 runs decreases from 1 .
74% | 2 .
51% to 0 .
08% | 0 .
14% when removing the attention mechanism from CondConv
| DyConv.
These observations indicate that the attention mechanism design plays the key role in dynamic convolution, and a more effective design may strike a good balance between model accuracy and size.
3. O MNI -D IMENSIONAL D YNAMIC C ONVOLUTION
In light of the above discussions, our ODConv introduces a multi-dimensional attention mechanism with a parallel strategy to learn diverse attentions for convolutional kernels along all four dimensions of the kernel space.
Fig.
1 provides a schematic comparison of CondConv, DyConv and ODConv.
( c ) .
𝑾 𝒊𝒎 𝜶 𝒔𝒊 ⊙ 𝑐 𝑖𝑛 𝑐 𝑖𝑛 𝑾 𝒊𝒎 𝑘 𝑘 𝜶 𝒔𝒊 𝑘 𝑘 𝑘 𝑘 (a) 1 1 𝑾 𝒊 𝜶 𝒇𝒊 ⊙ .
𝑐 𝑖𝑛 𝑐 𝑖𝑛 𝑐 𝑖𝑛 𝑘 𝑘 𝑐 𝑖𝑛 𝑘 𝑘 … … 𝑘 𝑘 𝑘 𝑘 … 𝜶 𝒇𝒊 𝑾 𝒊 𝑐 𝑜𝑢𝑡 𝑐 𝑜𝑢𝑡 𝑐 𝑜𝑢𝑡 .
1𝑐 𝑖𝑛 𝑐 𝑖𝑛 𝑘 𝑘 𝑾 𝒊𝒎 𝜶 𝒄𝒊 𝑘 𝑘 (b) 𝑐 𝑖𝑛 𝑾 𝒊𝒎 𝜶 𝒄𝒊 ⊙ (d) [𝑾 𝟏 … 𝑾 𝒏 ] 𝑐 𝑜𝑢𝑡 … 𝑐 𝑖𝑛 𝑘 𝑘 𝑐 𝑖𝑛 𝑘 𝑘 … 𝑐 𝑖𝑛 𝑘 𝑘 𝑐 𝑖𝑛 𝑘 𝑘 𝑛 … .
𝑛 11 11 … …
𝑐 𝑖𝑛 𝑘 𝑘 𝑐 𝑖𝑛 𝑘 𝑘 … 𝑐 𝑖𝑛 𝑘 𝑘 𝑐 𝑖𝑛 𝑘 𝑘 𝑛 … 𝜶 𝒘 [𝑾 𝟏 … 𝑾 𝒏 ] 𝜶 𝒘 ⊙
Formulation of ODConv.
Following the notations in Eq. 1, ODConv can be deﬁned as
[Equation]
where α wi ∈ R denotes the attention scalar for the convolutional kernel W i , which is the same to that in Eq.
1; α si ∈ R k × k , α ci ∈ R c in and α fi ∈ R
c
out denote three newly introduced attentions, which are computed along the spatial dimension, the input channel dimension and the output channel dimension of the kernel space for the convolutional kernel W i , respectively; (cid:12) denotes the multiplication operations along different dimensions of the kernel space.
Here, α si , α ci , α fi and α wi are computed with a multi-head attention module
π i ( x ) which will be clariﬁed later.
A Deep Understanding of ODConv.
In ODConv, for the convolutional kernel W i : (1) α si assigns different attention scalars to convolutional parameters (per ﬁlter) at k × k spatial locations; (2) α ci assigns different attention scalars to c in channels of each convolutional ﬁlter W mi ; (3) α fi assigns different attention scalars to c out convolutional ﬁlters; (4) α wi assigns an attention scalar to the whole convolutional kernel.
Fig.
2 illustrates the process of multiplying these four types of attentions to n convolutional kernels.
In principle, these four types of attentions are complementary to each other, and progressively multiplying them to the convolutional kernel W i in the location-wise, channel-wise, ﬁlter-wise and kernel-wise orders makes convolution operations be different w.r.t.
all spatial locations, all input channels, all ﬁlters and all kernels for the input x , providing a performance guarantee to capture rich context cues.
Therefore, ODConv can signiﬁcantly strengthen the feature extraction ability of basic convolution operations of a CNN.
Moreover, ODConv with one single convolutional kernel can compete with or outperform standard CondConv and DyConv, introducing substantially fewer extra parameters to the ﬁnal models.
Extensive experiments are provided to validate these advantages.
By comparing Eq. 1 and Eq. 2, we can clearly see that ODConv is a more generalized dynamic convolution.
Moreover, when setting n = 1 and all components of α s 1 , α c 1 and α w 1 to 1, ODConv with only ﬁlter-wise attention α f 1 will be reduced into: applying an SE variant conditioned on the input features to the convolutional ﬁlters, then followed by convolution operations (note the original SE (Hu et al., 2018b) is conditioned on the output features, and is used to recalibrate the output features themselves).
Such an SE variant is a special case of ODConv.
Implementation.
For ODConv, a critical question is how to compute four types of attentions α si , α ci , α fi and α wi for the convolutional kernel W i .
Following CondConv and DyConv, we also use a SE-typed attention module (Hu et al., 2018b) but with multiple heads as π i ( x ) to compute them, whose structure is shown in Fig.
Speciﬁcally, the input x is squeezed into a feature vector with the length of c in by channel-wise Global Average Pooling (GAP) operations ﬁrst.
Subsequently, there is a Fully Connected (FC) layer and four head branches.
A Rectiﬁed Linear Unit (ReLU) (Krizhevsky et al., 2012) comes after the FC layer.
The FC layer maps the squeezed feature vector to a lower dimensional space with the reduction ratio r (according to the ablative experiments, we set r = 1 /in all main experiments, avoiding high model complexity).
For four head branches, each has an FC layer with the output size of k × k , c in × 1 , c out × 1 and n × 1 , and a Softmax or Sigmoid function to generate the normalized attentions α si , α ci , α fi and α wi , respectively.
We adopt the temperature annealing strategy proposed in DyConv to facilitate the training process.
For easy implementation, we apply ODConv to all convolutional layers except the ﬁrst layer of each CNN architecture tested in our main experiments just like DyConv, and share α si , α ci and α fi to all convolutional kernels .
In the Experiments section, a comparison of the inference speed for different dynamic convolution methods is provided.
In the Appendix, we further provide a computational cost analysis of ODConv and an ablation study of applying ODConv to different layer locations.
E XPERIMENTS
In this section, we provide comprehensive experiments on two large-scale image recognition datasets with different CNN architectures to validate the effectiveness of ODConv, compare its performance with many attention based methods, and study the design of ODConv from different aspects.
4. I MAGE C LASSIFICATION ON I MAGE N ET
Our main experiments are performed on the ImageNet dataset (Russakovsky et al., 2015).
It has over 1.2 million images for training and 50,000 images for validation, including 1,000 object classes.
CNN Backbones.
We use MobileNetV2 (Sandler et al., 2018) and ResNet (He et al., 2016) families for experiments, covering both light-weight CNN architectures and larger ones.
Speciﬁcally, we choose ResNet18, ResNet50, ResNet101, and MobileNetV2 ( 1 . 0 × , 0 .
75 × , 0 .
5 × ) as the backbones.
Experimental Setup.
In the experiments, we consider existing dynamic convolution methods including CondConv (Yang et al., 2019), DyConv (Chen et al., 2020) and DCD (Li et al., 2021) as the key reference methods for comparisons on all CNN backbones.
On the ResNet backbones, we also compare our ODConv with many state-of-the-art methods using attention modules: (1) for output feature recalibration including SE (Hu et al., 2018b), CBAM (Woo et al., 2018) and ECA (Wang et al., 2020), and (2) for convolutional weight modiﬁcation including CGC (Lin et al., 2020),
WeightNet (Ma et al., 2020) and WE (Quader et al., 2020).
For fair comparisons, we use public codes of these methods unless otherwise stated, and adopt the popular training and test settings used in the community for implementing the experiments.
The models trained by all methods use the same settings including the batch size, the number of training epochs, the learning rate schedule, the weight decay, the momentum and the data processing pipeline.
Moreover, we do not use advanced training tricks such as mixup (Zhang et al., 2018a) and label smoothing (Szegedy et al., 2016), aiming to have clean performance comparisons.
Experimental details are described in the Appendix.
[Table]
Results Comparison on MobileNets.
Table 1 shows the results comparison on the MobileNetV ( 1 . 0 × , 0 .
75 × , 0 . 5 × ) backbones.
As CondConv and DyConv are primarily proposed to improve the performance of efﬁcient CNNs, they all bring promising top-1 gains to the light-weight MobileNetV2 ( 1 . 0 × , 0 .
75 × , 0 . 5 × ) backbones.
Comparatively, our ODConv ( 1 × ) with one single convolutional kernel performs better than CondConv ( 8 × ) with 8 convolutional kernels, and its performance is also on par with DyConv ( 4 × ) with 4 convolutional kernels.
Note that these competitive results of ODConv ( 1 × ) are obtained with signiﬁcantly lower numbers of extra parameters, validating that our ODConv can strike a better tradeoff between model accuracy and size.
Besides, ODConv ( 1 × ) also performs better than DCD in most cases.
ODConv ( 4 × ) always achieves the best results on all backbones.
The results comparison with an increase number of training epochs (from 150 to 300) can be found in the Appendix, from which we can observe similar performance trends.
Results Comparison on ResNets.
Table 2 shows the results comparison on the ResNet18 and
ResNet50 backbones which are much larger than the MobileNetV2 backbones.
We can obtain the following observations: (1) On the ResNet18 backbone, dynamic convolution methods (CondConv,
DyConv, DCD and our ODConv) and convolutional weight modiﬁcation methods (CGC, WeightNet and WE) mostly show better performance than output feature recalibration methods (SE, CBAM and ECA), although they all use attention mechanisms.
Comparatively, our ODConv ( 1 × ) with one single convolutional kernel outperforms the other methods both in model accuracy and size, bringing 2 .
85% top-1 gain to the baseline model.
ODConv ( 4 × ) gets the best results, yielding a top-1 gain of 3 . 72% ; (2) However, on the larger ResNet50 backbone, CondConv, DyConv and DCD show worse results than most of the other methods even though they have signiﬁcantly increased numbers of parameters.
Due to the increased parameter redundancy, adding more parameters to larger networks tends to be less effective in improving model accuracy, compared to small networks.
Thanks to the proposed multi-dimensional attention mechanism, ODConv can address this problem better, achieving superior performance both in model accuracy and size for larger backbones as well as light-weight ones.
To further validate the performance of ODConv on very deep and large networks, we apply ODConv to the ResNet101 backbone, and the results comparison is provided in Table 3.
Again, ODConv shows very promising results, yielding 1 .
57% top-1 gain with ODConv (1 × ) .
from the paper of WE (Quader et al., 2020) as its code is not publicly available.
Best results are bolded
[Table]
[Table]
[Table]
O BJECT D ETECTION ON MS-COCO
Given a backbone model pre-trained on the ImageNet classiﬁcation dataset, a critical question is whether the performance improvement by ODConv can be transferred to downstream tasks or not.
To explore it, we next perform comparative experiments on the object detection track of MS-COCO dataset (Lin et al., 2014).
The 2017 version of MS-COCO dataset contains 118,000 training images and 5,000 validation images with 80 object classes.
Experimental Setup.
We use the popular MMDetection toolbox (Chen et al., 2019) for experiments with the pre-trained ResNet50 and MobileNetV2 ( 1 . 0 × ) models as the backbones for the detector.
We select the mainstream Faster R-CNN (Ren et al., 2015) and Mask R-CNN (He et al., 2017) detectors with Feature Pyramid Networks (FPNs)
(Lin et al., 2017) as the necks to build the basic object detection systems.
For a neat comparison, the convolutional layers in the FPN necks still use regular convolutions, and we maintain the same data preparation pipeline and hyperparameter
[Table]
settings for all pre-trained models built with CondConv, DyConv and our ODConv, respectively.
Experimental details are described in the Appendix.
Results Comparison.
From the results shown in Table 4, we can observe similar performance improvement trends as on the ImageNet dataset.
For Faster R-CNN | Mask R-CNN with the pretrained ResNet50 backbone models, CondConv ( 8 × ) and DyConv ( 4 × ) show an AP improvement of 0 .
9% | 0 .
8% and 1 .
1% | 1 . 2% to the baseline model respectively, while our method performs much better, e.g., ODConv ( 1 × ) with one single convolutional kernel shows an AP improvement of 2 . 0% | 2 .
1% , respectively.
With the pre-trained MobileNetV2 ( 1 . 0 × ) backbone models, our ODConv ( 1 × ) performs obviously better than CondConv ( 8 × ), and its performance is on par with that of DyConv ( 4 × ) as their AP gap is only 0 .
2% for both two detectors, striking a better model accuracy and efﬁciency tradeoff.
Similar boosts to AP scores for small, medium and large objects are also obtained by three methods on both detectors.
ODConv ( 4 × ) always achieves the best AP scores.
4. A BLATION S TUDIES
Finally, we conduct a lot of ablative experiments on the ImageNet dataset, in order to have a better analysis of our ODConv.
[Table] [Table]
Reduction Ratio Selection.
Our ﬁrst set of ablative experiments is for the selection of the reduction ratio r used in the attention structure π i ( x ) .
From the results shown in Table 5, we can ﬁnd that under r = 1 / 4 , r = 1 / 8 and r = 1 / 16 , ODConv consistently obtains large performance improvements to the baseline ResNet18 model ( 2 . 85 ∼ 3 .
16% top-1 gain), and the extra MAdds are negligible.
Comparatively, ODConv with r = 1 / 16 strikes the best tradeoff between model accuracy and efﬁciency, and thus we choose it as our default setting.
[Table] [Table]
Convolutional Kernel Number.
Accordingly to the main experiments discussed beforehand, ODConv can strike a better tradeoff between model accuracy and size, compared to existing dynamic convolution methods.
For a better understanding of this advantage, we next perform the second set of ablative experiments on the ImageNet dataset, training the ResNet18 backbone based on ODConv with different settings of the convolutional kernel number n .
Table 6 shows the results.
It can be seen that ODConv with one single convolutional kernel brings near 3 . 0% top-1 gain to the ResNet baseline, and the gain tends to be saturated when the number of convolutional kernels is set to 8 .
Four Types of Attentions.
Note that our ODConv has four types of convolutional kernel attentions α si , α ci , α fi and α wi computed along all four dimensions of the kernel space, respectively.
Then, we perform another set of ablative experiments on the ResNet18 backbone with different combinations of them to investigate their dependencies.
Results are summarized in Table 7, from which the strong complementarity of four types of attentions α si , α ci , α fi and α wi can be clearly observed.
Inference Speed.
Besides the size and the MAdds for a CNN model, the inference speed at runtime is very important in the practical model deployment.
Table 8 provides a comparison of the inference speed for different dynamic convolution methods (both on a GPU and a CPU).
It can be seen that the models trained with our ODConv ( 1 × ) are faster than the counterparts trained with DyConv and
DCD on a CPU.
Comparatively, the models trained with CondConv show the fastest run-time speed both on a GPU and a CPU, this is because CondConv is merely added to the ﬁnal several blocks and the last fully connected layer as discussed in the Method section.
When adding ODConv ( 1 × ) to the same layer locations as for CondConv, we can obtain more accurate and efﬁcient models.
More Ablations.
In the Appendix, we provide more ablative experiments to study: (1) the effects of applying ODConv to different layer locations; (2) the importance of the temperature annealing strategy; (3) the choices of the activation functions for π i ( x ) ; (4) the inﬂuence of the attention sharing strategy; (5) the stability of the model training process; (6) other potentials of ODConv.
C ONCLUSION
In this paper, we present a new dynamic convolution design called Omni-dimensional Dynamic
Convolution (ODConv) to promote the representation power of deep CNNs.
ODConv leverages a multi-dimensional attention mechanism to learn four types of attentions for convolutional kernels along all four dimensions of the kernel space in a parallel manner, and progressively applying these attentions to the corresponding convolutional kernels can substantially strengthen the feature extraction ability of basic convolution operations of a CNN.
Experimental results on the ImageNet and
MS-COCO datasets with various prevailing CNN architectures validate its superior performance.
A A PPENDIX
In this section, we describe the supplementary materials including: (1) a computational cost analysis of ODConv; (2) implementation details for the experiments on the ImageNet and MS-COCO datasets; (3) more ablative experiments on the ImageNet dataset; (4) illustrative training and validation curves to compare the stability of the model training process with different dynamic convolution methods; (5) more experiments to study other potentials of ODConv.
A. C OMPUTATIONAL C OST OF ODC ONV
For a convolutional layer with ODConv, its computational cost can be easily calculated according to the design and the multiplication process of four types of attentions shown in Fig.
1 and Fig. 2, respectively.
Speciﬁcally, following the notations deﬁned in the Method section of the main paper, the extra MAdds of ODConv ( 1 × ) over regular convolution (which has hwk 2 c in c out MAdds, without consideration of the bias term) can be calculated as
[Equation]
For ODConv ( n × ), the extra MAdds over regular convolution can be calculated as
[Equation]
Compared to hwk 2 c in c out MAdds for regular convolution, the extra MAdds by ODConv are small.
A. E XPERIMENTAL D ETAILS
Experimental Details on ImageNet.
Recall that we use MobileNetV2 (Sandler et al., 2018) and
ResNet (He et al., 2016) families for experiments on the ImageNet dataset, covering both lightweight CNN architectures and larger ones.
For fair comparisons, we adopt the popular training settings used in the community to train the respective backbone models with all methods.
Speciﬁcally, for ResNet18, ResNet50 and ResNet101, all models are trained with SGD for 100 epochs.
We set the batch size as 256, the weight decay as 0.0001 and the momentum as 0.9.
The learning rate starts at 0.1, and is divided by 10 every 30 epochs.
Following DyConv (Chen et al., 2020), for our
ODConv, we also use dropout rate of 0.1 for ResNet18.
We use dropout rate of 0.2 for ResNet and ResNet101 for our ODConv.
For MobileNetV2 ( 1 .
× , 0 .
75 × , 0 . 5 × ), all models are trained with SGD for 150 epochs (we also have the experiments for training all models with 300 epochs in this Appendix).
We set the batch size as 256, the weight decay as 0.00004 and the momentum as 0.9.
The learning rate starts at 0.05, and is scheduled to arrive at zero within a single cosine cycle.
Following DyConv, for our ODConv, we also use dropout rate of 0.2 for MobileNetV2 ( 1 . 0 × ) and dropout rate of 0.1 for MobileNetV2 ( 0 .
75 × , 0 .
5 × )
Regarding the temperature annealing strategy used for DyConv and ODConv, the temperature reduces from 30 to 1 linearly in the ﬁrst 10 epochs for all models.
All experiments are performed on the servers having 8 GPUs.
We follow the standard protocols to train and evaluate each CNN backbone.
For training, images are resized to 256 ×ﬁrst, and then 224 × 224 crops are randomly sampled from the resized images or their horizontal ﬂips normalized with the per-channel mean and standard deviation values.
For evaluation, we report top-1 and top-5 recognition rates using the center image crops.
Experimental Details on MS-COCO.
Recall that we use the popular MMDetection toolbox (Chen et al., 2019) for experiments on the MS-COCO dataset with the pre-trained ResNet50 and MobileNetV2 ( 1 . 0 × ) models as the backbones for the detector.
We select the mainstream Faster RCNN (Ren et al., 2015) and Mask R-CNN
(He et al., 2017) detectors with Feature Pyramid Networks (FPNs) (Lin et al., 2017) as the necks to build the basic object detection systems.
For a neat and fair comparison, the convolutional layers in the FPN necks still use regular convolutions, and we maintain the same data preparation pipeline and hyperparameter settings for all pre-trained models built with CondConv, DyConv and our ODConv, respectively.
All experiments are performed on a server having 8 GPUs with a mini-batch size of 2 images per GPU.
We ﬁnetune these detectors on the MS-COCO training set following the 1 × learning rate schedule, which indicates a total of 12 epochs with the learning rate divided by 10 at the 8 th epoch and the 11 th epoch, respectively.
Following DyConv, the temperature annealing strategy is not used in the downstream experiments on the MS-COCO dataset, as it is prone to worse results.
In the validation, we report the standard
Average Precision (AP) under IOU thresholds ranging from 0.5 to 0.95 with an increment of 0.05.
We also keep AP scores for small, medium and large objects.
A. M ORE A BLATIVE E XPERIMENTS ON I MAGE N ET
Besides the ablative experiments described in the main paper, here we provide more ablative experiments, for a better understanding of our ODConv.
Effects of Applying ODConv to Different Layer Locations.
In the Experiments section of the main paper, we show that applying dynamic convolution to less layers leads to a faster runtime speed compared to applying it to more layers.
For easy implementation, we follow DyConv and apply ODConv to all convolutional layers except the ﬁrst layer of each CNN architecture tested in our main experiments.
However, it is necessary to study the effects of applying ODConv to different layer locations.
The performance comparison of adding ODConv to different layers of the MobileNetV ( 0 . 5 × ) backbone is given in Table 9, showing that our default setting (i.e., adding ODConv to all convolutional layers except the ﬁrst layer) has the largest gain.
This indicates that adding ODConv to more convolutional layers tends to have a model with higher recognition accuracy.
[Figure]
[Table]
Importance of Temperature Annealing Strategy.
Following DyConv [Table] we also use the temperature annealing strategy to facilitate the model training process of our ODConv.
To study its effect, we perform a set of ablative experiments on the ImageNet dataset [Table] training the ResNet18 backbone based on ODConv with or without using the temperature annealing strategy.
From the results shown in Table [Table] we can ﬁnd that the temperature annealing strategy is really important, which brings 1 .
06% | 1 .
04% top-1 improvement to the ResNet18 model based on ODConv ( 1 × ) | ODConv ( 4 × ).
[Table]
Choices of the Activation Functions.
Recall that in the structure of π i ( x ) for our ODConv, we use two different activation functions (either a Sigmoid function or a Softmax function) to compute four types of attentions α si , α ci , α fi and α wi .
It is also critical to compare the performance of different activation functions for ODConv.
Since the activation function choices for the channel dimension and the convolutional kernel dimension have been thoroughly discussed in the papers of SE and DyConv respectively, we adopt their suggestions for three related attentions deﬁned in our ODConv.
Here we perform another set of ablative experiments, focusing on the choices of the activation function for the spatial dimension.
With the ImageNet dataset, we use ResNet18 as the test backbone network to explore the effects of different activation functions for computing the attention of ODConv along the spatial dimension of the kernel space.
Table 11 summarizes the results, from which we can see that the Sigmoid function performs better than the Softmax function, and thus we use the Sigmoid function to compute the attention scalars along the spatial dimension for ODConv.
Attention Sharing Strategy.
As we discussed in the Method section, we share three attentions α si , α ci and α fi to all convolutional kernels for the easy implementation as well as for more efﬁcient
[Table] [Table]
training.
Actually, training separate α si , α ci and α fi for each additive convolutional kernel will bring further improved model accuracy, as can be seen from the ablative results in Table 12.
That is, when preferring to get more accurate models, the attention sharing strategy would be removed
[Table]
Training MobileNetV2 Backbones for 300 Epochs.
Recall that in the Experiments section of the main paper, we compare our ODConv with CondConv, DyConv and DCD on MobileNetV2 ( 1 .
× , 0 .
75 × , 0 . 5 × ), following the popular training settings used in the community, where all models are trained for 150 epochs.
To validate the effectiveness of our ODConv further, we also conduct another set of experiments using an increased number of training epochs (300 instead of 150) as adopted in DyConv (Chen et al., 2020).
All the other training settings remain the same to those for the experiments described in the main paper.
For a clean comparison of all methods, we do not use advanced training tricks such as mixup (Zhang et al., 2018a) and label smoothing (Szegedy et al., 2016).
Results are summarized in Table 13 where performance gains show a similar trend as those in Table 1 for the model training with 150 epochs.
Results in Fig.
3 further show that our ODConv gets a better trade-off between model accuracy and size even on the light-weight MobileNetV backbones compared to CondConv and DyConv.
[Table]
A. I LLUSTRATION OF M ODEL T RAINING AND V ALIDATION C URVES
Fig.
4 illustrates the training and validation accuracy curves of the ResNet18 models trained on the
ImageNet dataset with CondConv, DyConv, ODConv ( 1 × ) and ODConv ( 4 × ), respectively.
We can see that our ODConv shows consistent high top-1 gains throughout the training process compared to the other three dynamic convolution counterparts.
A. M ORE E XPERIMENTS FOR S TUDYING O THER P OTENTIALS OF ODC ONV
In this section we provide a lot of extra experiments conducted for studying other potentials of
ODConv.
Performance Comparison on A CNN Backbone Added with the SE Module.
Recall that SE (Hu et al., 2018b) performs attention on the output features of a convolutional layer, while our ODConv performs attention on the convolutional kernels, and thus they are likely to be complimentary.
To validate this, we perform another set of experiments on the ImageNet dataset with ResNet18 backbone.
In the experiments, we use the ResNet18+SE variant reported in Table 2 as the backbone, and train this backbone with CondConv, DyConv, DCD and our ODConv separately, adopting the same training settings used for the ResNet18 backbone.
Results are shown in Table 14.
It can be seen that our ODConv brings signiﬁcantly large performance improvement to the ResNet18 backbone already incorporating the SE module, showing its great combination potential.
[Table]
Performance Boost Using Heavy Data Augmentations and a Longer Training Schedule.
Recall that in the main paper, we do not use advanced training tricks such as mixup (Zhang et al., 2018a) and label smoothing (Szegedy et al., 2016), aiming to have clean performance comparisons.
To better explore the potential of our ODConv, it would be necessary to have a study about whether ODConv can also work well when using heavy data augmentations and a longer training schedule.
To this end, we perform a set of experiments on the ImageNet dataset with the ResNet backbone trained with ODConv (4 × ).
In the experiments, we set r=1/16 for our method, ﬁrst train
ResNet18 with label smoothing, mixup and (label smoothing+mixup) separately for 100 epochs (the other training settings are the same to those used for Table 2), and then train ResNet18 with (label smoothing+mixup) for 120 epochs instead of 100 epochs.
Detailed results are summarized in Table 15.
It can be seen that our ODConv works well when using aggressive data augmentations and a longer training schedule, showing further improved performance
[Table] [Table]
Feature Pooling Strategy.
Note that when computing four types of attentions in our ODConv, the inputs features are always globally averaged, reducing their spatial dimension to one pixel.
According to the ﬁrst three results of Table 7 in the main paper [Table] the spatial convolutional kernel attention α si brings the largest gain to the baseline model compared to the other two attentions ( α ci and α fi computed along the dimension of the input channels and the output channels, respectively).
Intuitively, at the ﬁrst glance, it may easily lead to a counterintuitive feeling as the input features used to predict α si (as well as the other three attentions) do not include spatial information after the global average pooling.
However, it should be noted that our ODConv uses the attentions generated from the input features to modify the convolutional kernels, which is quite different from popular selfcalibration attention mechanisms (e.g., SE (Hu et al., 2018b) and CBAM (Woo et al., 2018) use the channel attentions generated from the output features of a convolutional layer to recalibrate the output features themselves).
As our ODConv is not a self-calibration attention module, reducing the spatial dimension of the input features to one pixel (but the length of the reduced vector is large enough) will not affect its promising performance.
To validate this, we conduct another experiment on the ImageNet dataset with the ResNet18 backbone.
In the experiment, for α si , we pool the input features to have a size of 3 × 3 instead of 1 × 1, and show the results in Table 16.
It can be seen that preserving more spatial information only brings 0 .
06% | 0 .
02% gain to top-1 | top-5 accuracy, which proves the effectiveness of our current design to a large degree.
[Table]
Comparison of Training Cost.
Recall that in the main paper, we provide a comparison of the inference speed for different dynamic convolution methods.
Here, we further perform a set of experiments to compare the training cost of our ODConv, CondConv, DyConv and DCD.
Speciﬁcally, experiments are performed on the ImageNet dataset with the ResNet50 and MobileNetV2 backbones, using the same training settings as in the main paper.
Regarding the training cost, we report results in terms of three metrics (seconds per batch, minutes per epoch, and the total number of hours for the whole training) in Table 17.
Regarding different dynamic convolution methods, it can be seen that the training cost trend is somewhat similar to that for the runtime inference speed reported in Table 8.
Generally, the training cost for dynamic convolution methods is obviously heavier (about 2 × to 4 × ) than that for the baseline model, which is mainly due to the time-intensive back-propagation process for additive convolutional kernels weighted with input-dependent attentions.
seconds per batch, minutes per epoch, and the total number of hours for the whole training).
[Table]
A Deep Understanding of Learnt Attention Values.
Note that our ODConv leverages a multidimensional attention mechanism with a parallel strategy to learn four types of attentions α si , α ci , α fi , α wi for convolutional kernels along all four dimensions of the kernel space at any convolutional layer.
In principle, progressively multiplying these four types of attentions to the convolutional kernel in the location-wise, channel-wise, ﬁlter-wise and kernel-wise orders makes convolution operations be different w.r.t.
all spatial locations, all input channels, all ﬁlters and all kernels for each input sample, providing a performance guarantee to capture rich context cues.
To have a better understanding of this powerful attention mechanism, it is necessary to study the learnt attention values.
To this end, we use the well-trained ResNet18 models reported in Table 7 and all 50,000 images in the ImageNet validation set, and conduct a set of experiments to analyze the learnt attention values for α si , α ci , α fi , α wi and their full combination by providing lots of visualization examples (obtained with the Grad-CAM++ method (Chattopadhyay et al., 2018)) and the statistical distributions across different layers.
The detailed computational process and results are shown in Fig.
5 and Fig.
We can get the following observations: (1) Four types of attentions α si , α ci , α fi and α wi are complementary to each other by visualization examples, which echoes the conclusion from Table 7.
The most salient attention will combat the failure cases of the other three attentions, helping the model to produce accurate predications; (2) All four types of attentions demonstrate varying attention value distribution trends across different layers of the trained model, showing their capability to capture rich context cues; (3) Each attention has its own value distribution trend, which is more diverse for α si and α wi than for α ci and α fi , showing α si and α wi maybe more important to some degree.
Note that the dimension of attentions α si , α ci , α fi and α wi is quite different, e.g., 9 for α si with 3 × 3 convolutional kernels, 4 for α wi with n = 4 , and typically several hundred for α ci and α fi .
This is another perspective why they are different and complementary in their design nature regarding all four dimensions of the convolutional kernel space for any convolutional layer.
Limitations of ODConv.
On the one side, according to the results shown in Table 1, Table 2 and
Table 8, although our ODConv shows obviously better performance under the similar model size [Figure] (c) score [Figure]
guillotine crate 𝛼 𝑓𝑖 𝛼 𝑠𝑖 𝛼 𝑐𝑖 𝛼 𝑤𝑖 𝑎𝑙𝑙 Image GT (d) jean grille bow tie 𝛼 𝑓𝑖 𝛼 𝑠𝑖 𝛼 𝑐𝑖 𝛼 𝑤𝑖 𝑎𝑙𝑙 Image GT (
[Figure]
[Figure]
𝛼
[Figure]
𝛼
[Figure]
𝛼 𝑤𝑖
[Figure]
(e.g., ODConv (4 × ) vs. DyConv (4 × )), it leads to slightly increased FLOPs and introduces extra latency to the runtime inference speed, as the proposed four types of convolutional kernel attentions α si , α ci , α fi and α wi introduce a bit more learnable parameters.
Under the similar model size, the training cost of our ODConv is also heavier than reference methods (e.g., ODConv (4 × ) vs. DyConv (4 × )), according to the results shown in Table 17.
On the other side, although we provide a wide range of ablative studies to analyze the effect of different hyperparameters of our ODConv using the ResNet18 backbone with the ImageNet dataset, and apply the resulting combination of the hyperparameters to all backbone networks, it is not the optimal setting to different backbone networks.
This suggests the way to reduce the computational cost of our method, ﬁnding a proper combination to constrain the ﬁnal model to the target speed and accuracy for a particular circumstance.
Besides, the potential of applying ODConv to more deep and large backbones beyond ResNet101 has not been explored due to the constraint of our available computational resource.
[Figure]