D ILATED CONVOLUTION WITH LEARNABLE SPACINGS
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent works indicate that convolutional neural networks (CNN) need large receptive fields (RF) to compete with visual transformers and their attention mechanism.
In CNNs, RFs can simply be enlarged by increasing the convolution kernel sizes.
Yet the number of trainable parameters, which scales quadratically with the kernel’s size in the 2D case, rapidly becomes prohibitive, and the training is notoriously difficult.
This paper presents a new method to increase the RF size without increasing the number of parameters.
The dilated convolution (DC) has already been proposed for the same purpose.
DC can be seen as a convolution with a kernel that contains only a few non-zero elements placed on a regular grid.
Here we present a new version of the DC in which the spacings between the nonzero elements, or equivalently their positions, are no longer fixed but learnable via backpropagation thanks to an interpolation technique.
We call this method “Dilated Convolution with Learnable Spacings” (DCLS) and generalize it to the n-dimensional convolution case.
However, our main focus here will be on the 2D case for computer vision only.
We first tried our approach on ResNet50: we drop-in replaced the standard convolutions with DCLS ones, which increased the accuracy of ImageNet1k classification at iso-parameters, but at the expense of the throughput.
Next, we used the recent ConvNeXt state-of-the-art convolutional architecture and drop-in replaced the depthwise convolutions with DCLS ones.
This not only increased the accuracy of ImageNet1k classification but also of typical downstream and robustness tasks, again at iso-parameters but this time with negligible cost on throughput, as ConvNeXt uses separable convolutions.
Conversely, classic DC led to poor performance with both ResNet50 and ConvNeXt.
The code of the method is based on PyTorch and available.
I NTRODUCTION
The receptive field of a deep convolutional network is a crucial element to consider when dealing with recognition and downstream tasks in computer vision.
For instance, a logarithmic relationship between classification accuracy and receptive field size was observed in Araujo et al.
This tells us that large receptive fields are necessary for high-level vision tasks, but with logarithmically decreasing rewards and thus a higher computational cost to reach them.
Recent advances in vision transformers (Dosovitskiy et al., 2020) and in CNNs (Liu et al., 2022b;
Ding et al., 2022; Trockman & Kolter, 2022; Liu et al., 2022a) highlight the beneficial effect that a large convolution kernel can have, compared to the 3 × 3 kernels traditionally used in previous state-of-the-art CNN models (He et al., 2016).
However, when naively increasing the kernel size, the accuracy rapidly plateaus or even decreases.
For example, in ConvNeXt, the best accuracy was achieved by a 7 × 7 kernel (Liu et al., 2022b;a).
Using a structural re-parameterization trick, Ding et al.
(2022) demonstrated the benefit of increasing the kernel size up to 31 by 31.
Thereafter,
Liu et al.
(2022a) showed that there was still room for improvement by moving to 51 by 51, using the depthwise implicit matrix multiplication (gemm) method developed by Ding et al.
(2022) and
for which the implementation has been integrated into the open-sourced framework MegEngine (Megvii, 2020), in addition to a spatial separation of the depthwise kernel followed by an accumulation of the resulting activations.
Yet, all these improvements have a cost in terms of memory and computation and it does not seem possible to increase the size of the kernels indefinitely.
One of the first approaches that allow inflating the receptive field of a convolutional layer without increasing the number of learnable parameters nor the computational cost is called dilated convolution (DC).
DC or “atrous convolution” was first described in Holschneider et al.
and Shensa (1992), under the name “convolution with a dilated filter” before being referred to as “dilated convolution” in Yu & Koltun (2015).
The purpose of this approach is to inflate the convolutional kernel by regularly inserting spaces ( i.e. zeros) between the kernel elements, as depicted in Figure 2b.
The spacing between elements is thus constant, it is a hyper-parameter usually referred to as “dilation” or “dilation rate”.
Despite its early successes in classification since Yu et al.
(2017), and its even
[Figure]
more convincing results in semantic segmentation Sandler et al.
; Chen et al.
and object detection Lu et al.
(2019), DC has gradually fallen out of favor and has been confined to downstream tasks such as those described above.
Without much success, Ding et al.
(2022) tried to implement DC in their ReplKNet architecture.
Our own investigation on ResNet and ConvNeXt with standard dilated convolution (Section 4.2) will lead to a similar conclusion.
The failure of this method for classification tasks could be attributed to the great rigidity imposed by its regular grid as discussed in Wang & Ji (2018).
In this context, we propose DCLS (Dilated Convolution with Learnable Spacings), a new convolution method.
In DCLS, the positions of the non-zero elements within the convolutional kernels are learned in a gradient-based manner.
The inherent problem of non-differentiability due to the integer nature of the positions in the kernel is circumvented by interpolation (Fig. 2c).
DCLS is a differentiable method that only constructs the convolutional kernel.
To actually apply the method, we could either use the native convolution provided by PyTorch or a more advanced one such as the depthwise implicit gemm convolution method (Ding et al., 2022), using the constructed kernel.
DCLS comes in six sub-versions: 1D, 2D, 3D and what we call N-MD methods, namely: “2-1D, 3-1D and 3-2D” where a N-dimension kernel is used but positions are learned only along M dimension(s).
The main focus of this paper will be the 2D version for which we detail mathematical proofs, implementation specificities and results on image classification, downstream and robustness tasks.
Instead of having a grid of kernel elements like in standard and dilated convolutions, DCLS allows an arbitrary number of kernel elements (Fig. 2d).
We refer to this free tunable hyper-parameter as “kernel count”.
In this paper, we set it in order to be at iso or fewer parameters than the baselines we will compare ourselves to.
Conversely, we refer to the size of the kernel, or rather the maximum size in which the kernel elements are allowed to move inside the dilated kernel, as the “dilated kernel size”.
It is also a tunable hyper-parameter.
The positions of kernel elements in DCLS are randomly initialized, and are allowed to move throughout the learning process within the dilated kernel size limit.
We will then show how sharing positions across multiple blocks of a same convolution stage could further increase accuracy while reducing the number of learnable parameters.
This, together with other learning techniques, empirically and consistently improve the overall performance of the method.
They are summarized in
Section
[Figure]
K ERNEL CONSTRUCTION IN DCLS
Our method entails that we learn the float coordinates ( p 1 and p 2 in the 2D case for example) for every weight w of the dilated kernel (in addition to the actual weights themselves).
The positions or coordinates of the weights within the convolution kernel are conceptually integer parameters, yet in order to compute their derivatives, we should consider them as float parameters.
This problem of integer positions learning is smoothed by making use of a bilinear interpolation that will be described in equations 2, 3 and 4.
2. N OTATION AND PRELIMINARIES
We denote by ⌊ ⌋ the floor function and we define its derivative by the zero function.
[Equation]
We denote by m ∈ N ∗ the number of kernel elements inside the constructed kernel and we refer to it as the “kernel count”.
Moreover, we denote respectively by s 1 , s 2 ∈ N ∗ × N ∗ , the sizes of the constructed kernel along the x-axis and the y-axis.
The latter could be seen as the limits of the dilated kernel, and we refer to them as the “dilated kernel size”.
The s 1 × s 2 matrix space over R is defined as the set of all s 1 × s 2 matrices over R , and is denoted
M s 1 ,s 2 ( R ) .
The characters w , p 1 and p 2 respectively stand for the weight, the position of that weight along the x-axis (width) and its position along the y-axis (height) in the scalar case while the bold w = ( w i ) 1 ≤ i ≤ m , p 1 = ( p 1 i ) 1 ≤ i ≤ m and p 2 = ( p 2 i ) 1 ≤ i ≤ m respectively stand for the weight, the widthposition of that weight and its height-position in the vector case.
2. M ATHEMATICAL FORMULATION
The mathematical construction of the forward pass as well as the proofs for the derivations used in the backward pass are in Appendix 7.
This construction relies on bilinear interpolation and could be described by the following function:
[Equation]
With f defined as follows:
[Equation]
where ∀
i ∈ (cid:74) 1 .. s 1 (cid:75) , ∀ j ∈ (cid:74) 1 .. s 2 (cid:75) :
[Equation]
otherwise
[Equation]
4) and where the fractional parts are:
[Equation]
We invite the reader to look at Appendix 7 for more details on the mathematical proofs and derivatives that permits to calculate the gradients of a differentiable loss function with respect to the weights and their positions.
Those derivations lead to the DCLS kernel construction algorithm described for the 2D case in pseudo-code in Appendix 8.
Furthermore, the real code for the kernel construction in 1D, 2D, and 3D cases is included in Appendix 9.
This code is written in native
PyTorch language, with classical modules, and does not require any compilation or adaptation.
L EARNING TECHNIQUES
So far, we have seen how to implement the DCLS method.
We now turn to the techniques that allow us to get the most out of the method.
In what follows, we list
the training techniques that we have retained and for which we have noticed a consistent and empirical advantage on validation accuracy.
• Weight decay: weight decay is a regularization method widely used in a variety of deep learning models.
Though its beneficial effect on generalization, we noticed that when applied to the kernel positions in DCLS method, weight decay tends to ”artificially” over-concentrate the positions around the center of the kernel, resulting in poorer accuracy.
Therefore, we set this hyperparameter to 0 for the kernel positions and kept it unchanged for all the other parameters.
• Positions initialization: the DCLS positions tend to cluster around the center of the RF throughout the learning process (see Appendix 10).
In an attempt to facilitate learning, we chose an initial distribution close to the one obtained at the end of training, that is a centered normal law of standard deviation 0.5.
Yet in practice, the uniform law gives a similar performance.
• Positions clamping / overlapping: kernel elements that reach the dilated kernel size limit are clamped.
This is done at the end of every batch step to force kernel positions to stay within limits.
Agglutination around those limits can sometimes be observed and this indicates that the dilated kernel size is too small and should be enlarged.
Positions of different kernel weights (or their interpolations) could also overlap.
They are added together in such a case.
• Dilated kernel size tuning: we empirically tuned it using the remark above.
For simplicity, we used the same dilated kernel size in all the model layers (7 for ResNet-50-dcls and 17 for ConvNeXt-dcls; larger values did not bring any gain in accuracy).
Note that increasing the dilated kernel size has no cost on the number of trainable parameters, but has a cost on throughput, especially when using non-separable convolutions.
Besides, the convolution algorithm (which is the most time-consuming part of DCLS) that is used after the kernel construction (whether it is the native one or the depthwise implicit gemm one) does not leverage the kernel sparsity.
Thus, the kernel count does not impact the throughput; only the dilated kernel size does.
The
development of a convolution method that takes into account sparse kernels, such as Kundu et al.
(2019), could further help to reduce the throughput gap between DCLS convolution and standard dilated convolution.
• Kernel count tuning: as said before, we have set this hyper-parameter to the maximal integer value that allows us to be below the baselines to which we compare ourselves in terms of the number of trainable parameters.
Note that adding one element to the 2D-DCLS kernel leads to having three additional learnable parameters: the weight, its vertical and its horizontal position.
For simplicity, we used the same kernel count in all the model layers.
• Positions learning rate scaling: we found that kernel positions could benefit from a special scaling when it comes to the learning rate.
As their magnitude is different from regular weights, we scaled the learning rate of all the kernel positions by a factor of 5.
This is the best factor we found empirically.
Custom-made schedulers for positions have been tested, but scaling the learning rate while using the same scheduler as for the kernel weights remained the best choice.
Interestingly, we found that throughout learning, the average speed of the positions follows precisely the shape of the learning rate scheduler curve (see Appendix 11).
• Synchronizing positions: we shared the kernel positions across convolution layers with the same number of parameters (typically, those belonging to a same stage of the ConvNeXt/ResNet model), without sharing the weights.
Positions in this kind of stages were centralized in common parameters that accumulate the gradients.
This constraint has surprisingly enhanced the accuracy while reducing the number of extra parameters dedicated to positions (an ablation of this technique led to a 0.13% accuracy drop on ImageNet1k with ConvNeXt-T-dcls).
• Repulsive loss : following the work of Thomas et al.
on 3D cloud points, we implemented the repulsive loss for the DCLS kernel positions to discourage multiple elements from overlapping.
Despite a slight advantage with the ResNet-50-dcls model, this technique did not significantly improve the results with ConvNeXt-dcls.
• Depthwise implicit gemm : This method has been developed by
Ding et al.
and
integrated in Megvii (2020), it has for goal to modify the im2col algorithm as well as the matrix multiplication that follows, both used in the native 2D convolution method of PyTorch, by a very efficient algorithm which does not build explicitly the im2col tensor.
The advantage of this method is that it is much faster than the native one for large kernel sizes, without affecting the convolution speed for small kernel ones.
In our experiments, the largest dilated kernel size is 17, therefore this method is not absolutely necessary.
However, the DCLS user can choose to use it instead of the native method, which will improve the throughput.
R ESULTS AND DISCUSSION 4. S ETTINGS
We started with an exploratory study on ResNet-50, where we drop-in replaced all the 3 × 3 convolutions of the model by 2D-DCLS ones.
For that, we used a lightweight procedure named the “A3 configuration”, described in Wightman et al.
We then moved to the ConvNeXt models, where we limited our studies to its three first variants namely: the tiny, the small and the base models with input crops of size 224 × 224 (Liu et al., 2022b).
Here, we drop-in replaced all the depthwise convolutions of the model by DCLS ones.
We reconducted all the experiments, and evaluated the seed sensitivity for the ConvNeXt-dcls model by calculating the standard deviation of the top-1 accuracy on three different seeds, for the tiny variant.
We found the standard deviation to be ± 0 .
04 , which is compatible with what was found in Liu et al.
(2022b).
Given this reasonably low variability, the remaining experiments were done on one seed only.
Code and scripts to reproduce the training are available at 2 .
4. E MPIRICAL E VALUATIONS ON I MAGE N ET 1 K
In the following, we report the top-1 accuracies found on the ImageNet1k validation dataset (Deng et al., 2009), using ImageNet1k training dataset only.
Using ResNet-50.
Table 1 presents the results obtained for the ResNet-50 experiment using the
A3 configuration.
The aim of this study is not to exceed the state-of-the-art for this particular configuration, but rather to give a first experimental evidence of the relevance of the DCLS method with non-separable convolutions and with one of the most popular CNN architectures in computer vision.
We can observe that when using the standard dilated convolution, the results only get worse as we increase the dilation rate.
Moreover, increasing the kernel size ( 3 → 7 ) in the case of standard convolution increases the accuracy but at the expense of not only the throughput, which decreases, but also of the number of parameters, which triples.
With fewer parameters, the ResNet-50-dcls model surpasses the baseline but at the expense of the throughput.
This last disadvantage is due to the fact that ResNet-50 uses non-separable convolutions.
We will see in Table 2 that for the ConvNeXt model, this extra cost in throughput is minimized thanks to the depthwise separable convolution.
[Table]
The throughput was calculated at inference time, on image crops of size 224 × 224 using a single V100-32gb gpu.
When the model contains DCLS convolutions, we reported the kernel count and dilated kernel size.
Otherwise, the kernel size is reported and thus the kernel count is in fact the square of that parameter.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
38 M
[Table] [Table]
4 G
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
M
[Table] [Table]
6 G
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
4G
[Table] [Table]
[Table] [Table]
[Table]
4G
[Table] [Table]
[Table] [Table]
[Table]
4G
[Table] [Table]
[Table] [Table]
95 / 122 M 17 .
9 G 245 .
[Table] [Table]
[Table]
Using ConvNeXt.
We present numerically in Table 2 and graphically in Fig.
1, the results obtained for ConvNeXt using the settings for ImageNet1k training with input crops of size 224 × 224 , as described in Liu et al.
(2022b):
Table
1992) was used, and for all models in Table 2, we report the accuracy found with this technique.
We replaced ConvNeXt’s depthwise separable convolutions (of kernel size 7 × 7 ), by 2D-DCLS ones of dilated kernel size 17 × 17 and of kernel count equal to 34 for the tiny variant, and 40 for the small and base variants.
We used all the techniques previously described in Section 3 during training.
From Table 2, we highlight the fact that ConvNeXt with DCLS convolutions always surpasses the
ConvNeXt baseline in accuracy (gains ranging from 0.3 to 0.6) with the same number of parameters and only a little cost on throughput.
We believe that this gain in accuracy is remarkable, given that we only replaced the depthwise convolutional layers, which represent just about 1% of the total number of parameters and 2% of the total number of FLOPs in ConvNeXt models.
ConvNeXt model with a standard dilation of rate 2 performed poorly (see ConvNeXt-T-dil2).
SLaK model performs about as well as DCLS but with a higher cost on throughput and parameters count.
DCLS can be seen as a kernel reparametrization technique that reduces the number of trainable parameters, and thus regularizes large kernels.
For example, in the case of ConvNeXt-T-dcls, a 17 × 17 kernel ( 289 parameters) is parameterized by 34 triplets (x-position, y-position, weight), i.e.
102 parameters.
The kernels that could be represented by the DCLS reparametrization constitute a subset of all possible dense kernels.
In fact, by learning the suitable weights during training, a dense kernel could implement any DCLS one.
It may therefore be counter-intuitive that DCLS leads to higher accuracy than a dense 17 × 17 convolution layer (see ConvNeXt-T-ker17).
The problem with dense convolutional layers having large kernel sizes is that the number of trainable parameters is huge, which makes learning impractical.
Finally, we observe that after training, the DCLS position density is higher around the center of the RF (see Appendix 10), suggesting that the central region is the most important one (yet we experienced that reducing the dilated kernel size to values <degrades the accuracy, so the positions far from the center also matter).
Conversely, DC samples the whole RF uniformly, which is most likely sub-optimal, which could explain its poor performance (see ConvNeXt-T-dil2).
4. E MPIRICAL E VALUATION ON D OWNSTREAM AND R OBUSTNESS T ASKS
We now report the results found for semantic segmentation on the ADE20K dataset (Zhou et al., 2019) and for object detection on the COCO dataset (Lin et al., 2014) using ConvNeXt-dcls backbones.
Note that the depthwise implcit gemm algorithm was not used for those tasks as it led to throughput issues.
In addition, we present the results found for robustness tasks consisting of directly testing (without further tuning)
the previously obtained ConvNeXt-dcls backbones on the following robustness benchmarks: ImageNet-C/C/A/R/Sketch (Hendrycks & Dietterich, 2019;
Mintun et al., 2021; Hendrycks et al.
, 2021b;a; Wang et al., 2019).
Semantic segmentation on ADE20k.
The results obtained in semantic segmentation show an improvement in performance by the ConvNeXt-dcls tiny and base backbones with equal number of parameters and FLOPs (Table 3).
As in Liu et al.
(2021) and Bao et al.
(2021), we evaluated the mIoU with single scale testing and used the exact same configurations as in Liu et al.
(2022b).
[Table]
Object detection and segmentation on COCO.
All ConvNeXt-dcls backbones have shown a noticeable improvement in average accuracy on both the object detection and segmentation tasks on the COCO dataset, again at iso-parameters and iso-FLOPS (Table 4).
We only tested with Cascade
[Table]
Robustness Evaluation on ImageNet-C/C/A/R/Sketch.
ConvNeXt-dcls backbones show very good performances when it comes to robustness.
This is illustrated by the results obtained for the different benchmarks we have tried and for which we have reconducted the experiments.
All of them show a gain in classification accuracy with DCLS, except SK with the S model (Table 5)
[Table]
R ELATED WORK
One of the studies that motivated the DCLS method is that of the effective receptive field (ERF)
(Luo et al., 2016), which characterizes how much each input pixel in a receptive field can impact the output of a unit in the downstream convolutional layers of a neural network, leading to the notion of an effective receptive field.
Among the findings of this study, the most relevant ones for ours are that not all pixels in a receptive field contribute equally to an output response, the kernel center has a much larger impact, and that the effective receptive field size increases linearly with the square root of convolutional layers.
Given these findings, introducing a new degree of freedom by learning the positions of non-zero weights in dilated kernels might increase the expressive power of convolutional networks between and non available in Appendix
The work that relates the most to DCLS is that of deformable convolutions Dai et al.
(2017) where offsets from the regular dilated grid are optimized.
Deformable convolutions are used for several computer vision tasks such as object detection.
Even if both approaches share the use of a rather similar bilinear interpolation, DCLS remains very different from deformable convolution in several aspects: firstly, deformable convolutions require the application of a regular convolution to get the offsets (which are thus input-dependent) that are then passed to the actual deformable method.
Conversely, in DCLS method, a dilated kernel with learnable positions is constructed and then passed to the convolution operation, making the positions input-independent.
Secondly, deformable convolutions have been developed for plain convolutions, not for depthwise separable ones.
Finally, the number of extra learnable parameters in 2D deformable convolutions is the number of kernel elements in the preliminary convolution, precisely (channels in // groups, 2 ∗ K H ∗ K W , K H , K W ) with K H and K W being the kernel height and width, which establishes a strong dependence between the offsets and the input feature map.
Contrarily, in 2D-DCLS, the number of extra parameters dedicated to learning positions is simply twice the number of kernel weights (2, channels out, channels in // groups, K H , K W ).
We have not been able to compare our work to the deformable convolution v2 in ConvNeXt as the training becomes very slow (the throughput is divided by 4 for the ConvNeXt-tiny model).
In addition, we noticed that the training loss grew at the first epochs, which means that the method is not adapted as a drop-in replacement of the depthwise separable convolution in the ConvNeXt architecture.
DCLS is also similar to other input-independent kernel re-parameterization techniques, yet different from them.
For example in CKConv (Romero et al., 2021b) and FlexConv (Romero et al., 2021a), the kernel weights are not learned directly; what is learned is the continuous function that maps the positions to the weights.
In Jacobsen et al.
(2016), the kernel is modeled as a weighted sum of basis functions, which consist of centered Gaussian filters and their derivatives.
Pintea et al.
(2021) extended the approach by also learning the width of the Gaussians, which is equivalent to learning the optimal resolution.
Shelhamer et al.
(2019) proposed to factorize the kernel as the composition of a standard kernel with a structured gaussian one.
Finally, other methods such as(Worrall & Welling, 2019; Sosnovik et al., 2019; 2021a;b; Bekkers, 2019; Zhu et al., 2019), where the goal is to build a scale equivariant neural network, could be considered as similar to the approach.
One limitation of all these studies is that only small datasets were used (e.g., CIFAR), and whether they scale well to larger and more challenging datasets like ImageNet1k is unknown.
In addition, they cannot be used as a drop-in replacement for the depthwise separable convolution in ConvNeXt, at least in their current forms, so we could not benchmark them with DCLS.
C ONCLUSION
In this work, we proposed DCLS, a new dilated convolution method where the positions of non-zero kernel elements are made learnable via backpropagation.
The non-differentiability issue of learning discrete positions was circumvented by interpolation.
We demonstrated that DCLS often outperforms the standard and the dilated convolution.
We listed a number of techniques that improve the learning process of DCLS, in particular sharing the positions within stages was key.
We provided evidence that searching for optimal positions of weights within a dilated kernel can improve not only the accuracy in image classification, but also in downstream and robustness tasks, using existing CNN architectures, without increasing their number of parameters.
We reported a throughput overhead introduced by DCLS, but it remains marginal, provided that we use CNN architectures involving separable convolutions, which is the case for most modern CNNs, such as ConvNeXts.
Future work: So far we have shown that DCLS can be used to drop-in replace standard convolution layers in existing architectures.
We now would like to search for an architecture dedicated to DCLS, that would get the maximum benefit out of the method.
In addition, we will explore alternatives to bilinear interpolation, e.g. bicubic.
Recent CNN-related works such as RepLKNet, make use of large kernels using a re-parameterizing trick, we might explore this direction to further enhance
DCLS.
Finally, we wish to show that the method is also of interest in 1D and 3D use cases.
For 1D cases, one can think of audio waveform generation, where dilated 1D convolution filters with large receptive fields are key, as, for example, in the autoregressive WaveNet model (Oord et al., 2016), and in generative adversarial networks (Yamamoto et al., 2020; Greshler et al., 2021).
For 3D applications, video classification with architectures using 3D convolutions, e.g., X3D Feichtenhofer (2020), would be a good fit.
A PPENDIX : PROOFS AND DERIVATION FOR THE DCLS METHOD
In the following we show how to mathematically describe the DCLS kernel construction and how to explicitly calculate the gradients of the loss function with respect to weights and positions that are used in the backpropagation algorithm.
These gradients are useful to implement the DCLS method in a way that is compatible with the automatic differentiation of PyTorch.
7. N OTATION AND PRELIMINARIES
We denote by ⌊ ⌋ the floor function and we define its derivative by the zero function.
[Equation]
We denote by m ∈ N ∗ the number of kernel elements inside the constructed kernel and we refer to it as the “kernel count”.
Moreover, we denote respectively by s 1 , s 2 ∈ N ∗ × N ∗ , the sizes of the constructed kernel along the x-axis and the y-axis.
The latter could be seen as the limits of the dilated kernel, and we refer to them as the “dilated kernel size”.
The n × p matrix space over R is defined as the set of all n × p matrices over R , and is denoted
M n,p ( R ) .
The Frobenius inner product × F of two matrices A and B of M n,p ( R ) is defined by:
[Equation]
Where “tr” stands for the trace of the square matrix A T B .
The characters w , p 1 and p 2 respectively stand for the weight, the position of that weight along the x-axis (width) and its position along the y-axis (height) in the scalar case while the bold w = ( w i ) 1 ≤ i ≤ m , p 1 = ( p 1 i ) 1 ≤ i ≤ m and p 2 = ( p 2 i ) 1 ≤ i ≤ m respectively stand for the weight, the widthposition of that weight and its height-position in the vector case.
The proofs and algorithms that will be shown in the next subsections are made for the case of tensors with one input channel and one output channel.
Without loss of generality, those proofs and algorithms hold for the general case of 4D tensors and higher by considering and applying them channel-wise.
7. 2D-DCLS, SCALAR WEIGHT CASE
We begin by the case of a kernel containing only one element.
The function f that defines the kernel construction in the scalar weight case is as follows:
[Equation]
where ∀ i ∈ (cid:74) 1 .. s 1 (cid:75) , ∀ j ∈ (cid:74) 1 .. s 2 (cid:75) :
[Equation]
and where the fractional parts are:
[Equation]
The constructed kernel K is zero except for at most the 4 adjacent positions that represent the 2D interpolation of the single weight w .
Note that s 2 (cid:80)
[Equation]
We then define the scalar loss function as:
[Equation]
with g : M s 1 ,s 2 ( R ) → R a differentiable function that models the action of all the layers that will follow f in the model.
By applying the chain rule we obtain:
[Equation]
with
[Equation]
Let us put
[Equation]
and let us consider two column vectors x =
[Equation]
] T of R
[Equation]
1 and y =
[ yy· · · y s 2 ] T of R s 2 .
We have:
[Equation]
Since K is zero except for the 4 aforementioned positions
[Equation]
we have:
[Equation]
By deriving this expression with respect to w
[Equation]
p 1 and p 2 we obtain:
[Equation]
Because of the linearity of the differentiation in the three previous equations, we could write:
[Equation]
[Equation]
where G w , G p 1 , G p 2 , respectively stand for the s 1 by s 2 matrices described below and which have zeros everywhere except at the four positions of interpolation.
G w =
[Equation]
    · · ·  · · · . ..
[Equation]
r 1 )(1 − r
[Equation]
) r
[Equation] [Equation]
[Equation]
r 1 (1 − r
[Equation]
) r
[Equation]
r
[Equation]
[Equation] [Equation]
G p 1 =
⌊ p 2 ⌋ ⌊ p 2 ⌋ +     · · ·  · · · ...
− w (1 − r 2 ) − wr  ⌊ p 1 ⌋  w (1 − r 2 ) wr  ⌊ p 1 ⌋ + ...
G p 2 =
⌊ p 2 ⌋ ⌊ p 2 ⌋ +     · · ·  · · · ...
− w (1 − r 1 ) w (1 − r 1 )  ⌊ p 1 ⌋  − wrwr
⌊ p 1 ⌋ + ...
From the three equations (21), (22) and (23), we can identify
[Equation]
[Equation] [Equation]
Finally, we have:
[Equation]
In the next subsection, we will see how this result can be generalized to the vector case.
7. 2D-DCLS, GENERAL CASE
The general case is the one where the weights w =
[ ww· · · w m ] T and the positions p 1 = (cid:2) pp· · · p 1 m (cid:3) T , p 2 = (cid:2) pp· · · p 2 m (cid:3) T are stored in vectors, with the fractional parts r 1 = { p 1 } = p 1 − ⌊ p 1 ⌋ = (cid:2) rr· · · r 1 m (cid:3) T and r 2 = { p 2 } = p 2 − ⌊ p 2 ⌋ = (cid:2) rr· · · r 2 m (cid:3) T extended as well.
The function f defined in equation (7) is then extended to the function F defined as follows:
[Equation]
The constructed kernel K here is the result of a summation of the function f defined in (7) over the elements of weight and position vectors.
We then define the scalar loss function as in (10).
[Equation]
with g : M s 1 ,s 2 ( R ) → R a differentiable function that models the action of all the layers that will follow F in the model.
Let us put
[Equation]
As in (11), (12) and (13), by applying the chain rule we obtain:
[Equation]
with
[Equation]
Let us put this time
[Equation]
Using the definition of F
[Equation]
and by substituting the last equation in (36)
[Equation]
(37) and (38)
[Equation]
we have:
[Equation] [Equation]
And we know that:
[Equation]
which simplifies equations (41), (42) and (43) to ∀
i ∈
[Equation]
1 .. m (cid:75) :
[Equation]
We can notice that we are brought back to the scalar case, and we deduce from (30), (31) and (32) the gradients of the loss function with respect to weights and positions in the general case:
[Equation]
The results found for the general case are nothing but a component-wise application of the result obtained in the scalar case.
In addition, we show in Appendix 7.4, the extension to the 1D and 3D convolution cases.
7. 1D-DCLS, 3D-DCLS
We denote respectively by s 1 , s 2 , s 3 ∈ N ∗ × N ∗ × N ∗ , the sizes of the constructed kernel along the x-axis, y-axis and the z-axis.
Moreover, the n × p × q tensor space of third dimension is denoted R n × p × q .
The function f defined in (7) could be adapted in order to construct a suitable kernel for the 1D convolution in the scalar weight case as follows:
[Equation]
where ∀
i ∈ (cid:74) 1 .. s (cid:75) :
[Equation]
and where the fractional part is:
[Equation]
Following the same construction in Appendix 7.3 we can show that the gradients of the loss function with respect to weights and positions in the general 1D case are:
[Equation]
Furthermore, we define the function f 3 D , the suitable kernel construction function in the 3D convolution case, as such:
[Equation] [Equation]
where ∀
i ∈ (cid:74) 1 .. s 1 (cid:75) , ∀ j ∈ (cid:74) 1 .. s 2 (cid:75) , ∀ l ∈ (cid:74) 1 .. s 3 (cid:75) :
[Equation]
and where the fractional parts are:
[Equation]
We can show that the gradients of the loss function with respect to weights and positions in the general 3D case are:
[Equation] [Equation]
A PPENDIX : T HE 2D-DCLS KERNEL CONSTRUCTION ALGORITHM
In the following, we describe with pseudocode the forward and backward passes for kernel construction used in 2D-DCLS.
In practice, W , P 1 and P 2 are 3-D tensors of size ( channels out, channels in // groups, K count ), but the algorithms presented here are easily extended to this case by applying them channel-wise.
Algorithm
Input: W , P 1 , P 2 : vectors of dimension m
Output: K : the constructed kernel, of size ( s 1 × s 2 ) 1: K ←2: p 1 ← ⌊ P 1 ⌋ ; p 2 ← ⌊ P 2 ⌋ 3: R 1 ← P 1 − p 1 ;
R 2 ← P 2 − p4: save for backward ( p 1 , p 2 , R 1 , R 2 ) 5: for i =
→ m − 1 do 6:
K [ p 1 i , p 2 i ] += W i ∗ (1 − R 1 i ) ∗ (1 − R 2 i ) 7: K [ p 1 i + 1 , p 2 i ] += W i ∗ ( R 1 i ) ∗ (1 − R 2 i ) 8:
K [ p 1 i , p 2 i + 1] += W i ∗ (1 − R 1 i ) ∗
( R 2 i ) 9:
K [ p 1 i + 1 , p 2 i + 1] += W i ∗ ( R 1 i ) ∗ ( R 2 i ) 10: end for
Algorithm
Input: GradK = ∂Loss∂K : matrix of dimension ( s 1 × s 2 )
Output: ∂Loss∂W , ∂Loss∂P 1 , ∂Loss∂P 2 : vectors of dimension m 1: ∂Loss∂W ← 0 , ∂Loss∂P 1 ← 0 , ∂Loss∂P 2 ←2: p 1 , p 2 , R 1 , R 2 ← load saved ( ) 3: for i =
→ m − 1 do 4: ∂Loss ∂W
[ i ] += ∂Loss ∂K
[ p
[Equation]
, p
[Equation]
i ]
[Equation]
i )
[Equation] [Equation]
1 i
[Equation]
2 i
[Equation] [Equation]
R
i
[Equation]
R
[Equation]
5: ∂Loss
[Equation] [Equation]
] 7: end for
The ‘for’ loops in the algorithms are fully parallelized using GPU threads.
The 2D-DCLS convolution with kernel construction is then obtained by applying the classical 2D-convolution provided natively by PyTorch or any other method such as the depthwise implicit gemm convolution method
Ding et al.
(2022) using the constructed kernel.
When considering a concurrent execution of this pseudocode, the additions may result, in case of overlapping, in a replacement of the overlapped values instead of the desired accumulation.
This problem can be addressed by using atomic addition operations.
A PPENDIX : T HE DCLS KERNEL CONSTRUCTION ALGORITHM IN NATIVE P Y T ORCH
In the following, we describe with PyTorch code the DCLS construction module for 1D, 2D and 3D versions.
The DCLS convolution method is obtained by using the constructed kernel as a weight for the native torch.nn.
Conv { 1,2,3 } d, or another convolution method such as depthwise implicit gemm
Megvii (2020).
class ConstructKernel1d (Module): def __init__(self, out_channels, in_channels, groups, kernel_count, dilated_kernel_size): super().__init__() self.out_channels = out_channels
self.in_channels = in_channels self.groups
= groups self.dilated_kernel_size = dilated_kernel_size self.kernel_count =
kernel_count
I = torch.arange(0, dilated_kernel_size[0])
I = I.expand(out_channels, in_channels//groups, kernel_count,-1).permute(3,0,1,2) self.
I =
Parameter(I, requires_grad= False ) self.lim = torch.zeros(1)
self.lim[0]
= dilated_kernel_size[0]
self.lim = self.lim.expand(out_channels, in_channels//groups, kernel_count, -1).permute(3,0,1,2)
self.lim = Parameter(self.lim, requires_grad= False ) def forward(self, W, P):
P = P + self.lim //
Pr = P
P =
P.floor()
R =
(Pr - P).expand(self.dilated_kernel_size[0],-1,-1,-1,- class ConstructKernel2d (Module):
J = torch.arange(0, dilated_kernel_size[0]).expand(dilated_kernel_size[1],-1)
I=torch.arange(0, dilated_kernel_size[1]).expand(dilated_kernel_size[0],-1)
I=I.expand(out_channels, in_channels//groups,kernel_count,-1,-1).permute(3,4,0,1,2)
J=J.expand(out_channels, in_channels//groups,kernel_count,-1,-1).permute(4,3,0,1,2) self.
I =
Parameter(I, requires_grad= False ) self.
J = Parameter(J, requires_grad= False )
self.lim = torch.zeros(2) self.lim[0]
= dilated_kernel_size[0]
self.lim[1] = dilated_kernel_size[1];
self.lim = self.lim.expand(out_channels, in_channels//groups, kernel_count, -1).permute(3,0,1,2)
self.lim = Parameter(self.lim, requires_grad= False ) def forward(self, W, P):
P = P + self.lim //
Pr = P
P =
P.floor()
R =
(Pr - P).expand(self.dilated_kernel_size[0], self.dilated_kernel_size[1],-1,-1,-1,-1)
R1=R.select(2,0);P1=P.select(0,0)
R2=
R.select(2,1);P2=P.select(0,1)
R1R2=
R1*R cond1=(self.
I==P1)
cond2=(self.
J==P2) cond3=(self.
I==P
+1) cond4=(self.
J==P2+
) W1=torch.where(cond1*cond2,1.0,0.0) W2=torch.where(cond1*cond4,1.0,0.0) W3=torch.where(cond3*cond2,1.0,0.0) W4=torch.where(cond3*cond4,1.0,0.0) K=W1+R1R
(W1-W2-W3+W4)+R1*(W3 - W1)
+ R2*
(W
W1)
K = W * K
K = K.sum(4)
K = K.permute(2,3,0,1)
return K class ConstructKernel3d (Module): def __init__(self, out_channels, in_channels, groups, kernel_count, dilated_kernel_size): super().__init__() self.out_channels= out_channels self.in_channels = in_channels self.groups
= groups self.dilated_kernel_size = dilated_kernel_size self.kernel_count =
kernel_count
L = torch.arange(0, dilated_kernel_size[0]).expand(dilated_kernel_size[1], dilated_kernel_size[2],-1)
J = torch.arange(0, dilated_kernel_size[1]).expand(dilated_kernel_size[0], dilated_kernel_size[2],-1)
I=torch.arange(0, dilated_kernel_size[2]).expand(dilated_kernel_size[0], dilated_kernel_size[1],-1)
L = L.expand(out_channels, in_channels//groups,kernel_count,-1,-1,-1).permute(5,3,4,0,1,2)
I=I.expand(out_channels, in_channels//groups,kernel_count,-1,-1,-1).permute(3,4,5,0,1,2)
J=J.expand(out_channels, in_channels//groups,kernel_count,-1,-1,-1).permute(3,5,4,0,1,2) self.
L=
Parameter(L,requires_grad= False ) self.
I =
Parameter(I, requires_grad= False ) self.
J = Parameter(J, requires_grad= False )
self.lim = torch.zeros(3) self.lim[0]
= dilated_kernel_size[0]
self.lim[1] = dilated_kernel_size[1] self.lim[2]
= dilated_kernel_size[2]
self.lim = self.lim.expand(out_channels, in_channels//groups, kernel_count, -1).permute(3,0,1,2)
self.lim = Parameter(self.lim, requires_grad= False ) def forward(self, W, P):
P = P + self.lim //
Pr = P
P =
P.floor()
R =
(Pr - P).expand(self.dilated_kernel_size[0], self.dilated_kernel_size[1], self.dilated_kernel_size[2],-1,-1,-1,-1)
R1 = R.select(3,0); P1 = P.select(0,0)
R2 = R.select(3,1); P2 = P.select(0,1)
R3 = R.select(3,2); P3 = P.select(0,2) cond1 =
(self.
L == P1) cond2 =
(self.
I == P2) cond3 = (self.
J == P3) cond4 =
(self.
L == P1+1) cond5 =
(self.
I ==
P2+1) cond6 =
(self.
J == P3+1)
W1 =
torch.where(cond1*cond2*cond3, 1.0, 0.0)
W2 =
torch.where(cond4*cond2*cond3, 1.0, 0.0)
W3 =
torch.where(cond1*cond5*cond3, 1.0, 0.0)
W4 = torch.where(cond4*cond5*cond3, 1.0, 0.0)
W5 =
torch.where(cond1*cond2*cond6, 1.0, 0.0)
W6 = torch.where(cond4*cond2*cond6, 1.0, 0.0)
W7 = torch.where(cond1*cond5*cond6, 1.0, 0.0)
W8 = torch.where(cond4*cond5*cond6, 1.0, 0.0) # needs a better computing
K = W1 * (1 - R1)
* (1 - R2)
* (1 - R3)
K += W2 * R * (1 - R2)
* (1 - R3)
K += W3 * (1 - R1)
* R * (1 - R3)
K += W4 * R * R * (1 - R3)
K += W
(1 - R1)
* (1 - R2)
* R
K += W6 * R * (1 - R2)
* R
K += W
(1 - R1)
* R * R
K += W
R * R * R
K = W * K
K = K.sum(5)
K = K.permute(3,4,0,1,2) return K
A PPENDIX : H ISTOGRAMS OF POSITIONS
In the following, we show as histograms over training epochs, the distribution of the four kernel positions for the 2D-DCLS convolutions of the ConvNeXt-T-dcls model.
Note that there is no agglutination or edge effect around the kernel limits, and that the distributions are relatively stable, with a higher concentration around the center of the kernel.
Individual positions, however, are constantly moving; see animation at: https://wandb.ai/dcls/convnext/reports/Dcls-scatters-s0-b0-229-25-14-37-09---VmlldzoyNjkyMjkz?accessToken=e32u5xgd4a9bn46w0j d51kxvry2ciwedotq0im3haw1j1ue1a200jnjaf1jtbe .
A PPENDIX : S PEED CURVES AND LEARNING RATE SCHEDULE
Here, we plot the average speed curves of the four position tensors for the 2D-DCLS convolutions
V P of the ConvNeXt-T-dcls model as functions of the training epochs.
The general formula for the average speed of a DCLS position tensor of size ( cout, cin, m ) ∈ N ∗ 3 , at epoch t , is as follows:
[Equation]
V P ( t ) = cout · cin · m cout (cid:88) k
= cin (cid:88)
j = m (cid:88) i =
| P tijk − P t − 1 ijk |
At epoch 0, we can notice that the average speed is abnormally high, this is due to the fact that in the beginning, the positions are initialized randomly and we arbitrarily considered that V P (0) = 0 , thus the large speed gap at initialization.
Another peak can be seen at epoch 20, this one is due to the introduction of the repulsive loss (Thomas et al., 2019) at this precise epoch during training.
This last causes a momentary increase in the average speed of the DCLS positions.
In general, we can say that over epochs, the average DCLS positions follow the shape of the scheduler used in training.
A PPENDIX : E FFECTIVE RECEPTIVE FIELDS COMPARISON
In the following, we show the effective receptive fields (ERF) calculated respectively for ConvNeXtT-dcls, ConvNeXt-T with a standard dilated kernel of rate 2 and ConvNeXt-T models.
The input crops used here are of size 1024 × 1024 and the heatmaps are normalized (between 0 and 1). These
ERFs are obtained by finding (via backpropagation) the input pixels that most impact the activations of a pretrained model using samples of the dataset.
We observe that the ERF of ConvNeXt-T-dcls has a particular shape that resembles a square with more prominent diagonals and medians.
The
ERF of ConvNeXt-T with a standard dilated kernel is larger but with gridding artifacts.
In all plots, it seems that the center has more importance.
A PPENDIX : T IME MEASUREMENTS
We notice that when using the depthwise implicit gemm algorithm, DCLS forward is slightly slower than the PyTorch native Conv2d forward pass, but the DCLS backward pass is faster, in particular when increasing the model size.
forward and backward
We notice that when using the depthwise implicit gemm algorithm, the DCLS construction algorithm is negligible in time compared to the convolution method for large input map sizes.
For small map sizes (example 7x7), the DCLS construction algorithm takes as much time as the convolution.
When not using the depthwise implicit gemm algorithm but the PyTorch native Conv2d in DCLS, the construction algorithm time is significantly lower than the convolution time even for small input map sizes.
This is due to the fact the convolution time with the PyTorch native Conv2d is significantly higher than the one with the depthwise implicit gemm algorithm.