F LEX C ONV : C ONTINUOUS K ERNEL C ONVOLUTIONS WITH D IFFERENTIABLE K ERNEL S IZES
A BSTRACT
When designing Convolutional Neural Networks (CNNs), one must select the size of the convolutional kernels before training.
Recent works show CNNs beneﬁt from different kernel sizes at different layers, but exploring all possible combinations is unfeasible in practice
A more efﬁcient approach is to learn the kernel size during training
However , existing works that learn the kernel size have a limited bandwidth.
These approaches scale kernels by dilation and thus the detail they can describe is limited.
In this work we propose FlexConv a novel convolutional operation with which high bandwidth convolutional kernels of learnable kernel size can be learned at a ﬁxed parameter cost.
FlexNets model long-term dependencies without the use of pooling, achieve state-of-the-art performance on several sequential datasets, outperform recent works with learned kernel sizes and are competitive with much deeper ResNets on image benchmark datasets.
Additionally,
FlexNets can be deployed at higher resolutions than those seen during training
To avoid aliasing, we propose a novel kernel parameterization with which the frequency of the kernels can be analytically controlled.
Our novel kernel parameterization shows higher descriptive power and faster convergence speed than existing parameterizations.
This leads to important improvements in classiﬁcation accuracy.
I NTRODUCTION
The kernel size of a convolutional layer deﬁnes the region from which features are computed, and is a crucial choice in their design.
Commonly, small kernels (up to 7px) are used almost exclusively and are combined with pooling to model long term dependencies (Simonyan & Zisserman, 2014;
Szegedy et al., 2015;
He et al., 2016; Tan & Le, 2019).
Recent works indicate, however, that CNNs beneﬁt from using convolutional kernels ( i ) of varying size at different layers (Pintea et al., 2021;
Tomen et al., 2021), and ( ii ) at the same resolution of the data (Peng et al., 2017; Cordonnier et al., 2019; Romero et al., 2021).
Unfortunately, most CNNs represent convolutional kernels as tensors of discrete weights and their size must be ﬁxed prior to training.
This makes exploring different kernel sizes at different layers difﬁcult and time-consuming due to ( i ) the large search space, and ( ii ) the large number of weights required to construct large kernels.
A more efﬁcient way to tune different kernel sizes at different layers is to learn them during training.
Existing methods deﬁne a discrete weighted set of basis functions, e.g., shifted Delta-Diracs (Fig. 2b,
Dai et al.
or Gaussian functions (Fig.
2c, Jacobsen et al.
; Shelhamer et al.
Pintea et al.
During training they learn dilation factors over the basis functions to increase the kernel size, which crucially limits the bandwidth of the resulting kernels.
In this work, we present the Flexible Size Continuous Kernel Convolution (FlexConv), a convolutional layer able to learn high bandwidth convolutional kernels of varying size during training (Fig. 1).
Instead of using discrete weights, we provide a continuous parameterization of convolutional kernels via a small neural network (Romero et al., 2021).
This parameterization allows us to model continuous functions of arbitrary size with a ﬁxed number of parameters.
By multiplying the response of the neural network with a Gaussian mask, the size of the kernel can be learned during training (Fig. 2a).
This allows us to produce detailed kernels of small sizes (Fig.
FlexConvs can be deployed at higher resolutions than those observed during training, simply by using a more densely sampled grid of kernel indices.
However, the high bandwidth of the kernel can lead
FlexConv to learn kernels that show aliasing at higher resolutions, if the kernel bandwidth exceeds the Nyquist frequency.
To solve this problem, we propose to parameterize convolutional kernels as
Multiplicative Anisotropic Gabor Networks (MAGNets). MAGNets are a new class of Multiplicative
Filter Networks (Fathony et al., 2021) that allows us to analyze and control the frequency spectrum of the generated kernels.
We use this analysis to regularize FlexConv against aliasing.
With this regularization, FlexConvs can be directly deployed at higher resolutions with minimal accuracy loss.
Furthermore, MAGNets provide higher descriptive power and faster convergence speed than existing continuous kernel parameterizations (Schütt et al., 2017; Finzi et al., 2020; Romero et al., 2021).
This leads to important improvements in classiﬁcation accuracy (Sec. 4).
Our experiments show that CNNs with FlexConvs, coined FlexNets , achieve state-of-the-art across several sequential datasets, match performance of recent works with learnable kernel sizes with less compute, and are competitive with much deeper ResNets (He et al., 2016) when applied on image benchmark datasets.
Thanks to the ability of FlexConvs to generalize across resolutions, FlexNets can be efﬁciently trained at low-resolution to save compute, e.g., 16 × 16 CIFAR images, and be deployed on the original data resolution with marginal accuracy loss, e.g., 32 × 32 CIFAR images.
In summary, our contributions are:
• We introduce the Flexible Size Continuous Kernel Convolution (FlexConv), a convolution operation able to learn high bandwidth convolutional kernels of varying size end-to-end.
• Our proposed Multiplicative Anisotropic Gabor Networks (MAGNets) allow for analytic control of the properties of the generated kernels.
This property allows us to construct analytic alias-free convolutional kernels that generalize to higher resolutions, and to train FlexNets at low resolution and deploy them at higher resolutions.
Moreover, MAGNets show higher descriptive power and faster convergence speed than existing kernel parameterizations.
• CNN architectures with FlexConvs (FlexNets) obtain state-of-the-art across several sequential datasets, and match recent works with learnable kernel size on CIFAR-10 with less compute.
R ELATED W ORK
Adaptive kernel sizes.
Loog & Lauze (2017) regularize the scale of convolutional kernels for ﬁlter learning.
For image classiﬁcation, adaptive kernel sizes have been proposed via learnable pixel-wise offsets (Dai et al., 2017), learnable padding operations (Han et al., 2018), learnable dilated Gaussian functions (Shelhamer et al., 2019;
Xiong et al., 2020; Tabernik et al., 2020; Nguyen, 2020) and scalable Gaussian derivative ﬁlters (Pintea et al., 2021; Tomen et al., 2021; Lindeberg, 2021).
These approaches either dilate discrete kernels (Fig. 2b), or use discrete weights on dilated basis functions (Fig. 2c).
Using dilation crucially limits the bandwidth of the resulting kernels.
In contrast, FlexConvs are able to construct high bandwidth convolutional kernels of varying size with a ﬁxed parameter count.
Larger kernels are obtained simply by passing more positions to the kernel network (Fig. 1).
Continuous kernel convolutions.
Discrete convolutional kernel parameterizations assign an independent weight to each speciﬁc position in the kernel.
Continuous convolutional kernels, on the [Figure] [Figure] other hand view convolutional kernels as continuous functions parameterized via a small neural network MLP ψ ∶ R D
→ R N out × N in , with D the data dimensionality.
This deﬁnes a convolutional kernel for which arbitrary input positions can be queried.
Continuous kernels have primarily been used to handle irregularly-sampled data locally , e g., molecular data (Simonovsky & Komodakis, 2017;
Schütt et al., 2017) and point-clouds (Thomas et al., 2018;
Wang et al., 2018; Shi et al., 2019).
Recently, Romero et al.
(2021) introduced the Continuous Kernel Convolution (CKConv) as a tool to model long-term dependencies.
CKConv uses a continuous kernel parameterization to construct convolutional kernels as big as the input signal with a constant parameter cost.
Contrarily, FlexConvs jointly learn the convolutional kernel as well as its size.
This leads to important advantages in terms of expressivity (Fig. 3), convergence speed and compute costs of the operation.
Implicit neural representations.
Parameterizing a convolutional kernel via a neural network can be seen as learning an implicit neural representation of the underlying convolutional kernel (Romero et al., 2021).
Implicit neural representations construct continuous data representations by encoding data in the weights of a neural network (Park et al., 2019; Sitzmann et al., 2020; Fathony et al., 2021).
We replace the SIREN (Sitzmann et al., 2020) kernel parameterization used in Romero et al.
(2021) by our Multiplicative Anisotropic Gabor Networks : a new class of Multiplicative Filter Networks (Fathony et al., 2021).
MFNs allow for analytic control of the resulting representations, and allow us to construct analytic alias-free convolutional kernels.
The higher expressivity and convergence speed of MAGNets lead to accuracy improvements in CNNs using them as kernel parameterization.
M ETHOD
In this section, we introduce our approach.
First, we introduce FlexConv and the Gaussian mask.
Next, we introduce our Multiplicative Anisotropic Gabor Networks (MAGNets) and provide a description of our regularization technique used to control the spectral components of the generated kernel.
3. F LEXIBLE S IZE C ONTINUOUS K ERNEL C ONVOLUTION (F LEX C ONV )
To learn the kernel size during training, FlexConvs deﬁne their convolutional kernels ψ as the product of the output of a neural network MLP ψ with a Gaussian mask of local support.
The neural network
MLP ψ parameterizes the kernel, and the Gaussian mask parameterizes its size (Fig. 1).
Anisotropic Gaussian mask.
Let G ( x ; µ X ,σ 2X )∶= exp {− 12 σ − 2X ( x − µ X ) 2 } be a Gaussian function parameterized by a mean-variance tuple ( µ X ,σ 2X ) .
The anisotropic Gaussian mask is deﬁned as:
[Equation]
By learning ( µ X ,σ 2X ) and ( µ Y ,σ 2Y ) independently, anisotropic non-centered windows can be learned.
3. M ULTIPLICATIVE A NISOTROPIC G ABOR N ETWORKS (MAGN ETS )
In this section, we formalize our proposed parameterization for the kernel MLP ψ .
We start by introducing Multiplicative Filter Networks (Fathony et al., 2021), and present our MAGNets next.
Multiplicative Filter Networks (MFNs).
Recently, Fathony et al.
(2021) proposed to construct implicit neural representations as the linear combination of exponentially many basis functions g :
[Equation]
where { θ ( l ) , W ( l ) , b ( l ) } depict the learnable parameters of the bases and the afﬁne transformations, and N , N hid depict the number of output and hidden channels, respectively.
Depending on the selection of g , MFNs obtain approximations comparable to those of SIRENs (Sitzmann et al., 2020) with faster convergence rate.
The most successful instantiation of MNFs are the Multiplicative Gabor
Network (MGN): MFNs constructed with isotropic Gabor functions as basis g (in Eq. 2):
[Equation]
Note that, by setting N = N out × N in , an MFN can parameterize a convolutional kernel with N in input and
N out output channels.
Fathony et al.
(2021) show that MFNs are equivalent to a linear combination of exponentially many basis functions g .
This allows us to analytically derive properties of MFN representations, and plays a crucial role in the derivation of alias-free MAGNets (Sec. 3.3).
Multiplicative Anisotropic Gabor Networks (MAGNets).
Our MAGNet formulation is based on the observation that isotropic Gabor functions, i.e., with equal γ for the horizontal and vertical directions, are undesirable as basis for the construction of MFNs.
Whenever a frequency is required along a certain direction, an isotropic Gabor function automatically introduces that frequency in both directions.
As a result, other bases must counteract this frequency in the direction where the frequency is not required, and thus the capacity of the MFN is not used optimally (Daugman, 1988).
Following the original formulation of the 2D Gabor functions (Daugman, 1988), we alleviate this limitation by using anisotropic Gabor functions instead:
[Equation]
The resulting Multiplicative Anisotropic Gabor Network (MAGNet) obtains better control upon frequency components introduced to the approximation, and demonstrates important improvements in terms of descriptive power and convergence speed (Sec. 4).
MAGNet initialization.
Fathony et al.
(2021) proposes to initialize MGNs by drawing the size of the Gaussian envelopes, i.e., the γ ( l ) term, from a Gamma ( α ⋅ L − 1 ,β ) distribution at every layer l ∈ [ 1 ,.., L − 1 ] .
We observe however that this initialization does not provide much variability on the initial extension of the Gaussian envelopes and in fact, most of them cover a large portion of the space at initialization.
To stimulate diversity, we initialize the { γ ( l ) X , γ ( l ) Y } terms by a Gamma ( αl − 1 ,β ) distribution at the l -th layer.
We observe that our proposed initialization consistently leads to better accuracy than the initialization of Fathony et al.
(2021) across all tasks considered.
(Sec. 4).
3. A NALYTIC A LIAS - FREE MAGN ETS
FlexConvs can be deployed at higher resolutions than those observed during training, simply by sampling the underlying continuous representation of the kernel more densely, and accounting for the change in sampling rate.
Consider a D -dimensional input signal f r ( 1 ) with resolution r ( 1 ) .
FlexConv learns a kernel ψ r ( 1 ) that can be inferred at a higher resolution r ( 2 ) (Romero et al., 2021):
[Equation]
Note however, that Eq.
9 holds approximately .
This is due to aliasing artifacts which can appear if the frequencies in the learned kernel surpass the Nyquist criterion of the target resolution.
Consequently, an anti-aliased parameterization is vital to construct kernels that generalize well to high resolutions.
Towards alias-free implicit neural representations.
We observe that SIRENs as well as unconstrained MFNs and MAGNets exhibit aliasing when deployed on resolutions higher than the training resolution, which hurts performance of the model.
An example kernel with aliasing is shown in Fig.
To combat aliasing, we would like to control the representation learned by MAGNets.
MAGNets –and
MFNs in general– construct implicit neural representations that can be seen as a linear combination of basis functions .
This property allows us to analytically derive and study the properties of the resulting neural representation.
Here, we use this property to derive the maximum frequency of
MAGNet-generated kernels, so as to regularize MAGNets against aliasing during training.
We analytically derive the maximum frequency of a MAGNet, and penalize it whenever it exceeds the
Nyquist frequency of the training resolution.
We note that analytic derivations are difﬁcult for other implicit neural representations, e.g., SIRENs, due to stacked layer-wise nonlinearities.
Maximum frequency of MAGNets.
The maximum frequency component of a MAGNet is given by:
[Equation]
where L corresponds to the number of layers, W ( l ) g , γ ( l ) X , γ ( l ) Y to the MAGNet parameters as deﬁned in Eq. 8, and σ cut = 2 ⋅ stdev to the cut-off frequency of the Gaussian envelopes in the Gabor ﬁlters.
A formal treatment as well as the derivations can be found in Appx.
A.1.
Effect of the FlexConv mask.
The Gaussian mask used to localize the response of the MAGNet also has an effect on the frequency spectrum.
Hence, the maximum frequency of a FlexConv kernel is:
[Equation]
Here, σ X ,σ Y correspond to the mask parameters (Eq. 1).
Intuitively, multiplication with the mask blurs in the frequency domain, as it is equivalent to convolution with the Fourier transform of the mask.
Aliasing regularization of FlexConv kernels.
With the analytic derivation of f + FlexConv we penalize the generated kernels to have frequencies smaller or equal to their Nyquist frequency
f Nyq ( k ) via:
[Equation]
Here, k depicts the size of the FlexConv kernel before applying the Gaussian mask, and is equal to the size of the input signal.
In practice, we implement Eq.
by regularizing the individual MAGNet layers, as is detailed in Appx.
A.2.
To verify our method, Fig.
8 (Appx. A.1) shows that the frequency components of FlexNet kernels are properly regularized for aliasing.
E XPERIMENTS
We evaluate FlexConv across classiﬁcation tasks on sequential and image benchmark datasets, and validate the ability of MAGNets to approximate complex functions.
A complete description of the datasets used is given in Appx.
B. Appx.
D.2 reports the parameters used in all our experiments.
4. W HAT KIND OF FUNCTIONS CAN MAGN ETS APPROXIMATE
Bandwidth of methods with learnable sizes.
First, we compare the bandwidth of MAGNet against
N-Jet (Pintea et al., 2021)
by optimizing each to ﬁt simple targets: (i) Gabor ﬁlters of known frequency, (ii) random noise and (iii) an a 11 × 11 AlexNet kernel from the ﬁrst layer (Krizhevsky et al., 2012).
Fig.
4 shows that, even with 9 orders of Gaussian derivatives, N-Jets cannot ﬁt high frequency signals in large kernels.
Crucially, N-Jet models require many Gaussian derivative orders to model high frequency signals in large kernels: a hyperparameter which proportionally increases their inference
[Figure]
MAGNet-S: a small MAGNet of size akin to N-Jets, still does well on
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
time and parameter count.
MAGNets, on the other hand, accurately model large high frequency signals.
This allows FlexNets to learn large kernels with high frequency components.
Expressivity of MLP parameterizations.
Next [Table] we compare the descriptive power and convergence speed of MAGNets, Gabor MFNs, Fourier MFNs and SIRENs for image approximation.
To this end, we ﬁt the images in the Kodak dataset (Kodak, 1991) with each of these methods.
Our results (Tab. 5) show that MAGNets outperform all other methods, and converge faster to good approximations.
4. C LASSIFICATION T ASKS
Network speciﬁcations.
Here, we specify our networks for all our classiﬁcation experiments.
We parameterize all our convolutional kernels as the superposition of a 3-layer MAGNet and a learnable anisotropic Gaussian mask.
We construct two network instances for sequential and image datasets respectively: FlexTCNs and FlexNets.
Both are constructed by taking the structure of a baseline network –TCN (Bai et al., 2018a) or CIFARResNet (He et al., 2016)–, removing all internal pooling layers, and replacing convolutional kernels by FlexConvs.
The FlexNet architecture is shown in
Fig.
10 and varies only in the number of channels and blocks, e.g., FlexNet-16 has 7 blocks.
Akin to
Romero et al.
we utilize the Fourier theorem to speed up convolutions with large kernels.
Mask initialization.
We initialize the FlexConv masks to be small.
Preliminary experiments show this leads to better performance, faster execution, and faster training convergence.
For sequences, the mask center is initialized at the last kernel position to prioritize the last information seen.
Time series and sequential data.
First we evaluate FlexTCNs on sequential classiﬁcation datasets, for which long-term dependencies play an important role.
We validate our approach on intrinsic
[Table] [Table]
(Le et al., 2015)
(Chang et al., 2017) [-noise-padded CIFAR
-]noise- CIFAR10 (Chang et al., 2019), well as series
(CT (Bagnall et al., 2018)
Warden,
Our results are summarized in Tables 1 and 2.
FlexTCNs with two residual blocks obtain state-ofthe-art results on all tasks considered.
In addition, depth further improves performance.
FlexTCNimproves the current state-of-the-art on sCIFAR10 and npCIFAR10 by more than 6%.
On the difﬁcult SC_raw dataset –with sequences of length 16000–, FlexTCN-6 outperform the previous state-of-the-art by 20.07%: a remarkable improvement.
Furthermore, we conduct ablation studies by changing the parameterization of MLP ψ , and switching off the learnable kernel size ("CKTCNs") and considering global kernel sizes instead.
CKTCNs and
FlexTCNs with MAGNet kernels outperform corresponding models with all other kernel parameterizations: SIRENs (Sitzmann et al., 2020), MGNs and MFNs (Fathony et al., 2021).
Moreover, we see a consistent improvement with respect to CKCNNs (Romero et al., 2021) by using learnable kernel sizes.
This shows that both MAGNets and learnable kernel sizes contribute to the performance of
FlexTCNs.
Note that in 1D, MAGNets are equivalent to MGNs.
However, MAGNets consistently perform better than MGNs.
This improvement in accuracy is a result of our MAGNet initialization.
Image classiﬁcation.
Next, we evaluate FlexNets for image classiﬁcation on CIFAR-10 (Krizhevsky et al., 2009).
Additional experiments on Imagenet-32, MNIST and STL-10 can be found in Appx.
C.
Table 3 shows our results on CIFAR-10.
FlexNets are competitive with pooling-based methods such as CIFARResNet (He et al., 2016) and outperform learnable kernel size method DCNs (Tomen et al., 2021).
In addition, we compare using N-Jet layers of order three (as in Pintea et al.
(2021)) in
FlexNets against using MAGNet kernels.
We observe that N-Jet layers lead to worse performance, and are signiﬁcantly slower than FlexConv layers with MAGNet kernels.
The low accuracy of N-Jet layers is likely to be linked to the fact that FlexNets do not use pooling.
Consequently, N-Jets are forced to learn large kernels with high-frequencies, which we show N-Jets struggle learning in Sec. 4.1.
To illustrate the effect of learning kernel sizes, we also compare FlexNets against FlexNets with large and small discrete convolutional kernels (Tab. 3).
Using small kernel sizes is parameter efﬁcient, but is not competitive with FlexNets.
Large discrete kernels on the other hand require a copious amount of parameters and lead to signiﬁcantly worse performance.
These results indicate that the best solution is somewhere in the middle and varying kernel sizes can learn the optimal kernel size for the task at hand.
Similar to the sequential case, we conduct ablation studies on image data with learnable, nonlearnable kernel sizes and different kernel parameterizations.
Table 3 shows that FlexNets outperform
CKCNNs with corresponding kernel parameterizations.
In addition, a clear difference in performance is apparent for MAGNets with respect to other parameterizations.
These results corroborate that both
MAGNets and FlexConvs contribute to the performance of FlexNets.
Moreover, Tab. 3 illustrates the effect of the two contributions of MAGNet over MGN: anisotropic Gabor ﬁlters, and our improved initialization.
Our results in image data are in unison with our previous results for sequential data (Tabs. 1, 2) and illustrate the value of the proposed improvements in MAGNets. [Figure] f [Figure]
F l e x C
[Figure]
4. A LIAS - FREE F LEX N ETS
Regularizing the FlexConv mask.
Though including f + w gauss in the frequency analysis of MAGNets is crucial for the accuracy of the derivation, including the FlexConv mask in aliasing regularization is undesirable, as it steers the model to learn large kernels in order to minimize the loss (see Eq. 25).
However, excluding the mask from regularization could compromise the ability of FlexNet to generalize to higher resolutions.
Here, we experiment with this trade-off.
[Table]
Figure 5 shows accuracy change between ten source and target resolution combinations on
CIFAR-10, both for including and excluding the FlexConv mask in the aliasing regularization.
We train at the source resolution forepochs, before testing the model at the target resolution with the upsampling described in
Sec. 3.3.
Next, we adjust f Nyq ( k ) to the target resolution
[Table]
and ﬁnetune each model for 100 epochs at the target resolution.
We ﬁnd that regularizing just f + MAGNet yields a trade-off.
It increases the accuracy difference between low and high resolution inference, but also increases the ﬁne-tune accuracy at the target resolution.
We therefore choose to, by default, regularize f + MAGNet only.
Results of our alias-free FlexNet training on CIFAR-10 are in Table 4.
We observe that the performance of a FlexNet trained without aliasing regularization largely breaks down when the dataset is upscaled.
However, with our aliasing regularization most of the performance is retained.
Comparatively, FlexNet retains more of the source resolution performance than FlexNets with N-Jet layers, while baselines degrade drastically at the target resolution.
Fig.
8 shows the effect of aliasing regularization on the frequency components of FlexConv.
Training at lower resolutions saves compute.
We can train alias-free FlexNets at lower resolutions.
To verify that this saves compute, we time the ﬁrst 32 batches of training a FlexNet-7 on CIFAR-10.
We compare against training on 16 × 16 images (downsampled before training).
On 16x16 images, each batch takes 179ms ( ± 7ms).
On 32x32 images, each batch takes 222ms ( ± 9ms).
Therefore, we save 24% training time when training FlexNets alias-free at half the native CIFAR-10 resolution.
D ISCUSSION
Learned kernel sizes match conventional priors.
Commonly, CNNs use architectures of small kernels and pooling layers.
This allows convolutions to build a progressively growing receptive ﬁeld.
With learnable kernel sizes, FlexNet could learn a different prior over receptive ﬁelds, e.g., large kernels ﬁrst, and small kernels next.
However, FlexNets learn to increase kernel sizes progressively (Fig. 6), and match the network design that has been popular since AlexNet (Krizhevsky et al., 2012).
Mask initialization as a prior for feature importance.
The initial values of the FlexConv mask can be used to prioritize information at particular input regions.
For instance, initializing the center of mask on the ﬁrst element of sequential FlexConvs can be used to prioritize information from the far past.
This prior is advantageous for tasks such as npCIFAR10.
We observe that using this prior on npCIFAR10 leads to much faster convergence and better results (68.33% acc.
w/ FlexTCN-2).
MAGNet regularization as prior induction.
MAGNets allow for analytic control of the properties of the resulting representations.
We use this property to generate alias-free kernels.
However, other desiderata could be induced, e.g., smoothness, for the construction of implicit neural representations.
Beneﬁts of cropping and the inﬂuence of PyTorch .
Dynamic cropping adjust the computational cost of the convolutions on the ﬂy.
For a signal of size M and a cropped kernel size k , this incurs in savings from O ( M 2 D ) to O ( M D k D ) relative to using global kernel sizes ( O ( M 4 ) to O
( M 2 k 2 ) in 2D).
We test this theoretical speed up in a controlled environment for the Speech Commands and CIFAR10 datasets.
Cropping reduces the per-epoch run time by a factor of 11.8 x and 5.5 x for Speech
Commands and CIFAR-10, respectively.
Interestingly, however, both run times become similar if the ﬂag torch.backends.cudnn.benchmark is activated, with global kernel sizes being sometimes faster.
This is because this ﬂag tells PyTorch to optimize the convolution algorithms used under the hood, and some of these CUDA algorithms seem to be faster than our masking strategy on Python .
L IMITATIONS
Dynamic kernel sizes: computation and memory cost of convolutions with large kernels.
Performing convolutions with large convolutional kernels is a compute-intensive operation.
FlexConvs are initialized with small kernel sizes and their inference cost is relatively small at the start of training.
However, despite the cropping operations used to improve computational efﬁciency (Figs. 1, 3, Tab. 3), the inference time may increase to up to double as the learned masks increase in size.
At the cost of more memory, convolutions can be sped up by performing them in the frequency domain.
However, we observe that this does not bring gains for the image data considered because FFT convolutions are faster very convolutional kernels pixels).
Remaining accuracy drop in alias-free FlexNets.
Some drop in accuracy is still observed when using alias-free FlexNets at a higher test resolutions (Tab. 4).
Although more evidence is needed, this may be caused by aliasing effects introduced by ReLU (Vasconcelos et al., 2021), or changes in the activation statistics of the feature maps passed to global average pooling (Touvron et al., 2019).
C ONCLUSION
We propose FlexConv, a convolutional operation able to learn high bandwidth convolutional kernels of varying size during training at a ﬁxed parameter cost.
We demonstrate that FlexConvs are able to model long-term dependencies without the need of pooling, and shallow pooling-free FlexNets achieve state-of-the-art performance on several sequential datasets, match performance of recent works with learned kernel sizes with less compute, and are competitive with much deeper ResNets on image benchmark datasets.
In addition, we show that our alias-free convolutional kernels allow
FlexNets to be deployed at higher resolutions than seen during training with minimal precision loss.
Future work.
MAGNets give control over the bandwidth of the kernel.
We anticipate that this control has more uses, such as ﬁghting sub-sampling aliasing (Zhang, 2019; Kayhan & Gemert, 2020; Karras et al., 2021).
With the ability to upscale FlexNets to different input image sizes comes the possibility of transfer learning representations between previously incompatible datasets, such as CIFAR-10 and
Imagenet.
In a similar vein, the automatic adaptation of FlexConv to the kernel sizes required for the task at hand may make it possible to generalize the FlexNet architecture across different tasks and datasets.
Neural architecture search (Zoph & Le, 2016) could see beneﬁts from narrowing the search space to exclude kernel size and pooling layers.
In addition, we envisage additional improvements from structural developments of FlexConvs such as attentive FlexNets.
R EPRODUCIBILITY S TATEMENT
We hope to inspire others to use and reproduce our work.
We publish the source code of this work, for which the link is provided in Sec. 4.2.
Sec. 4 and Appx.
D.1 detail FlexNet, its hyperparameters and optimization procedure.
The full derivation of the aliasing regularization objective is included in
Appx. A.1.
We report means over multiple runs for many experiments, to ensure the reported results are fair and reproducible, and do not rely on tuning of the random seed.
All datasets used in our experiments are publicly available.
If any questions remain, we welcome one and all to contact the corresponding author.
A CKNOWLEDGMENTS
We thank Nergis Tömen for her valuable insights regarding signal processing principles for FlexConv, and Silvia-Laura Pintea for explanations and access to code of her work Pintea et al. We thank Yerlan Idelbayev for the use of the CIFARResNet code. This work is co-supported by the Qualcomm Innovation Fellowship granted to David W. Romero.
David W. Romero sincerely thanks Qualcomm for his support.
David W. Romero is ﬁnanced as part of the Efﬁcient Deep Learning (EDL) programme (grant number P16-25), partly funded by the Dutch
Research Council (NWO).
Robert-Jan Bruintjes is ﬁnanced by the Dutch Research Council (NWO) (project VI.Vidi.192.100).
All authors sincerely thank everyone involved in funding this work.
This work was partially carried out on the Dutch national infrastructure with the support of SURF
Cooperative.
We used Weights & Biases (Biewald, 2020) for experiment tracking and visualizations.
A A LIAS - FREE F LEX C ONV REGULARIZATION
In this section we provide the complete derivation and analysis for our FlexConv regularization against aliasing.
First, we derive the analytic maximum frequency component of a FlexConv kernel.
Next, we compute the Nyquist frequency of a FlexConv kernel, and subsequently show how to combine the previous results into a regularization term to train alias-free FlexConvs.
A. A NALYZING THE FREQUENCY SPECTRUM OF F LEX C ONV
In order to make FlexConv alias-free (Sec. 3.3), we need to compute the maximum frequency component of the kernels generated by a MAGNet, so that we can regularize it during training.
In this section we analytically derive this maximum frequency component from the parameters of the
MAGNet.
Recall that MAGNets generate a kernel ψ ( x,y ) through of a succession of anisotropic Gabor ﬁlters and linear layers (Sec. 3.2, Eqs.
[Equation]
y ] ; θ ( l ) ) =
exp ( − 1 2 [( γ ( l ) X ( x − µ ( l ) X )) 2 + ( γ ( l ) Y ( y − µ ( l ) Y )) 2 ]) Sin ( W ( l ) g [ x,y ] + b ( l ) g ) θ ( l ) = { γ ( l ) X ∈ R N hid , γ ( l ) Y ∈ R N hid , µ ( l ) X ∈ R N hid , µ ( l ) Y ∈ R N hid , W ( l ) g ∈ R N hid × 2 , b ( l ) g ∈ R N hid }
To analyse the maximum frequency component f + MAGNet , we analyse the frequency components of the Gabor ﬁlters used in MAGNet, and retain their maximum.
We then plug the found frequency component into the analysis of Fathony et al.
(2021) to show how the frequency responses of Gabor ﬁlters and linear layers interact in MFNs.
Finally, we add the effect of the FlexConv Gaussian mask to our analysis to obtain the maximum frequency component ot the ﬁnal FlexConv kernel f + FlexConv .
Sine term in Gabor ﬁlters.
In a Gabor ﬁlter, the sine term is multiplied with a Gaussian envelope.
The frequency (in radians) of a sine function of the form Sin ( w T [ x,y ] + b ) is given by w .
We divide by 2 π to convert the frequency units to Hertz, for compatibility with the rest of the analysis.
For 2D inputs, the maximum frequency component of the sine function correspond to the largest frequency in the two input dimensions:
[Equation] [Equation]
The sine terms in MAGNets have multiple output channels:
Sin ( W g ⋅ [ x,y ] + b ( l ) g ) .
Effectively, we compute the sine term independently for each channel:
[Equation]
[Figure]
Gaussian term in Gabor ﬁlters.
In a Gabor ﬁlter, a Gaussian envelope modulates a sine term.
Let us assume for now that the Gaussian envelope is isotropic, rather than anisotropic as in MAGNets, and has single-channel output.
By applying the convolution theorem, the sine term is equivalently convolved with the Fourier transform of the Gaussian envelope in the frequency domain.
Since the Fourier transform of a Gaussian envelope is another Gaussian envelope, the application of a
Gaussian envelope amounts to blurring with a Gaussian kernel in the frequency domain.
The size of the envelope in the Fourier domain σ F can be derived from the standard deviation of the Guassian envelope in the spatial domain σ T as follows:
[Equation]
Gaussian blurs induce impulse signals to have a long tail.
Consequently, we must deﬁne a cutoff point for this tail in terms of standard deviations to derive the maximum added frequency induced by the blur.
We describe the cutoff point as σ cut ∈ N .
Typical choices for σ cut are known as the empirical , or the "68-95-99.7" rule (Hald, 2007).
We choose a standard of two standard deviations, i.e., σ cut = 2 , which covers 95% of the mass of the Gaussian envelope.
For an isotropic Gabor ﬁlter with γ = σ − 1T , the maximum frequency of its Gaussian envelope f + env is:
[Equation]
Anisotropic envelopes.
Our analysis so far assumes an isotropic Gaussian envelope in the Gabor ﬁlter.
However, we need to account for the anisotropic Gaussian envelopes in MAGNets.
Anisotropic ﬁlters have not one but two γ parameters: { γ X ,γ Y } .
The smallest of these will contribute most to f + env , as it will blur the most, so it is sufﬁcient to compute f
+ env only using the smallest of the two γ terms:
[Equation]
The other assumption we made before was to work with single-channel outputs.
MAGNets however use multi-channel outputs with independent Gaussian terms.
The maximum frequency of multichannel Gaussian envelopes is given by:
[Equation]
where the subscript i indexes the channels of the multi-channel Gaussian envelopes.
Maximum frequency component of anisotropic Gabor ﬁlters.
Finally, the maximum frequency component of the i -th channel of an anisotropic Gabor ﬁlter g is given by:
[Equation]
Figure 9 illustrates the frequency spectrum of an example Gabor ﬁlter.
Maximum frequency component of a MAGNet.
Fathony et al.
(2021) characterize the expansion of each term of the isotropic Gabor layers in MFNs in the ﬁnal MFN output.
In Eq.
, Fathony et al.
(2021) demonstrate that the MFN representation contains a set of sine frequencies ω given by:
[Equation]
Here, the indexes i 1 ,i 2 , ⋯ ,i L − 1 range over all possible indices of each hidden unit of each layer of an MFN, and s 2 , ⋯ ,s L ∈ {− 1 , + 1 } range over all 2 L − 1 possible binary signs.
In other words,
Fathony et al.
(2021) demonstrate that the representation of an MFN at a particular layer contains an exponential combination of all possible positive and negative combinations of the frequencies of the sine terms in each hidden unit at each layer in the MFN up to the current layer.
The original analysis uses these terms to argue that MFNs model exponentially many terms through a linear amount of layers.
For our purpose of computing the frequency response of the MAGNet generated kernel, we can plug our derivation of the frequencies of the Gabor ﬁlter
f
Gabor into ω to compute the frequency spectrum of the generated kernel:
[Equation]
[Figure]
) =
[Figure]
( x | 0 . 0 , 0 . 1)
h ( x ) = sin(50 .
3 x ) g ( x ) =
N ( x | 0 . 0 , 0 .
1)sin(50 .
3 x )
[Figure]
As stated before
[Equation]
we are only interested in the maximum frequency in the frequency spectrum.
We can therefore simplify Eq.
18 in two ways.
First
[Equation]
we simplify over MAGNet layers by taking the maximum value of the spectrum, which is the sum over all layers using only the positive binary signs in s L (Eq. 19).
Next, we simplify over channel indices by retaining only the channel index that results in the highest frequency (Eq. 20).
The maximum frequency of a MAGNet is shown in Eq.
[Equation]
[Equation]
[Equation]
Effect of the Gaussian mask in the frequency components of a FlexConv.
FlexConvs attenuate the MAGNet output with a Gaussian mask.
The Gaussian mask (Eq. 1) works analogously to the
Gaussian envelope term in the Gabor ﬁlter: it blurs the frequency components of the generated kernel with standard deviation σ F .
Therefore, we can reuse our derivation for the Gaussian envelope of the
Gabor ﬁlter (Eq. 15).
The maximum frequency component of a FlexConv kernel is given by:
[Equation]
Visualization of regularized kernels.
Fig.
8 shows example kernels from FlexNets trained with aliasing regularization.
The frequency domain plots conﬁrm the accuracy of our frequency component regularization.
A. R EGULARIZING THE FREQUENCY RESPONSE OF F LEX C ONV
Nyquist frequency of a FlexConv kernel.
Given the sampling rate f s of the kernel, we can compute its Nyquist frequency f
Nyq as:
[Equation]
To compute the sampling rate, we note that the kernel coordinates input to our MAGNet stretch over a [− 1 , 1 ] D domain.
For a kernel of length k , we therefore sample one point in every f s = k − 12 units.
Knowing the sampling rate in terms of the kernel size allows us to express the Nyquist frequency in terms of the (pre-masked) kernel size:
[Equation]
Note that the kernel size in a FlexConv is initialized to be equal to the resolution of the data, if it is odd.
For even resolutions, it corresponds to the resolution of the data plus one.
Constructing the regularization term.
We train FlexConv with a regularization term on the frequency response of the generated kernel to ensure that aliasing effects do not distort the performance of the model when it is inferred at a higher resolution.
This section details the implementation of the regularization function.
From the parameters of each FlexConv module, we compute f + FlexConv according to Eq. 22.
For the amount of standard deviations to use in determining f + env
(Eq. 15)
we use h = 2 .
From the kernel size k of the FlexConv module we compute f Nyq ( k ) according to Eq.
We then apply an L2 regularizer over the amount that f + FlexConv exceeds f Nyq ( k ) : [Equation]
[Equation]
We weight L HF by λ = 0 .
1 when adding it to our loss function.
Improved implementation.
Eq.
25 contains a sum over the L layers of the MAGNet.
In practice, we prefer to regularize each layer l ∈ L separately, so that the gradients of the regularization of different layers are not dependent on each other.
We therefore implement the anti-aliasing regularization by regularizing each MAGNet layer independently, and spreading the f + env term from the gaussian mask uniformly over all MAGNet layers:
[Equation]
In the code, we refer to this method as the together method, versus the summed method of Eq.
In preliminary experiments, we observed improved performance of anti-aliasing training when using the together method.
All of our experiments anti-aliasing experiments therefore use the together setting.
B D ATASET D ESCRIPTION B. I MAGE F ITTING D ATASETS
Kodak dataset.
The Kodak dataset (Kodak, 1991) consists of 24 natural images of size 768 × 512 .
This dataset is a popular benchmark used for compression and image ﬁtting methods.
B. S EQUENTIAL D ATASETS
Sequential and Permuted MNIST.
The sequential MNIST dataset (sMNIST) (Le et al., 2015)takes the 28 × 28 images from the original MNIST dataset (LeCun et al., 1998), and presents them as a sequence of 784 pixels.
The goal of this task is to perform digit classiﬁcation given the representation of the last sequence element of a sequential model.
Consequently, good predictions require the model to preserve long-term dependencies up to 784 steps in the past.
The permuted MNIST dataset (pMNIST) additionally changes the order of all the sMNIST sequences by a random permutation.
Consequently, models can no longer rely on local features to construct good feature representations.
As a result, the classiﬁcation problem becomes more difﬁcult, and the importance of long-term dependencies more pronounced.
Sequential and Noise-Padded CIFAR10.
The sequential CIFAR10 dataset (sCIFAR10) (Chang et al., 2017) takes the 32 × 32 images from the original CIFAR10 dataset (Krizhevsky et al., 2009) and presents them as a sequence of 1,024 pixels.
The goal of this task is to perform image classiﬁcation given the representation of the last sequence element of a sequential model.
This task is more difﬁcult than sMNIST, as a larger memory horizon is required to solve the task and more complex structures and intra-class variations are present in the data (Bai et al., 2018b).
The noise-padded CIFAR10 dataset (npCIFAR10) (Chang et al., 2019) ﬂattens the images from the original CIFAR10 dataset (Krizhevsky et al., 2009) along their rows to create a sequence of length 32, and 96 channels (32 rows × 3 channels).
Next, these sequences are concatenated with 968 entries of noise to form the ﬁnal sequences of length 1000.
As for sCIFAR10, the goal of the task is to perform image classiﬁcation given the representation of the last sequence element of a sequential model.
CharacterTrajectories.
The CharacterTrajectories dataset is part of the UEA time series classiﬁcation archive (Bagnall et al., 2018).
It consists of 2858 time series of length 182 and 3 channels representing the x,y positions, and the tip force of a pen while writing Latin alphabet characters in a single stroke.
The goal is to classify out of 20 classes the written character using the time series data.
Speech Commands.
The Speech Commands dataset (Warden, 2018) consists of 105,809 onesecond audio recordings of 35 spoken words sampled at 16 kHz .
Following Kidger et al.
, we extract 34975 recordings from ten spoken words to construct a balanced classiﬁcation problem.
We refer to this dataset as SpeechCommands_raw , or SC_raw for short.
Furhtermore, we utilize the preprocessing steps of Kidger et al.
and extract mel-frequency cepstrum coefﬁcients from the raw data.
The resulting dataset, abreviated SC , consists of time series of length 101, and 20 channels.
B. I MAGE B ENCHMARK D ATASETS
MNIST.
The MNIST hadwritten digits datset (LeCun & Cortes, 2010) consists of 70,000 grayscale handwritten digits of size 28 × 28 , divided into a training and test sets of 60,000 and 10, images, respectively.
The goal of the task is to classify these digits as one of the ten possible digits ( 0 , 1 ,.. 8 , 9 ) .
CIFAR-
The CIFAR-10 dataset (Krizhevsky et al., 2009) consists of 60,000 natural images from 10 classes of size 32 × 32 , divided into training and test sets of 50,000 and 10,000 images, respectively.
STL-10.
The STL-10 dataset (Coates et al., 2011) is a subset of the ImageNet dataset (Krizhevsky et al., 2012) consisting of 5,000 natural images from 10 classes of size 96 × 96 , divided into trainint and test sets of 4,500 and 500 images, respectively.
ImageNet- k .
The Imagenet- k (Chrabaszcz et al., 2017) dataset is derived from the ImageNet dataset
Russakovsky et al.
by downsampling all samples to a resolution k ∈ [ 64 , 32 , 16 , 8 ] .
The dataset contains 1000 classes with 1,281,167 training samples and 50,000 validation samples.
[Table]
a
CIFARResNet-44 w/ FlexConv is a CIFARResNet-44 where all convolutional layers are replaced with
FlexConv with learned kernel size, except for the shortcut connections of the strided convolutional layers, which are
[Table]
C A DDITIONAL E XPERIMENTS C. I MAGE C LASSIFICATION
CIFAR-10.
Tab. 6 shows all results for our CIFAR-10 experiments, including more ablations.
ImageNet-32.
Results for the ImageNet-32 experiment are shown in Table 7.
FlexNets are slightly worse than CIFARResNet-
(He et al., 2016) with slightly less parameters.
However, the results
[Table] [Table]
[Table]
reported by Chrabaszcz et al.
(2017) for Wide ResNets (Zagoruyko & Komodakis, 2016) outperform
FlexNets by a signiﬁcant margin.
Alias-free ImageNet-32.
We report results for alias-free FlexNets on ImageNet- k
(Chrabaszcz et al., 2017) in Table 8, to verify the results of alias-free training at a larger scale.
We ﬁnd that FlexConv and N-Jet both mostly retain classiﬁcation accuracy between source and target resolution, while
CIFARResNet-32 degrades drastically.
MNIST and STL-10.
We additionally report results on MNIST (Tab. 9) and STL-10 (Tab. 10.
We choose these dataset for the difference in image sizes of the training data.
On MNIST, though performance on MNIST is quite saturated, we are competitive with state of the art methods.
On
STL-10 we are signiﬁcantly worse than the baseline CIFARResNet from (Luo et al., 2020), though with signiﬁcantly less parameters.
We were not able to prepare a more relevant baseline for this experiment.
D E XPERIMENTAL D ETAILS D. F LEX N ET
We propose an image classiﬁcation architecture named FlexNet (Fig. 10), consisting of a stack of
FlexConv blocks followed by a global average pooling layer and a linear layer.
FlexNets are named "FlexNet- L " where L indicates the amount of layers in the architecture.
FlexBlock.
Each FlexBlock consists of two FlexConvs with BatchNorm (Ioffe & Szegedy, 2015) and dropout (Srivastava et al., 2014)
( d = 0 . 2 ) as well as a residual connection.
The width of a block i is determined by scaling a base amount c by progressively increasing factors: c i =
[ c,c × 1 . 5 ,c × 1 . 5 ,c × 2 . 0 ,c × 2 . 0 ]
( i ) .
The default conﬁguration of FlexNet uses c = 22 .
In FlexNet-N-Jet models, we scale c to match the amount of parameters of the FlexNet in the comparison.
[Table]
FlexConv initialization.
We initialize the FlexConv mask variances small, at σ 2X ,σ 2Y = 0 .
For initializing MAGNet, we initialize the Gaussian envelopes as discussed in Sec. 3.2.
We initialize the linear layer weights by the same Gamma distribution as used for the enveloped, modulated by a scaling factor of 25 . 6 .
We found that this value of the scaling factor, rather than a higher one, helped in reducing the performance of alias-free models.
We initialize the bias of the linear layers by U(− π,π ) .
CIFAR-10.
In FlexNet-16 models for CIFAR-10 we use c = 24 to approximate the parameter count of CIFARResNets in the experiment.
D. O PTIMIZATION
We use Adam (Kingma & Ba, 2014) to optimize FlexNet.
Unless otherwise speciﬁed, we use a learning rate of 0 .
01 with a cosine annealing scheme (Loshchilov & Hutter, 2016) with ﬁve warmup epochs.
We use a different learning rate of 0 .
1 × the regular learning rate for the FlexConv Gaussian mask parameters.
We do not use weight decay, unless otherwise speciﬁed.
Kodak.
We overﬁt on each image of the dataset for 20,000 iterations.
To this end, we use a learning rate of 0.01 without any learning rate scheme.
We observe that SIRENs diverge with this learning rate and thus, reduce the learning rate to 0.001 for these models.
CIFAR-10.
We train for 350 epochs with a batch size of 64.
We use the data augmentation from He et al.
(2016) when training CIFAR-10: a four pixel padding, followed by a random 32 pixel crop and a random horizontal ﬂip.
ImageNet-32.
We train for 350 epochs with a batch size of 2048.
We use the same data augmentation as used for CIFAR-10.
We do use a weight decay of 1e − 5 for ImageNet-32 training.
Sequential and Permuted MNIST.
We train for 200 epochs with a batch size of 64 and a learning rate of 0.01.
We use a weight decay of 1e − 5 .
Sequential and Noise-Padded CIFAR-10.
For sequential CIFAR-10, we train for 200 epochs with a batch size of 64, a learning rate of 0.001 and a weight decay of 1e − 5 .
For noise-padded CIFAR-10, we train for 300 epochs with a batch size of 32, a learning rate of 0.01 and no weight decay.
Speech Commands and CharTrajectories.
We train for 300 epochs with a batch size of 32 and a learning rate of 0.001.
For CharTrajectories, we use a weight decay of 1e − 5 .
D. R OTATED G AUSSIAN MASKS
MAGNets use anisotropic Gaussian terms in the Gabor ﬁlters, which yields improvements in descriptive power and convergence speed (Sec. 3.2).
For the same reason, we explore making the anisotropic
FlexConv Gaussian mask steerable, by including an additional vector of learnable angle parameters φ
( l ) ∈
R N hid that rotates the Gaussian masks.
Although preliminary experiments show rotated masks lead to slight additional improvements, the computational overhead required to rotate the masks is large.
Consequently, we do not consider rotated Gaussian masks in our ﬁnal experiments.