L EARNING STRIDES IN CONVOLUTIONAL NEURAL NETWORKS
A BSTRACT
Convolutional neural networks typically contain several downsampling operators, such as strided convolutions or pooling layers, that progressively reduce the resolution of intermediate representations.
This provides some shift-invariance while reducing the computational complexity of the whole architecture.
A critical hyperparameter of such layers is their stride: the integer factor of downsampling.
As strides are not differentiable, ﬁnding the best conﬁguration either requires crossvalidation or discrete optimization (e g. architecture search) which rapidly become prohibitive as the search space grows exponentially with the number of downsampling layers.
Hence, exploring this search space by gradient descent would allow ﬁnding better conﬁgurations at a lower computational cost.
This work introduces DiffStride, the ﬁrst downsampling layer with learnable strides.
Our layer learns the size of a cropping mask in the Fourier domain, that effectively performs resizing in a differentiable way.
Experiments on audio and image classiﬁcation show the generality and effectiveness of our solution: we use DiffStride as a drop-in replacement to standard downsampling layers and outperform them.
In particular we show that introducing our layer into a ResNet-18 architecture allows keeping consistent high performance on CIFAR10, CIFAR100 and ImageNet even when training starts from poor random stride conﬁgurations.
Moreover, formulating strides as learnable variables allows us to introduce a regularization term that controls the computational complexity of the architecture.
We show how this regularization allows trading off accuracy for efﬁciency on ImageNet
I NTRODUCTION
Convolutional neural networks (CNNs) (Fukushima, 1980; LeCun et al., 1989) have been the most widely used neural architecture across a wide range of tasks, including image classiﬁcation (Krizhevsky et al., 2012;
He et al., 2016a; Huang et al., 2017; Bello et al., 2021), audio pattern recognition (Kong et al., 2020), text classiﬁcation (Conneau et al., 2017), machine translation (Gehring et al., 2017) and speech recognition (Amodei et al., 2016; Sercu et al., 2016; Zeghidour et al., 2018).
Convolution layers, which are the building block of CNNs, project input features to a higher-level representation while preserving their resolution.
When composed with non-linearities and normalization layers, this allows for learning rich mappings at a constant resolution, e.g. autogressive image synthesis (van den Oord et al., 2016).
However, many tasks infer high-level low-resolution information (identity of a speaker (Muckenhirn et al., 2018), presence of a face (Chopra et al., 2005)) by integrating over low-level, high-resolution measurements (waveform, pixels).
This integration requires extracting the right features, discarding irrelevant information over several downsampling steps.
To that end, pooling layers and strided convolutions aggressively reduce the resolution of their inputs, providing several beneﬁts.
First, they act as a bottleneck that forces features to focus on information relevant to the task at hand.
Second, pooling layers such as low-pass ﬁlters (Zhang, 2019) improve shift-invariance.
Third, a reduced resolution implies a reduced number of ﬂoating-point operations and a higher receptive ﬁeld in the subsequent layers.
Pooling layers can usually be decomposed into two basic steps: (1) computing local statistics densely over the whole input (2) sub-sampling these statistics by an integer striding factor.
Past work has mostly focused on improving (1), by proposing better alternatives to max and average pooling that avoid aliasing (Zhang, 2019; Fonseca et al., 2021), preserve the important local details (Saeedan et al., 2018), or adapt to the training data distribution (Gulcehre et al., 2014; Lee et al., 2016).
Observing that integer strides reduce resolution too quickly (e.g. a (2 , 2) striding reduces the output size by 75% ), Graham (2014) proposed fractional max-pooling, that allows for fractional (i.e. rational) strides, allowing for integration of more downsampling layers into a network.
Similarly,
Rippel et al.
(2015) introduce spectral pooling which, by cropping its inputs in the Fourier domain, performs downsampling with fractional strides while emphasizing lower frequencies.
While fractional strides give more ﬂexibility in designing downsampling layers, they increase the size of an already gigantic search space.
Indeed, as strides are hyperparameters, ﬁnding the best combination requires cross-validation or architecture search (Zoph & Le, 2017; Baker et al., 2017;
Tan et al., 2019), which rapidly become infeasible as the number of conﬁgurations grows exponentially with the number of downsampling layers.
This led Zoph & Le (2017) not to search for strides in most of their experiments.
Talebi & Milanfar (2021) and Jin et al.
(2021) proposed a neural network that learns a resizing function for natural images, but the scaling factor (i.e. the stride) still required cross-validation.
Thus, the nature of strides as hyperparameters — rather than trainable parameters — hinders the discovery of convolutional architectures and learning strides by backpropagation would unlock a virtually inﬁnite search space.
In this work, we introduce DiffStride, the ﬁrst downsampling layer that learns its strides jointly with the rest of the network.
Inspired by Rippel et al.
, DiffStride casts downsampling in the spatial domain as cropping in the frequency domain.
However, and unlike Rippel et al. (2015), rather than cropping with a ﬁxed bounding box controlled by a striding hyperparameter, DiffStride learns the size of its cropping box by backpropagation.
To do so, we propose a 2D version of an attention window with learnable size proposed by Sukhbaatar et al.
for language modeling.
On ﬁve audio classiﬁcation tasks, using DiffStride as a drop-in replacement to strided convolutions improves performance overall while providing interpretability on the optimal per-task receptive ﬁeld.
By integrating DiffStride into a ResNet-18 (He et al., 2016a), we show on CIFAR (Krizhevsky et al., 2009) and ImageNet (Deng et al., 2009) that even when initializing strides randomly, our model converges to the best performance obtained with the properly cross-validated strides of He et al.
(2016a).
Moreover, casting strides as learnable parameters allows us to propose a regularization that directly minimizes computation and memory usage.
We release our implementation of DiffStride 1 .
M ETHODS
We ﬁrst provide background on spatial and spectral pooling, and propose DiffStride for learning strides of downsampling layers.
We focus on 2D CNNs since they are generic enough to be used for image (LeCun et al., 1989; Krizhevsky et al., 2012;
He et al., 2016a) and audio (Amodei et al., 2016; Kong et al., 2020) processing (taking time-frequency representations as inputs).
However, these methods are equally applicable to the 1D (e.g. time-series) and 3D (e.g. video) cases.
2. N OTATIONS
Let x ∈ R H × W , its Discrete Fourier Transform (DFT)
y
= F ( x ) ∈ C H × W is obtained through the decomposition on a ﬁxed set of basis ﬁlters (Lyons, 2004):
[Equation]
The DFT transformation is linear and its inverse is given by its conjugate F ( . )
− 1 = F ( . ) ∗ .
The
Fourier transform of a real-valued signal x ∈ R H × W being conjugate symmetric (Hermitiansymmetry), we can reconstruct x from the positive half frequencies for the width dimension and omit the negative frequencies ( y mn =
y ∗ ( H − m ) mod H, ( W − n ) mod W ).
In addition, the DFT and its inverse are differentiable with regard to their inputs and the derivative of the DFT (resp.
inverse DFT) is its conjugate linear operator, i.e. the inverse DFT (resp.
DFT).
More formally, if we consider
L : C H × W −→ R as a loss taking as input the Fourier representation y , we can compute the gradient of L with regard to x , by using the inverse DFT:
[Equation]
We denote by L the total number of convolution layers in a CNN architecture and each layer is indexed by l .
The ◦ symbol represents the element-wise product between two tensors, (cid:98) .
(cid:99) is the ﬂoor operation and ⊗ the outer product between two vectors.
S represents the stride parameters, and sg is the stop gradient operator (Bengio et al., 2013; Yin et al., 2019), deﬁned has the identity function during forward pass and with zero partial derivatives.
2. D OWNSAMPLING IN CONVOLUTIONAL NEURAL NETWORKS
A basic mechanism for downsampling representations in a CNN is strided convolutions which jointly convolve inputs and ﬁnite impulse response ﬁlters and downsample the output.
Alternatively, one can disentangle both operations by ﬁrst applying a non-strided convolution followed by a pooling operation that computes local statistics (e.g. using an average, max (Boureau et al., 2010)) before downsampling.
In both settings, downsampling does not beneﬁt from the global structure of its inputs and can discard important information (Hinton, 2014; Saeedan et al., 2018).
Moreover, and as observed by Graham (2014), the integer nature of strides only allows for drastic reductions in resolution: a 2D-convolution with strides S =
(2 , 2) reduces the dimension of its inputs by 75% .
Furthermore, stride conﬁgurations are cumbersome to explore as the number of stride combinations grows exponentially with the number of downsampling layers.
This means that cross-validation can only explore a limited subset of the stride hyperparameter conﬁgurations.
This limitation is likely to translate into lower performance, as Section 3.2 shows that an inappropriate choice of strides for a
ResNet-18 architecture can account for a drop of > 18% in accuracy on CIFAR-100.
2. S PECTRAL POOLING
Energy of natural signals is typically not uniformly distributed in the frequency domain, with signals such as sounds (Singh & Theunissen, 2003), images (Ruderman, 1994) and surfaces (Kuroki et al., 2018) concentrating most of the information in the lower frequencies.
Rippel et al.
(2015) build on this observation to introduce spectral pooling which alleviates the loss of information of spatial pooling, while enabling fractional downsizing factors.
Spectral pooling also preserves low frequencies without aliasing, a known weakness of spatial/temporal convnets (Zhang, 2019; Ribeiro & Sch¨on, 2021).
We consider an input x ∈ R H × W and strides S =
( S h , S w ) ∈ [1 , H ) ×
[1 , W ) .
First, the DFT is computed y = F ( x ) ∈ C H × W and for simplicity we assume that the center of this matrix is the
DC component — the zero frequency.
Then, a bounding box of size (cid:98) HS h (cid:99) × (cid:98) WS w (cid:99) crops this matrix around its center to produce ˜ y ∈ C (cid:98) HSh (cid:99)×(cid:98) WSw (cid:99) .
Finally, this output is brought back to the spatial domain with an inverse DFT: ˜ x
= F − 1 (˜ y ) ∈ R (cid:98) H S h (cid:99)×(cid:98) W S w (cid:99) .
In practice, x is typically a multichannel input (i.e. x ∈ R H × W × C ) and the same cropping is applied to all channels.
Moreover, since x is real-valued and thanks to Hermitian symmetry (see Section 2.1 for more details), only the positive half of the DFT coefﬁcients are computed, which allows saving computation and memory while ensuring that the output ˜ x remains real-valued.
Unlike spatial pooling that requires integer strides, spectral pooling only requires integer output dimensions, which allows for much more ﬁne-grained downsizing.
Moreover, spectral pooling acts as a low-pass ﬁlter over the entire input, only keeping the lower frequencies i.e. the most relevant information in general and avoiding aliasing (Zhang, 2019).
However, and similarly to spatial pooling, spectral pooling is differentiable with respect to its inputs but not with respect to its strides.
Thus, one still needs to provide S as hyperparameters for each downsampling layer.
In this case, the search space is even bigger than with spatial pooling since strides are not constrained to integer values anymore.
2. D IFF S TRIDE
To address the difﬁculty of searching stride parameters, we propose DiffStride, a novel downsampling layer that allows spectral pooling to learn its strides through backpropagation.
To downsample x ∈ R H × W , DiffStride performs cropping in the Fourier domain similarly to spectral pooling.
However, instead of using a ﬁxed bounding box, DiffStride learns the box size via backpropagation.
The learnable box W is parametrized by the shape of the input, a smoothness factor R and the strides.
We design this mask W as the outer product between two differentiable 1D masking functions (depicted in the lower right corner of Figure 1), one along the horizontal axis and one along the vertical axis.
These 1D masks are directly derived from the adaptive attention span introduced by Sukhbaatar et al.
(2019) to learn the attention span of self-attention models for natural language processing.
Exploiting the conjugate symmetry of the coefﬁcients, we only consider positive frequencies along the horizontal axis, while we mirror the vertical mask around frequency zero.
Therefore, the two masks are deﬁned as follows:
[Equation]
where S =
( S h , S w ) are the strides and R an hyperparameter that controls the smoothness of the mask.
We build the 2D differentiable mask W as the outer product between the two 1D masks:
[Equation]
We use W in two ways: (1) we apply it to the Fourier representation of the inputs via an elementwise product, which performs low-pass ﬁltering (2) we crop the Fourier coefﬁcients where the mask is zero (i.e. the output has dimensions (cid:98) HS h + 2 × R (cid:99) × (cid:98) WS w + 2 × R (cid:99) ).
The ﬁrst step is differentiable with respect to strides S , however the cropping operation is not.
Therefore, we apply a stop gradient operator (Bengio et al., 2013) to the mask before cropping.
This way, gradients can ﬂow to the strides through the differentiable low-pass ﬁltering operation, but not through the non-differentiable cropping.
Finally, the cropped tensor is transformed back into the spatial domain using an inverse DFT.
All these steps are summarized by Algorithm 1 and illustrated on a single channel image in the Figure 1.
During training we constrain strides S
=
( S h , S w ) to remain in [1 , H ) ×
[1 , W ) .
When x is a multi-channel input (i.e. x ∈ R H × W × C ), we learn the same strides S for all channels to ensure uniform spatial dimensions across channels.
In spatial and spectral pooling, strides are typically
Inputs : Input x ∈ R H × W , strides S
=
( S h , S w ) ∈ [1 , H ) ×
[1 , W ) , smoothness factor R .
Output: Downsampled output ˜ x ∈ R (cid:98) HSh +2 × R (cid:99)×(cid:98) WSw +2 × R (cid:99) 1 y ←− F ( x ) (cid:46) Project input to the Fourier domain.
2 mask ←− W ( S h , S w , H, W, R ) (cid:46) Construct the mask.
See Equation 5.
[Figure]
(cid:46) Apply the mask as a low-pass ﬁlter.
[Figure]
y
[Figure]
←−
[Figure]
( y
[Figure]
, sg (mask))
(cid:46)
Crop the tensor with the mask after stopping gradients.
[Figure]
( y cropped )
(cid:46) Return to the spatial domain.
[Figure]
tied along the spatial axes (i.e. S w = S h ), which we can also do in DiffStride by sharing a single learnable stride on both dimensions.
However, our experiments in Section 3 show that learning speciﬁc strides for the vertical and horizontal axis is beneﬁcial, not only when processing timefrequency representations of audio, but also — more surprisingly— when classifying natural images.
Adding an hyperparameter R to each downsampling layer would conﬂict with the goal of removing strides as hyperparameters.
Thus, not only we use a single R value for all layers, but we found no signiﬁcant impact of this choice and all our experiments use R = 4 .
While we focus on the 2D case, using a single 1D mask allows deriving DiffStride in 1D CNNs, while performing the outer product between three 1D masks allows applying DiffStride to 3D inputs.
2.4. R ESIDUAL B LOCK WITH D IFF S TRIDE
Unlike systems that only feed outputs of the l th layer to the ( l + 1) th (Krizhevsky et al., 2012),
ResNets (He et al., 2016a;b) introduce skip-connections that operate in parallel to the main branch.
ResNets stack two types of blocks: (1) identity blocks that maintain the input channel dimension and spatial resolution and (2) shortcut blocks that increase the output channel dimension while reducing the spatial resolution with a strided convolution (see Figure 2a).
We integrate DiffStride into these shortcut blocks by replacing strided convolutions by convolutions without strides followed by DiffStride.
Besides, sharing DiffStride strides between the main and residual branches ensures that their respective outputs have identical spatial dimensions and can be summed (See Figure 2b).
2.4. R EGULARIZING COMPUTATION AND MEMORY COST WITH D IFF S TRIDE
The number of activations in a network depends on the strides and learning these parameters gives control over the space and time complexity of an architecture in a differentiable manner.
This contrasts with previous work, as measures of complexity such as the number of ﬂoating-point operations (FLOPs) are typically not differentiable with respect to the parameters of a model and searching for efﬁcient architectures is done via high-level exploration (e.g. introducing separable convolutions (Howard et al., 2017)), architecture search (Howard et al., 2019; Tan & Le, 2019) or using continuous relaxations of complexity (Paria et al., 2020).
A standard 2D convolution with a square kernel of size k 2 and C (cid:48) output channels has a computational cost of k 2 × C × C (cid:48)
× H × W when operating on x ∈ R H × W × C .
Its memory usage— in terms of the number of activations to store— is C (cid:48)
× H × W .
Considering a ﬁxed number of channels and kernel size, both the computational complexity and memory usage of a convolution layer are thus linear functions of its input size H × W .
This illustrates our argument made in Section 1 that downsampling does not only improve performance by discarding irrelevant information, but also reduces the complexity of the upper layers.
More importantly, in the context of DiffStride the input size H l × W l of layer l is determined as follows:
H l × W l
= (cid:98)
[Equation]
which is differentiable with respect to the strides at the previous layer S l − 1 .
Furthermore, it also depends on spatial dimensions at the previous layer H l − 1 × W l − 1 , which themselves are a function of S l − 2 .
By induction over layers, the total computational cost and memory usage are proportional to (cid:80) l
= L l =1 (cid:81) li = 1 S ih × S iw .
Since in the context of DiffStride the kernel size and number of channels remain constant during training, we can directly regularize our model towards time and space efﬁciency by adding the following regularizer to our training loss:
[Equation]
where λ is the regularization weight.
In Section 3.2, we show that training on ImageNet with different values for λ allows us to trade-off accuracy for efﬁciency in a smooth fashion.
E XPERIMENTS
We evaluate DiffStride on eight classiﬁcation tasks, both on audio and images.
For each comparison, we keep the same architecture and replace strided convolutions by convolutions with no stride followed by DiffStride.
To avoid the confounding factor of downsampling in the Fourier domain, we also compare our approach to the spectral pooling of Rippel et al.
(2015), which only differs from
DiffStride by the fact that its strides are not learnable.
3. A UDIO CLASSIFICATION
Experimental setup We perform single-task and multi-task audio classiﬁcation on 5 tasks: acoustic scene classiﬁcation (Heittola et al., 2018), birdsong detection (Stowell et al., 2018), musical instrumental classiﬁcation and pitch estimation on the NSynth dataset (Engel et al., 2017) and speech command classiﬁcation (Warden, 2018).
The statistics of the datasets are summarized in Table A.1.
The audio sampled at 16 kHz is decomposed into log-compressed mel-spectrograms with 64 channels, computed with a window of 25 ms every 10 ms .
A 2D-CNN, based on (Tagliasacchi et al., 2019) takes these spectrograms as inputs and alternates blocks of strided convolutions along time (( 3 × 1 ) kernel) and frequency (( 1 × 3 ) kernel).
Each strided convolution is followed by a ReLU (Glorot et al., 2011) and batch normalization (Ioffe & Szegedy, 2015).
The sequence of channels dimensions is deﬁned as (64 , 128 , 256 , 256 , 512 , 512) and the strides are initialized as ((2 , 2) , (2 , 2) , (1 , 1) , (2 , 2) , (1 , 1) , (2 , 2) for all downsampling methods.
The output of the CNN passes through a global max-pooling and feeds into a single linear classiﬁcation layer for single-task, and multiple classiﬁcation layers for multi-task classiﬁcation.
As examples vary in length, we train models on random 1 s windows with ADAM (Kingma & Ba, 2015) and a learning rate of 10 − 4 for 1 M batches, with batch size 256.
Evaluation is run by splitting full sequences into 1 s non-overlapping windows and averaging the logits over windows.
Results Table 1 summarizes the results for single-task and multi-task audio classiﬁcation.
In both settings, DiffStride improves over strided convolutions and spectral pooling, with strided convolutions only outperforming DiffStride for acoustic scene classiﬁcation in the single task setting.
Table 2 shows the strides learned by the ﬁrst layer of DiffStride, which downsamples mel-spectrograms along frequency and time axes.
Learning allows the strides to deviate from their initialization ( (2 , 2) ) and to adapt to the task at hand.
Converting strides to cut-off frequencies shows that the learned strides fall in a range showed by behavioral studies and direct neural recordings (Hullett et al., 2016;
Flinker et al., 2019) to be necessary for e.g. speech intelligibility at 25 Hz (Elliott & Theunissen, 2009).
Moreover, DiffStride learns different strides for the time and frequency axes.
Table A. shows the beneﬁts of learning a per-dimension value rather than sharing strides.
Another notable phenomenon is the per-task discrepancy on NSynth, with the pitch estimation requiring faster spectral modulations (as represented by a higher cutt-off frequency along the frequency axis).
Finally, multi-task models do not converge to the mean of strides, but rather to a higher value that passes more frequencies not to negatively impact individual tasks.
[Table]
[Table]
3. I MAGE CLASSIFICATION
Experimental setup We use the ResNet-18 (He et al., 2016a) architecture, comparing the original strided convolutions (see Figure 2a) to spectral pooling and DiffStride (both as in Figure 2b).
We randomly sample 6 striding conﬁgurations for the three shortcut blocks of the ResNet-18, each stride being sampled in [1 , 3] , with (2 , 2 , 2) being the conﬁguration of the original ResNet of He et al.
(2016a).
The horizontal and vertical strides are initialized equally at start.
These random conﬁgurations simulate cross-validation of stride conﬁgurations to: (1) showcase the sensitivity of the architecture to these hyperparameters, (2) test our hypothesis that DiffStride can beneﬁt from learning its strides to recover from a poor initialization.
On Imagenet, as inputs are bigger than
CIFAR we also allow the ﬁrst ResNet-18 identity block to learn its strides which are 1 by default.
We ﬁrst benchmark the three methods on the two CIFAR datasets (Krizhevsky, 2009).
CIFAR consists of 32 × 32 images labeled in 10 classes with 6000 images per class.
We use the ofﬁcial split, with 50,000 images for training and 10,000 images for testing.
CIFAR100 uses the same images as CIFAR10, but with a more detailed labelling with 100 classes.
We also compare the
ResNet-18 architectures on the ImageNet dataset (Deng et al., 2009), which contains 1,000 classes.
The models are trained on the ofﬁcial training split of the Imagenet dataset (1.28M images) and we report our results on the validation set (50k images).
Here, we evaluate performance in terms of top-1 and top-5 accuracy.
We train on all datasets with stochastic gradient descent (SGD) (Bottou et al., 1998) with a learning rate of 0 . 1 , a batch size of 256 and a momentum (Qian, 1999) of 0 . 9 .
On CIFAR, we train models for 400 epochs dividing the learning rate by 10 at 200 epochs and again by 10 at 300 epochs, with a weight decay of 5 .
10 − 3 .
For CIFAR, we apply random cropping on the input images and left-right random ﬂipping.
On ImageNet, we train with a weight decay of 1 . 10 −for 90 epochs, dividing the learning rate by 10 at epochs 30, 60 and 80.
We apply random cropping on the input images as in (Szegedy et al., 2015) and left-right random ﬂipping.
Results We report the results on the CIFAR datasets and Imagenet in Tables 3 and 4 respectively, with the accuracy of our baseline ResNet-18 (ﬁrst row, Strided Conv.) being consistent with previous work (Bianco et al., 2018).
First, we observe that strides are indeed critical hyperparameters for the performance of a standard ResNet-18 on the three datasets, with the accuracy on CIFAR dropping from 66 .
8% average to 48 .
2% between the best and worst conﬁgurations.
Remarkably, spectral pooling is much more robust to bad initializations than strided convolutions, even though
[Table]
((Foret et al., 2020)) with an accuracy of 99 .
[Figure]
its strides are also ﬁxed.
However, DiffStride is overall much more robust to poor choices of strides, converging consistently to a high accuracy on the three datasets, with a variance over initializations that is lower by an order of magnitude.
This shows that backpropagation allows DiffStride to ﬁnd a better conﬁguration during training avoiding a cross-validation which would require 6,561 experiments for testing all combinations of strides in [1 , 3] on Imagenet.
Tables A.5 and A.6 conﬁrm these observations on the EfﬁcientNet-B0 (Tan & Le, 2019) architecture.
Learning dynamics and equivalence classes Figure 3 illustrates the learning dynamics of DiffStride on CIFAR10.
Figure 3a plots the strides as a function of the epoch for a run with the baseline (2,2,2) conﬁguration as initialization.
The strides all deviate from their initialization while converging rapidly, with the lower layers keeping more information while higher layers downsample more drastically.
Interestingly, we discover equivalence classes: despite converging to the same accuracy (as reported in Table 3) the various initializations yield very diverse strides conﬁgurations at convergence, both in terms of total striding factor (deﬁned as the product of strides, see Figure 3c) and of repartition of downsampling factors along the architecture (see Figure 3b).
We obtain similar conclusions on CIFAR100 and Imagenet (see Figures A.1 and A.2).
In the non-regularized case, it could seem counter-intuitive that minimizing the training loss yields positive stride updates, i.e. dropping more information through cropping.
It highlights that loss optimization is a tradeoff between preserving information (no striding, no cropping) and downscaling such that the next convolution kernel accesses a wider spatial context.
Regularizing the complexity The existence of equivalence classes suggests that DiffStride can ﬁnd more computationally efﬁcient conﬁgurations for a same accuracy.
We thus train DiffStride on ImageNet using the complexity regularizer deﬁned in Equation 6, with λ varying between 0 .and 10 , always initializing strides with the baseline ((1 , 1) , (2 , 2) , (2 , 2) , (2 , 2)) .
Figure 4 plots accuracy versus computational complexity (as measured by the value of the regularization term at convergence) of DiffStride.
For comparison, we also plot the models with strided convolutions with the random initializations of Table 4, showing that DiffStride ﬁnds conﬁgurations with a lower computational cost for the same accuracy.
Some of these are quite extreme, e.g. with λ =a model converges to strides ((10 . 51 , 32 . 23) , (1 . 20 , 2 . 68) , (1 . 20 , 2 . 04) , (1 . 96 , 4 . 53)) for a 58 .
[Table]
[Figure]
top-1 accuracy.
When training a ResNet-18 with strided convolutions using the closest integer strides (i.e. ((11 , 32) , (1 , 3) , (1 , 2) , (2 , 5)) ), the model converges to a 24 .
54% top-1 accuracy.
This suggests that performing pooling in the spectral domain is more robust to aggressive downsampling, which corroborates the remarkable advantage of spectral pooling over strided convolutions when using poor strides choices in Tables 3 and 4 despite both models having ﬁxed strides.
Limitations Pooling in the spectral domain comes at higher computational cost than strided convolutions as it requires (1) computing a non-strided convolution and (2) a DFT and its inverse (see
Table A.2).
This could be alleviated by computing the convolution in the Fourier domain as an element-wise multiplication and summation over channels.
Further improvements could be obtained by replacing the DFT by a real-valued counterpart, such as the Hartley transform (Zhang & Ma, 2018), which would remove the need for complex-valued operations that may be poorly optimized in deep learning frameworks.
We also observe no beneﬁts of DiffStride when training
DenseNets (Huang et al., 2017), see Tables A.3 and A.4.
We hypothesize that this is due to the limited number of downsampling layers, which reduces the space of stride conﬁgurations to a few, equivalent ones when sampling strides in [1; 3] .
Finally, some hardware (e.g. TPUs) require a static computation graph.
As DiffStride changes the spatial dimensions of intermediate representations— and thus the computation graph— between each gradient update, we currently only train on GPUs.
C ONCLUSION AND FUTURE WORK
We introduce DiffStride the ﬁrst downsampling layer with learnable strides.
We show on audio and image classiﬁcation that DiffStride can be used as a drop-in replacement to strided convolutions, removing the need for cross-validating strides.
As we observe that our method discovers multiple equally-accurate stride conﬁgurations, we introduce a regularization term to favor the most computationally advantageous.
In future work, we will extend the scope of applications of DiffStride, to e.g. 1D and 3D architectures.
Moreover, learning strides by backpropagation opens new avenues in designing adaptive convolutional architectures, such as multi-scale models that would learn to operate at various scales in parallel by using independent branches with separate instances of DiffStride, or by predicting strides parameters of DiffStride on a per-example basis. A CKNOWLEDGMENTS
Authors thank F´elix de Chaumont Quitry for the useful discussions and assistance throughout this project, as well as the reviewers of ICLR 2022 for their feedback that helped improving this manuscript.
Authors also thank Oren Rippel, Jasper Snoek, and Ryan P. Adams for their inspiring work on spectral pooling.
R EPRODUCIBILITY STATEMENT
We describe DiffStride in details in the text as well as with Algorithm 1 and Figure 1.
We mention all relevant hyperparameters to reproduce our experiments, as well as describe audio datasets in A.1.
Moreover, we release Tensorﬂow 2.0 code for training a Pre-Act ResNet-18 with strided convolutions, spectral pooling or DiffStride on CIFAR10 and CIFAR100, with DiffStride being implemented as a stand-alone, reusable Keras layer.
This open-source code can be found at https://github.com/google-research/diffstride .
A A PPENDIX
[Table]
[Equation]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Equation] [Figure] [Equation] [Figure]
[Figure]
S
[Figure]
S lw
[Figure]
Analysis of strides learned on CIFAR100 and ImageNet
In Figure A.1 (Figure A.2), we show the distributions of learned strides and the global striding factor at convergence on CIFAR100 (Imagenet), starting from random stride initializations.
On CIFAR100, we observe equivalence classes, i [Figure] e. model that learns various stride conﬁgurations for a same accuracy.
On Imagenet, even though we also observe a signiﬁcant variance of the global striding factor, models tend to downsample only in the upper layers.
Striding late in the architecture comes at a higher computational cost, which furthermore justiﬁes regularizing DiffStride to reduce complexity as shown in Section 3.2.
Time and space complexity in practice While Figure 4 reports theoretical estimates of computational complexity based on stride conﬁgurations, both spectral pooling and DiffStride require computing a DFT and its inverse.
Moreover, DiffStride requires accumulating gradients with respect to the strides during training.
Table A.2 reports the duration and peak memory usage of the multitask architecture described in 3.1, for a single batch.
Replacing strided convolutions with spectral pooling increases the wall time by 32% due to the DFT and inverse DFT, while the peak memory usage is almost unaffected.
DiffStride furthermore increases the wall time (by 43% w.r.t strided convolutions) as the backward pass is more expensive.
Similarly, it almost doubles the peak memory usage.
However, in inference, DiffStride does not need to compute and store gradients w.r.t.
the strides, thus the time and space complexity become identical to that of spectral pooling.
[Table]
DenseNet experiments on CIFAR We also evaluate DiffStride in DenseNet (Huang et al., 2017), especially the DenseNet-BC architecture with a depth of 121 and a growth rate of 32.
The DenseNet architecture halves spatial dimensions during transition blocks.
We replace the 2D average pooling in the transition blocks by spectral pooling or DiffStride.
The considered architecture for DenseNet has two downsampling steps.
We run a similar experiment as in 3.2 with random strides between the dense blocks on the two CIFAR datasets.
We observe that initializing strides randomly does not affect the performance of the standard Densenet-BC architecture with average pooling.
Consequently,
DiffStride does not improve over alternatives.
[Table] [Table]
EfﬁcientNet experiments on CIFAR We evaluate DiffStride in an EfﬁcientNet-B0 architecture (Tan & Le, 2019), a lightweight model discovered by architecture search.
This architecture has seven strided convolutions.
Unlike Tan & Le (2019), we do not pre-train on ImageNet, but rather train from scratch on CIFAR, which explains the lower accuracy of the baseline.
As the model has seven downsampling layers, we rescale the images from 32 × 32 to 128 × 128 , and only sample strides in [1; 2] .
We run a similar experiment as in 3.2 with random strides on the two CIFAR datasets.
Consistently with the results obtained with a ResNet-18, spectral pooling is much more robust to poor strides than strided convolutions, with DiffStride outperforming all alternatives
[Table]
[Table]
Ablation study: learning a per-dimension stride or a shared one We perform multi-task audio classiﬁcation with either learning a single stride value for each DiffStride layer
[Table]
or a different one for the time and frequency axes.
The overall performance across tasks is improved when learning a different stride value for each dimension (See Table A.7)
[Table]
the multi-task (one model for all tasks) with DiffStride, when learning a single stride value (Shared stride) per layer, or a different one for each dimension (Different strides).