U NDERSTANDING THE C OVARIANCE S TRUCTURE OF C ONVOLUTIONAL F ILTERS
A BSTRACT
Neural network weights are typically initialized at random from univariate distributions, controlling just the variance of individual weights even in highlystructured operations like convolutions.
Recent ViT-inspired convolutional networks such as ConvMixer and ConvNeXt use large-kernel depthwise convolutions whose learned ﬁlters have notable structure; this presents an opportunity to study their empirical covariances.
In this work, we ﬁrst observe that such learned ﬁlters have highly-structured covariance matrices, and moreover, we ﬁnd that covariances calculated from a small network may be used to effectively initialize a variety of larger networks of different depths, widths, patch sizes, and kernel sizes, indicating a degree of model-independence to the covariance structure.
Motivated by this ﬁnding, we then propose a learning-free multivariate initialization scheme for convolutional ﬁlters using a simple, closed-form construction of their covariance.
Models using our initialization outperform those using traditional univariate initializations, and typically meet or exceed the performance of those initialized from the covariances of learned ﬁlters; in some cases, this improvement can be achieved without training the depthwise convolutional ﬁlters at all .
Our code is available at https://github.com/locuslab/convcov .
I NTRODUCTION
Early work in deep learning for vision demonstrated that the convolutional ﬁlters in trained neural networks are often highly-structured, in some cases being qualitatively similar to ﬁlters known from classical computer vision (Krizhevsky et al., 2017).
However, for many years it became standard to replace large-ﬁlter convolutions with stacked small-ﬁlter convolutions, which have less room for any notable amount of structure.
But in the past year, this trend has changed with inspiration from the long-range spatial mixing abilities of vision transformers.
Some of the most prominent new convolutional neural networks, such as ConvNeXt and ConvMixer, once again use large-ﬁlter convolutions.
These new models also completely separate the processing of the channel and spatial dimensions, meaning that the now-single-channel ﬁlters are, in some sense, more independent from each other than in previous models such as ResNets.
This presents an opportunity to investigate the structure of convolutional ﬁlters.
In particular, we seek to understand the statistical structure of convolutional ﬁlters, with the goal of more effectively initializing them.
Most initialization strategies for neural networks focus simply on controlling the variance of weights, as in Kaiming (He et al., 2015) and Xavier (Glorot & Bengio, 2010) initialization, which neglect the fact that many layers in neural networks are highly-structured, with interdependencies between weights, particularly after training.
Consequently, we study the covariance matrices of the parameters of convolutional ﬁlters, which we ﬁnd to have a large degree of perhaps-interpretable structure.
We observe that the covariance of ﬁlters calculated from pretrained models can be used to effectively initialize new convolutions by sampling ﬁlters from the corresponding multivariate Gaussian distribution.
We then propose a closed-form and completely learning-free construction of covariance matrices for randomly initializing convolutional ﬁlters from Gaussian distributions.
Our initialization is highly effective , especially for larger ﬁlters, deeper models, and shorter training times; it usually outperforms both standard uniform initialization techniques and our baseline technique of initializing by sampling from the distributions of pre-trained ﬁlters, both in terms of ﬁnal accuracy and time-toconvergence.
Models using our initialization often see gains of over 1% accuracy on CIFAR-10 and short-training ImageNet classiﬁcation; it also leads to small but signiﬁcant performance gains on full-scale, ≈ 80% -accuracy ImageNet training.
Indeed, in some cases our initialization works so well that it outperforms uniform initialization even when the ﬁlters aren’t trained at all .
And our initialization is almost completely free to compute.
Related work Saxe et al.
(2013) proposed to replace random i.i.d.
Gaussian weights with random orthogonal matrices, a constraint in which weights depend on each other and are thus, in some sense, “multivariate”; Xiao et al.
(2018) also proposed an orthogonal initialization for convolutions.
Similarly to these works, our initialization greatly improves the trainability of deep (depthwise) convolutional networks, but is much simpler and constraint-free, being just a random sample from a multivariate Gaussian distribution.
Zhang et al.
(2022) suggests that the main purpose of pretraining may be to ﬁnd a good initialization, and crafts a mimicking initialization based on observed, desirable information transfer patterns.
We similarly initialize convolutional ﬁlters to be closer to those found in pre-trained models, but do so in a completely random and simpler manner.
Romero et al.
(2021) proposes an analytic parameterization of variable-size convolutions, based in part on
Gaussian ﬁlters; while our covariance construction is also analytic and built upon Gaussian ﬁlters, we use them to specify the distribution of ﬁlters.
Our contribution is most advantageous for large-ﬁlter convolutions, which have become prevalent in recent work: ConvNeXt (Liu et al., 2022b) uses 7 × 7 convolutions, and ConvMixer (Trockman & Kolter, 2022) uses 9 × 9 ; taking the trend a step further, Ding et al.
(2022) uses 31 × 31 , and Liu et al.
(2022a) uses 51 × 51 sparse convolutions.
Many other works argue for large-ﬁlter convolutions (Wang et al., 2022; Chen et al., 2022; Han et al., 2021).
Preliminaries This work is concerned with depthwise convolutional ﬁlters, each of which is parametrized by a k × k matrix, where k (generally odd) denotes the ﬁlter’s size.
Our aim is to study distributions that arise from convolutional ﬁlters in pretrained networks, and to explore properties of distributions whose samples produce strong initial parameters for convolutional layers.
More speciﬁcally, we hope to understand the covariance among pairs of ﬁlter parameters for ﬁxed ﬁlter size k .
This is intuitively expressed as a covariance matrix Σ ∈ R k 2 × k 2 with block structure: Σ has k × k blocks, where each block [Σ i,j ] ∈ R k × k corresponds to the covariance between ﬁlter pixel i, j and all other k 2 − 1 ﬁlter pixels.
That is, [Σ i,j ] (cid:96),m =
[Σ (cid:96),m ] i,j gives the covariance of pixels i, j and (cid:96), m .
In practice, we restrict our study to multivariate Gaussian distributions, which by convention are considered as distributions over n -dimensional vectors rather than matrices, where the distribution
N ( µ, Σ (cid:48) ) has a covariance matrix Σ (cid:48) ∈ S n + where Σ (cid:48)
i,j = Σ (cid:48)
j,i represents the covariance between vector elements i and j .
To align with this convention when sampling ﬁlters, we convert from our original block covariance matrix representation to the representation above by simple reassignment of matrix entries, given by
[Equation]
In this form, we may now easily generate a ﬁlter F ∈ R k × k by drawing a sample f ∈ R k 2 from
N ( µ, Σ (cid:48) ) and assigning F i,j := f ki + j .
In this paper, we assume covariance matrices are in the block form unless we are sampling from a distribution, where the conversion between forms is assumed.
Scope We restricted our study to the large-ﬁlter depthwise convolutions found in new ViT-style
CNNs, namely the popular ConvMixer and ConvNeXt architectures.
These networks consist of a patch embedding layer followed by alternating spatial- and channel-mixing steps.
Both use depthwise convolution for spatial mixing, but ConvMixer uses pointwise convolution (equivalently, linear layers) for spatial mixing while ConvNeXt uses MLPs.
ConvMixer uses no internal downsampling, while ConvNeXt includes several downsampling stages.
Unlike normal convolutions, the ﬁlters in depthwise convolutions act on each input channel separately rather than summing features over input channels.
The depth of networks throughout the paper is synonymous with the number of depthwise convolutional layers.
All networks investigated use a ﬁxed ﬁlter size throughout the network, though the methods we present could easily be extended to the non-uniform case.
Further, all methods presented do not concern the biases of convolutional layers.
[Figure]
[Table]
d
[Table] [Table]
T HE C OVARIANCES OF T RAINED C ONVOLUTIONAL F ILTERS AND T HEIR T RANSFERABILITY A CROSS A RCHITECTURES
In this section, we propose a simple starting point in our investigation of convolutional ﬁlter covariance structure: using the distribution of ﬁlters from pre-trained models to initialize ﬁlters in new models, a process we term covariance transfer .
In the simplest case, we use a pre-trained model with exactly the same architecture as the model to be initialized; we then show that we can actually transfer ﬁlter covariances across very different models.
Basic method.
We use i ∈ 1 , . . .
, D to denote the i th depthwise convolutional layer of a model with D layers.
We denote the j ∈ 1 , . . .
, H ﬁlters of the i th pre-trained layer of the model by F ij for a model with H convolutional ﬁlters in a particular layer ( i.e., hidden dimension H ) and F (cid:48) to denote the ﬁlters of a new, untrained model.
Then the empirical covariance of the ﬁlters in layer i is
[Equation]
with the mean µ i computed similarly.
Then the new model can be initialized by drawing ﬁlters from the multivariate Gaussian distribution with parameters
µ i , Σ i :
[Equation]
Note that in this section, we use the means of the ﬁlters in addition to the covariances to deﬁne the distributions from which to initialize.
However, we found that the mean can be assumed to be zero with little change in performance, and we focus solely on the covariance in later sections.
Experiment design.
We test our initialization methods primarily on ConvMixer since it is simple and exceptionally easy to train on CIFAR-10.
We use FFCV (Leclerc et al., 2022) for fast data loading using our own implementations of fast depthwise convolution and RandAugment (Cubuk et al., 2020).
To demonstrate the performance of our methods across a variety of training times, we train for 20, 50, or 200 epochs with a batch size of 512, and we repeat all experiments with three random seeds.
For all experiments, we use a simple triangular learning rate schedule (see
Appendix A.1) with the AdamW optimizer, a learning rate of .01, and weight decay of .01 as in
Trockman & Kolter (2022).
Most of our CIFAR experiments use a ConvMixer-256/8 with either patch size 1 or 2; a ConvMixerH / D has precisely D depthwise convolutional layers with H ﬁlters each, ideal for testing our initial covariance transfer techniques.
We train ConvMixers using popular ﬁlter sizes 3, 7, and 9, as well as 15.
We also test our methods on ConvNeXt (Liu et al., 2022b), which includes downsampling unlike
ConvMixer; we use a patch size of 1 or 2 with ConvNeXt rather than the default 4 to accomodate relatively small CIFAR-10 images, and the default 7 × 7 ﬁlters.
For most experiments, we provide two baselines for comparison: standard uniform initialization, the standard in PyTorch (He et al., 2015), as well as directly transferring the learned ﬁlters from a pre-trained model to the new model.
In most cases, we expect new random initializations to fall between the performance of uniform and direct transfer initializations.
For our covariance transfer experiments, we trained a variety of reference models from which to compute covariances; these are all trained for the full 200 epochs using the same settings as above.
Frozen ﬁlters.
Cazenavette et al. noticed that ConvMixers with 3 × 3 ﬁlters perform well
[Figure]
even when the ﬁlters are frozen ; that is, the ﬁlter weights remain unchanged over the course of training, receiving no gradient updates.
As we are initializing ﬁlters from the distribution of trained ﬁlters, we suspect that additional training may not be completely necessary.
Consequently, in all experiments we investigate both models with thawed ﬁlters as well as their frozen counterparts.
Freezing ﬁlters removes one of the two gradient calculations from depthwise convolution, resulting in substantial training speedups as kernel size increases (see Figure 2).
ConvMixer-512/12 with kernel size 9 ×is around 20% faster, while 15 × 15 is around 40% faster.
Further, good performance in the frozen ﬁlter setting suggests that an initialization technique is highly effective.
2. R ESULTS
The simplest case of covariance transfer (from exactly the same architecture) is a fairly effective initialization scheme for convolutional ﬁlters.
In Fig. 3, note that this case of covariance transfer (group
B ) results in somewhat higher accuracies than uniform initialization (group A ), particularly for 20epoch training; it also substantially improves the case for frozen ﬁlters.
Across all trials, the effect of using this initialization is higher for larger kernel sizes.
In Fig.
8, we show that covariance transfer ( gold ) initially increases convergence, but the advantange over uniform initialization quickly fades.
As expected, covariance transfer tends to fall between the performance of direct transfer , where we directly initialize using the ﬁlters of the pre-trained model, and default uniform initialization (see group D in Fig.
3 and the green curves in Fig.
However, we acknowledge that it is not appealing to pre-train models just for an initialization technique with rather marginal gains, so we explore the feasibility of covariance transfer from smaller models, both in terms of width and depth.
Narrower models.
We ﬁrst see if it’s possible to train a narrower reference model to calculate ﬁlter covariances to initialize a wider model; for example, using a ConvMixer-32/8 to initialize a
ConvMixer-256/8.
In Figure 4, we show that the optimal performance surprisingly comes from the covariances of a smaller model .
For ﬁlter sizes sizes greater than 3, the covariance transfer performance increases with width until width 32, and then decreases for width 256 for both the thawed and frozen cases.
We plot this method in Fig.
3 (group C ), and note that it almost uniformly exceeds the performance of covariance transfer from the same-sized model.
Note that the method does not change; the covariances are simply calculated from a smaller sample of ﬁlters.
Shallower models.
Covariance transfer from a shallow model to a deeper model is somewhat more complicated, as there is no longer a one-to-one mapping between layers.
Instead, we linearly interpolate the covariance matrices to the desired depth (see Appendix A.1 for more details).
Surprisingly, we ﬁnd that this technique is also highly effective: for example, for a 32-layer-deep
ConvMixer, the optimal covariance transfer result is from an 8-layer-deep ConvMixer, and 4-deep models are also quite effective (see Figure 4).
Different patch sizes.
Similarly, it is straightforward to transfer covariances between models with different patch sizes.
We ﬁnd that initializing ConvMixers with 1 × 1 patches from ﬁlter covariances of ConvMixers with 2 × 2 patches leads to a decrease in performance relative to using a reference model of the correct patch size; however, using the ﬁlters of a 1 × 1 patch-size ConvMixer to initialize a 2 × 2 patch size ConvMixer increases performance (see group b vs. group B in Fig. 9).
Yet, in both cases, the performance is better than uniform initialization.
[Figure]
covariance transfer ( B-D ), and our custom initialization results ( E )
[Figure]
Different ﬁlter sizes.
Covariance transfer between models with different ﬁlter sizes is more challenging, as the covariance matrices have different sizes.
In the block form, we mean-pad or clip each block to the target ﬁlter size, and then bilinearly interpolate over the blocks to reach a correctly-sized covariance matrix.
This technique is still better than uniform initialization for ﬁlter sizes larger than 3 (which naturally has very little structure to transfer), especially in the frozen case (see Fig. 9)
Discussion.
We have demonstrated that it is possible to initialize ﬁlters from the covariances of pre-trained models of different widths, depths, patch sizes, and kernel sizes; while some of these techniques perform better than others, they are almost all better than uniform initialization.
Our ob- servations indicate that the optimal choice of reference model is narrower or shallower, and perhaps with a smaller patch size or kernel size.
We also found that covariance transfer from ConvMixers trained on ImageNet led to greater performance still (Appendix A).
This suggests that the best covariances for ﬁlter initialization may be quite unrelated to the target model, i.e., model independent.
D.I.Y. F ILTER C OVARIANCES
Ultimately, the above methods for initializing convolutional ﬁlters via transfer are limited by the necessity of a trained network from which to form a ﬁlter distribution, which must be accessible at initialization.
We thus use observations on the structure of ﬁlter covariance matrices to construct our own covariance matrices from scratch.
Using our construction, we propose a depth-dependent but simple initialization strategy for convolutional ﬁlters that greatly outperforms previous techniques.
Visual observations.
Filter covariance matrices in pre-trained ConvMixers and ConvNeXts have a great deal of structure, which we observe across models with different patch sizes, architectures, and data sets; see Fig.
1 and 32 for examples.
In both the block and rearranged forms of the covariance matrices, we noticed clear repetitive structure, which led to an initial investigation on modeling covariances via Kronecker factorizations; see Appendix A for experimental results.
Beyond this, we ﬁrst note that the overall variance of ﬁlters tends to increase with depth, until breaking down towards the last layer.
Second, we note that the blocks of the covariances often have a static negative component in the center, with a dynamic positive component whose position mirrors that of the block itself.
Finally, the covariance of ﬁlter parameters is greater in their center, i.e., covariance matrices are at ﬁrst centrally-focused and become more diffuse with depth.
These observations agree with intuition about the structure of convolutional ﬁlters: most ﬁlters have the greatest weight towards their center, and their parameters are correlated with their neighbors.
Constructing covariances.
With these observations in mind, we propose a construction of covariance matrices.
We ﬁx the (odd) ﬁlter size k ∈ N + , let 1 ∈ R k × k be the all-ones matrix, and, as a building block for our initialization, use unnormalized Gaussian-like ﬁlters Z σ ∈ R k × k with a single variance parameter σ , deﬁned elementwise by
[Equation]
Such a construction produces ﬁlters similar to those observed in the blocks of the Layer #5 covariance matrix in Fig.
To capture the dynamic component that moves according to the position of its block, we deﬁne the block matrix C ∈ R k 2 × k 2 with k × k blocks by
[Equation]
where the Shift operation translates each element of the matrix i and j positions forward in their respective dimensions; see Appendix E for details.
We then deﬁne two additional components, both constructed from Gaussian ﬁlters: a static component S = 1 ⊗ Z σ ∈ R k 2 × k 2 and a blockwise mask component M =
Z σ ⊗ 1 ∈ R k 2 × k 2 , which encodes higher variance as pixels approach the center of the ﬁlter.
Using these components and our intuition, we ﬁrst consider ˆΣ =
M (cid:12) ( C − 12 S ) , where (cid:12) is an elementwise product.
While this adequately represents what we view to be the important structural components of ﬁlter covariance matrices, it does not satisfy the property
[Σ i,j ] (cid:96),m =
[Σ (cid:96),m ] i,j ( i.e., covariance matrices must be symmetric, accounting for our block representation).
Consequently, we instead calculate its symmetric part, using the notation as follows to denote a “block-transpose”:
[Equation]
Equivalently, this is the perfect shufﬂe permutation such that ( X ⊗ Y ) B =
Y ⊗ X with X, Y ∈ R k × k .
First, we note that C B = C due to the deﬁnition of the shift operation used in Eq.
5 (see Appendix E).
Then, noting that S B = M and M B = S by the previous rule, we deﬁne our construction
Σ =  
M (cid:12) (cid:32)
C −
S (cid:33) +
C (cid:12)
S  
[Figure]
V a r i a n
[Figure]
increases quadratically with depth according to a schedule which can be chosen through visual inspection of pre-trained models or through grid search.
Here we use the parameters
[Equation]
of Σ
to be the symmetric part of ˆΣ (where C, S, M are implicitly parameterized by the σ of Z σ ): Σ = 12 (ˆΣ + ˆΣ
[Equation]
= 12 (cid:2) M (cid:12) ( C
[Equation]
S )
+ ( M (cid:12) (
[Equation]
S
[Equation]
B (cid:3) (7) = 12 (cid:2)
[Equation]
(cid:12) (
[Equation] [Equation]
+ (
[Equation]
(cid:12)
[Equation] [Equation]
)) (cid:3) =
[Equation]
(cid:12) ( C
[Equation] [Equation]
+ S (cid:12)
( C − 12 M ) (8) =
[ M (cid:12) ( C − S )
+ S (cid:12) C ] .
While Σ is now symmetric (in the rearranged form of Eq. 1), it is not positive semi-deﬁnite, but can easily be projected to S k 2 + , as is often done automatically by multivariate Gaussian procedures.
We illustrate our construction in Fig. 5, and provide an implementation in Fig. 30.
Completing the initialization.
As explained in Fig.
1, we observed that in pre-trained models, the ﬁlters become more “diffuse” as depth increases; we capture this fact in our construction by increasing the parameter σ with depth according to a simple quadratic schedule; let d be the percentage depth, i.e., d = i − 1 D − 1 for the i th convolutional layer of a model with D total such layers.
Then for layer i , we parameterize our covariance construction by a variance schedule :
[Equation]
where σ 0 , v σ , a σ jointly describe how the covariance evolves with depth.
Then, for each layer i ∈ 1 , . . .
, D , we compute d = i − 1 D − 1 and initialize the ﬁlters as F i,j ∼ N (0 , Σ (cid:48)
σ ( d ) )
for j ∈ 1 , .
, H .
We illustrate our complete initialization scheme in Figure 6.
R ESULTS
In this section, we present the performance of our initialization within ConvMixer and ConvNeXt on
CIFAR-10 and ImageNet classiﬁcation, ﬁnding it to be highly effective, particularly for deep models with large ﬁlters.
Our new initialization overshadows our previous covariance transfer results.
Settings of initialization hyperparameters σ 0 , v σ , and a σ were found and ﬁxed for CIFARexperiments, while two such settings were used for ImageNet experiments.
Appendix C.1 contains full details on our (relatively small) hyperparameter searches and experimental setups, as well as empirical evidence that our method is robust to a large swath of hyperparameter settings.
Additional experiments on more datasets and baseline initializations may be found in Appendix B.
4. CIFAR-10 R ESULTS
Thawed ﬁlters.
In Fig.
3, we show that large-kernel models using our initialization (group E ) outperform those using uniform initialization (group A ), covariance transfer (groups B, C ), and even those directly initializing via learned ﬁlters (group D ).
For 2 × 2 -patch models (200 epochs), relative to uniform, our initialization causes up to a 1.1% increase in accuracy for ConvMixer-256/8, and up to 1.6% for ConvMixer-256/24.
The effect size increases with the the ﬁlter size, and is often more prominent for shorter training times.
Results are similar for 1 × 1 -patch models, but with a smaller increase for 7 × 7 ﬁlters ( 0 .
15% vs. 0 .
Our initialization has the same effects for ConvNeXt (Fig. 7).
However, our method works poorly for 3 × 3 ﬁlters, which we believe have fundamentally different structure than larger ﬁlters; this setting is better-served by our original covariance transfer techniques.
In addition to improving the ﬁnal accuracy, our initialization also drastically speeds up convergence of models with thawed ﬁlters (see Fig.
8), particularly for deeper models.
A ConvMixer-256/16 with 2 × 2 patches using our initialization reaches 90% accuracy in approximately 50% fewer epochs than uniform initialization, and around 25% fewer than direct learned ﬁlter transfer.
The same occurs, albeit to a lesser extent, for 1 × 1 patches—but note that for this experiment we used the same initialization parameters for both patch sizes to demonstrate robustness to parameter choices.
[Figure]
Frozen ﬁlters.
Our initialization leads to even more surprising effects in models with frozen ﬁlters.
In Fig.
3, we see that frozenﬁlter 2 × 2 -patch models using our initialization often exceed the performance of their uniform, thawed-ﬁlter counterparts by a signiﬁcant margin of 0.4% – 2.0% for 200 epochs, and an even larger margin of 0.6% – 5.0% for 20 epochs (for large ﬁlters).
That is, group E (frozen ) consistently outperforms groups A-D (thawed) , and in some cases even group E (thawed) , especially for the deeper 24-layer ConvMixer.
While this effect breaks down for 1 × 1 patch models, such frozen-ﬁlter models still see accuracy increases of 0.6%– 3.5%.
However, the effect can still be seen for 1 × 1 -patch ConvNeXts (Fig. 7).
Also note that frozenﬁlter models can be up to 40% faster to train (see Fig. 2), and may be more robust (Cazenavette et al.).
[Figure]
4. I MAGE N ET E XPERIMENTS
Our initialization performs extremely well on CIFAR-10 for large-kernel models, almost always helping and rarely hurting.
Here, we explore if the performance gains transfer to larger-scale ImageNet models.
We observe in Fig.
32, Appendix F that ﬁlter covariances for such models have ﬁnergrained structure than models trained on CIFAR-10, perhaps due to using larger patches.
Nonetheless, our initialization leads to quite encouraging improvements in this setting.
Experiment design.
We used the “A1” training recipe from Wightman et al.
(2021), with crossentropy loss, fewer epochs, and a triangular LR schedule as in Trockman & Kolter (2022).
We primarily demonstrate our initialization for 50-epoch training, as the difference between initializations is most pronounced for lower training times.
We also present two full, practical-scale 150-epoch experiments on large models.
We also included covariance transfer experiments in Appendix F.
[Table]
Thawed ﬁlters.
On models trained for 50 epochs with thawed ﬁlters, our initialization improves the ﬁnal accuracy by 0 .
4% − 3 .
8% (see Table 1).
For the relatively-shallow ConvMixer-512/12 on which we tuned the initialization parameters, we see a gain of just 0 .
4% ; however, when increasing the depth to 24 or 32, we see larger gains of 1 . 8% and 3 .
8% , respectively, and a similar trend among the wider ConvMixer-1024 models.
Our initialization also boosts the accuracy of the 18layer ConvNeXt-Tiny from 76 . 0% to 77 .
1% ; however, it decreased the accuracy of the smaller, 12-layer ConvNeXt-Atto.
This is perhaps unsurprising, seeing as our initialization seems to be more helpful for deep models, and we used hyperparameters optimized for a model with a substantially different patch and ﬁlter size.
Our initialization is also beneﬁcial for more-practical 150-epoch training, boosting accuracy by around 0 .
1% on both ConvMixer-1536/24 and ConvNeXt-Tiny (see Table 1, bottom rows).
While the effect is small, this demonstrates that our initialization is still helpful even for longer training times and very wide models.
We expect that within deeper models and with slightly more parameter tuning, our initialization could lead to still larger gains in full-scale ImageNet training.
Frozen ﬁlters.
Our initialization is extremely helpful for models with frozen ﬁlters.
Using our initialization, the difference between thawed and frozen-ﬁlter models decreases with increasing depth, i.e., it leads to 2 − 11% improvements over models with frozen, uniformly-initialized ﬁlters.
For
ConvMixer-1024/32, the accuracy improves from 64 .
9% to 73 .
1% , which is over 1% better than the corresponding thawed, uniformly-initialized model , and only 2% from the best result using our initialization.
This mirrors the effects we saw for deeper models on our earlier CIFAR-10 experiments.
We see a similar effect for ConvNeXt-Tiny, with the frozen version using our initialization achieving 75 .
2% accuracy vs. the thawed 76 . 0% .
In other words, our initialization so effectively captures the structure of convolutional ﬁlters that it is hardly necessary to train them after initialization; one beneﬁt of this is that it substantially speeds up training for large-ﬁlter convolutions.
C ONCLUSION
In this paper, we proposed a simple, closed-form, and learning-free initialization scheme for large depthwise convolutional ﬁlters.
Models using our initialization typically reach higher accuracies more quickly than uniformly-initialized models.
We also demonstrated that our random initialization of convolutional ﬁlters is so effective, that in many cases, networks perform nearly as well (or even better) if the resulting ﬁlters do not receive gradient updates during training.
Moreover, like the standard uniform initializations generally used in neural networks, our technique merely samples from a particular statistical distribution, and it is thus almost completely computationally free.
In summary, our initialization technique for the increasingly-popular large-kernel depthwise convolution operation almost always helps, rarely hurts, and is also free.
A A DDITIONAL CIFAR R ESULTS
[Figure]
[Figure] [Figure]
[Figure]
Covariance structure.
As a ﬁrst step towards modeling the structure of ﬁlter covariances, we replaced covariances with their Kroneckerfactorized counterparts using the rearranged form of the covariance matrix deﬁned in Eq.
(1), i.e., Σ =
A ⊗ A where A ∈ S k
+ .
Surprisingly, this slightly improved performance over unfactorized covariance transfer (see Fig. 12), suggesting that ﬁlter covariances are not only eminently transferrable for initialization, but that their core structure may be simpler than meets the eye.
Kronecker factorizations were computed via gradient descent minimizing the mean squared error.
Empirical Covariance
Our Covariance
Layer #1 of
Learned Filters
Sampled Filters Sampled Filters
Empirical Covariance
Our Covariance
Layer #4 of
Learned Filters
Sampled Filters Sampled Filters
Empirical Covariance
Our Covariance
Layer #8 of
Learned Filters
Sampled Filters Sampled Filters
A. A DDITIONAL E XPERIMENT D ETAILS
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
B A DDITIONAL B ASELINES AND D ATASETS
[Figure]
r a c y (
[Figure]
C
D E
A B C D E A
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
10 and CIFAR-100.
That is, one can often train models for fewer epochs when using our initialization to achieve results comparable to those from using other initialization techniques
[Figure]
C H YPERPARAMETER G RID S EARCHES & E XPERIMENTAL S ETUP
CIFAR-10 hyperparameter search.
We chose an initial setting of our method’s three hyperparameters via visual inspection, and then reﬁned them via small-scale grid searches.
For CIFARexperiments, we searched over parameters for ConvMixer-256/8 with frozen 9 × 9 ﬁlters trained for 20 epochs, and chose σ 0 = .
08 , v σ = .
37 , a σ = 2 .
9 for 2 × 2 -patch models, and found the optimal parameters for 1 × 1 -patch models to be approximately doubled.
However, note that our initialization is quite robust to different parameter settings, with the difference from our doubling choice being less than 0 .
1% (see Figure 23).
We used the same parameters across all kernel sizes, as well as for ConvNeXt, a choice which is likely sub-optimal; our search only serves as a rough heuristic.
ImageNet-1k hyperparameter search.
We did a small grid search using a ConvMixer-512/ with 14 × 14 patches and 9 × 9 ﬁlters trained for 10 epochs on ImageNet-1k (see Appendix F), from which we chose two candidate settings: σ 0 = .
15 , v σ = .
5 , a σ = .
25 for frozen-ﬁlter models and
σ 0 = .
15 , v σ = 0 .
25 , a σ = 1 . 0 for thawed models.
We use these parameters for all the
ImageNet experiments, even for models with different patch and kernel sizes ( e.g., ConvNeXt).
This demonstrates that hyperparameter tuning is optional for our technique; its transferability is not surprising given our results in Sec. 2.
C. CIFAR-10 G RID S EARCHES
[Table]
Acceleration: 8.00 (Patch Size 2 x 2) 0.
[Table] [Table]
Acceleration
[Table]
1.44 (Patch Size 1 x
[Table]
Acceleration: 8.00 (Patch Size 1 x 1) 0.
[Figure] [Figure]
[Table]
Acceleration: 8.00 (Patch Size 1 x 1) 0.
[Table]
Acceleration: 8.00 (Patch Size 1 x 1) 0.
[Figure] [Figure]
C.
I MAGE N ET G RID S EARCHES
[Table]
[Figure]
r
[Figure]
n c e
[Figure]
[Figure]
D I MPLEMENTATIONS D. T ERSE IMPLEMENTATION
[Figure]
[Figure]
wconv.data = torch.tensor(filters.reshape(c,1,ks,ks),dtype=wconv.dtype,device=wconv.device)
# Find depthwise convolutional layersconvs =
[x for x in model.modules() if isinstance(x, nn.Conv2d)
\and len(x.weight.shape) =
= 4 and x.weight.shape[1]
== 1] # Initialize them according to variance schedulefor i, conv in enumerate(convs):
Initialize(conv.weight, i / (len(convs) - 1), 0.16, 0.32, 2.88)
E S HIFT F UNCTION D EFINITION & P ROOF
For a given matrix Z ∈ R k × k (say, a Gaussian kernel centered at 0, 0—the top left of the ﬁlter), we assume the shift operator is deﬁned as follows:
[Equation]
Then, if
[Equation]
12) and the operation ( . )
B is deﬁned by
[Equation]
13) and [ C i
[Equation] [Equation]
this shows that [ C i,j ] (cid:96),m =
[ C (cid:96),m ] i,j for all 1 ≤ i, j, (cid:96), m ≤ k (i.e., C is “block-symmetric”), which shows C = C B .
[Figure]
A DDITIONAL I MAGE N ET E XPERIMENTS
tend to have more struc
, including a “checkerboard” pattern each
[Figure]
[Table]
N o r m a li z e d P e r - B
l
o
c k
MAGE
[Table] [Table]
[Table] [Table]
the same arch 33.
Stats from 1 / 8 th -width arch 29.
Filters transferred from same arch 31.
Our init (.15, .3, .5) 37.
Our init (.15, .5, .25) 31.
Our init (.15, .25, 1.0) 38.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
G A DDITIONAL CIFAR-10 T ABLES
[Figure]
T a b l e 10 :
C
I F
A R10 r e s u lt s f o r
C onv
M i x e r256 /w it h p a t
c h s i ze 2 .
B o l d d e no t e s t h e h
i gh
e s t p e r g r oup , a nd b l u e b o l d d e no t e s t
h e s ec
ond h
i gh e s t .
T HA
W E D
F R O ZE
N
F ilt .
S i ze # E p s
U n i f
o r m
C ov .
t r a n s f e r
C ov .
t r a n s f e r ( 18 w i d t h )
D i r ec t t r a n s f e r
O u r
i n it
U n i f
o r m
[Table]
t r a
[Table]
r
[Table] [Table]
a
[Table] [Table]
±
[Table] [Table]
±
[Table] [Table]
±
[Table] [Table]
T a b l e 11 :
C
I F
A R10 r e s u lt s f o r
C onv
M i x e r256 / w it h p a t
c h s i ze 2 .
B o l d d e no t e s t h e h
i gh
e s t p e r g r oup , a nd b l u e b o l d d e no t e s t
h e s ec
ond h
i gh e s t .
T HA
W E D
F R O ZE
N
F ilt .
S i ze # E p s
U n i f
o r m
C ov .
t r a n s f e r
C ov .
t r a n s f e r ( 18 w i d t h )
D i r ec t t r a n s f e r
O u r
i n it
U n i f
o r m
[Table]
t r a n s f
[Table]
r
[Table]
t r a
[Table]
s
[Table]
r (
[Table]
i
[Table] [Table]
[Table] [Table]
±
[Table] [Table]
±
[Table]
±
[Table]
T a b l e 12 :
C
I F
A R10 r e s u lt s f o r
C onv
M i x e r256 /w it h p a t
c h s i ze 1 .
B o l d d e no t e s t h e h
i gh
e s t p e r g r oup , a nd b l u e b o l d d e no t e s t
h e s ec
ond h
i gh e s t .
T HA
W E D
F R O ZE
N
F ilt .
S i ze # E p s
U n i f
o r m
C ov .
t r a n s f e r
C ov .
t r a n s f e r ( 18 w i d t h )
[Table]
i
[Table]
t t r a
[Table]
s f e r
[Table]
u r i n it
[Table]
n
[Table]
T a b l e 1 3 :
C
I F
A R10 r e s u lt s f o r
C onv
N e X t - a tt o w it h p
a t c h s i ze 1 .
B o l d d e no t e s t h e h
i gh
e s t p e r g r oup , a nd b l u e b o l d d e no t e s t
h e s ec
ond h
i gh e s t .
T HA
W E D
F R O ZE
N
F ilt .
S i ze # E p s
U n i f
o r m
[Table]
ov .
t r a n s f e r
C ov .
t r a n s f e r (
[Table]
i d t h )
[Table]
i
[Table]
r i n
[Table]
± .91 .±
.92 .±
.87 .±
.89 .± .89 .±
.90 .±
.92 .± .
T a b l e 1 4 :
C
I F
A R10 r e s u lt s f o r
C onv
N e X t - a tt o w it h p
a t c h s i ze 2 .
B o l d d e no t e s t h e h
i gh
e s t p e r g r oup , a nd b l u e b o l d d e no t e s t
h e s ec
ond h
i gh e s t .
T HA
W E D
F R O ZE
N
F ilt .
S i ze # E p s
U n i f
o r m
C ov .
t r a n s f e r
C ov .
t r a n s f e r ( 18 w i d t h )
D i r ec t t r a n s f e r
O u r
i n it
U n i f
o r m
C ov .
t r a n s f e r
C ov .
t r a n s f e r ( 18 w i d t h )
D i r ec t t r a n s f e r
O u r i n it  72 .± .82 .± .
1 .± .83 .±
.81 .±
.72 .±
.80 .±
.79 .±
.83 .±
.79 .± .
83 .±
.86 .± .
6 .± .86 .±
.85 .±
.77 .±
[Table]