Patches Are All You Need?
Anonymous authors Paper under double-blind review
Abstract
Although convolutional networks have been the dominant architecture for vision tasks for many years, recent experiments have shown that Transformer-based models, most notably the Vision Transformer (ViT), may exceed their performance in some settings.
However, due to the quadratic runtime of the self-attention layers in Transformers, ViTs require the use of patch embeddings, which group together small regions of the image into single input features, in order to be applied to larger image sizes.
This raises a question: Is the performance of ViTs due to the inherently-more-powerful Transformer architecture, or is it at least partly due to using patches as the input representation?
In this paper, we present some evidence for the latter: speciﬁcally, we propose the ConvMixer, an extremely simple model that is similar in spirit to the ViT and the even-more-basic MLP-Mixer in that it operates directly on patches as input, separates the mixing of spatial and channel dimensions, and maintains equal size and resolution throughout the network.
In contrast, however, the ConvMixer uses only standard convolutions to achieve the mixing steps.
Despite its simplicity, we show that the ConvMixer outperforms the
ViT, MLP-Mixer, and some of their variants for similar parameter counts and data set sizes, in addition to outperforming classical vision models such as the ResNet.
Our code is available at https://github.com/tmp-iclr/convmixer .
Introduction
[Figure]
For many years, convolutional neural networks have been the dominant architecture for deep learning systems applied to computer vision tasks.
But recently, architectures based upon Transformer models, e.g. , the so-called Vision
Transformer architecture (Dosovitskiy et al., 2020), have demonstrated compelling performance in many of these tasks, often outperforming classical convolutional architectures, especially for large data sets.
An understandable assumption, then, is that it is only a matter of time before Transformers become the dominant architecture for vision domains, just as they have for language processing.
In order to apply Transformers to images, however, the representation had to be changed: because the computational cost of the self-attention layers used in Transformers would scale quadratically with the number of pixels per image if applied naively at the per-pixel level, the compromise was to ﬁrst split the image into multiple “patches”, linearly embed them, and then apply the transformer directly to this collection of patches.
In this work, we explore the question of whether, fundamentally, the strong performance of vision transformers may result more from this patch-based representation than from the Transformer architecture itself.
We develop a very simple convolutional architecture which we dub the “ConvMixer” due to its similarity to the recently-proposed MLP-Mixer (Tolstikhin et al., 2021).
This architecture is similar to the Vision Transformer (and MLP-Mixer) in many respects: it directly operates on patches, it maintains an equal-resolution-and-size representation throughout all layers, it does no downsampling of the representation at successive layers, and it separates “channel-wise mixing”
Residual connection +
D ep t h
w i s e
C on v o l u t i on
G E L U
B a t c h N o r m
G E L U
B a t c h N o r m
P o i n t w i s e
C on v o l u t i on
ConvMixer Layer depth
G l oba l A v e r age P oo li ng
F u ll y - C onne c t ed
C l a ss
G E L U
B a t c h N o r m
P a t c h
E m bedd
i ng in PyTorch; see Appendix D for more implementations.
from the “spatial mixing” of information.
But unlike the Vision Transformer and MLP-Mixer, our architecture does all these operations via only standard convolutions.
The chief result we show in this paper is that this ConvMixer architecture, despite its extreme simplicity (it can be implemented in ≈ 6 lines of dense PyTorch code), outperforms both “standard” computer vision models such as ResNets of similar parameter counts and some corresponding Vision
Transformer and MLP-Mixer variants, even with a slate of additions intended to make those architectures more performant on smaller data sets.
Importantly, this is despite the fact that we did not design our experiments to maximize accuracy nor speed , in contrast to the models we compared against.
Our results suggest that, at least to some extent, the patch representation itself may be a critical component to the “superior” performance of newer architectures like Vision Transformers.
While these results are naturally just a snapshot, and more experiments are required to exactly disentangle the eﬀect of patch embeddings from other factors, we believe that this provides a strong “convolutionalbut-patch-based” baseline to compare against for more advanced architectures in the future.
A Simple Model: ConvMixer
Our model, dubbed ConvMixer , consists of a patch embedding layer followed by repeated applications of a simple fully-convolutional block.
We maintain the spatial structure of the patch embeddings, as illustrated in Fig.
Patch embeddings with patch size p and embedding dimension h can be implemented as convolution with c in input channels, h output channels, kernel size p , and stride p :
[Equation]
The ConvMixer block itself consists of depthwise convolution ( i.e. , grouped convolution with groups equal to the number of channels, h ) followed by pointwise ( i.e. , kernel size 1 × 1 ) convolution.
As we will explain in Sec. 3, ConvMixers work best with unusually large kernel sizes for the depthwise convolution.
Each of the convolutions is followed by an activation and post-activation BatchNorm:
[Equation]
After many applications of this block, we perform global pooling to get a feature vector of size h , which we pass to a softmax classiﬁer.
See Fig.
3 for an implementation of ConvMixer in PyTorch.
Design parameters.
An instantiation of ConvMixer depends on four parameters: (1) the “width” or hidden dimension h ( i.e. , the dimension of the patch embeddings), (2) the depth d , or the number of repetitions of the ConvMixer layer, (3) the patch size p which controls the internal resolution of the
Current “Most Interesting” ConvMixer Conﬁgurations vs. Other Simple Models
[Table]
model, (4) the kernel size k of the depthwise convolutional layer.
We name ConvMixers after their hidden dimension and depth, like ConvMixer- h/d .
We refer to the original input size n divided by the patch size p as the internal resolution ; note, however, that ConvMixers support variable-sized inputs.
Motivation.
Our architecture is based on the idea of mixing , as in Tolstikhin et al.
In particular, we chose depthwise convolution to mix spatial locations and pointwise convolution to mix channel locations .
A key idea from previous work is that MLPs and self-attention can mix distant spatial locations, i.e. , they can have an arbitrarily large receptive ﬁeld.
Consequently, we used convolutions with an unusually large kernel size to mix distant spatial locations.
While self-attention and MLPs are theoretically more ﬂexible, allowing for large receptive ﬁelds and content-aware behavior, the inductive bias of convolution is well-suited to vision tasks and leads to high data eﬃciency.
By using such a standard operation, we also get a glimpse into the eﬀect of the patch representation itself in contrast to the conventional pyramid-shaped, progressivelydownsampling design of convolutional networks.
Experiments
Training setup.
We primarily evaluate ConvMixers on ImageNet-1k classiﬁcation without any pretraining or additional data.
We added ConvMixer to the timm framework (Wightman, 2019) and trained it with nearly-standard settings: we used RandAugment (Cubuk et al., 2020), mixup (Zhang et al., 2017), CutMix (Yun et al., 2019), random erasing (Zhong et al., 2020), and gradient norm clipping in addition to default timm augmentation.
We used the AdamW (Loshchilov & Hutter, 2018) optimizer and a simple triangular learning rate schedule.
Due to limited compute, we did absolutely no hyperparameter tuning on ImageNet and trained for fewer epochs than competitors.
Consequently, our models could be over- or under-regularized, and the accuracies we report likely underestimate the capabilities of our model.
Results.
A ConvMixer-1536/20 with 52M parameters can achieve 81.4% top-1 accuracy on ImageNet, and a ConvMixer-768/32 with 21M parameters 80.2% (see Table 1).
Wider ConvMixers seem to converge in fewer epochs, but are memory- and compute-hungry.
They also work best with large kernel sizes: ConvMixer-1536/20 lost ≈ 1% accuracy when reducing the kernel size from k =to k = 3 (we discuss kernel sizes more in Appendix A & B).
ConvMixers with smaller patches are substantially better in our experiments, similarly to Sandler et al.
(2019); we believe larger patches require deeper ConvMixers.
With everything held equal except increasing the patch size from 7 to 14, ConvMixer-1536/20 achieves 78.9% top-1 accuracy but is around 4 × faster.
We trained one model with ReLU to demonstrate that GELU (Hendrycks & Gimpel, 2016), which is popular in recent isotropic models, isn’t necessary.
Comparisons.
Our model and ImageNet1k-only training setup closely resemble that of recent patchbased models like DeiT (Touvron et al., 2020). Due to ConvMixer’s simplicity, we focus on comparing to only the most basic isotropic patch-based architectures adapted to the ImageNet-1k setting, namely DeiT and ResMLP.
Attempting a fair comparison with a standard baseline, we trained ResNets using exactly the same parameters as ConvMixers; while this choice of parameters is suboptimal (Wightman et al., 2021), it is likely also suboptimal for ConvMixers, since we did no hyperparameter tuning .
Looking at Table 1 and Fig.
1, ConvMixers achieve competitive accuracies for a given parameter budget:
ConvMixer-1536/20 outperforms both ResNet-152 and ResMLP-B24 despite having substantially fewer parameters and is competitive with DeiT-B. ConvMixer-768/32 uses just a third of the parameters of ResNet-152, but is similarly accurate. Note that unlike ConvMixer, the DeiT and
ResMLP results involved hyperparameter tuning, and when substantial resources are dedicated to tuning ResNets, including training for twice as many epochs, they only outperform an equivalently-sized
ConvMixer by ≈ 0 . 2% (Wightman et al., 2021). However, ConvMixers are substantially slower at inference than the competitors, likely due to their smaller patch size; hyperparameter tuning and optimizations could narrow this gap.
For more discussion and comparisons, see Table 2 and
Appendix A.
CIFAR-10 Experiments.
We also performed smaller-scale experiments on CIFAR-10, where ConvMixers achieve over 96% accuracy with as few as 0.7M parameters, demonstrating the data eﬃciency of the convolutional inductive bias.
Details of these experiments are presented in Appendix B.
Related Work
Isotropic architectures.
Vision transformers have inspired a new paradigm of “isotropic” architectures, i.e. , those with equal size and shape throughout the network, which use patch embeddings for the ﬁrst layer.
These models look similar to repeated transformer-encoder blocks (Vaswani et al., 2017) with diﬀerent operations replacing the self-attention and MLP operations.
For example, MLP-Mixer (Tolstikhin et al., 2021) replaces them both with MLPs applied across diﬀerent dimensions ( i.e. , spatial and channel location mixing); ResMLP (Touvron et al., 2021a) is a data-eﬃcient variation on this theme.
CycleMLP (Chen et al., 2021), gMLP
(Liu et al., 2021a), and vision permutator (Hou et al., 2021), replace one or both blocks with various novel operations.
These are all quite performant, which is typically attributed to the novel choice of operations.
In contrast, Melas-Kyriazi (2021) proposed an MLP-based isotropic vision model, and also hypothesized patch embeddings could be behind its performance.
ResMLP tried replacing its linear interaction layer with (small-kernel) convolution and achieved good performance, but kept its MLP-based cross-channel layer and did not explore convolutions further.
As our investigation of ConvMixers suggests, these works may conﬂate the eﬀect of the new operations (like self-attention and MLPs) with the eﬀect of the use of patch embeddings and the resulting isotropic architecture.
A study predating vision transformers investigates isotropic (or “isometric”)
MobileNets (Sandler et al., 2019), and even implements patch embeddings under another name.
Their architecture simply repeats an isotropic MobileNetv3 block.
They identify a tradeoﬀ between patch size and accuracy that matches our experience, and train similarly performant models (see Appendix A, Table 2).
However, their block is substantially more complex than ours; simplicity and motivation sets our work apart.
Patches aren’t all you need.
Several papers have increased vision transformer performance by replacing standard patch embeddings with a diﬀerent stem: Xiao et al.
(2021) and Yuan et al.
(2021a) use a standard convolutional stem, while Yuan et al.
(2021b) repeatedly combines nearby patch embeddings.
However, this conﬂates the eﬀect of using patch embeddings with the eﬀect of adding convolution or similar inductive biases e.g. , locality.
We attempt to focus on the use of patches.
CNNs meet ViTs.
Many eﬀorts have been made to incorporate features of convolutional networks into vision transformers and vice versa.
Self-attention can emulate convolution (Cordonnier et al., 2019) and can be initialized or regularized to be like it (d’Ascoli et al., 2021); other works simply add convolution operations to transformers (Dai et al., 2021; Guo et al., 2021), or include downsampling to be more like traditional pyramid-shaped convolutional networks (Wang et al., 2021).
Conversely, self-attention or attention-like operations can supplement or replace convolution in ResNet-style models (Bello et al., 2019;
Ramachandran et al., 2019; Bello, 2021).
While all of these attempts have been successful in one way or another, they are orthogonal to this work, which aims to emphasize the eﬀect of the architecture common to most ViTs by showcasing it with a less-expressive operation.
Conclusion
We presented ConvMixers, an extremely simple class of models that independently mixes the spatial and channel locations of patch embeddings using only standard convolutions.
We also highlighted that using large kernel sizes, inspired by the large receptive ﬁelds of ViTs and MLP-Mixers, provides a substantial performance boost.
While neither our model nor our experiments were designed to maximize accuracy or speed, i.e. , we did not search for good hyperparameters, ConvMixers outperform the Vision Transformer and MLP-Mixer, and are competitive with ResNets, DeiTs, and ResMLPs.
We provided evidence that the increasingly common “isotropic” architecture with a simple patch embedding stem is itself a powerful template for deep learning.
Patch embeddings allow all the downsampling to happen at once, immediately decreasing the internal resolution and thus increasing the eﬀective receptive ﬁeld size, making it easier to mix distant spatial information.
Our title, while an exaggeration, points out that attention isn’t the only export from language processing into computer vision: tokenizing inputs, i.e. , using patch embeddings, is also a powerful and important takeaway.
While our model is not state-of-the-art, we ﬁnd its simple patch-mixing design to be compelling.
We hope that ConvMixers can serve as a baseline for future patch-based architectures with novel operations, or that they can provide a basic template for new conceptually simple and performant models.
Future work.
We are optimistic that a deeper ConvMixer with larger patches could reach a desirable tradeoﬀ between accuracy, parameters, and throughput after longer training and more regularization and hyperparameter tuning, similarly to how Wightman et al.
(2021) enhanced ResNet performance through carefully-designed training regimens.
Low-level optimization of large-kernel depthwise convolution could substantially increase throughput, and small enhancements to our architecture like the addition of bottlenecks or a more expressive classiﬁer could trade simplicity for performance.
Due to its large internal resolution and isotropic design, ConvMixer may be especially well-suited for semantic segmentation, and it would be useful to run experiments on this task with a ConvMixer-like model and on other tasks such as object detection.
More experiments could be designed to more clearly extricate the eﬀect of patch embeddings from other architectural choices.
In particular, for a more in-depth comparison to ViTs and MLP-Mixers, which excel when trained on very large data sets, it is important to investigate the performance of ConvMixers in the regime of large-scale pre-training.
A note on paper length.
Expecting more text in this paper?
Wondering if it’s a workshop paper we hastily submitted to ICLR?
No.
This paper presents a simple idea, one where we genuinely believe that a short paper presentation is more eﬀective.
Do we really need exactly 8 (now 9? 10?)
pages to describe every machine learning architecture and algorithm in existence?
We proposed an incredibly simple architecture and made a very simple point that we think is worth more discussion: patches work well in convolutional architectures.
We think that four and a half pages is more than enough space for this.
The details of the experiments and architectures are in the appendix for those who want to read through it all.
A Comparison to other models
Comparison with
[Table]
[Table] G 81. ConvMixer-1536/   49. G 80.
ConvMixer-
[Table]
G 78.
ConvMixer-768/32 •  21.
R 80. ConvMixer [Table] G 79.
ConvMixer [Table] G 77. ConvMixer-512/   5.
G 73.
ConvMixer-512/12 •  4.
G 72.
ConvMixer
[Table]
R 74.
ConvMixer-1024/20 •  24.
G 76.
ResNet
[Table]
R 79.
ResNet
[Table]
[Table]
R 78.
ResNet- – 25.
R 76.
DeiT-B † – 86.
G – –
DeiT-S † – 22.
G – –
DeiT-Ti † – 5.
G – –
DeiT-B • –
G 81.
DeiT-S • –
G 79.
DeiT-Ti • – 5.
G 72.
ResMLP-S12/8 • – 22.
G 79.
ResMLP-B24/8 • –
G 81.
ResMLP-B  –
G 81.
Swin-S • – G 83. Swin-T • – G 81.
ViT-B/16 • –
G 77.
Mixer-B/16 • –
G 76.
Isotropic MobileNetv3 •   – R – 80. Isotropic MobileNetv3 •   – R – 77.
Experiment overview.
We did not design our experiments to maximize accuracy: We chose “common sense” parameters for timm and its augmentation settings, found that it worked well for a ConvMixer-1024/12, and stuck with them for the proceeding experiments.
We admit this is not an optimal strategy, however, we were aware from our early experiments on CIFAR-10 that results seemed robust to various small changes.
We did not have access to suﬃcient compute to attempt to tune hyperparameters for each model: e.g. , larger ConvMixers could probably beneﬁt from more regularization than we chose, and smaller ones from less regularization.
Keeping the parameters the same across ConvMixer instances seemed more reasonable than guessing for each.
However, to some extent, we changed the number of epochs per model: for earlier experiments, we merely wanted a “proof of concept”, and used only 90–100 epochs.
Once we saw potential, we increased this to 150 epochs and trained some larger models, namely ConvMixer-1024/20 with p = 14 patches and ConvMixer-1536/20 with p = 7 patches.
Then, believing that we should explore deeper-but-less-wide ConvMixers, and knowing from CIFAR-10 that the deeper models converged more slowly, we trained ConvMixer-768/32s with p = 14 and p = 7 for 300 epochs.
Of course, training time was a consideration:
ConvMixer-1536/20 took about 9 days to train (on 10 × RTX8000s) 150 epochs, and ConvMixer-768/32 is over twice as fast, making 300 epochs more feasible.
If anything, we believe that in the worst case, the lack of parameter tuning in our experiments resulted in underestimating the accuracies of ConvMixers.
Further, due to our limited compute and the fact that large models (particularly ConvMixers) are expensive to train on large data sets, we generally trained our models for fewer epochs than competition like DeiT and ResMLP (see Table 2).
A note on throughput.
We measured throughput using batches of 64 images in half precision on a single RTX
GPU, averaged over 20 such batches.
In particular, we measured CUDA execution time rather than “wall-clock” time.
We noticed discrepancies in the relative throughputs of models, e.g. , Touvron et al.
(2020) reports that ResNet-152 is 2 × faster than DeiT-B, but our measurements show that it is only 1 . 25 × faster.
We therefore speculate that our throughputs may underestimate the performance of ResNets and ConvMixers relative to the transformers.
The diﬀerence may be due to using RTX8000 rather than V100 GPUs, or other low-level diﬀerences.
Our throughputs were similar for batch sizes 32 and 128.
ResNets.
As a simple baseline to which to compare ConvMixers, we trained three standard ResNets using exactly the same training setup and parameters as ConvMixer-1536/20.
Despite having fewer parameters and being architecturally much simpler, ConvMixers substantially outperform these
ResNets in terms of accuracy.
A possible confounding factor is that ConvMixers use GELU, which may boost performance, while ResNets use ReLU.
In an attempt to rule out this confound, we used
ReLU in a later ConvMixer-768/32 experiment and found that it still achieved competitive accuracy.
We also note that the choice of ReLU vs. GELU was not important on CIFAR-10 experiments (see
Table 3).
However, ConvMixers do have substantially less throughput.
DeiTs.
We believe that DeiT is the most reasonable comparison in terms of vision transformers: It only adds additional regularization, as opposed to architectural additions in the case of CaiT (Touvron et al., 2021b), and is then essentially a “vanilla” ViT modulo the distillation token (we don’t consider distilled architectures).
In terms of a ﬁxed parameter budget, ConvMixers generally outperform
DeiTs.
For example, ConvMixer-1536/20 is only 0.43% less accurate than DeiT-B despite having over 30M fewer parameters; ConvMixer-768/32 is 0.36% more accurate than DeiT-S despite having 0.9M fewer parameters; and ConvMixer-512/16 is 0.39% more accurate than DeiT-Ti for nearly the same number of parameters.
Admittedly, none of the ConvMixers are very competitive in terms of throughput, with the closest being the ConvMixer-512/16 which is 5 × slower than DeiT-Ti.
A confounding factor is the diﬀerence in patch size between DeiT and ConvMixer; DeiT uses p =while ConvMixer uses p = 7 .
This means DeiT is substantially faster.
However, ConvMixers using larger patches are not as competitive.
While we were not able to train DeiTs with larger patch sizes, it is possible that they would outperform ConvMixers on the parameter count vs. accuracy curve; however, we tested their throughput for p = 7 , and they are even slower than ConvMixers.
Given the diﬀerence between convolution and self-attention, we are not sure it is salient to control for patch size diﬀerences.
DeiTs were subject to more hyperparameter tuning than ConvMixers, as well as longer training times.
They also used stochastic depth while we did not, which can in some cases contribute percent diﬀerences in model accuracy (Touvron et al., 2021a).
It is therefore possible that further hyperparameter tuning and more epochs for ConvMixers could close the gap between the two architectures for large patches, e.g. , p = 16 .
ResMLPs.
Similarly to DeiT for ViT, we believe that ResMLP is the most relevant MLP-Mixer variant to compare against.
Unlike DeiT, we can compare against instances of ResMLP with similar patch size: ResMLP-B24/8 has p = 8 patches, and underperforms ConvMixer-1536/20 by 0.37%, despite having over twice the number of parameters; it also has similarly low throughput.
ConvMixer-768/ also outperforms ResMLP-S12/8 for millions fewer parameters, but 3 × less throughput.
ResMLP did not signiﬁcantly improve in terms of accuracy for halving the patch size from 16 to 8, which shows that smaller patches do not always lead to better accuracy for a ﬁxed architecture and regularization strategy ( e.g. , training a p = 8 DeiT may be challenging). Swin Transformers.
While we intend to focus on the most basic isotropic, patch-based architectures for fair comparisons with ConvMixer, it is also interesting to compare to a more complicated model that is closer to state-of-the-art.
For a similar parameter budget, ConvMixer is around 1.2-1.6% less accurate than the Swin Transformer, while also being 4-6 × slower.
However, considering we did not attempt to tune or optimize our model in any way, we ﬁnd it surprising that an exceedingly simple patch-based model that uses only plain convolution does not lag too far behind Swin Transformer.
Isotropic MobileNets.
These models are closest in design to ours, despite using a repeating block that is substantially more complex than the ConvMixer one.
Despite this, for a similar number of parameters, we can get similar performance.
Notably, isotropic MobileNets seem to suﬀer less from larger patch sizes than ConvMixers, which makes us optimistic that suﬃcient parameter tuning could lead to more performant large-patch ConvMixers.
Other models.
We included ViT and MLP-Mixer instances in our table, though they are not competitive with ConvMixer, DeiT, or ResMLP, even though MLP-Mixer has comparable regularization to
ConvMixer.
That is, ConvMixer seems to outperform MLP-Mixer and ViT, while being closer to complexity to them in terms of design and training regime than the other competitors, DeiT and ResMLP.
Kernel size.
While we found some evidence that larger kernels are better on CIFAR-10, we wanted to see if this ﬁnding transferred to ImageNet.
Consequently, we trained our best-performing model,
ConvMixer-1536/20, with kernel size k = 3 rather than k = 9 .
This resulted in a decrease of 0.94% top-1 accuracy, which we believe is quite signiﬁcant relative to the mere 2.2M additional parameters.
However, k = 3 is substantially faster than k = 9 for spatial-domain convolution; we speculate that low-level optimizations could close the performance gap to some extent, e.g. , by using implicit instead of explicit padding.
Since large-kernel convolutions throughout a model are unconventional, there has likely been low demand for such optimizations.
B Experiments on CIFAR-
Residual connections.
We experimented with leaving out one, the other, or both residual connections before settling on the current conﬁguration, and consequently chose to leave out the second residual connection.
Our baseline model without the connection achieves 95.88% accuracy, while including the connection reduces it to 94.78%.
Surprisingly, we see only a 0 .
31% decrease in accuracy for removing all residual connections .
We acknowledge that these ﬁndings for residual connections may not generalize to deeper ConvMixers trained on larger data sets.
[Table]
Normalization.
Our model is conceptually similar to the vision transformer and MLP-Mixer, both of which use LayerNorm instead of BatchNorm.
We attempted to use LayerNorm instead [Table] and saw a decrease in performance of around 1% as well as slower convergence (see Table 3).
However, this was for a relatively shallow model, and we cannot guarantee that LayerNorm would not hinder
ImageNet-scale models to an even larger degree.
We note that the authors of ResMLP also saw a relatively small increase in accuracy for replacing LayerNorm with BatchNorm [Table] but for a larger- scale experiment (Touvron et al., 2021a).
We conclude that BatchNorm is no more crucial to our architecture than other regularizations or parameter settings ( e.g. , kernel size).
Having settled on an architecture, we proceeded to adjust its parameters h, d, p, k as well as weight decay on CIFAR-10 experiments.
(Initially, we took the unconventional approach of excluding weight decay since we were already using strong regularization in the form of RandAug and mixup.)
We acknowledge that tuning our architecture on CIFAR-10 does not necessarily generalize to performance on larger data sets, and that this is a limitation of our study.
B.1 Results
ConvMixers are quite performant on CIFAR-10, easily achieving > 91% accuracy for as little as 100 , 000 parameters, or > 96% accuracy for only 887 , 000 parameters (see Table 4).
With additional reﬁnements e.g. , a more expressive classiﬁer or bottlenecks, we think that ConvMixer could be even more competitive.
For all experiments, we trained for 200 epochs on CIFAR-10 with RandAug, mixup, cutmix, random erasing, gradient norm clipping, and the standard augmentations in timm .
We remove some of these augmentations in Table 3, ﬁnding that RandAug and random scaling (“default” in timm ) are very important, each accounting for over 3% of the accuracy.
Scaling ConvMixer.
We adjusted the hidden dimension h and the depth d , ﬁnding that deeper networks take longer to converge while wider networks converge faster.
That said, increasing the width or the depth is an eﬀective way to increase accuracy; a doubling of depth incurs less compute than a doubling of width.
The number of parameters in a ConvMixer is given exactly by:
[Equation]
including aﬃne scaling parameters in BatchNorm layers, convolutional kernels, and the classiﬁer.
Kernel size.
We initially hypothesized that large kernels would be important for ConvMixers, as they would allow the mixing of distant spatial information similarly to unconstrained MLPs or selfattention layers.
We tried to investigate the eﬀect of kernel size on CIFAR-10: we ﬁxed the model to be a ConvMixer-256/8, and increased the kernel size by 2s from 3 to 15.
Using a kernel size of 3, the ConvMixer only achieves 93.61% accuracy.
Simply increasing it to 5 gives an additional 1.50% accuracy, and further to 7 an additional 0.61%.
The gains afterwards are relatively marginal, with kernel size 15 giving an additional 0.28% accuracy.
It could be that with more training iterations or more regularization, the eﬀect of larger kernels would be more pronounced.
Nonetheless, we concluded that ConvMixers beneﬁt from larger-than-usual kernels, and thus used kernel sizes 7 or 9 in most of our later experiments.
It is conventional wisdom that large-kernel convolutions can be “decomposed” into stacked smallkernel convolutions with activations between them, and it is therefore standard practice to use k =convolutions, stacking more of them to increase the receptive ﬁeld size with additional beneﬁts from nonlinearities.
This raises a question: is the beneﬁt of larger kernels in ConvMixer actually better than simply increasing the depth with small kernels?
First, we note that deeper networks are generally harder to train, so by increasing the kernel size independently of the depth, we may recover some of the beneﬁts of depth without making it harder for signals to “propagate back” through the network.
To test this, we trained a ConvMixer-256/10 with k = 3 (698K parameters) in the same setting as a ConvMixer-256/8 with k = 9 (707K parameters), i.e. , we increased depth in a smallkernel model to roughly match the parameters of a large-kernel model.
The ConvMixer-256/ achieved 94.29% accuracy (1.5% less), which provides more evidence for the importance of larger kernels in ConvMixers.
Next, instead of ﬁxing the parameter budget, we tripled the depth (using the intuition that 3 stacked k = 3 convolutions have the receptive ﬁeld of a k = 9 convolution), giving a
ConvMixer-256/24 with 1670K parameters, and got 95.16% accuracy, i.e. , still less.
Patch size.
CIFAR-10 inputs are so small that we initially only used p = 1 , i.e. , the patch embedding layer does little more than compute h linear combinations of the input image.
Using p = 2 , we see a reduction in accuracy of about 0.80%; this is a worthy tradeoﬀ in terms of training and inference time.
Further increasing the patch size leads to rapid decreases in accuracy, with only 92.61% for p = 4 .
Since the “internal resolution” is decreased by a factor of p when increasing the patch size, we assumed that larger kernels would be less important for larger p .
We investigated this by again increasing the kernel size from 3 to 11 for ConvMixer-256/8 with p = 2 : however, this time, the improvement going from 3 to 5 is only 1.13%, and larger kernels than 5 provide only marginal beneﬁt.
Weight decay.
We did many of our initial experiments with minimal weight decay.
However, this was not optimal: by tuning weight decay, we can get an additional 0.15% of accuracy for no cost.
Consequently, we used weight decay (without tuning) for our larger-scale experiments on ImageNet.
Tiny ConvMixers trained on CIFAR-10.
[Table]
1 × 10 −95.
1 × 10 −96.
1 × 10 −95. 1 × 10 −95.
1 × 10 −95.
[Table]
when p [Table] [Table] Adding weight decay to the above     1 × 10 −94.
1 × 10 −95.
1 × 10 −95. 1 × 10 −95.
1 × 10 −95.
C Weight Visualizations
[Figure]
In Figure 4 and 5, we visualize the (complete) weights of the patch embedding layers of a ConvMixer1536/20 with p = 14 and a ConvMixer-768/32 with p = 7 , respectively.
Much like Sandler et al.
(2019), the layer consists of Gabor-like ﬁlters as well as “colorful globs” or rough edge detectors.
The ﬁlters seem to be more structured than those learned by MLP-Mixer (Tolstikhin et al., 2021); also unlike MLP-Mixer, the weights look much the same going from p = 14 to p = 7 : the latter simply looks like a downsampled version of the former.
It is unclear, then, why we see such a drop in accuracy for larger patches.
However, some of the ﬁlters essentially look like noise, maybe suggesting a need for more regularization or longer training, or even more data.
Ultimately, we cannot read too much into the learned representations here.
In Figure 6, we plot the hidden convolutional kernels for successive layers of a ConvMixer.
Initially, the kernels seem to be relatively small, but make use of their allowed full size in later layers; there is a clear hierarchy of features as one would expect from a standard convolutional architecture.
Interestingly, Touvron et al.
(2021a) saw a similar eﬀect for ResMLP, where earlier layers look like small-kernel convolution, while later layers were more diﬀuse, despite these layers being representated by an unconstrained matrix multiplication rather than convolution.
D Implementation
[Figure] [Figure]
[Figure] [Figure]
[Figure]
self.fn(
[Figure] [Figure]
ConvMixer(
[Figure]
_size
[Figure] [Figure]
_size=
[Figure]
n_classes
[Figure] [Figure]
[Figure]
nn
[Figure] [Figure]
](x)+x})
[Figure]
S(A(C(3,h,p,p)),*[S(R(A(C(h,h,k,
[Figure]
=h,padding=k//2))),
[Figure]
(C(h,h
[Figure]
))) for
[Figure]
in range(
[Figure]
[Figure] [Figure] [Figure]
(cid:44)
[Figure]
All you need to do to run this is from torch.nn import * .
This section presents an expanded (but still quite compact) version of the terse ConvMixer implementation that we presented in the paper.
The code is given in Figure 7.
We also present an even more terse implementation in Figure 8, which to the best of our knowledge is the ﬁrst model that achieves the elusive dual goals of 80%+ ImageNet top-1 accuracy while also ﬁtting into a tweet.