Sequencer: Deep LSTM for Image Classification
Abstract
In recent computer vision research, the advent of the Vision Transformer (ViT) has rapidly revolutionized various architectural design efforts: ViT achieved stateof-the-art image classification performance using self-attention found in natural language processing, and MLP-Mixer achieved competitive performance using simple multi-layer perceptrons.
In contrast, several studies have also suggested that carefully redesigned convolutional neural networks (CNNs) can achieve advanced performance comparable to ViT without resorting to these new ideas.
Against this background, there is growing interest in what inductive bias is suitable for computer vision.
Here we propose Sequencer , a novel and competitive architecture alternative to ViT that provides a new perspective on these issues.
Unlike ViTs,
Sequencer models long-range dependencies using LSTMs rather than self-attention layers.
We also propose a two-dimensional version of Sequencer module, where an LSTM is decomposed into vertical and horizontal LSTMs to enhance performance.
Despite its simplicity, several experiments demonstrate that Sequencer performs impressively well: Sequencer2D-L, with 54M parameters, realizes 84.6% topaccuracy on only ImageNet-1K. Not only that, we show that it has good transferability and the robust resolution adaptability on double resolution-band. Our source code is available at https://github.com/okojoalg/sequencer .
Introduction
[Figure]
models are trained on IN-1K at resolution 224 2 from scratch. The de-facto standard for computer vision has been convolutional neural networks (CNNs) [39, 64, 22, 65, 66, 9, 29, 67].
However, inspired by the many breakthroughs in natural language processing (NLP) achieved by Transformers [75, 35, 57], applications of Transformers for computer vision are now being actively studied.
In particular, Vision Transformer (ViT)
[16] is a pure Transformer applied to image recognition and achieves performance competitive with CNNs.
Various studies triggered by ViT have shown that the state-of-the-art (SOTA) performance can be achieved for a wide range of vision tasks using self-attention alone [79, 48, 73, 47, 15], without convolution.
The reason for this success is thought to be due to the ability of self-attention to model long-range dependencies.
However, it is still unclear how essential the self-attention is to the effectiveness of Transformers for vision tasks.
Indeed, the MLP-Mixer [70] based only on multi-layer perceptrons (MLPs) is proposed as an appealing alternative to Vision Trans- formers (ViTs).
In addition, some studies [49, 14] have shown that carefully designed CNNs are still competitive enough with Transformers in computer vision.
Therefore, identifying which architectural designs are inherently effective for computer vision tasks is of great interest for current research [83].
This paper provides a new perspective on this issue by proposing a novel and competitive alternative to these vision architectures.
We propose the Sequencer architecture, that uses the long short-term memory (LSTM) [27] rather than the self-attention for sequence modeling.
The macro-architecture design of Sequencer follows ViTs, which iteratively applies token mixing and channel mixing, but the self-attention layer is replaced by one based on LSTMs.
In particular, Sequencer uses bidirectional LSTM (BiLSTM) [63] as a building block.
While simple BiLSTM shows a certain level of performance, Sequencer can be further improved by using ideas similar to Vision Permutator (ViP) [28].
The key idea in ViP is to process the vertical and horizontal axes in parallel.
We also introduce two BiLSTMs for top/bottom and left/right directions in parallel.
This modification improves the efficiency and accuracy of Sequencer because this structure reduces the length of the sequence and yields a spatially meaningful receptive field.
When pre-trained on ImageNet-1K (IN-1K) dataset, our new attention-free architecture outperforms advanced architectures such as Swin [48] and ConvNeXt [49] of comparable size, see Figure 1.
It also outperforms other attention-free and CNN-free architectures such as MLP-Mixer [70] and
GFNet [61], making Sequencer an attractive new alternative to the self-attention mechanism in vision tasks.
This study also aims to propose novel architecture with practicality by employing LSTM for spatial pattern processing.
Notably, Sequencer exhibits robust resolution adaptability, which strongly prevents accuracy degradation even when the input’s resolution is increased double during inference.
Moreover, fine-tuning Sequencer on high-resolution data can achieve higher accuracy than Swin-B [48] and Sequencer is also useful for semantic segmentation.
On peak memory, Sequencer tends to be more economical than ViTs and recent CNNs for high-resolution input.
Although Sequencer requires more FLOPs than other models due to recursion, the higher resolution improves the relative efficiency of peak memory, enhancing the accuracy/cost trade-off at a high-resolution regime.
Therefore,
Sequencer also has attractive properties as a practical image recognition model.
Related works
Inspired by the success of Transformers in NLP [75, 35, 57, 58, 3, 60], various applications of self-attention have been studied in computer vision.
For example, in iGPT [6], an attempt was made to apply autoregressive pre-training with causal self-attention [57] to image classification.
However, due to the computational cost of pixel-wise attention, it could only be applied to low-resolution images, and its ImageNet classification performance was significantly inferior to the SOTA.
ViT [16], on the other hand, quickly brought Transformer’s image classification performance closer to SOTA with its idea of applying bidirectional self-attention [35] to image patches rather than pixels.
Various architectural and training improvements [72, 84, 79, 90, 48, 73, 5] have been attempted for ViT [16].
In this paper, we do not improve self-attention itself but propose a completely new module for image classification to replace it.
The extent to which attention-based cross-token communication inherently contributes to ViT’s success is not yet well understood, starting with MLP-Mixer [70], which completely replaced ViT’s self-attention with MLP, various MLP-based architectures [71, 46, 28, 69, 68, 13] have achieved competitive performance on the ImageNet dataset.
We refer to these architectures as global MLPs (GMLPs) because they have global receptive fields.
This series of studies cast doubt on the need for self-attention.
From a practical standpoint, however, these MLP-based models have a drawback: they need to be finetuned to cope with flexible input sizes during inference by modifying the shape of their token-mixing MLP blocks.
This resolution adaptability problem has been improved in CycleMLP [7], for example, by the idea of realizing a local kernel with a cyclic MLP.
There are similar ideas such as [82, 81, 42, 21] which are collectively referred to as local MLPs (LMLPs).
Besides the MLP-based idea, several other interesting self-attention alternatives have been found.
GFNet [61] uses Fourier transformation of the tokens and mixes the tokens by global filtering in the frequency domain.
PoolFormer [83], on the other hand, achieved competitive performance with only local pooling of tokens, demonstrating that simple local operations are also a suitable alternative.
Our proposed Sequencer is a new alternative to self-attention that differs from both of the above, and
Sequencer is an attempt to realize token mixing in vision architectures using only LSTM.
It achieved competitive performance with SOTA on the IN-1K benchmark, especially with an architecture that can flexibly adapt to higher resolution.
The idea of spatial axis decomposition has been used several times in neural architecture in computer vision.
For example, SqueezeNeXt [17] decomposes a 3x3 convolution layer into 1x3 and 3x1 convolution layers, resulting in a lightweight model.
Criss-cross attention [31] reduces memory usage and computational complexity by restricting the attention to only vertical and horizontal portions.
Current architectures such as CSwin [15], Couplformer [40], ViP [28], RaftMLP [69], SparseMLP [68], and
MorphMLP [86] have included similar ideas to improve efficiency and performance.
In the early days of deep learning, there were attempts to use RNNs for image recognition.
The earliest study that applied RNNs to image recognition is [19].
The primary difference between our study and [19] is that we utilize a usual RNN in place of a 2-multi-dimensional RNN(2MDRNN).
The 2MDRNN requires H + W sequential operations; The LSTM requires H sequential operations, where H and W are height and width, respectively.
For subsequent work on image recognition using 2MDRNNs, see [20, 32, 4, 43].
[4] proposed an architecture in which information is collected from four directions (upper left, lower left, upper right, and lower right) by RNNs for understanding natural scene images.
[43] proposed a novel 2MDRNN for semantic object parsing that integrates global and local context information, called LG-LSTM.
The overall architecture design is structured to input deep ConvNet features into the LG-LSTM, unlike Sequencer which stacks LSTMs.
ReNet [77] is most relevant to our work; ReNet [77] uses a 4-way LSTM and non-overlapping patches as input.
In this respect, it is similar to Sequencer.
Meanwhile, there are three differences.
First, Sequencer is the first MetaFormer [83] realized by adopting LSTM as the token mixing block.
Sequencer also adopts a larger patch size than ReNet [77].
The benefit of adopting these designs is that we can modernize LSTM-based vision architectures and fairly compare LSTM-based models with ViT. As a result, our results provide further evidence for the extremely interesting hypothesis MetaFormer [83].
Second, the way vertical BiLSTMs and horizontal BiLSTMs are connected is different.
Our work connects them in parallel, allowing us to gather vertical and horizontal information simultaneously.
On the other hand, in ReNet [77], the output of the horizontal BiLSTM is used as input to the vertical
BiLSTM.
Finally, we trained Sequencer on large datasets such as ImageNet, whereas ReNet [77] is limited to small datasets as MNIST [41], CIFAR-10 [38], and SVHN [54], and has not shown the effectiveness of LSTM for larger datasets.
ReSeg [76] applied ReNet to semantic segmentation.
RNNs have been applied not only to image recognition, but also to generative models: PixcelRNN [74] is a pixel-channel autoregressive generative model of images using Row RNN, which consists of a 1D-convolution and a usual RNN, and Diagonal BiLSTM, which is computationally expensive.
In NLP, attempts have been made to avoid the computational cost of attention by approximating causal self-attention with recurrent neural network (RNN) [34] or replacing it with RNN after training [33].
In particular, in [34], an autoregressive pixel-wise image generation task is experimented with an architecture where the attentions in iGPT are approximated by RNNs.
These studies are specific to unidirectional Transformers, in contrast to our token-based Sequencer which is the bidirectional analog of them.
Method
In this section, we briefly recap the preliminary background on LSTM and further describe the details of the proposed architectures. 3.1 Preliminaries: Long short-term memory
LSTM [27] is a specialized recurrent neural network (RNN) for modeling long-term dependencies of sequences. Plain LSTM has an input gate i t that controls the storage of inputs, a forget gate f t that controls the forgetting of the former cell state c t − 1 and an output gate o t that controls the cell output h t from the current cell state c t .
Plain LSTM is formulated as follows: [Equation] where σ is the logistic sigmoid function and ⊙ is Hadamard product. G l o b a l A v e r a g e P oo l i n g
L i n e a r L a y e r N o r m
P a t c h E m b e dd i n g
I n p u t
S e q u e n c e r B l o c k
P a t c h M e r g i n g
S e q u e n c e r B l o c k
P W L i n e a r
S e q u e n c e r B l o c k
P W L i n e a r
S e q u e n c e r B l o c k
[Figure]
BiLSTM [63] is profitable for sequences where mutual dependencies are expected.
A BiLSTM consists of two plain LSTMs.
Let −→ x be the input series and ←− x be the rearrangement of −→ x in reverse order.
−−→ h for and ←−−− h back are the outputs obtained by processing −→ x and ←− x with the corresponding
LSTMs, respectively.
Let −−−→ h back be the output ←−−− h back rearranged in the original order, and the output of BiLSTM is obtained as follows:
[Equation]
Assume that both −−→ h for and −−−→ h back have the same hidden dimension D , which is hyperparameter of
BiLSTM.
Accordingly, vector h has dimension 2 D .
3.2 Sequencer architecture
Overall architecture
In the last few years, ViT and its many variants based on self-attention [16, 72, 48, 91] have attracted much attention in computer vision.
Following these, several works [70, 71, 46, 28] have been proposed to replace self-attention with MLP.
There have also been studies of replacing self-attention with a hard local induced bias module [7, 83] and with a global filter [61] using the fast Fourier transform algorithm (FFT) [10].
This paper continues this trend and attempts to replace the self-attention layer with LSTM [27]: we propose a new architecture aiming at memory saving by mixing spatial information with LSTM, which is memory-economical compared to ViT, parameter-saving, and has the ability to learn long-range dependencies.
Figure 2a shows the overall structure of Sequencer architecture.
Sequencer architecture takes nonoverlapping patches as input and projects them onto the feature map.
Sequencer block, which is a core component of Sequencer, consists of the following sub-components: (1) BiLSTM layer can mix spatial information more memory-economically for high-resolution images than Transformer layer and more globally than CNN.
(2) MLP for channel-mixing as well as [16, 70].
Sequencer block is called Vanilla Sequencer block when plain BiLSTM layers are used as BiLSTM layers as Figure 2d and Sequencer2D block when BiLSTM2D layers are used as Figure 2e.
We define BiLSTM2D layer later.
The output of the last block is sent to the linear classifier via the global average pooling layer, as in most other architectures.
BiLSTM2D layer We propose the BiLSTM2D layer as a technique to mix 2D spatial information efficaciously.
It has two plain BiLSTMs: a vertical BiLSTM and a horizontal one.
For an input
X ∈ R H × W × C , { X : ,w, : ∈ R H × C } Ww =1 is viewed as a set of sequences, where H is the number of tokens in the vertical direction, W is the number of sequences in the horizontal direction, and C is the channel dimension.
All sequences X : ,w, : are input into the vertical BiLSTM with shared weights and hidden dimension D :
[Equation]
In a very similar manner, { X h, : , : ∈ R W × C } Hh
=
1 is viewed as a set of sequences, and all sequences
X h, : , : are input into the horizontal BiLSTM with shared weights and hidden dimension D as well:
[Equation]
We combine { H ver: ,w, : ∈ R H × 2 D } Ww =1 into H ver ∈ R W × H × 2 D and { H hor h, : , : ∈ R W × 2 D } Hh =1 into
H hor ∈ R W × H × 2 D .
They are then concatenated and processed point-wisely in a fully-connection layer.
These processes are formulated as follows:
[Equation]
where FC( · ) denotes the fully-connected layer with weight W ∈ R C × 4 D .
The PyTorch -like pseudocode is shown in Appendix B.1.
BiLSTM2D is more memory-economical and throughput-efficiency than multi-head-attention of
ViT for high-resolution input.
BiLSTM2D involves ( WC + HC ) / 2 dimensional cell states, while a multi-head-attention involves h ∗ ( HW ) 2 dimensional attention map where h is a number of heads.
Thus, as H and W increase, the memory cost of an attention map increases more rapidly than the cost of a cell state.
On throughput, the computational complexity of self-attention is O ( W 4 C ) , whereas the computational complexity of BiLSTM is O ( WC 2 ) where we assume W = H for simplicity.
There are O ( W ) sequential operations for BiLSTM2D. Therefore, assuming we use a sufficiently efficient LSTM cell implementation, such as official PyTorch LSTMs we are using, the increase of the complexity of self-attention is much more rapid than BiLSTM2D. It implies a lower throughput of attention compared to BiLSTM2D. See an experiment in Section 4.5.
Architecture variants For comparison between models of different depths consisting of Sequencer2D blocks, we have prepared three models with different depths: 18, 24, and 36.
The names of the models are Sequencer2D-S , Sequencer2D-M , and Sequencer2D-L , respectively.
The hidden dimension is set to D =
C/ 4 .
Details of these models are provided in Appendix B.2.
As shown in subsection 4.1, these architectures outperform typical models.
Interestingly, however, subsection 4.3 shows that replacing Sequencer2D block with the simpler Vanilla Sequencer block maintains moderate accuracy.
We denote such a model as Vanilla Sequencer.
Note that some of the explicit positional information is lost in the Vanilla Sequencer because the model treats patches as a 1D sequence.
[-LMLP 
-]
[-PoolFormer-S36 [83] LMLP 
-]
[-PoolFormer-M36 [83] LMLP 
-]
Experiments
In this section, we compare Sequencers with previous studies on the IN-1K benchmark [39].
We also carry out ablation studies, transfer learning studies, and analysis of the results to demonstrate the effectiveness of Sequencers.
We adopt PyTorch [56] and timm [80] library to implement models in the conduct of all experiments.
See Appendix B for more setup details.
4.1 Scratch training on IN-1K
We utilize IN-1K [39], which has 1000 classes and contains 1,281,167 training images and 50, validation images.
We adopt AdamW optimizer [50].
Following the previous study [72], we adopt the base learning rate batchsize512 × 5 × 10 − 4 .
The batch sizes for Sequencer2D-S, Sequencer2D-M, and Sequencer2D-L are 2048, 1536, and 1024, respectively.
As a regularization method, stochastic depth [30] and label smoothing [66] are employed.
As data augmentation methods, mixup [87], cutout [12], cutmix [85], random erasing [88], and randaugment [11] are applied. [Table] as above RegNetY-8GF [59]
CNN 22439M 8.0G 211/ 1776/ 81. as above T2T-ViT t -19 [84] Trans.
22439M 9.8G 197/654 3520/ 82.
as above
CycleMLP-B3 [7]
LMLP 22438M 6.9G 100/ 2326/ 82.
as above
PoolFormer-S36 [83] LMLP 22431M 5.2G 213/ 2187/ 81.
as above
GFNet-H-S [61]
FFT 22432M 4.5G 227/ 1740/ 81.
as above
Sequencer2D-M
Seq.
22438M 11.1G 83/ 2311/ 82.
as above
RegNetY-12GF [59]
CNN 22446M 12.0G 199/ 2181/ 82.
as above
ConvNeXt-S [49]
CNN 22450M 8.7G 212/ 2265/ 83.
as above
Swin-S [48]
{+Trans. 
+}
M 8.7G 165/ 2635/ 83.
as above
Mixer-B/16 [70]
GMLP 22459M 12.7G 338/ 1864/ 76.
as above
ViP-M/7 [28] GMLP 22455M 16.3G 130/ 3095/ 82. as above CycleMLP-B4 [7]
LMLP 22452M 10.1G 70/ 3272/ 83. as above PoolFormer-M36 [83] LMLP 22456M 9.1G 171/
[Table]
Table 1 shows the results that are comparing the proposed models to others with a comparable number of parameters to our models, including models with local and global receptive fields such as CNNs,
ViTs [Table]
and MLP-based and FFT-based models.
Sequencers has the disadvantage that its throughput is slower than other models because it uses RNNs.
In the scratch training on IN-1K, however, they outperform these recent comparative models in accuracy across their parameter bands.
In particular,
Seqeuncer2D-L is competitive with recently discussed models with comparable parameters such as
ConvNeXt-S [49] and Swin-S [48], with accuracy outperformance of 0.3% and 0.2%, respectively
[Table]
Table 1 demonstrates that Sequencers don’t have good throughput during the training and inference. The training throughput is about three times the inference throughput for all these models. Compared to other models, both measured inference and training time are not good.
4.2 Fine-tuning on IN-1K
In this fine-tuning study, Sequencer2D-L pre-trained on IN-1K at 224 2 resolution is fine-tuned on IN-1K at 392 2 resolution.
We compare it with the other models fine-tuned on IN-1K at {+
 [Table]+}
resolution. Since 392 is divisible by 14, the input at this resolution can be split into patches without padding.
However, note that this is not the case with a resolution of 384 2 . As Table 1 indicates [Table]
even when higher-resolution Sequencer is fine-tuned [Table] it is competitive with the latest models such as ConvNeXt [49], Swin [48] [Table]
and GFNet [61].
4.3 Ablation studies
This subsection presents ablation studies based on Sequencer2D-S for further understanding of
Sequencer.
We seek to clarify the effectiveness and validity of the Sequencers architecture in terms of the importance of each component, bidirectional necessaries, setting of the hidden dimension, and the comparison with simple BiLSTM.
We show where and how relevant the components of BiLSTM2D are: The BiLSTM2D is composed of vertical BiLSTM, horizontal BiLSTM, and channel fusion elements.
We want to see the validity of vertical BiLSTM, horizontal BiLSTM, and channel fusion.
For this purpose, we examine the removal of channel fusion and vertical or horizontal BiLSTM.
Table 2a shows the results.
Removing channel fusion shows that the performance degrades from 82.3% to 81.6%.
Furthermore, the additional removal of vertical or horizontal BiLSTM exposes a 6.0% or 6.6% performance drop, respectively.
Hence, each component discussed here is necessary for Sequencer2D.
We show that the bidirectionality for BiLSTM2D is important for Sequencer.
We compare
Sequencer2D-S with a version that replaces the vertical and horizontal BiLSTMs with vertical and horizontal unidirectional LSTMs.
Table 2b shows that the unidirectional model is 2.6% less accurate than the bidirectional model.
This result attests to the significance of using not unidirectional
LSTM but BiLSTM.
It is important to set the hidden dimension of LSTM to a reasonable size.
As described in subsection 3.2, Sequencer2D sets the hidden dimension D of BiLSTM to D =
C/ 4 , but this is not necessary if the model has channel fusions.
Table 2d compares Sequencer2D-S with the model with increased
D .
Although accuracy is 0.3% improved, FLOPs increase by 65%, and the number of parameters increases by 60%.
Namely, the accuracy has not improved for the increase in FLOPs.
Moreover, the increase in dimension causes overfitting, which is discussed in Appendix C.3.
Vanilla Sequencer can also achieve accuracy that outperforms MLP-Mixer [70], but is not as accurate as Sequencer2D.
Following experimental result supports the claim.
We experiment with the
Sequencer2D-S variants, where Vanilla Sequencer blocks replace the Sequencer2D blocks, called
VSequencer-S(H), with incomplete positional information.
In addition, we experiment with a variant of VSequencer-S(H) without the hierarchical structure, which we call VSequencer-S. VSequencerS(PE) is VSequencer-S using ViTs-style learned positional embedding (PE) [16].
Table 2c indicates effectiveness for combination of LSTM and ViTs-like architecture.
Surprisingly, even with Vanilla
[Table] Sequencer and Vanilla Sequencer(H) without PE, the performance reduction from Sequencer2D-S is only 4.3% and 3.5%, respectively.
According to these results, there is no doubt that Vanilla Sequencer using BiLSTMs is significant enough, although not as accurate as Sequencer2D.
All LSTMs in the BiLSTM2D layer can be replaced with other recurrent networks such as gated recurrent units (GRUs) [8] or tanh-RNNs to define BiGRU2D layer or BiRNN2D layer.
We also trained these models on IN-1K, so see Table 2e for the results.
The table suggests that all of these variants, including RNN-cell, work well.
Also, tanh-RNN performs slightly worse than others, probably due to its lower ability to model long-range dependence.
4.4 Transfer learning and semantic segmentation
Sequencers perform well on IN-1K, and they have good transferability.
In other words, they have satisfactory generalization performance for a new domain, which is shown below.
We utilize the commonly used CIFAR-10 [38], CIFAR-100 [38], Flowers-102 [55], and Stanford Cars [37] for this experiment.
See the references and Appendix B.4 for details on the datasets.
The results of the proposed model and the results in previous studies of models with comparable capacity are presented in Table 3.
In particular, Sequencer2D-L achieves results that are competitive with CaiT-S-36 [73] and EfficientNet-B7 [67].
We experiment for semantic segmentation on ADE20K[89] dataset.
See Appendix C.4 for details on the setup.
Sequencer outperforms PVT [79] and PoolFormer [83] with similar parameters; compared to PoolFormer, mIoU is about 6 pts higher.
We have investigated a commonly object detection model with Sequencer as the backbone.
Its performance is not much different from the case of ResNet [22] backbone.
Its improvement is the future work.
See Appendix C.5.
4.5 Analysis and visualization
In this subsection, we investigate the properties of Sequencer in terms of resolution adaptability and efficiency.
Furthermore, effective receptive field (ERF) [51] and visualization of the hidden states provides insight into the question of how Sequencer recognizes images.
One of the attractive properties of Sequencer is its flexible adaptability to the resolution, with minimal impact on accuracy even when the resolution of the input image is varied from one-half to twice.
In comparison, architectures like MLP-Mixer [70] have a fixed input resolution, and GFNet [61] requires interpolation of weights in the Fourier domain when inputting images with a resolution different from training images.
We evaluate the resolution adaptability of models comparatively by inputting different resolution images to each model, without fine-tuning, with pre-trained weights on IN-1K at the resolution of 224 2 .
Figure 3a compares absolute top-1 accuracy on IN-1K, and
Figure 3b compares relative one to the input image with the resolution of 224 2 .
By increasing the resolution by 28 for Sequencer2D-S and by 32 for other models, we avoid padding and prevent the effect of padding on accuracy.
Compared to DeiT-S [72], GFNet-S [61], CycleMLP-B2 [7], and ConvNeXt-T [49], Sequencer-S’s performance is more sustainable.
The relative accuracy is consistently better than ConvNeXt [49], which is influential in the lower-resolution band, and, at 448 2 resolution, 0.6% higher than CycleMLP [7], which is influential in the double-resolution band.
It is noteworthy that Sequencer continues to maintain high accuracy on double resolution.
The higher the input resolution, the higher memory-efficiency and throughput of Sequencers when compared to DeiT [72].
Figure 3 shows the efficiency of Sequencer2D-S when compared to DeiT-S and ConvNeXt-T [49].
Memory consumption increases rapidly in DeiT-S and ConvNeXt-T with increasing input resolution, but more gradual increase in Sequencer2D-S.
The result strongly implies that it has more practical potential as the resolution increases than the ViTs.
At a resolution of 224 2 , it is behind DeiT in throughput, but it stands ahead of DeiT when images with a resolution of 896are input. [Figure] Resolution [Figure]
k M
[Figure]
m [Figure] l [Figure]
[Figure]
[Figure]
h r
[Figure]
h p
[Figure]
t
[Figure]
m a g
[Figure]
/ s
[Figure]
[Figure]
In general, CNNs have localized, layer-by-layer expanding receptive fields and ViTs without shifted windows capture global dependencies, working the self-attention mechanism.
In contrast, in the case of Sequencer, it is not clear how information is processed in Sequencer block.
We calculated
ERF [51] for ResNet-50 [22], DeiT-S [72] and Sequencer2D-S as shown in Figure 5.
ERFs of
Sequencer2D-S form a cruciform shape in all layers.
The trend distinguishes it from well-known models such as DeiT-S and ResNet-50.
More remarkably, in shallow layers, Sequencer2D-S has a wider ERF than ResNet-50, although not as wide as DeiT.
This observation confirms that LSTMs in
Sequencer can model long-term dependencies as expected and that Sequencer recognizes sufficiently long vertical or horizontal regions.
Thus, it can be argued that Sequencer recognizes an image in a very different way than CNNs or ViTs.
For more details on ERF and additional visualization, see
Appendix D.
Moreover we also visualized a hidden state of vertical and horizontal BiLSTM, and a feature map after channel fusion, and the results are visualized in Figure 4.
It demonstrates that our Sequencer has the hidden states interact with each other over the vertical and horizontal directions.
The closer tokens are in position, the stronger their interaction tends to be; the farther tokens are in position, the less their interaction tends to be. [Figure]
Conclusions
We propose a novel and simple architecture that leverages LSTM for computer vision.
It is demonstrated that new modeling with LSTM instead of the self-attention layer can achieve competitive performance with current state-of-the-art models.
Our experiments show that Sequencer has a good memory-resource/accuracy and parameter/accuracy tradeoffs, comparable to the main existing methods.
Despite the impact of recursion on throughput, we have demonstrated benefits over it.
We believe that these results raise a number of interesting issues.
Improving Sequencer’s poor throughput is one example.
Moreover, we expect that investigating the internal mechanisms of our model using methods other than ERF will further our understanding of how this architecture works.
In addition, it would be important to analyze in more detail the features learned by Sequencer in comparison to other architectures.
We hope this will lead to a better understanding of the role of various inductive biases in computer vision.
Furthermore, we expect that our results trigger further study beyond the domain or research area.
Especially, it would be a very interesting open question to see if such a design works with time-series data in vision such as video or in a multi-modal problem setting combined with another modality such as video with audio. Acknowledgments and Disclosure of Funding
Our colleagues at AnyTech Co., Ltd. provided valuable comments on the early versions and encouragement. We thank them for their cooperation. In particular, We thank Atsushi Fukuda for organizing discussion opportunities. We also thank people who support us, belonging to Graduate School of
Artificial Intelligence and
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] The throughputs for our models are a bit inferior because of using RNNs, see Section 4.1 and 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Appendix A.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Included in supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] We follow the typical practice used in our baseline methodology no error bars are reported.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] They are included only if they are of essential significance to each experiment/item.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
The impact of this study on society has both positive and negative aspects. Here we discuss each. On the positive side, our proposal would promote modeling methods using LSTMs in computer vision.
This study takes image patches as tokens and models their relationships with LSTMs. Although LSTMs have been used in computer vision, designing image recognition with a module that includes LSTMs in the spatial direction as the main elements, as our study does, is new.
It is exciting to see if this design benefits computer vision tasks other than image classification.
Thus, our study would be an impetus for further research on its application to various computer vision tasks.
On the other side, our architecture may increase the carbon dioxide footprint: the study of new architectures for vision, such as Sequencer, requires iterative training of models for long periods to optimize the model’s design.
In particular, Sequencer is not a FLOPs-friendly design, and the amount of carbon dioxide emitted during training is likely to be high.
Therefore, considering the environmental burden caused by the training of Sequencers, research to reduce the computational cost of Sequencers is also desired by society.
B Implementation details
In this section, implementation details are supplemented.
We describe the pseudocode of the
BiLSTM2D layer, the architecture details, settings for training on IN-1K, and introduce settings for transfer learning.
B.1 Pseudocode
Algorithm
[Table]
B.2 Architecture details
This subsection describes Sequencer’s architecture.
The architectural details are shown in Tableand 5.
Sequencer2D-S is based on a ViP-S/7-like architecture.
We intend to directly compare the BiLSTM2D layer in Sequencer2D, which has a similar structure, with the Permute-MLP layer in ViP-S/7.
Tableis a summary of the architecture.
In keeping with ViP, the first stage of Sequencers involves patch embedding with a 7x7 kernel.
The second stage of Sequencers performs patch embedding with a 2x2 kernel, but the following two stages have no downsampling.
The classifier of Sequencers then continues with layer normalization (LN) [1], followed by global average pooling and a linear layer.
The number of blocks of Sequencer2D-S, Sequencer2D-M, and Sequencer2D-L correspond to
ViP-S/7, ViP-M/7, and ViP-L/7, respectively.
However, as described in Appendix C, we configure the dimension of the block to be different from ViP-M/7 and ViP-L/7 for Sequencer2D-M and
Sequencer2D-L, respectively, because high dimension causes over-fitting.
VSequencer is a bit different from Sequencer2D in that it is non-hierarchical architecture.
Tabledefine that no downsampling is performed in the second stage, instead of downsampling with 14x kernel for patch embedding in the first stage.
In addition, we match the dimension of the blocks in the first stage to the dimension of the subsequent blocks.
Following the overall architecture, we describe the details of the modules not mentioned in the main text.
Sequencer2D block and the Vanilla Sequencer block use LNs [1] for the normalization layers.
We follow previous studies for the channel MLPs of these blocks and employ MLPs with Gaussian
Error Linear Units (GELUs) [25] as the activation function; the ratio of increasing dimension in
MLPs is uniformly 3x, as shown in Table 4 and 5.
B.3 IN-1K settings
On IN-1K dataset [39], we utilize the hyper-parameters displayed in Table 6 to scratch train models in subsections 4.1 and 4.3.
All Sequencer variants, including the models in the ablation study, follow almost the same settings for pre-training.
However, the stochastic depth rate and batch size are adjusted depending on the model variant.
The models in the ablation study are Sequencer2D-S based because of the following Sequencer2D-S settings.
The fine-tuning Sequencer2D-L ↑ 392 2 in subsection 4.2 has slightly different hyper-parameters than the pre-training models.
There are changes in the settings for the number of epochs and learning rate because it uses trained weights, so there is no need to increase these hyper-parameters.
In addition, we used crop ratio 0.875 during testing in the pre-training models instead of crop ratio 1.0 in the fine-tuning model.
B.4 Transfer learning settings
Details of the datasets used for transfer learning in subsection 4.4 are shown in Table 7.
This summary includes for each dataset CIFAR-10 [38], CIFAR-100 [38], Flowers-102 [55], and Stanford Cars [37], the number of training images, test images, and number of categories are listed.
Table 6 demonstrates the hyperparameters used in transfer learning with these datasets.
The training epochs are especially adjusting to the datasets and changing them.
The reason for this is attributable to the different sizes of the datasets.
C More results
This section discusses additional results that could not be addressed in the main text.
The contents of the experiment consist of three parts: an evaluation of robustness in subsection C.1, an evaluation of generalization performance in subsection C.2, and a discussion of over-fitting in subsection C.3.
C.1 Robustness
In this subsection, we evaluate the robustness of Sequencer.
There are two main evaluation methods, benchmark datasets and adversarial attacks.
Evaluation with benchmark datasets reveals nice robustness of Sequencer.
The evaluation results are summarized in Table 8.
We test our models, trained on only IN-1K, on several datasets such as
ImageNet-A/R/Sketch/C (IN-A/R/Sketch/C)
[26, 23, 78, 24] to evaluate robustness.
We evaluate our models on IN-C with mean corruption error (mCE), and on other datasets with top-1 accuracy.
This result leads us to suggest that for models with a similar number of parameters, Sequencer is conquered by Swin and is robust enough to be competitive with ConvNeXt.
Table 9 shows detail evaluation on IN-C.
According to the results, it is understood that Sequencer is more immune to corruptions other than Noise than Swin and ConvNeXt, and, in particular, the model is less sensitive to weather conditions.
Sequencers are tolerant of principal adversarial attacks.
We evaluate robustness using the single-step attack algorithm FGSM [18] and multi-step attack algorithm PGD [52].
Both algorithms give a perturbation of max magnitude 1 .
For PGD, we choose steps 5 and step size 0 .
This setup is based on RVT [53].
Table 9 indicates that Sequencer2D-L defeats in both FGSM and PGD compared to other models.
Thus, Sequencer has an advantage over conventional models, such as RVT, which tout robustness on these adversarial attacks.
[Table]
C.2 Generalization ability
The generalization ability of Sequencers is also impressive.
We evaluate our models on ImageNetReal/V2 (IN-Real/V2)
[2, 62] to test their generalization performance: IN-Real is a re-labeled dataset of the IN-1K validation set, and IN-V2 is the dataset that re-collects the IN-1K validation set.
Tableshows the results of evaluating the top-1 accuracy on both datasets.
We reveal an understanding of the Sequencer’s excellent generalization ability.
C.3 Over-fitting
Wide Sequencers tend to be over-trained.
We scratch-train Sequencer2D-Lx1.3, which has 4/3 times the dimension of each layer of Sequencer2D-L, on IN-1K.
The training utilizes the same conditions as Sequencer2D-L. Consequently, as Table 10 shows, Sequencer2D-Lx1.3 has 0.8% less accuracy than Sequencer2D-L. Figure 6 illustrates the cross-entropy evolution and top-1 accuracy on IN-1K validation set for the two models.
On the one hand, cross-entropy decreased on Sequencer2D-L in the last 100 epochs.
On the other hand, Sequencer2D-Lx1.3 is increasing.
Thus, widening Sequencer is counterproductive for training.
C.4 Semantic segmentation
We evaluate models with Sequencer as the backbone for a semantic segmentation task.
We trained and evaluated on ADE20K dataset [89], a well-known scene parsing benchmark.
The dataset consists of the training set with about 20k images and the validation set with about 2k, covering 150 fine-grained semantic classes.
We employed Sequencer as the backbone of SemanticFPN [36] to train and evaluate semantic segmentation.
The training adopts a batch size of 32 and AdamW [50] with the initial learning rate of 2e-4, decay in the polynomial decay schedule with a
[Table]
[Table]
[Figure]
[Figure]
Layer Norm., Global Average Pooling, Linear
[Figure]
of training.
These settings follow Metaformer [83].
Table 3 of the result indicates that Sequencer has the generalization for segmentation is comparable to other leading models.
C.5 Object Detection
We evaluate Sequencer on COCO benchmark [45].
The dataset consists of 118k training images and 5k validation images.
Sequencer with ImageNet pre-trained weights is employed as the backbone of
RetinaNet [44].
Following [44], we employ AdamW, batch size of 16, and 1 × training schedule.
Table 11 shows that Sequencer is not suited for existing standard object detection models such as
[Table]
[Table]
RetinaNet.
It shows no improvement trend for model scaling.
It also struggles to detect small objects, making RNN-based object detection models an issue to consider in the future.
C.6 More studies
Method of merge As shown in Figure 2, " concatenate " is used to merge the vertical BiLSTM and horizontal BiLSTM outputs but " add " can also be used.
See Table 12a for the result of the experiment.
D Effective receptive field
This section covers in detail the effective receptive fields (ERFs) [51] used in the visualization in subsection 4.5.
First, we explain how the visualized effective receptive fields are obtained.
Second, we present other visualization results not addressed in the main text.
The ERF’s calculations in this paper are based on [14].
D.1 Calculation of visualized ERFs
The ERF [51] is a technique for calculating the pixels that contribute to the center of a output feature maps of a neural network.
Let I ∈ R n × h × w × c be a input image collection and O ∈ R n × h ′ × w ′ ×
c
′ be the output feature map collection.
The center of the output feature map can be expressed as
O : , ⌊ h ′ / 2 ⌋ , ⌊ w ′ / 2 ⌋ , : , where ⌊·⌋ is the floor function.
Each element of the derivative of O i, ⌊ h ′ / 2 ⌋ , ⌊ w ′ / 2 ⌋ ,j to I , i.e., ∂ ( (cid:80) i,j O i, ⌊ h ′ / 2 , ⌋⌊ w ′ / 2 ⌋ ,j ) ∂ I , represents to what extent the center of the output feature map
[Table]
[Table]
changes for each perturbation of each pixel in each input image.
Adding these together for all images and channels, we can calculate the average pixel contribution for all input images, which can be activated with a Rectified Linear Unit (ReLU) to get the positively contributing pixel values
P ∈ R n × h × w × c , defined by
[Equation]
Furthermore, the score S ∈ R h × w is calculated by
[Equation]
and S is called the effective receptive field.
Next, define a visualized effective receptive field based on the effective receptive field.
We want to compare the effective receptive fields across models.
We, therefore, calculate the score S model for each model and rescale S model from 0 to 1 across the models.
The tensor calculated in this way is called the visualized effective receptive field.
The derivatives used in these definitions are efficient if they take advantage of the auto-grad mechanism.
Indeed, we also relied on the automatic auto-grad function on PyTorch [56] to calculate the effective receptive fields.
[Table]
[Table]
[Table]
D.2 More visualization of ERFs
We introduce additional visualization and concrete visualization method.
We experiment with visualization using input images of two different resolutions.
We visualize the effective receptive fields of Sequencer2D-S and comparative models by using 224resolution images.
The method is applied to the following models for comparing: ResNet-50 [22],
ConvNeXt-T [49], CycleMLP-B2 [7], DeiT-S [72], Swin-T [48], GFNet-S [61]
[Table]
and ViP-S/7 [28].
The object to be visualized
For example, in the case of Sequencer2D-S, the effective receptive fields are calculated for the output of each Sequencer block.
We are rescaling within a value between 0 and 1 for the whole to effective receptive fields for each model block.
The effective receptive fields of Sequencer2D-S and comparative models are then visualized using input images with a resolution of 448 2 .
The reason for running experiments is to verify how the receptive field is affected when the input resolution is increased compared to the 224 2 resolution input image.
Sequencer2D-S compare with ResNet-50 [22], ConvNeXt-T [49], CycleMLP-B2 [7],
DeiT-S [72], and GFNet-S [28].
The method of visualization of the effective receptive field follows the case of input images with a resolution of 224 2 .
Sequencer has very distinctive cruciform ERFs in all layers.
Table 7, 8, 9, 10, 11, 12, 13, andillustrates this fact for 224 2 resolution input images.
Furthermore, as shown in Table 15, 16, 17, 18 and 19, we observe the same trend when the double resolution.
The ERFs are structurally quite different from the ERFs other than ViP, which have a similar structure.
ViP’s ERFs have, on average, some also coverage except for the cruciforms.
In contrast, Sequencer’s ERFs are limited to the cruciform and its neighborhood.
It is interesting to note that Sequencer, with its characteristic ERFs, achieves high accuracy.
It will be helpful for future architecture development because of the possibility of creating Sequencer-like
ERFs outside of LSTM.
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]