Sequencer: Deep LSTM for Image Classification
Anonymous Author(s)
AffiliationAddress email
Abstract
In recent computer vision research, the advent of the Vision Transformer (ViT)has rapidly revolutionized various architectural design efforts: ViT achieved stateof-the-art image classification performance using self-attention found in naturallanguage processing, and MLP-Mixer achieved competitive performance usingsimple multi-layer perceptrons.
In contrast, several studies have also suggested thatcarefully redesigned convolutional neural networks (CNNs) can achieve advancedperformance comparable to ViT without resorting to these new ideas.
Againstthis background, there is growing interest in what inductive bias is suitable forcomputer vision.
Here we propose Sequencer , a novel and competitive architecturealternative to ViT that provides a new perspective on these issues.
Unlike ViTs,
Sequencer models long-range dependencies using LSTMs rather than self-attentionlayers.
We also propose a two-dimensional version of Sequencer module, wherean LSTM is decomposed into vertical and horizontal LSTMs to enhance performance.
Despite its simplicity, several experiments demonstrate that Sequencerperforms impressively well: Sequencer2D-L, with 54M parameters, realizes 84.6%top-1 accuracy on only ImageNet-1K. Not only that, we show that it has goodtransferability and the robust resolution adaptability on double resolution-band.
Introduction
[Figure]
The de-facto standard for computer vision has beenconvolutional neural networks (CNNs) [39, 64, 22,65 , 66, 9, 29, 67].
However, inspired by the manybreakthroughs in natural language processing (NLP)achieved by Transformers [75, 35, 57], applicationsof Transformers for computer vision are now beingactively studied.
In particular, Vision Transformer(ViT)
[16] is a pure Transformer applied to imagerecognition and achieves performance competitivewith CNNs.
Various studies triggered by ViT haveshown that the state-of-the-art (SOTA) performancecan be achieved for a wide range of vision tasks using self-attention alone [79, 48, 73, 47, 15], withoutconvolution.
The reason for this success is thought to be due tothe ability of self-attention to model long-range dependencies.
However, it is still unclear how essential the self-attention is to the effectiveness of
Transformers for vision tasks.
Indeed, the MLP-Mixer [70] based only on multi-layer perceptrons(MLPs) is proposed as an appealing alternative to Vision Transformers (ViTs).
In addition, somestudies [49, 14] have shown that carefully designed CNNs are still competitive enough with Transformers in computer vision.
Therefore, identifying which architectural designs are inherently effectivefor computer vision tasks is of great interest for current research [83].
This paper provides a newperspective on this issue by proposing a novel and competitive alternative to these vision architectures.
We propose the Sequencer architecture, that uses the long short-term memory (LSTM) [27] rather thanthe self-attention for sequence modeling.
The macro-architecture design of Sequencer follows ViTs,which iteratively applies token mixing and channel mixing, but the self-attention layer is replacedby one based on LSTMs.
In particular, Sequencer uses bidirectional LSTM (BiLSTM) [63] as abuilding block.
While simple BiLSTM shows a certain level of performance, Sequencer can be furtherimproved by using ideas similar to Vision Permutator (ViP) [28].
The key idea in ViP is to process thevertical and horizontal axes in parallel.
We also introduce two BiLSTMs for top/bottom and left/rightdirections in parallel.
This modification improves the efficiency and accuracy of Sequencer becausethis structure reduces the length of the sequence and yields a spatially meaningful receptive field.
When pre-trained on ImageNet-1K (IN-1K) dataset, our new attention-free architecture outperformsadvanced architectures such as Swin [48] and ConvNeXt [49] of comparable size, see Figure 1.
It also outperforms other attention-free and CNN-free architectures such as MLP-Mixer [70] and
GFNet [61], making Sequencer an attractive new alternative to the self-attention mechanism in visiontasks.
This study also aims to propose novel architecture with practicality by employing LSTM for spatialpattern processing.
Notably, Sequencer exhibits robust resolution adaptability, which stronglyprevents accuracy degradation even when the input’s resolution is increased double during inference.
Moreover, fine-tuning Sequencer on high-resolution data can achieve higher accuracy than Swin-B[48] and Sequencer is also useful for semantic segmentation.
On peak memory, Sequencer tends to bemore economical than ViTs and recent CNNs for high-resolution input.
Although Sequencer requiresmore FLOPs than other models due to recursion, the higher resolution improves the relative efficiencyof peak memory, enhancing the accuracy/cost trade-off at a high-resolution regime.
Therefore,
Sequencer also has attractive properties as a practical image recognition model.
Related works
Inspired by the success of Transformers in NLP [75, 35, 57, 58, 3, 60], various applications ofself-attention have been studied in computer vision.
For example, in iGPT [6], an attempt was madeto apply autoregressive pre-training with causal self-attention [57] to image classification.
However,due to the computational cost of pixel-wise attention, it could only be applied to low-resolutionimages, and its ImageNet classification performance was significantly inferior to the SOTA.
ViT [16],on the other hand, quickly brought Transformer’s image classification performance closer to SOTAwith its idea of applying bidirectional self-attention [35] to image patches rather than pixels.
Variousarchitectural and training improvements [72, 84, 79, 90, 48, 73, 5] have been attempted for ViT [16].
In this paper, we do not improve self-attention itself but propose a completely new module for imageclassification to replace it.
The extent to which attention-based cross-token communication inherently contributes to ViT’ssuccess is not yet well understood, starting with MLP-Mixer [70], which completely replaced ViT’sself-attention with MLP, various MLP-based architectures [71, 46, 28, 69, 68, 13] have achievedcompetitive performance on the ImageNet dataset.
We refer to these architectures as global MLPs(GMLPs) because they have global receptive fields.
This series of studies cast doubt on the need forself-attention.
From a practical standpoint, however, these MLP-based models have a major drawback:they cannot cope with flexible input sizes during inference.
This resolution adaptability problem hasbeen improved in CycleMLP [7], for example, by the idea of realizing a local kernel with a cyclic
MLP.
There are similar ideas such as [82, 81, 42, 21] which are collectively referred to as local MLPs(LMLPs).
Besides the MLP-based idea, several other interesting self-attention alternatives have been found.
GFNet [61] uses Fourier transformation of the tokens and mixes the tokens by global filteringin the frequency domain.
PoolFormer [83], on the other hand, achieved competitive performance withonly local pooling of tokens, demonstrating that simple local operations are also a suitable alternative.
Our proposed Sequencer is a new alternative to self-attention that differs from both of the above, and
Sequencer is an attempt to realize token mixing in vision architectures using only LSTM.
It achievedcompetitive performance with SOTA on the IN-1K benchmark, especially with an architecture thatcan flexibly adapt to higher resolution.
The idea of spatial axis decomposition has been used several times in neural architecture in computervision.
For example, SqueezeNeXt [17] decomposes a 3x3 convolution layer into 1x3 and 3x1 convolution layers, resulting in a lightweight model.
Criss-cross attention [31] reduces memory usage andcomputational complexity by restricting the attention to only vertical and horizontal portions.
Currentarchitectures such as CSwin [15], Couplformer [40], ViP [28], RaftMLP [69], SparseMLP [68], and
MorphMLP [86] have included similar ideas to improve efficiency and performance.
In the early days of deep learning, there were attempts to use RNNs for image recognition.
Theearliest study that applied RNNs to image recognition is [19].
The primary difference between ourstudy and [19] is that we utilize a usual RNN in place of a 2-multi-dimensional RNN(2MDRNN).
The 2MDRNN requires H + W sequential operations; The LSTM requires H sequential operations,where H and W are height and width, respectively.
For subsequent work on image recognition using2MDRNNs, see [20, 32, 4, 43].
[4] proposed an architecture in which information is collected fromfour directions (upper left, lower left, upper right, and lower right) by RNNs for understanding naturalscene images.
[43] proposed a novel 2MDRNN for semantic object parsing that integrates global andlocal context information, called LG-LSTM.
The overall architecture design is structured to inputdeep ConvNet features into the LG-LSTM, unlike Sequencer which stacks LSTMs.
ReNet [77] ismost relevant to our work; ReNet [77] uses a 4-way LSTM and non-overlapping patches as input.
Inthis respect, it is similar to Sequencer.
Meanwhile, there are three differences.
First, Sequencer isthe first MetaFormer [83] realized by adopting LSTM as the token mixing block.
Sequencer alsoadopts a larger patch size than ReNet [77].
The benefit of adopting these designs is that we canmodernize LSTM-based vision architectures and fairly compare LSTM-based models with ViT. As aresult, our results provide further evidence for the extremely interesting hypothesis MetaFormer [83].
Second, the way vertical BiLSTMs and horizontal BiLSTMs are connected is different.
Our workconnects them in parallel, allowing us to gather vertical and horizontal information simultaneously.
On the other hand, in ReNet [77], the output of the horizontal BiLSTM is used as input to the vertical
BiLSTM.
Finally, we trained Sequencer on large datasets such as ImageNet, whereas ReNet [77] islimited to small datasets as MNIST [41], CIFAR-10 [38], and SVHN [54], and has not shown theeffectiveness of LSTM for larger datasets.
ReSeg [76] applied ReNet to semantic segmentation.
RNNshave been applied not only to image recognition, but also to generative models: PixcelRNN [74]is a pixel-channel autoregressive generative model of images using Row RNN, which consists of a1D-convolution and a usual RNN, and Diagonal BiLSTM, which is computationally expensive.
In NLP, attempts have been made to avoid the computational cost of attention by approximating causalself-attention with recurrent neural network (RNN) [34] or replacing it with RNN after training [33].
In particular, in [34], an autoregressive pixel-wise image generation task is experimented with anarchitecture where the attentions in iGPT are approximated by RNNs.
These studies are specificto unidirectional Transformers, in contrast to our token-based Sequencer which is the bidirectionalanalog of them.
Method
In this section, we briefly recap the preliminary background on LSTM and further describe the detailsof the proposed architectures.
G l o b a l A v e r a g e P oo l i n g
L i n e a r
L a y e r N o r m
P a t c h E m b e dd i n g
I n p u t
S e q u e n c e r B l o c k
P a t c h M e r g i n g
S e q u e n c e r B l o c k
P W L i n e a r
S e q u e n c e r B l o c k
P W L i n e a r
S e q u e n c e r B l o c k
[Figure]
3.1 Preliminaries: Long short-term memory
LSTM [27] is a specialized recurrent neural network (RNN) for modeling long-term dependencies ofsequences.
Plain LSTM has an input gate i t that controls the storage of inputs, a forget gate f t thatcontrols the forgetting of the former cell state c t − 1 and an output gate o t that controls the cell outputh t from the current cell state c t .
Plain LSTM is formulated as follows:
i
[Equation]
where σ is the logistic sigmoid function and ⊙ is Hadamard product.
BiLSTM [63] is profitable for sequences where mutual dependencies are expected.
A BiLSTMconsists of two plain LSTMs.
Let −→ x be the input series and ←− x be the rearrangement of −→ x in reverseorder.
−−→ h for and ←−−− h back are the outputs obtained by processing −→ x and ←− x with the corresponding
LSTMs, respectively.
Let −−−→ h back be the output ←−−− h back rearranged in the original order, and the outputof BiLSTM is obtained as follows:
[Equation]
Assume that both −−→ h for and −−−→ h back have the same hidden dimension D , which is hyperparameter of
BiLSTM.
Accordingly, vector h has dimension 2 D .
3.2 Sequencer architecture
Overall architecture
In the last few years, ViT and its many variants based on self-attention [16,72, 48, 91] have attracted much attention in computer vision.
Following these, several works [70,71, 46, 28] have been proposed to replace self-attention with MLP.
There have also been studies ofreplacing self-attention with a hard local induced bias module [7, 83] and with a global filter [61]using the fast Fourier transform algorithm (FFT) [10].
This paper continues this trend and attempts to replace the self-attention layer with LSTM [27]: we propose a new architecture aiming at memorysaving by mixing spatial information with LSTM, which is memory-economical compared to ViT,parameter-saving, and has the ability to learn long-range dependencies.
Figure 2a shows the overall structure of Sequencer architecture.
Sequencer architecture takes nonoverlapping patches as input and projects them onto the feature map.
Sequencer block, which is acore component of Sequencer, consists of the following sub-components: (1) BiLSTM layer can mixspatial information more memory-economically for high-resolution images than Transformer layerand more globally than CNN.
(2) MLP for channel-mixing as well as [16, 70].
Sequencer block iscalled Vanilla Sequencer block when plain BiLSTM layers are used as BiLSTM layers as Figure 2dand
Sequencer2D block when BiLSTM2D layers are used as Figure 2e.
We define BiLSTM2D layerlater.
The output of the last block is sent to the linear classifier via the global average pooling layer,as in most other architectures.
BiLSTM2D layer We propose the BiLSTM2D layer as a technique to mix 2D spatial informationefficaciously.
It has two plain BiLSTMs: a vertical BiLSTM and a horizontal one.
For an input
X ∈ R H × W × C , { X : ,w, : ∈ R H × C } Ww =1 is viewed as a set of sequences, where H is the number oftokens in the vertical direction, W is the number of sequences in the horizontal direction, and C isthe channel dimension.
All sequences X : ,w, : are input into the vertical BiLSTM with shared weightsand hidden dimension D :
H ver: ,w, : = BiLSTM
( X : ,w, : ) .
In a very similar manner, { X h, : , : ∈ R W × C } Hh
=
1 is viewed as a set of sequences, and all sequences
X h, : , : are input into the horizontal BiLSTM with shared weights and hidden dimension D as well:
[Equation]
We combine { H ver: ,w, : ∈ R H × 2 D } Ww =1 into H ver ∈ R W × H × 2 D and { H hor h, : , : ∈ R W × 2 D } Hh =1 into
H hor ∈ R W × H × 2 D .
They are then concatenated and processed point-wisely in a fully-connectionlayer.
These processes are formulated as follows:
[Equation]
where FC( · ) denotes the fully-connected layer with weight W ∈ R C × 4 D .
The PyTorch -likepseudocode is shown in Appendix B.1.
BiLSTM2D is more memory-economical and throughput-efficiency than multi-head-attention of
ViT for high-resolution input.
BiLSTM2D involves ( WC + HC ) / 2 dimensional cell states, while amulti-head-attention involves h ∗ ( HW ) 2 dimensional attention map where h is a number of heads.
Thus, as H and W increase, the memory cost of an attention map increases more rapidly than the costof a cell state.
On throughput, the computational complexity of self-attention is O ( W 4 C ) , whereasthe computational complexity of BiLSTM is O ( WC 2 ) where we assume W = H for simplicity.
There are O ( W ) sequential operations for BiLSTM2D. Therefore, assuming we use a sufficientlyefficient LSTM cell implementation, such as official PyTorch LSTMs we are using, the increase ofthe complexity of self-attention is much more rapid than BiLSTM2D. It implies a lower throughputof attention compared to BiLSTM2D. See an experiment in Section 4.5.
Architecture variants For comparison between models of different depths consisting of Sequencer2D blocks, we have prepared three models with different depths: 18, 24, and 36.
The namesof the models are Sequencer2D-S , Sequencer2D-M , and Sequencer2D-L , respectively.
The hiddendimension is set to D =
C/ 4 .
Details of these models are provided in Appendix B.2.
As shown in subsection 4.1, these architectures outperform typical models.
Interestingly, however,subsection 4.3 shows that replacing Sequencer2D block with the simpler Vanilla Sequencer blockmaintains moderate accuracy.
We denote such a model as Vanilla Sequencer.
Note that some of theexplicit positional information is lost in the Vanilla Sequencer because the model treats patches as a1D sequence.
[Table]
above
CycleMLP-B3 [7]
LMLP
[Table]
above
PoolFormer-S36 [83] LMLP
[Table]
above
GFNet-H-S [61]
[Table]
above
Sequencer2D-M
[Table]
above
RegNetY-12GF [59]
[Table]
above
ConvNeXt- [Table]
[Table]
above
Swin-
[Table] [Table]
Mixer-B/16 [70]
[Table]
ViP-M/7 [28]
[Table]
CycleMLP-B4 [7]
[Table]
above
PoolFormer-M36 [83] LMLP
[Table]
above
GFNet-H-B [61]
[Table]
above
Sequencer2D-L
[Table]
above
ConvNeXt-B ↑ [49]
[Table]
Swin-B ↑ [48]
[Table]
Experiments
In this section, we compare Sequencers with previous studies on the IN-1K benchmark [39].
We alsocarry out ablation studies, transfer learning studies, and analysis of the results to demonstrate theeffectiveness of Sequencers.
We adopt PyTorch [56] and timm [80] library to implement models inthe conduct of all experiments.
See Appendix B for more setup details.
4.1 Scratch training on IN-1K
We utilize IN-1K [39], which has 1000 classes and contains 1,281,167 training images and 50,000validation images.
We adopt AdamW optimizer [50].
Following the previous study [72], we adoptthe base learning rate batchsize512 × 5 × 10 − 4 .
The batch sizes for Sequencer2D-S, Sequencer2D-M,and Sequencer2D-L are 2048, 1536, and 1024, respectively.
As a regularization method, stochasticdepth [30] and label smoothing [66] are employed.
As data augmentation methods, mixup [87],cutout [12], cutmix [85], random erasing [88], and randaugment [11] are applied.
Table 1 shows the results that are comparing the proposed models to others with a comparable numberof parameters to our models, including models with local and global receptive fields such as CNNs,
ViTs, and MLP-based and FFT-based models.
Sequencers has the disadvantage that its throughput
[Table]
is slower than other models because it uses RNNs.
In the scratch training on IN-1K
[Table]
however, theyoutperform these recent comparative models in accuracy across their parameter bands.
In particular,
Seqeuncer2D-L is competitive with recently discussed models with comparable parameters such as
ConvNeXt-S [49] and Swin-S [48]
[Table]
with accuracy outperformance of 0.3% and 0.2%, respectively.
4.2 Fine-tuning on IN-1K
In this fine-tuning study, Sequencer2D-L pre-trained on IN-1K at 224 2 resolution is fine-tunedon IN-1K at 392 2 resolution.
We compare it with the other models fine-tuned on IN-1K at 384 2resolution.
Since 392 is divisible by 14, the input at this resolution can be split into patches withoutpadding.
However, note that this is not the case with a resolution of 384 2 .
As Table 1 indicates, even when higher-resolution Sequencer is fine-tuned, it is competitive with thelatest models such as ConvNeXt [49], Swin [48], and GFNet [61].
4.3 Ablation studies
This subsection presents ablation studies based on Sequencer2D-S for further understanding of
Sequencer.
We seek to clarify the effectiveness and validity of the Sequencers architecture in termsof the importance of each component, bidirectional necessaries, setting of the hidden dimension, andthe comparison with simple BiLSTM.
We show where and how relevant the components of BiLSTM2D are: The BiLSTM2D is composedof vertical BiLSTM, horizontal BiLSTM, and channel fusion elements.
We want to see the validity ofvertical BiLSTM, horizontal BiLSTM, and channel fusion.
For this purpose, we examine the removalof channel fusion and vertical or horizontal BiLSTM.
Table 2a shows the results.
Removing channelfusion shows that the performance degrades from 82.3% to 81.6%.
Furthermore, the additionalremoval of vertical or horizontal BiLSTM exposes a 6.0% or 6.6% performance drop, respectively.
Hence, each component discussed here is necessary for Sequencer2D.
We show that the bidirectionality for BiLSTM2D is important for Sequencer.
We compare
Sequencer2D-S with a version that replaces the vertical and horizontal BiLSTMs with verticaland horizontal unidirectional LSTMs.
Table 2b shows that the unidirectional model is 2.6% lessaccurate than the bidirectional model.
This result attests to the significance of using not unidirectional
LSTM but BiLSTM.
It is important to set the hidden dimension of LSTM to a reasonable size.
As described in subsection3.2, Sequencer2D sets the hidden dimension D of BiLSTM to D =
C/ 4 , but this is not necessary ifthe model has channel fusions.
Table 2d compares Sequencer2D-S with the model with increased
D .
Although accuracy is 0.3% improved, FLOPs increase by 65%, and the number of parametersincreases by 60%.
Namely, the accuracy has not improved for the increase in FLOPs.
Moreover, theincrease in dimension causes overfitting, which is discussed in Appendix C.3.
Vanilla Sequencer can also achieve accuracy that outperforms MLP-Mixer [70], but is not as accurate as Sequencer2D.
Following experimental result supports the claim.
We experiment with the
Sequencer2D-S variants, where Vanilla Sequencer blocks replace the Sequencer2D blocks, called
VSequencer-S(H), with incomplete positional information.
In addition, we experiment with a variantof VSequencer-S(H) without the hierarchical structure, which we call VSequencer-S. VSequencerS(PE) is VSequencer-S using ViTs-style learned positional embedding (PE) [16].
Table 2c indicateseffectiveness for combination of LSTM and ViTs-like architecture.
Surprisingly, even with Vanilla
Sequencer and Vanilla Sequencer(H) without PE, the performance reduction from Sequencer2D-S isonly 4.3% and 3.5%, respectively.
According to these results, there is no doubt that Vanilla Sequencerusing BiLSTMs is significant enough, although not as accurate as Sequencer2D.
All LSTMs in the BiLSTM2D layer can be replaced with other recurrent networks such as gatedrecurrent units (GRUs) [8] or tanh-RNNs to define BiGRU2D layer or BiRNN2D layer.
We alsotrained these models on IN-1K, so see Table 2e for the results.
The table suggests that all of thesevariants, including RNN-cell, work well.
Also, tanh-RNN performs slightly worse than others,probably due to its lower ability to model long-range dependence.
4.4 Transfer learning and semantic segmentation
Sequencers perform well on IN-1K, and they have good transferability.
In other words, they havesatisfactory generalization performance for a new domain, which is shown below.
We utilize thecommonly used CIFAR-10 [38], CIFAR-100 [38], Flowers-102 [55], and Stanford Cars [37] forthis experiment.
See the references and Appendix B.4 for details on the datasets.
The results of theproposed model and the results in previous studies of models with comparable capacity are presentedin Table 3.
In particular, Sequencer2D-L achieves results that are competitive with [-CaiT-S-
 [71]and-]CaiT-S-36 [73]and EfficientNet-B7 [67].
We experiment for semantic segmentation on ADE20K[89] dataset.
See Appendix C.4 for details onthe setup.
Sequencer outperforms PVT [79] and PoolFormer [83] with similar parameters; comparedto PoolFormer, mIoU is about 6 pts higher.
We have investigated a commonly object detection model with Sequencer as the backbone.
Itsperformance is not much different from the case of ResNet [22] backbone.
Its improvement is thefuture work.
See Appendix C.5.
4.5 Analysis and visualization
In this subsection, we investigate the properties of Sequencer in terms of resolution adaptability andefficiency.
Furthermore, effective receptive field (ERF) [51] and visualization of the hidden statesprovides insight into the question of how Sequencer recognizes images.
One of the attractive properties of Sequencer is its flexible adaptability to the resolution, with minimalimpact on accuracy even when the resolution of the input image is varied from one-half to twice.
In comparison, architectures like MLP-Mixer [70] have a fixed input resolution, and GFNet [61]requires interpolation of weights in the Fourier domain when inputting images with a resolutiondifferent from training images.
We evaluate the resolution adaptability of models comparatively byinputting different resolution images to each model, without fine-tuning, with pre-trained weightson IN-1K at the resolution of 224 2 .
Figure 3a compares absolute top-1 accuracy on IN-1K, and
Figure 3b compares relative one to the input image with the resolution of 224 2 .
By increasing theresolution by 28 for Sequencer2D-S and by 32 for other models, we avoid padding and prevent [Table] the effect of padding on accuracy.
Compared to DeiT-S [72], GFNet-S [61], CycleMLP-B2 [7],and ConvNeXt-T [49], Sequencer-S’s performance is more sustainable.
The relative accuracy isconsistently better than ConvNeXt [49], which is influential in the lower-resolution band, and, at448 2 resolution, 0.6% higher than CycleMLP [7], which is influential in the double-resolution band.
It is noteworthy that Sequencer continues to maintain high accuracy on double resolution.
The higher the input resolution, the higher memory-efficiency and throughput of Sequencers whencompared to DeiT [72].
Figure 3 shows the efficiency of Sequencer2D-S when compared to DeiT-Sand ConvNeXt-T [49].
Memory consumption increases rapidly in DeiT-S and ConvNeXt-T withincreasing input resolution, but more gradual increase in Sequencer2D-S.
The result strongly impliesthat it has more practical potential as the resolution increases than the ViTs.
At a resolution of 224 2 ,it is behind DeiT in throughput, but it stands ahead of DeiT when images with a resolution of 896 2are input.
In general, CNNs have localized, layer-by-layer expanding receptive fields, and ViTs without shiftedwindows capture global dependencies, working the self-attention mechanism.
In contrast, in thecase of Sequencer, it is not clear how information is processed in Sequencer block.
We calculated
ERF [51] for ResNet-50 [22], DeiT-S [72], and Sequencer2D-S as shown in Figure 5.
ERFs of
Sequencer2D-S form a cruciform shape in all layers.
The trend distinguishes it from well-knownmodels such as DeiT-S and ResNet-50.
More remarkably, in shallow layers, Sequencer2D-S has awider ERF than ResNet-50, although not as wide as DeiT.
This observation confirms that LSTMs in
Sequencer can model long-term dependencies as expected and that Sequencer recognizes sufficientlylong vertical or horizontal regions.
Thus, it can be argued that Sequencer recognizes an image in avery different way than CNNs or ViTs.
For more details on ERF and additional visualization, see
Appendix D.
Moreover we also visualized a hidden state of vertical and horizontal BiLSTM, and a feature mapafter channel fusion, and the results are visualized in Figure 4.
It demonstrates that our Sequencerhas the hidden states interact with each other over the vertical and horizontal directions.
The closertokens are in position, the stronger their interaction tends to be; the farther tokens are in position, theless their interaction tends to be.
Conclusions
We propose a novel and simple architecture that leverages LSTM for computer vision.
It is demonstrated that modeling long-range dependencies by self-attention is not necessarily essential in computer vision and that competitive performance can be achieved with LSTM.
Our experiments showthat Sequencer has a good memory-resource/accuracy and parameter/accuracy tradeoffs, comparableto the main existing methods.
Despite the impact of recursion on throughput, we have demonstrated [Figure] [Figure] [Figure]
[Figure]
[Figure] [Figure] benefits over it. We believe that these results raise a number of interesting issues.
Improving
Sequencer’s poor throughput is one example.
Moreover, we expect that investigating the internalmechanisms of our model using methods other than ERF will further our understanding of how thisarchitecture works.
In addition, it would be important to analyze in more detail the features learnedby Sequencer in comparison to other architectures.
We hope this will lead to a better understandingof the role of various inductive biases in computer vision.
Furthermore, we expect that our resultstrigger further study beyond the domain or research area.
Especially, it would be a very interestingopen question to see if such a design works with time-series data in vision such as video or in amulti-modal problem setting combined with another modality such as video with audio.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] The throughputs for our models
are a bit inferior because of using RNNs, see Section 4.1 and 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Appendix A. (d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] Included in
supplemental material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[No] We follow the typical practice used in our baseline
methodology no error bars are reported.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[N/A] They are included only if they are
of essential significance to each experiment/item.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]