S4ND: Modeling Images and Videos as Multidimensional Signals Using State Spaces
Anonymous Author(s)
AffiliationAddress email
Abstract
Visual data such as images and videos are typically modeled as discretizationsof inherently continuous, multidimensional signals.
Existing continuous-signalmodels attempt to exploit this fact by modeling the underlying signals of visual(e.g., image) data directly.
However, they have not yet been able to achievecompetitive performance on practical vision tasks such as large-scale image andvideo classification.
Building on a recent line of work on deep state space models(SSMs), we propose S4ND, a new multidimensional SSM layer that extends SSMs’continuous-signal modeling ability to multidimensional data including images andvideos.
We show that S4ND can model large-scale visual data in 1D, 2D, and 3Das continuous multidimensional signals and demonstrate strong performance bysimply swapping Conv2D and self-attention layers with S4ND layers in existingstate-of-the-art models.
On ImageNet-1k, S4ND exceeds the performance of a
ViT baseline by 1 . 5% accuracy when training with a 1D sequence of patches, andmatches ConvNeXt when modeling images in 2D. For videos, S4ND improves onan inflated 3D ConvNeXt in activity classification on HMDB-51 by 4% accuracy.
S4ND implicitly learns global, continuous convolutional kernels that are resolutioninvariant by construction, providing an inductive bias that enables generalizationacross multiple resolutions.
By developing a simple bandlimiting modificationto S4 to overcome aliasing, S4ND achieves strong zero-shot (unseen at test time)resolution performance, e.g. achieving 88 .
7% accuracy on CIFAR-10 when trainedon 16 × 16 and tested on 32 × 32 images.
When trained with progressive resizing, S4ND comes within ∼ 1% of a high-resolution model while training 22% faster.
Introduction
Modeling visual data such as images and videos is a canonical problem in deep learning.
In the lastfew years, many modern deep learning backbones that achieve strong performance on benchmarkslike ImageNet [46] have been proposed.
These backbones are diverse, and include 1 D sequencemodels such as the Vision Transformer (ViT) [10], which treats images as sequences of patches, and2 D and 3 D models that use local convolutions over images and videos (ConvNets) [31, 20, 49, 51,52, 36, 19, 26, 42, 55, 12].
A commonality among modern vision models capable of achieving state-of-the-art (SotA) performance is that they treat visual data as discrete pixels rather than continuous-signals.
However, imagesand videos are discretizations of multidimensional and naturally continuous signals, sampled ata fixed rate in the spatial and temporal dimensions.
Ideally, we would want approaches that arecapable of recognizing this distinction between data and signal, and directly model the underlyingcontinuous-signals.
This would give them capabilities like the ability to adapt the model to datasampled at different resolutions.
[Figure] ( 1/∆ 002. Discretize $ A natural approach to building such models is to parameterize and learn continuous convolutionalkernels, which can then be sampled differently for data at different resolutions [13, 43, 47, 16, 17].
Among these, deep state space models (SSM) [16], in particular S4 [17], have achieved SotA resultsin modeling sequence data derived from continuous-signals, such as audio [14].
However, a keylimitation of SSMs is that they were developed for 1 D signals, and cannot directly be applied to visualdata derived from multidimensional “ND” signals.
Given that 1 D SSMs outperform other continuousmodeling solutions for sequence data [17], and have had preliminary success on image [17] andvideo classification [25], we hypothesize that they may be well suited to modeling visual data whenappropriately generalized to the setting of multidimensional signals.
Our main contribution is S4ND, a new deep learning layer that extends S4 to multidimensionalsignals.
The key idea is to turn the standard SSM (a 1 D ODE) into a multidimensional PDE governedby an independent SSM per dimension.
By adding additional structure to this ND SSM, we showthat it is equivalent to an ND continuous convolution that can be factored into a separate 1 D SSMconvolution per dimension.
This results in a model that is efficient and easy to implement, using thestandard 1 D S4 layer as a black box.
Furthermore, it can be controlled by S4’s parameterization,allowing it to model both long-range dependencies, or finite windows with a learnable window sizethat generalize conventional local convolutions [0].
We show that S4ND can be used as a drop-in replacement in strong modern vision architectureswhile matching or improving performance in 1 D, 2 D, and 3 D. With minimal change to the trainingprocedure, replacing the self-attention in ViT with S4-1D improves top- 1 accuracy by 1 . 5% , andreplacing the convolution layers in a 2 D ConvNeXt backbone [36] preserves its performance on
ImageNet-1k [7].
Simply inflating this pretrained S4-2D-ConvNeXt backbone to 3 D improves videoactivity classification results on HMDB-51 [32] by 4 points over the pretrained ConvNeXt baseline.
Additionally, we propose a low-pass bandlimiting modification to S4 that encourages the learnedconvolutional kernels to be smooth.
While S4ND can be used at any resolution, performance sufferswhen moving between resolutions due to aliasing artifacts in the kernel, an issue also noted by priorwork on continuous models [43].
While S4 was capable of transferring between different resolutionson audio data [17], visual data presents a greater challenge due to the scale-invariant properties ofimages in space and time [45], as sampled images with more distant objects are more likely to containpower at frequencies above the Nyquist cutoff frequency.
Motivated by this, we propose a simplecriteria that masks out frequencies in the S4ND kernel that lie above the Nyquist cutoff frequency.
The continuous-signal modeling capabilities of S4ND open the door to new training recipes, suchas the ability to train and test at different resolutions.
On the standard CIFAR-10 [30] and Celeb-A [37] datasets, S4ND degrades by as little as 1 . 3% when upsampling from low- to high-resolutiondata (e.g. 128 × 128 → 160 × 160 ), and can be used to facilitate progressive resizing to speed uptraining by 22% with ∼ 1% drop in final accuracy compared to training at the high resolution alone.
We also validate that our new bandlimiting method is critical to these capabilities, with ablationsshowing absolute performance degradation of up to 20%+ without it.
Related Work
Image Classification.
There is a long line of work in image classification, with much of the 2010sdominated by ConvNet backbones [31, 20, 49, 51, 52].
Recently, Transformer backbones, such as
ViT [10], have achieved SotA performance on images using self-attention over a sequence of 1 Dpatches [34, 33, 54, 61, 9].
Their scaling behavior in both model and dataset training size is believedto give them an inherent advantage over ConvNets [10], even with minimal inductive bias.
Liu et al.[36] introduce ConvNeXt, which modernizes the standard ResNet architecture [20] using moderntraining techniques, matching the performance of Transformers on image classification.
We select abackbone in the 1 D and 2 D settings, ViT and ConvNeXt, to convert into continuous-signal models byreplacing the multi-headed self-attention layers in ViT and the standard Conv2D layers in ConvNeXtwith S4ND layers, maintaining their top-1 accuracy on large-scale image classification.
S4 & Video Classification.
To handle the long-range dependancies inherent in videos, [25] used1D S4 for video classification on the Long-form Video Understanding dataset [58].
They firstapplied a Transformer to each frame to obtain a sequence of patch embeddings for each videoframe independently, followed by a standard 1D S4 to model across the concatenated sequence ofpatches.
This is akin to previous methods that learned spatial and temporal information separately[27], for example using ConvNets on single frames, followed by an LSTM [22] to aggregate temporal information.
In contrast, modern video architectures such as 3 D ConvNets and Transformers[19, 26, 42, 55, 12, 29, 35, 58, 2, 1] show stronger results when learning spatiotemporal featuressimultaneously, which the generalization of S4ND into multidimensions now enables us to do.
Continuous-signal Models.
Visual data are discretizations of naturally continuous signals thatpossess extensive structure in the joint distribution of spatial frequencies, including the properties ofscale and translation invariance.
For example, an object in an image generates correlations betweenlower and higher frequencies that arises in part from phase alignment at edges [41].
As an objectchanges distances in the image, these correlations remain the same but the frequencies shift.
Thisrelationship can potentially be learned from a coarsely sampled image and then applied at higherfrequency at higher resolution.
A number of continuous-signal models have been proposed for the visual domain to learn theseinductive biases, and have led to additional desirable properties and capabilities.
A classic exampleof continuous-signal driven processing is the fast Fourier transform, which is routinely used forfiltering and data consistency in computational and medical imaging [8].
Using a continuous function,
NeRF represents a static scene as a continuous function, allowing them to render scenes smoothlyfrom multiple viewpoints [39].
CKConv [44] learns a continuous representation to create kernels ofarbitrary size for several data types including images, with additional benefits such as the ability tohandle irregularly sampled data.
FlexConv [43] extends this work with a learned kernel size, andshow that images can be trained at low resolution and tested at high resolution if the aliasing problemis addressed.
S4 [17] increased abilities to model long-range dependancies using continuous kernels,allowing SSMs to achieve SotA on sequential CIFAR [30].
However, these methods including 1D
S4 have been applied to relatively low dimensional data, e.g., time series, and small image datasets.
S4ND is the first continuous-signal model applied to high dimensional visual data with the ability tomaintain SotA performance on large-scale image and video classification.
Progressive Resizing.
Training times for large-scale image classification can be quite long, atrend that is exacerbated by the emergence of foundation models [3].
A number of strategies haveemerged for reducing overall training time.
Fix-Res [53] trains entirely at a lower resolution, andthen fine-tunes at the higher test resolution to speed up training in a two-stage process.
Mix-andMatch [23] randomly samples low and high resolutions during training in an interleaved manner.
Aneffective method to reduce training time on images is to utilize progressive resizing.
This involvestraining at a lower resolution and gradually upsampling in stages.
For instance, fastai [11] utilizedprogressive resizing to train an ImageNet in under 4 hours.
EfficientNetV2 [52] coupled resizing with a progressively regularization schedule, increasing the regularization as well to maintain accuracy.
In
EfficientNetV2 and other described approaches, the models eventually train on the final test resolution.
As a continuous-signal model, we demonstrate that S4ND is naturally suited to progressive resizing,while being able to generalize to unseen resolutions at test time.
Preliminaries
State space models.
S4 investigated state space models, which are linear time-invariant systems thatmap signals u ( t ) (cid:55)→ y ( t ) and can be represented either as a linear ODE (equation (1)) or convolution(equation (2)).
Its parameters are A ∈ C N × N and B , C ∈ C N for a state size N .
[Equation]
Basis functions.
For the clearest intuition, we think of the convolution kernel as a linear combination(controlled by C ) of basis kernels K n ( t ) (controlled by A , B )
[Equation]
Discretization.
The SSM (1) is defined over a continuous-time axis and produces continuous-timeconvolution kernels (2)(3).
Given a discrete input sequence u 0 , u 1 , . . .
sampled uniformly from anunderlying signal u ( t ) at a step size ∆ (i.e. u k = u ( k ∆) ), the kernel can be sampled to match the rateof the input.
Note that instead of directly sampling the kernel, standard discretization rules should beapplied to minimize the error from the discrete to the continuous-time kernel [17].
For inputs given atdifferent resolutions, the model can then simply change its ∆ value to compute the kernel at differentresolutions.
We note that the step size ∆ does not have to be exactly equal to a “true sampling rate” of theunderlying signal, but only the relative rate matters.
Concretely, the discrete-time kernel dependsonly on the product ∆ A and ∆ B , and S4 learns separate parameters ∆ , A , B .
S4.
S4 is a special SSM with prescribed ( A , B ) matrices that define well-behaved basis functions,and an algorithm that allows the convolution kernel to be computed efficiently.
Variants of S4 existthat define different basis functions, such as simple diagonal SSMs [18], or one that defines truncated
Fourier functions K n ( t ) = sin(2 πnt ) I [0 , 1] [0] (Fig. 1).
These versions of S4 have easy-to-interpretbasis functions that will allow us to control the frequencies in the kernel (Section 4.2).
Method
We describe the proposed S4ND model for the 2D case only, for ease of notation and presentation.
The results extend readily to general dimensions; full statements and proofs for the general case are in
Appendix B. Section 4.1 describes the multidimensional S4ND layer, and Section 4.2 describes oursimple modification to restrict frequencies in the kernels.
Fig.
1 illustrates the complete S4ND layer.
4.1 S4ND
We begin by generalizing the (linear time-invariant) SSM (1) to higher dimensions.
Notationally, we denote the individual time axes with superscripts in parentheses.
Let u =
u
( t (1) , t (2) )and y =
y ( t (1) , t (2) ) be the input and output which are signals R 2 → C , and x =( x (1) ( t (1) , t (2) ) , x (2) ( t (1) , t (2) ))
∈ C N (1) × N (2) be the SSM state of dimension N (1) × N (2) , wherex ( τ ) :
R 2 → C N ( τ ) .
Definition 1 (Multidimensional SSM) .
Given parameters A ( τ ) ∈ C N ( τ ) × N ( τ ) , B ( τ ) ∈
C N ( τ ) × 1 ,
C ∈ C N (1) × N (2) , the 2D SSM is the map u
(cid:55)→ y defined by the linear PDE with initial condition
[Equation]
Note that Definition 1 differs from the usual notion of multidimensional SSM, which is simply amap from u ( t ) ∈
C n (cid:55)→ y ( t ) ∈
C m for higher-dimensional n, m > 1 but still with 1 time axis.
However, Definition 1 is a map from u ( t 1 , t 2 ) ∈ C 1 (cid:55)→ y ( t 1 , t 2 ) ∈ C 1 for scalar input/outputsbut over multiple time axes.
When thinking of the input u ( t (1) , t (2) ) as a function over a 2D grid,
Definition 1 can be thought of as a simple linear PDE that just runs a standard 1D SSM over eachaxis independently.
Analogous to equation (2), the 2D SSM can also be viewed as a multidimensional convolution.
Theorem 1.
(4) is a time-invariant system that is equivalent to a 2D convolution y
=
K ∗ u by thekernel
[Equation]
[Equation] This kernel is a linear combination of the N (1) × N (2) basis kernels { K (1) n (1) ( t (1) ) ⊗ K (1) n (2) ( t (2) ) :
n (1) ∈
[ N (1) ] , n (2) ∈
[ N (2) ] } where K ( τ ) are the standard 1D SSM kernels (3) for each axis .
However, a limitation of this general form is that the number of basis functions N (1) × N (2) × . .
.grows exponentially in the dimension, increasing the parameter count (of C ) and overall computationdramatically.
This can be mitigated by factoring C as a low-rank tensor.
Theorem 2.
Suppose that C ∈ C N (1) × N (2) is a low-rank tensor C
= (cid:80) ri =1 C (1) i ⊗ C (2)
i whereeach C ( τ ) i ∈ C N ( τ ) .
Then the kernel (5) also factors as a tensor product of 1D kernels
K
[Equation]
In our experiments, we choose C as a rank-1 tensor, but the rank can be freely adjusted to tradeoffparameters and computation for expressivity.
Using the equivalence between (1) and (2), Theorem 2 also has the simple interpretion as defining an independent 1D SSM along each axis of themultidimensional input.
4.2 Resolution Change and Bandlimiting
SSMs in 1D have shown strong performance in the audio domain, and can nearly preserve fullaccuracy when tested zero-shot on inputs sampled at very different frequencies [17].
This capabilityrelies simply on scaling ∆ by the relative change in frequencies (i.e., if the input resolution is doubled,halve the SSM’s ∆ parameter).
However, sampling rates in the spatial domain are often much lowerthan temporally, leading to potential aliasing when changing resolutions.
A standard technique toavoid aliasing is to apply a low-pass filter to remove frequencies above the Nyquist cutoff frequency.
For example, when A is diagonal with n -th element a n , each basis function has simple form
K n ( t ) =
e t a n B n .
Note that the frequencies are mainly controlled by the imaginary part of a n .
Wepropose the following simple method: for any n such that a n · ∆ < 12 α , mask out the correspondingcoefficient of the linear combination C n (equation (3)).
Here α is a hyperparameter that controlsthe cutoff; theoretically, α = 1 . 0 corresponds to the Nyquist cutoff if the basis functions are puresinusoids.
However, due to the decay e ℜ ( a n ) arising from the real part as well as approximationsarising from using finite-state SSMs, α often has to be set lower empirically.
Experiments
We evaluate S4ND on large-scale image classification in Section 5.1 in the 1D and 2D settings,followed by activity classification in videos in Section 5.2 in the 3D setting, where using S4ND as a drop-in replacement for standard deep learning layers matches or improves performance in allsettings.
In Section 5.3, we performed controlled ablations to highlight the benefits of S4ND as acontinuous-signal model in images.
5.1 S4ND in 1D & 2D: Large-scale Image Classification
First, we show that S4ND is a drop-in replacement for existing visual modeling layers such as 1Dself-attention and 2D local convolutions, with no degradation in top- 1 performance when used inmodern backbones such as ViT [10] and ConvNeXt [36] on ImageNet-1k [7].
Baselines and Methodology.
We consider large-scale image classification on the ImageNet-1kdataset, which consists of 1000 classes and 1 .
3 M images.
We start with two strong baselines: ViT-B( base , 88 M parameters) for processing images in the 1D setting and ConvNeXt-T ( tiny , 28 .
4 M) inthe 2D setting.
(We omit the postfix “B” and “T” for brevity).
More recent works using Transformerson images have surpassed ViT, but we focus on the original ViT model to highlight specifically thedrop-in capability and performance difference in self-attention vs. S4ND layers.
We first swap theself-attention layers in ViT with S4ND layers, and call this model S4ND-ViT.
Notably, we simplify
ViT by removing the positional encodings, as S4ND does not require injecting this inductive bias.
Similarly, we swap the depthwise Conv2D layers in the ConvNeXt blocks with S4ND layers, whichwe call S4ND-ConvNeXt.
These result in similar parameter counts compared to their baseline models,with S4ND-ViT at 88 .
8 M and S4ND-ConvNeXt at 30 M.
Training.
For all ImageNet models, we train from scratch with no outside data and adopt thetraining procedure from [54, 60], which uses the AdamW optimizer [38] for 300 epochs, cosine decaylearning rate, weight decay 0.05, and aggressive data augmentations including RandAugment [5],
Mixup [62], and AugMix [21].
We add RepeatAug [24] for ConvNeXt and S4ND-ConvNeXt.
Theinitial learning rate for ViT (and S4ND-ViT) is 0 .
001 , while for ConvNeXt (and S4ND-ConvNeXt) itis 0 . 0001 .
See Appendix C.1 for additional training procedure details.
[Table]
Results.
Table 1 shows top- 1 accuracy resultsfor each model on ImageNet.
After reproducing the baselines, S4ND-ViT was able to mod- [Table] erately boost performance by + 1 . 5 % over ViT,while S4ND-ConvNeXt matched the original
ConvNeXt’s performance.
This indicates that
S4ND is a strong primitive that can replace self-
[Table]
attention and standard 2D convolutions in prac- [Table] tical image settings with large-scale data.
5.2 S4ND in 3D: Video Classification
Next, we demonstrate the flexible capabilitiesof S4ND in settings involving pretraining andeven higher-dimensional signals.
We use the activity recognition dataset HMDB-51 [32] whichinvolves classifying videos in 51 activity classes.
Baselines and Methodology.
Prior work demonstrated that 2D CNNs (e.g. pretrained on ImageNet)can be adapted to 3D models by 2D to 3D kernel inflation (I3D [4]), in which the 2D kernels arerepeated temporally N times and normalized by 1 /N .
Our baseline, which we call ConvNeXt-I3D,uses the 2D ConvNeXt pretrained on ImageNet (Section 5.1) with I3D inflation.
We additionally testthe more modern spatial-temporal separated 3D convolution used by S3D [59] and R(2+1)D
[56],which factor the 3D convolution kernel as the outer product of a 2D (spatial) by 1D (temporal) kernel.
Because of its flexible factored form (Section 4.1), S4ND automatically has these inflation capabilities.
We inflate the pretrained S4ND-ConvNeXt simply by loading the pretrained 2D model weights forthe spatial dimensions, and initializing the temporal kernel parameters
A (3) , B (3) , C (3) from scratch.
We note that this model is essentially identical to the baseline ConvNeXt-S3D except that eachcomponent of the factored kernels use standard 1D S4 layers instead of 1D local convolutions. [Table]
Finally, by varying the initialization of these parameters, we can investigate additional factors affectingmodel training; in particular, we also run an ablation on the kernel timescales ∆ .
Training.
Our training procedure is minimal, using only RGB frames (no optical flow).
We sampleclips of 2 seconds with 30 total frames at 224 × 224 , followed by applying RandAugment; weperformed a small sweep of the RandAugment magnitude for each model.
All models are trainedwith learning rate 0 .
0001 and weight decay 0 .
Additional details are included in Appendix C.2.
Results.
Results are presented in Table 2.
Our baselines are much stronger than prior work in thissetting, 8% top- 1 accuracy higher than the original I3D model in the RGB frames only setting, andconfirming that separable kernels (S3D) perform at least as well as standard inflation ( +0 . 53% ).
S4ND-ConvNeXt-3D improves over the baseline ConvNeXt-I3D by +4 .
0% with no difference inmodels other than using a temporal S4 kernel.
This even exceeds the performance of I3D whentrained on optical flow.
[Table]
Finally, we show how S4ND’sparameters can control for factors such as the kernel length(Table 2).
Note that our temporal kernels K 3 ( t 3 ) are alwaysfull length (30 frames in thiscase), while standard convolution kernels are shorter temporally and require setting thewidth of each layer manually
asa hyperparameter [59].
S4 layers have a parameter ∆ that canbe interpreted such that 1∆ is the expected length of the kernel.
By simply adjusting this hyperparameter, S4ND can be essentially initialized with length- 1 temporal kernels that can automaticallylearn to cover the whole temporal length if needed.
We hypothesize that this contributes to S4ND’simproved performance over baselines.
5.3 Continuous-signal Capabilities for Images
Images are often collected at varied resolutions, even with the same hardware, so it is desirable thatmodels generalize to data sampled at different resolutions.
We show that S4ND inherits this capabilityas a continuous-signal model, with strong zero-shot performance when changing resolutions, andthe ability to train with progressively resized multi-resolution data.
We ablate that our proposedbandlimiting modification is critical to achieving strong performance when changing resolutions.
Setup.
We focus on image classification on 2 D benchmark datasets: a dataset with low-resolutionimages (CIFAR-10) and one with higher-resolution images (Celeb-A).
For each dataset, we specifya base image resolution ( base ), and two lower resolutions ( mid , low ), summarized in Table 3.
Tohighlight that S4ND’s continuous capabilities are independent of backbone, we experiment with [-
 [Table] different-]2different 2D backbones: an isotropic, fixed-width model backbone on CIFAR-10 [30] and a small
ConvNeXt backbone on Celeb-A [37].
For each backbone, we compare S4ND’s performance to
Conv2D layers as a standard, widely used baseline.
Additional details can be found in Appendix C.3.
[Table] We first verify that S4ND achieves comparable test classification performance with baseline Conv2Dmodels.
Table 1 shows that we exceed the performance of Conv2D on both tasks, with S4NDimproving over Conv2D models by 0 .
4% on CIFAR-10 and 0 .
3% on Celeb-A.
Zero-Shot Resolution Change.
We train S4ND and Conv2D models at either low or mid resolution,and test them at the base resolution for each dataset.
We also compare to FlexConv [43] on CIFAR-10,which is the current SotA for zero-shot resolution change.
Compared to training and testing at thebase resolution, we expect that Conv2D should degrade more strongly than S4ND, since it cannotadapt its kernel appropriately to the changed resolution.
Table 4 shows that S4ND outperforms
Conv2D on mid → base by 15+ points and low → base by 40+ points on CIFAR-10, and 9+ pointsand 3+ points on Celeb-A.
In fact, S4ND yields better performance on low → base (a more difficulttask) than Conv2D does from mid → base (an easier task), improving by 1 .
1% on CIFAR-10 and3% on Celeb-A.
Compared to FlexConv on CIFAR-10 mid → base , S4ND improves zero-shotperformance by 5+ points, setting a new SotA.
Progressively Resized Training.
We provide an exploration of training with progressive resizing [11,52] i.e. training in multiple stages at different resolutions.
The only change we make from standardtraining is to reset the learning rate scheduler at the beginning of each stage (details in Appendix C.3).
We compare S4ND and the Conv2D baseline with progressive resizing in Table 5.
For CIFAR-10, we train with a low → base , 80 − 20 epoch schedule, and perform within ∼ 1% ofan S4ND model trained with base resolution data while speeding up training by 21 . 8% .
We note that
Conv2D attains much higher speedups as a consequence of highly optimized implementations, whichwe discuss in more detail in Section 6.
For Celeb-A, we explore flexibly combining the benefits ofboth progressive resizing and zero-shot testing, training with a low → mid , 16 − 4 epoch schedulethat uses no base data.
We outperform Conv2D by 7 . 5%+ , and attain large speedups of 50%+ overtraining at the base resolution.
Effect of Bandlimiting.
Bandlimiting in S4ND is critical to generalization at different resolutions.
We analyze the effect of the bandlimiting parameter α on CIFAR-10 performance when doing zeroshot resolution change.
We additionally vary the choice of basis function K n ( t ) used in the SSM.
Fig.
2 shows that zero-shot performance on base degrades for larger values of α i.e. for cutoffsthat do not remove high frequencies that violate the Nyquist cutoff.
As we would expect, this holdsregardless of the choice of basis function.
In Section 5.3, we visualize learned kernels with andwithout bandlimiting, showing that bandlimiting improves smoothness.
Appendix C.3 includesadditional experiments that analyze α on Celeb-A, and when doing progressively resized training.
Discussion
Summary.
We introduced S4ND, a multidimensional extension of S4 that models visual dataas continuous valued signals.
S4ND is the first continuous model that matches SotA baselines onlarge-scale 1 D and 2 D image classification on ImageNet-1k, as well as outperforming a strongpretrained model in a 3 D video classification setting.
As a continuous-signal model, S4ND inheritsuseful properties that are absent from standard visual modeling approaches, such as zero-shot testingon unseen resolutions without a significant performance drop. [Figure] [Table]
Limitations.
A limitation of S4ND is its training speed in high dimensions.
In the 1 D image setting,
S4ND-ViT has similar training speed to ViT; however, in the 2 D setting, the S4ND-ConvNeXt was2 × slower than the baseline ConvNeXt.
We remark that vanilla local convolutions have been heavilyoptimized for years, and we expect that layers such as S4ND can be substantially sped up with moreoptimized implementations.
Our core computational primitives accounting for 65% of our runtime(FFT, pointwise operations, inverse FFT) are all bottlenecked by reading from and writing to GPUmemory [50].
With a more optimized implementation that fuses these operations [57] (i.e., loadingthe input once from GPU memory, perform all operations, then write result back to GPU memory),we expect to speed these up by 3 × .
Further discussion can be found in Appendix D.
Future work.
We presented a first step in using continuous-signal models in images and videos, andbelieve this opens the door to new capabilities and directions.
For example, recent video benchmarkdatasets are significantly larger than the HMDB-51 dataset used in our experiments [28, 15, 6, 40, 48],and can be explored.
In addition, we demonstrated capabilities in zero-shot resolution spatially ,but even less work has been done on zero-shot testing temporally .
This opens up an exciting newdirection of work that could allow models to be agnostic to different video sampling rates as well,capable of testing on higher unseen sampling rates, or irregular (non-uniform) sampling rates.
Asmodels become larger and combine multiple modalities, S4ND has shown strong promise in beingable to better model underlying continuous-signals and create new capabilities across visual data,audio, time-series (e.g., wearable sensors, health data), and beyond.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
[Table]
(b) Did you include complete proofs of all theoretical results?
[Yes]
[Table]
If you ran experiments...
[Table] (a) Did you include the code, data, and instructions needed to reproduce the main exper- [Table]
imental results (either in the supplemental material or as a URL)?
[Yes] The main
[Table] method can be implemented based on the released S4 repository, and we will publically [Table]
release our code.
All hyperparameters and experimental details are reported in the [Table] Appendix. [Table]
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
[Table]
were chosen)?
[Yes]
Yes
[Table] (c) Did you report error bars (e.g., with respect to the random seed after running experi- [Table]
ments multiple times)?
[Yes] Full experimental results including standard deviations [Table] can be found in the Appendix.
[Table] (d) Did you include the total amount of compute and the type of resources used (e g., type [Table]
of GPUs, internal cluster, or cloud provider)?
[Yes] Reported in the Appendix.
[Table]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
[Table] (a) If your work uses existing assets, did you cite the creators?
[N/A]
[Table] (b) Did you mention the license of the assets? [N/A] [Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[-u ( t (1) , · , · ) is a function from R 
-]
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with [-no outside data using 8 Nvidia A
 GPUs. For both ViT and ConvNeXt experiments, we follow the procedure from T2T-ViT-]human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A](b
) Did
you
describe any potential participant risks, with links to Institutional
Review
Board (IRB approvals
[-On CIFAR-10, we use a bandlimit α of 0 . 10 or 0 . 20 for the first stage, depending on whether the firststage was training at 8 × 8 ( low ) or 16 × 
 ( mid ) respectively.-], if applicable?
[N/A
c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A