E FFECTIVELY M ODELING T IME S ERIES WITH S IMPLE D ISCRETE S TATE S PACES
A BSTRACT
Time series modeling is a well-established problem, which often requires that methods (1) expressively represent complicated dependencies
(2) forecast long horizons and (3) efficiently train over long sequences.
State space models (SSMs are classical models for time series
, and prior works combine SSMs with deep learning layers for efficient sequence modeling
However we find fundamental limitations with these prior approaches, proving their SSM representations cannot express autoregressive time series processes.
We thus introduce
S PACE T IME , a new state-space time series architecture that improves all three criteria.
For expressivity, we propose a new SSM parameterization based on the companion matrix —a canonical representation for discrete-time processes—which enables
S PACE T IME ’s SSM layers to learn desirable autoregressive processes.
For long horizon forecasting, we introduce a “closed-loop” variation of the companion
SSM which enables S PACE T IME to predict many future time-steps by generating its own layer-wise inputs.
For efficient training and inference, we introduce an algorithm that reduces the memory and compute of a forward pass with the companion matrix.
With sequence length ℓ and state-space size d , we go from ˜ O
( dℓ )
na
¨ ıvely to ˜ O
( d + ℓ ) .
In experiments, our contributions lead to state-of-the-art results on extensive and diverse benchmarks with best or second-best AUROC on
ECG and speech time series classification and best MSE on
16 Informer forecasting tasks.
Furthermore
, we find
S PACE T IME (1) fits
AR( p ) processes that prior deep SSMs fail on, (2) forecasts notably more accurately on longer horizons than prior state-of-the-art and
3) speeds
up training on real-world ETTh1 data by 73% and 80% relative wall-clock time over Transformers and LSTMs.
I NTRODUCTION
Time series modeling is a well-established problem, with tasks such as forecasting and classification motivated by many domains such as healthcare, finance, and engineering (Shumway et al., 2000).
However, effective time series modeling presents several challenges
: • First, methods should be expressive enough to capture complex, long-range, and autoregressive dependencies.
Time series data often reflects higher order dependencies, seasonality, and trends, which govern how past samples determine future samples (Chatfield, 2000).
This motivates many classical approaches that model these properties (Box et al., 1970; Winters, 1960), alongside expressive deep learning mechanisms such as attention (Vaswani et al., 2017) and fully connected layers that model interactions between every sample in an input sequence (Zeng et al., 2022).
• Second, methods should be able to forecast a wide range of long horizons over various data domains.
Reflecting real world demands, popular forecasting benchmarks evaluate methods on 34 different tasks (Godahewa et al., 2021) and 24 − 960 time-step horizons Zhou et al.
Furthermore, as testament to accurately learning time series processes, forecasting methods should ideally also be able to predict future time-steps on horizons they were not explicitly trained on.
Finally, methods should be efficient with training and inference.
Many time series applications require processing very long sequences, e.g. , classifying audio data with sampling rates up to 16 ,
Hz (Warden, 2018).
To handle such settings—where we still need large enough models that
can expressively model this data—training and inference should ideally scale subquadratically with sequence length and model size in time and space complexity.
Unfortunately, existing time series methods struggle to achieve all three criteria.
Classical methods ( c.f. , ARIMA (Box et al., 1970), exponential smoothing (ETS) (Winters, 1960)) often require manual data preprocessing and model selection to identify expressive-enough models.
Deep learning methods commonly train to predict specific horizon lengths, i.e. , as direct multi-step forecasting (Chevillon, 2007), and we find this hurts their ability to forecast longer horizons (Sec. 4.2.2).
They also face limitations achieving high expressivity and efficiency.
Fully connected networks (FCNs) in Zeng et al. (2022) scale quadratically in O ( ℓh ) space complexity (with input length ℓ and forecast length h ).
Recent Transformer-based models reduce this complexity to O ( ℓ + h ) , but do not always outperform the above FCNs on forecasting benchmarks (Liu et al., 2022; Zhou et al., 2021).
We thus propose S PACE T IME , a deep state- space architecture for effective time series modeling.
To achieve this, we focus on improving each criteria via three core contributions:
For expressivity, our key idea and building block is a linear layer that models time series processes as state-space models (SSMs) via the companion matrix (Fig. 1).
We start with SSMs due to their connections to both classical time series analysis (Kalman
Hamilton
and recent deep learning advances (Gu et al., 2021a).
Classically, many time series models such as ARIMA and exponential smoothing (ETS) can be expressed as SSMs (Box et al., 1970;
Winters, 1960).
Meanwhile, recent state-of-the-art deep sequence models (Gu et al., 2021a) have used SSMs to outperform Transformers and LSTMs on challenging long-range benchmarks (Tay et al., 2020).
Their primary innovations show how to formulate SSMs as neural network parameters that are practical to train.
However, we find limitations with these deep SSMs for time series data.
While we build on their advances, we prove that these prior SSM representations (Gu et al.
, 2021b;a; Gupta, 2022) cannot capture autoregressive processes fundamental for time series.
We thus specifically propose the companion matrix representation for its expressive and memory-efficient properties.
We prove that the companion matrix SSM recovers fundamental autoregressive (AR) and smoothing processes modeled in classical techniques such as ARIMA and ETS, while only requiring O ( d ) memory to represent an O ( d 2 ) matrix.
Thus, S PACE T IME inherits the benefits of prior SSM-based sequence models, but introduces improved expressivity to recover fundamental time series processes simply through its layer weights.
For forecasting long horizons, we introduce a new “closed-loop” view of SSMs.
Prior deep SSM architectures either apply the SSM as an “open-loop” (Gu et al., 2021a), where fixed-length inputs necessarily generate same-length outputs, or use closed-loop autoregression where final layer outputs are fed through the entire network as next-time-step inputs (Goel et al., 2022).
We
describe issues with both approaches in Sec. 3.2, and instead achieve autogressive forecasting in a deep network with only a single SSM layer.
We do so by explicitly training the SSM layer to predict its next time-step inputs , alongside its usual outputs.
This allows the SSM to recurrently generate its own future inputs that lead to desired outputs— i.e. , those that match an observed time series—so we can forecast over many future time-steps without explicit data inputs.
For efficiency, we introduce an algorithm for efficient training and inference with the companion matrix SSM.
We exploit the companion matrix’s structure as a “shift plus low-rank” matrix, which allows us to reduce the time and space complexity for computing SSM hidden states and outputs from ˜ O ( dℓ ) to ˜ O
( d + ℓ ) in SSM state size d and input sequence length ℓ .
In experiments, we find S PACE T IME consistently obtains state-of-the-art or near-state-of-the-art results, achieving best or second-best AUROC on 6 out of 7 ECG and audio speech time series classification tasks, and best mean-squared error (MSE) on 14 out of 16 Informer benchmark forecasting tasks (Zhou et al., 2021).
S PACE T IME also sets a new best average ranking across 34 tasks on the Monash benchmark (Godahewa et al., 2021).
We connect these gains with improvements on our three effective time series modeling criteria.
For expressivity, on synthetic ARIMA processes S PACE T IME learns AR processes that prior deep SSMs cannot.
For long horizon forecasting,
S PACE T IME consistently outperforms prior state-of-the-art on the longest horizons by large margins.
S PACE T IME also generalizes better to new horizons not used for training.
For efficiency, on speed benchmarks S PACE T IME obtains 73% and 80% relative wall-clock speedups over parametermatched Transformers and LSTMs respectively, when training on real-world ETTh1 data.
P RELIMINARIES
Problem setting.
We evaluate effective time series modeling with classification and forecasting tasks.
For both tasks, we are given input sequences of ℓ “look-back” or “lag” time series samples u t − ℓ : t − 1 = ( u t − ℓ , . . .
, u t − 1 ) ∈
R ℓ × m for sample feature size m .
For classification, we aim to classify the sequence as the true class y out of possible classes Y .
For forecasting, we aim to correctly predict H future time-steps over a “horizon” y t,t
+ H − 1 = ( u t , . . .
, u t + H − 1 ) ∈
R H × m .
State-space models for time series.
We build on the discrete-time state-space model (SSM), which maps observed inputs u k to hidden states x k , before projecting back to observed outputs y k via
[Equation]
where A ∈ R d × d , B ∈ R d × m , C ∈ R m ′
× d , and D ∈ R m ′
× m .
For now, we stick to single-input single-output conventions where m, m ′ = 1 , and let D = 0 .
To model time series in the single SSM setting, we treat u and y as copies of the same process, such that
[Equation]
We can thus learn a time series SSM by treating A , B , C as black-box parameters in a neural net layer, i.e. , by updating A , B , C via gradient descent s.t . with input u k and state x k at timestep k , following (3) predicts ˆ y k +1 that matches the next time-step sample
y k
+1 =
u k +1 .
This
SSM framework and modeling setup is similar to prior works (Gu et al., 2021b;a), which adopt a similar interpretation of inputs and outputs being derived from the “same” process, e.g. , for language modeling.
Here we study and improve this framework for time series modeling.
As extensions, in
Sec.
3.1.1 we show how (1) and (2) express univariate time series with the right A representation.
In
Sec.
3.1.2 we discuss the multi-layer setting, where layer-specific u and y now differ, and we only model first layer inputs and last layer outputs as copies of the same time series process.
M ETHOD : S PACE T IME
We now present S PACE T IME , a deep architecture that uses structured state-spaces for more effective time-series modeling.
S PACE T IME is a standard multi-layer encoder-decoder sequence model, built as a stack of repeated layers that each parametrize multiple SSMs.
We designate the last layer as the “decoder”, and prior layers as “encoder” layers.
Each encoder layer processes an input time series sample as a sequence-to-sequence map.
The decoder layer then takes the encoded sequence representation as input and outputs a prediction (for classification) or output sequence (for forecasting).
Below we expand on our contributions that allow S PACE T IME to improve expressivity, long-horizon forecasting, and efficiency of time series modeling.
In Sec. 3.1, we present our key building block, a layer that parametrizes the companion matrix SSM (companion SSM) for expressive and autoregressive modeling.
In Sec. 3.2, we introduce a specific instantiation of the companion SSM to flexibly forecast over long horizons.
In Sec. 3.3, we provide an efficient inference algorithm that allows
S PACE T IME to train and predict over long sequences in sub-quadratic time and space complexity.
3. T HE M ULTI -SSM S PACE T IME LAYER
We discuss our first core contribution and key building block of our model, the S PACE T IME layer, which captures the companion SSM ’s expressive properties, and prove that the SSM represents multiple fundamental processes.
To scale up this expressiveness in a neural architecture, we then go over how we represent and compute multiple SSMs in each S PACE T IME layer.
We finally show how the companion SSM’s expressiveness allows us to build in various time series data preprocessing operations in a S PACE T IME layer via different weight initializations of the same layer architecture.
3.1. E XPRESSIVE S TATE -S PACE M ODELS WITH THE C OMPANION M ATRIX
For expressive time series modeling, our SSM parametrization represents the state matrix A as a companion matrix.
Our key motivation is that A should allow us to capture autoregressive relationships between a sample u k and various past samples
u k − 1 , u k − 2 , . .
, u k − n .
Such dependencies are a basic yet essential premise for time series modeling; they underlie many fundamental time series processes, e.g. , those captured by standard ARIMA models.
For example, consider the simplest version of this, where u k is a linear combination of p prior samples (with coefficients ϕ 1 , . . .
, ϕ p )
[Equation]
i.e. , a noiseless, unbiased AR ( p ) process in standard ARIMA time series analysis (Box et al., 1970).
To allow (3) to express (4), we need the hidden state x k to carry information about past samples.
However, while setting the state-space matrices as trainable neural net weights may suggest we can learn arbitrary task-desirable A and B via supervised learning, prior work showed this could not be done without restricting A to specific classes of matrices (Gu et al., 2021b; Gupta, 2022).
Fortunately, we find that a class of relatively simple A matrices suffices.
We propose to set A ∈ R d × d as the d × d companion matrix , a square matrix of the form: ( Companion Matrix )
[Equation]
[Equation]
otherwise (5)
Then simply letting state dimension d = p , assuming initial hidden state x 0 = 0 , and setting a :=
[ aa. . .
a d − 1 ] T = 0 , B = [ 1 0 . . .
0 ] T , C =
[ ϕ. . .
ϕ p ] allows the discrete SSM in (1, 2) to recover the AR ( p ) process in (4).
We next extend this result in
Proposition 1, proving in App.
B that setting A as the companion matrix allows the SSM to recover a wide range of fundamental time series and dynamical system processes beyond the AR ( p ) process.
Proposition 1.
A companion state matrix SSM can represent ARIMA (Box et al., 1970), exponential smoothing (Winters, 1960; Holt, 2004), controllable linear time–invariant systems (Chen, 1984).
As a result, by training neural network layers that parameterize the companion SSM, we provably enable these layers to learn the ground-truth parameters for multiple time series processes.
In addition, as we only update a ∈ R d (5), we can efficiently scale the hidden-state size to capture more expressive processes with only O ( d ) parameters.
Finally, by learning multiple such SSMs in a single layer, and stacking multiple such layers, we can further scale up expressivity in a deep architecture.
Prior SSMs are insufficient.
We further support the companion SSM by proving that existing related SSM representations used in Gu et al.
(2021a)
; Gupta (2022); Smith et al.
; Alcaraz and Strodthoff (2022) cannot capture the simple yet fundamental AR ( p ) process.
Such works, including S4 and S4D, build on the Linear State-Space Layer (LSSL) (Gu et al., 2021b), and cannot represent AR processes due to their continuous-time or diagonal parametrizations of A .
Proposition 2.
No class of continuous-time LSSL SSMs can represent the noiseless AR ( p ) process.
We defer the proof to App.
B.1.
In Sec.
4.2.1, we support this analysis with empirical results, showing that these prior SSMs fit synthetic AR processes less accurately than the companion SSM.
This suggests the companion matrix resolves a fundamental limitation in related work for time series.
3.1. L AYER A RCHITECTURE AND M ULTI -SSM C OMPUTATION
Architecture.
To capture and scale up the companion SSM’s expressive and autoregressive modeling capabilities, we model multiple companion SSMs in each S PACE T IME layer’s weights.
S PACET IME layers are similar to prior work such as LSSLs, with A , B , C as trainable weights, and D added back as a skip connection.
To model multiple SSMs, we add a dimension to each matrix.
For s SSMs per S PACE T IME layer, we specify weights A ∈ R s × d × d ,
B ∈ R d × s , and C ∈ R s × d .
Each slice in the s dimension represents an individual SSM.
We thus compute s outputs and hidden states in parallel by following (1) and (2) via simple matrix multiplications on standard GPUs.
To model dependencies across individual SSM outputs, we optionally follow each S PACE T IME layer with a one-layer nonlinear feedforward network (FFN).
The FFN thus mixes the m outputs across a
S PACE T IME layer’s SSMs, allowing subsequent layers to model dependencies across SSMs.
Computation.
To compute the companion SSM, we could use the recurrence in (1).
However, this sequential operation is slow on modern GPUs, which parallelize matrix multiplications.
Luckily, as described in Gu et al.
(2021a)
we can also compute the SSM as a 1-D convolution .
This enables parallelizable inference and training.
To see how, note that given a sequence with at least k inputs and hidden state x 0 = 0 , the hidden state and output at time-step k by induction are:
[Equation]
We can thus compute hidden state x k and output y k as 1-D convolutions with “filters” as
[Equation]
So when we have inputs available for each output ( i.e. , equal-sized input and output sequences) we can obtain outputs by first computing output filters F y (8), and then computing outputs efficiently with the Fast Fourier Transform (FFT).
We thus compute each encoder SSM as a convolution.
For now we note two caveats.
Having inputs for each output is not always true, e.g. , with long horizon forecasting.
Efficient inference also importantly requires that F y can be computed efficiently, but this is not necessarily trivial for time series: we may have long input sequences with large k .
Fortunately we later provide solutions for both.
In Sec. 3.2, we show how to predict output samples many time-steps ahead of our last input sample via a “closed-loop” forecasting SSM.
In Sec. 3.
we show how to compute both hidden state and output filters efficiently over long sequences via an efficient inference algorithm that handles the repeated powering of A k .
3.1. B UILT - IN D ATA P REPROCESSING WITH C OMPANION SSM S
We now show how beyond autoregressive modeling, the companion SSM also enables S PACE T IME layers to do standard data preprocessing techniques used to handle nonstationarities.
Consider differencing and smoothing, two classical techniques to handle nonstationarity and noise:
[Equation]
We explicitly build these preprocessing operations into a S PACE T IME layer by simply initializing companion SSM weights.
Furthermore, by specifying weights for multiple SSMs, we simultaneously perform preprocessing with various orders in one forward pass.
We do so by setting a = 0 and
B = [1 , 0 , . .
. , 0] T , such that SSM outputs via the convolution view (6) are simple sliding windows / 1 -D convolutions with filter determined by C .
We can then recover arbitrary n -order differencing or average smoothing via C weight initializations, e.g. , (see App.
D.7.1 for more examples),
C =
[Equation] [Equation]
[Equation] [Equation]
L ONG H ORIZON F ORECASTING WITH C LOSED - LOOP SSM S
We now discuss our second core contribution, which enables long horizon forcasting.
Using a slight variation of the companion SSM, we allow the same constant size S PACE T IME model to forecast over many horizons.
This forecasting SSM recovers the flexible and stateful inference of RNNs, while retaining the faster parallelizable training of computing SSMs as convolutions.
Challenges and limitations.
For forecasting, a model must process an input lag sequence of length ℓ and output a forecast sequence of length h , where h ̸ = ℓ necessarily.
Many state-of-the-art neural nets thus train by specifically predicting h -long targets given ℓ -long inputs.
However, in Sec. 4.2.
we find this hurts transfer to new horizons in other models, as they only train to predict specific horizons.
Alternatively, we could output horizons autoregressively through the network similar to stacked RNNs as in S A S HI M I (Goel et al., 2022) or DeepAR (Salinas et al., 2020).
However, we find this can still be relatively inefficient, as it requires passing states to each layer of a deep network.
Closed-loop SSM solution.
Our approach is similar to autoregression, but only applied at a single
S PACE T IME layer.
We treat the inputs and outputs as distinct processes in a multi-layer network, and add another matrix K to each decoder SSM to model future input time-steps explicitly.
¯ u
= (¯ u 0 , . .
, ¯ u ℓ − 1 ) be the input sequence to a decoder SSM and u = ( u 0 , . .
. , u ℓ − 1 ) be the original input sequence, we jointly train A , B , C , K such that x k +1 =
A x k + B ¯ u k , and ˆ y k
[Equation]
(where ˆ y k +1 =
y k
+1 =
u k +1 )
[Equation]
(where ˆ¯ u k +1 =
¯
u k +1 )
We thus train the decoder S PACE T IME layer to explicitly model its own next time-step inputs with
A , B , K , and model its next time-step outputs ( i.e. , future time series samples) with A , B , C .
For forecasting, we first process the lag terms via (11) and (12) as convolutions
[Equation]
for k ∈ [0 , ℓ − 1] .
To forecast h future time-steps, with last hidden state x ℓ we first predict future input ˆ¯ u ℓ via (12).
Plugging this back into the SSM and iterating for h − 1 future time-steps leads to
[Equation]
We can thus use Eq.
15 to get future outputs without sequential recurrence, using the same FFT operation as for Eq. 8, 9.
This flexibly recovers O ( ℓ + h ) time complexity for forecasting h future time-steps, assuming that powers ( A + BK ) h are taken care of.
Next, we derive an efficient matrix powering algorithm to take care of this powering and enable fast training and inference in practice.
3. E FFICIENT I NFERENCE WITH THE C OMPANION SSM
We finally discuss our third contribution, where we derive an algorithm for efficient training and inference with the companion SSM.
To motivate this section, we note that prior efficient algorithms to compute powers of the state matrix A were only proposed to handle specific classes of A , and do not apply to the companion matrix (Gu et al., 2021a
Goel et al.,
Gu et al
Recall from Sec.
3.1.2 that for a sequence of length ℓ , we want to construct the output filter F y
=
( CB , . . .
, CA ℓ − 1 B ) , where A is a d × d companion matrix and B , C are d × 1 and 1 × d matrices.
Na¨ıvely, we could use sparse matrix multiplications to compute powers CA j B for j = 0 , . . . , ℓ −sequentially.
As A has O ( d ) nonzeros, this would take O ( ℓd ) time.
We instead derive an algorithm that constructs this filter in O ( ℓ log ℓ + d log d ) time.
The main idea is that rather than computing the filter directly, we can compute its spectrum (its discrete Fourier transform) more easily, i.e. , ˜ F y [ m ] :=
F ( F y )
= (cid:80) ℓ − 1 j =
CA j ω mj B
=
C
( I − A ℓ )( I − A ω m ) − 1 B , m = 0 , 1 , . .
ℓ − 1 .
where ω = exp( − 2 πi/ℓ ) is the ℓ -th root of unity.
This reduces to computing the quadratic form of the resolvent ( I − A ω m ) − 1 on the roots of unity (the powers of ω ).
Since A is a companion matrix, we can write A as a shift matrix plus a rank-1 matrix, A = S + ae Td .
Thus Woodbury’s formula reduces this computation to the resolvent of a shift matrix ( I − S ω m ) − 1 , with a rank-1 correction.
This resolvent can be shown analytically to be a lower-triangular matrix consisting of roots of unity, and its quadratic form can be computed by the Fourier transform of a linear convolution of size d .
Thus one can construct F yk by linear convolution and the FFT, resulting in O ( ℓ log ℓ + d log d ) time.
We validate in Sec.
4.2.3 that Algorithm 1 leads to a wall-clock time speedup of 2 × compared to computing the output filter na ¨ ıvely by powering A .
In App.
B.2, we prove the time complexity
O ( ℓ log ℓ + d log d ) and correctness of Algorithm 1.
We also provide an extension to the closed-loop
SSM, which can also be computed in subquadratic time as A + BK is a shift plus rank-2 matrix.
E XPERIMENTS
We test S PACE T IME on a broad range of time series forecasting and classification tasks.
In Sec. 4.1, we evaluate whether S PACE T IME ’s contributions lead to state-of-the-art results on standard benchmarks.
To help explain S PACE T IME ’s performance and validate our contributions, in Sec. 4.
we then evaluate whether these gains coincide with empirical improvements in expressiveness (Sec. 4.2.1), forecasting flexibility (Sec. 4.2.2), and training efficiency (Sec. 4.2.3).
4. M AIN R ESULTS : T IME S ERIES F ORECASTING AND C LASSIFICATION
For forecasting, we evaluate S PACE T IME on 40 forecasting tasks from the popular Informer (Zhou et al., 2021) and Monash (Godahewa et al., 2021) benchmarks, testing on horizons 8 to 960 timesteps long.
For classification, we evaluate S PACE T IME on seven medical ECG or speech audio classification tasks, which test on sequences up to 16,000 time-steps long.
For all results, we report mean evaluation metrics over three seeds.
✗ denotes the method was computationally infeasible on allocated GPUs, e.g. , due to memory constraints (same resources for all methods; see App.
C for details).
App.
C also contains additional dataset, implementation, and hyperparameter details.
Informer (forecasting).
We report univariate time series forecasting results in Table 1, comparing against recent state-of-the-art methods (Zeng et al., 2022; Zhou et al., 2022a), related state-space models (Gu et al., 2021a), and other competitive deep architectures.
We include extended results on additional horizons and multivariate forecasting in App. D.2.
We find S PACE T IME obtains lowest
MSE and MAE on 14 and 11 forecasting settings respectively, 3 × more than prior state-of-the-art.
S PACE T IME also outperforms S4 on 15 / 16 settings, supporting the companion SSM representation.
Monash (forecasting).
We also evaluate on 32 datasets in the Monash forecasting benchmark (Godahewa et al., 2021), spanning domains including finance, weather, and traffic.
For space, we report
[Table]
ETT m
[Table]
ETT m
[Table] [Table]
[Table]
results in Table 20 (App. D.3).
We compare against 13 classical and deep learning baselines.
S PACET IME achieves best RMSE on 7 tasks and sets new state-of-the-art average performance across all 32 datasets.
S PACE T IME ’s relative improvements also notably grow on long horizon tasks (Fig. 6).
ECG (multi-label classification).
Beyond forecasting, we show that S PACE T IME can also perform state-of-the-art time series classification.
To classify sequences, we use the same sequence model architecture in Sec. 3.1.
Like prior work (Gu et al., 2021a), we simply use the last-layer FFN to project from number of SSMs to number of classes, and mean pooling over length before a softmax to output class logits.
In Table 2, we find that S PACE T IME obtains best or second-best AUROC on five out of six tasks, outperforming both general sequence models and specialized architectures.
Speech Audio (single-label classification).
We further test S PACE T IME on long-range audio classification on the Speech Commands dataset (Warden, 2018).
The task is classifying raw audio sequences of length 16,000 into 10 word classes.
We use the same pooling operation for classification as in ECG.
S PACE T IME outperforms domain-specific architectures, e.g. , WaveGan-D (Donahue et al., 2018) and efficient Transformers, e.g. , Performer (Choromanski et al., 2020)
(Table 3).
4. I MPROVEMENT ON C RITERIA FOR E FFECTIVE T IME S ERIES M ODELING
For further insight into S PACE T IME ’s performance, we now validate that our contributions improve expressivity (4.2.1), forecasting ability (4.2.2), and efficiency (4.2.3) over existing approaches
. 4.2. E XPRESSIVITY
To first study S PACE T IME ’s expressivity, we test how well S PACE T IME can fit controlled autoregressive processes.
To validate our theory on S PACE T IME ’s expressivity gains in Sec. 3.1, we compare against recent related SSM architectures such as S4 (Gu et al., 2021a) and
S4D (Gu et al., 2022).
For evaluation, we generate noiseless synthetic AR( p ) sequences.
We test if models learn the true process by inspecting whether the trained model weights recover transfer functions specified by the
AR coefficients (Oppenheim, 1999).
We use simple 1-layer 1-SSM models, with state-space size equal to AR p , and predict one time-step given p lagged inputs (the smallest sufficient setting).
In Fig.
3 we compare the trained forecasts and transfer functions (as frequency response plots) of
S PACE T IME , S4, and S4D models on a relatively smooth AR(4) process and sharp AR(6) process.
[Figure]
[Figure]
ETTh1 NLinear
[Figure]
ETTh2 NLinear
[Figure]
Our results support the relative expressivity of S PACE T IME ’s companion matrix SSM.
While all models accurately forecast the AR(4) time series, only S PACE T IME recovers the ground-truth transfer functions for both, and notably forecasts the AR(6) process more accurately (Fig.
3c, d).
4.2. L ONG -H ORIZON F ORECASTING
To next study S PACE T IME ’s improved long horizon forecasting capabilities, we consider two additional long horizon tasks.
First, we test on much longer horizons than prior settings ( c.f. , Table 1).
Second, we test a new forecasting ability: how well methods trained to forecast one horizon transfer to longer horizons at test-time.
For both, we use the popular Informer ETTh datasets.
We compare
S PACE T IME with NLinear—the prior state-of-the-art on longer-horizon ETTh datasets—an FCN that learns a dense linear mapping between every lag input and horizon output (Zeng et al., 2022).
We find S PACE T IME outperforms NLinear on both long horizon tasks.
On training to predict long horizons, S PACE T IME consistently obtains lower MSE than NLinear on all settings (Table 4).
On transferring to new horizons, S PACE T IME models trained to forecast 192 time-step horizons transfer more accurately and consistently to forecasting longer horizons up to 576 time-steps (Fig. 4).
This suggests S PACE T IME ’s autoregressive forecasting more convincingly learns the time series process; rather than only fitting to the specified horizon, the same model can generalize to new horizons.
4.2. E FFICIENCY
[Figure]
To finally study if our companion matrix algorithm enables efficient training on long sequences, we conduct two speed benchmarks.
We (1) compare the wall-clock time per training epoch of S PACE T IME to standard sequence models, e.g. , LSTMs and Transformers, with similar pararemeter counts, and (2) empirically test our theory in
Sec. 3.3, which suggests S PACE T IME trains near-linearly with sequence length and state dimension.
For (1), we use ETTh1 data with lag and horizon 720 time-steps long.
For (2), we use synthetic data, scaling sequences from 100 − 2000 time-steps long.
On (1) we find S PACE T IME reduces clock time on ETTh1 by 73% and 80% compared to Transformers and LSTMs (Table 5).
Our efficient algorithm (Sec. 3.3) is also important; it speeds up training by 2 × , and makes S PACE T IME ’s training time competitive with efficient models such as S4.
On (2), we find S PACE T IME also scales near-linearly with input sequence length, achieving 91% faster training time versus similarly recurrent LSTMs (Fig. 5).
C ONCLUSION
We introduce S PACE T IME , a state-space time series model.
We achieve high expressivity by modeling SSMs with the companion matrix, long-horizon forecasting with a closed-loop SSM variant, and efficiency with a new algorithm to compute the companion SSM.
We validate S PACE T IME ’s proposed components on extensive time series forecasting and classification tasks.
E THICS S TATEMENT
A main objective of our work is to improve the ability to classify and forecast time series, which has real-world applications in many fields.
These applications may have high stakes, such as classifying abnormalities in medical time series.
In these situations, incorrect predictions may lead to harmful patient outcomes.
It is thus critical to understand that while we aim to improve time series modeling towards these applications, we do not solve these problems.
Further analysis and development into where models fail in time series modeling is necessary, including potentials intersections with research directions such as robustness and model biases when aiming to deploy machine learning models in real world applications.
R EPRODUCIBILITY
We include code to reproduce our main results in Table 1 in the supplementary material.
We provide training hyperparameters and dataset details for each benchmark in Appendix C, discussing the Informer forecasting benchmark in Appendix C.1, the Monash forecasting benchmark in Appendix C.2, and the ECG and speech audio classification benchmarks in Appendix C.3.
We provide proofs for all propositions and algorithm complexities in Appendix B.
A CKNOWLEDGEMENTS
We thank Albert Gu, Yining Chen, Ke Alexander Wang, and Rose Wang for helpful discussions and feedback.
We also gratefully acknowledge the support of NIH under No. U54EB020405 (Mobilize),
NSF under Nos.
CCF
(Beyond Sparsity), CCF1563078 (Volume to Velocity), and(RTML); US DEVCOM ARL under No.
W911NF-21-2-0251 (Interactive Human-AI Teaming);
ONR under No. N
(Unifying Weak Supervision); ONR N00014-20-1-2480: Understanding and Applying Non-Euclidean Geometry in Machine Learning; N000142012275 (NEPTUNE); NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi,
BASF, Accenture, Ericsson, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, the HAIGCP Cloud Credits for Research program, the Stanford Data Science Initiative (SDSI) and members of the Stanford DAWN project: Facebook, Google and VMWare.
The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon.
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of NIH, ONR, or the U.S. Government
A R ELATED W ORK A. C LASSICAL A PPROACHES
Classical approaches in time series modeling include the Box-Jenkins method (Box and Jenkins, 1968), exponential smoothing (Hyndman et al., 2008; Winters, 1960), autoregressive integrated moving average (ARIMA) (Box et al., 1970), and state-space models (Hamilton, 1994).
In such approaches, the model is usually manually selected based analyzing time series features (e.g., seasonality and order of non-stationarity), where the selected model is then fitted for each individual time series.
While classical approaches may be more interpretable than recent deep learning techniques, the domain expertise and manual labor needed to succesfully apply them renders them infeasible to the common setting of modeling thousands, or millions, of time series.
A. D EEP L EARNING A PPROACHES
Recurrent models.
Common deep learning architectures for modeling sequence data are the family of recurrent neural networks, which include GRUs (Chung et al., 2014), LSTMs (Hochreiter and
Schmidhuber, 1997), and DeepAR (Salinas et al., 2020).
However, due to the recurrent nature of
RNNs, they are slow to train and may suffer from vanishing/exploding gradients, making them difficult to train (Pascanu et al., 2013).
Deep State Space models.
Recent work has investigated combining the expressive strengths of
SSMs with the scalable strengths of deep neural networks (Rangapuram et al., 2018; Gu et al., 2021a).
Rangapuram et al.
(2018) propose to train a global RNN that transforms input covariates to sequence-spcific SSM parameters; however, one downside of this approach is that they inherit the drawbacks of RNNs.
More recent approaches, such as LSSL (Gu et al., 2021b), S4 (Gu et al., 2021a), S4D (Gu et al., 2022), and S5 (Smith et al., 2022), directly parameterize the layers of a neural network with multiple linear SSMs, and overcome common recurrent training drawbacks by leveraging the convolutional view of SSMs.
While deep SSM models have been shown great promise in time series modeling, we show in our work – which builds off deep SSMs – that current deep SSM approaches are not able to capture autoregressive processes due to their continuous nature.
Neural differential equations as nonlinear state spaces.
(Chen et al., 2018) parametrizes the vector field of continuous–time autonomous systems.
These models, termed Neural Differential
Equations (NDEs) have seen extensive application to time series and sequences, first by Rubanova et al.
(2019) and then by Kidger et al.
; Morrill et al.
; Massaroli et al.
(2021) with the notable extension to Neural Controlled Differential Equations (Neural CDEs).
Neural CDEs can be considered the continuous–time, nonlinear version of state space models and RNNs (Kidger, 2022).
Rather than introducing nonlinearity between linear state space layers, Neural CDEs model nonlinear systems driven by a control input.
The NDE framework has been further applied by Poli et al.
to model graph time series via Neural Graph Differential Equations .
In Queiruga et al.
(2020), a continuous-depth ResNet generalization based on ODEs is proposed, and in Kim et al.
(2021) numerical techniques to enable learning of stiff dynamical systems with Neural ODEs are investigated.
The idea of parameterizing the vector field of a differential equation with a neural network, popularized by NDEs, can be traced back to earlier works (Funahashi and Nakamura, 1993; Zhang et al., 2014; Weinan, 2017).
Transformers.
While RNNs and its variants have shown some success at time series modeling, a major limitation is their applicability to long input sequences.
Since RNNs are recurrent by nature, they require long traversal paths to access past inputs, which leads to vanishing/exploding gradients and as a result struggle with capturing long-range dependencies.
To counteract the long-range dependency problem with RNNs, a recent line of work considers Transformers for time series modeling.
The motivation is that due to the attention mechanism, a Transformer can directly model dependencies between any two points in the input sequence, independently of how far apart the points are.
However, the high expressivity of the attention mechanism comes at the cost of the time and space complexity being quadratic in sequence length, making
Transformers infeasible for very long sequences.
As a result, many works consider specialized
Transformer architectures with sparse attention mechanisms to bring down the quadratic complexity.
For example, Beltagy et al.
propose LogSparse self-attention, where a cell attends to a subset of past cells (as opposed to all cells), where closer cells are attended to more frequently, proportional to the log of their distance, which brings down complexity from O ( ℓ 2 ) to O ( ℓ (log ℓ ) 2 ) .
Zhou et al.
propose ProbSparse self-attention, which achieves O ( ℓ log ℓ ) time and memory complexity, where they propose a generative style decoder to speed inference.
Liu et al.
(2022) propose a pyramidal attention mechanism which shows linear time and space complexity with sequence length.
Autoformer (Wu et al., 2021) suggests more specialization is needed in time series with a decomposition forecasting architecture, which extracts long-term stationary trend from the seasonal series and utilizes an auto-correlation mechanism, which discovers the period-based dependencies.
Zhou et al.
(2022b) believes previous attempts of Transformer-based architectures do not capture global statistical properties, and to do so requires an attention mechanism in the frequency domain.
Conformer (Gulati et al., 2020) stacks convolutional and self-attention modules into a shared layer to combine the strengths of local interactions from convolutional modules and global interactions from self-attention modules.
Perceiver AR (Hawthorne et al., 2022) builds on the Perceiver architecture, which reduces the computational complexity of transformers by performing self-attention in a latent space, and extends Perceiver’s applicability to causal autoregressive generation.
While these works have shown exciting progress on time series forecasting, their proposed architectures are specialized to handle specific time series settings (e.g., long input sequences, or seasonal sequences), and are commonly trained to output a fixed target horizon length
(Zhou et al., 2021), i.e. , as direct multi-step forecasting (DMS) Chevillon (2007).
Thus, while effective at specific fore- casting tasks, their setups are not obviously applicable to a broad range of time series settings (such as forecasting arbitrary horizon lengths, or generalizing to classification or regression tasks).
Moreover, Zeng et al.
(2022) showed that simpler alternatives to Transformers, such as data normalization plus a single linear layer (NLinear), can outperform these specialized Transformer architectures when similarly trained to predict the entire fixed forecasting horizons.
Their results suggest that neither the attention mechanism nor the proposed modifications of these time series
Transformers may be best suited for time series modeling.
Instead, the success of these prior works may just be from learning to forecast the entire horizon with fully connected dependencies between prior time-step inputs and future time-step outputs, where a fully connected linear layer is sufficient.
Other deep learning methods.
Other works also investigate pure deep learning architectures with no explicit temporal components, and show these models can also perform well on time series forecasting.
Oreshkin et al.
(2019) propose N-BEATS, a deep architecture based on backward and forward residual links.
Even simpler, Zeng et al.
(2022) investigate single linear layer models for time series forecasting.
Both works show that simple architectures are capable of achieving high performance for time series forecasting.
In particular, with just data normalization, the NLinear model in Zeng et al.
(2022) obtained state-of-the-art performance on the popular Informer benchmark Zhou et al.
Given an input sequence of past lag terms and a target output sequence of future horizon terms, for every horizon output their model simply learns the fully connected dependencies between that output and every input lag sample.
However, FCNs such as NLinear also carry inefficient downsides.
Unlike Transformers and SSM-based models, the number of parameters for
FCNs scales directly with input and output sequence length, i.e. , O ( ℓh ) for ℓ inputs and h outputs.
Meanwhile, S PACE T IME shows that the SSM can improve the modeling quality of deep architectures, while maintaining constant parameter count regardless of input or output length.
Especially when forecasting long horizons, we achieve higher forecasting accuracy with smaller models.
B P ROOFS AND T HEORETICAL D ISCUSSION B. E XPRESSIVITY R ESULTS
Proposition 1.
An SSM with a companion state matrix can represent
i. ARIMA (Box et al., 1970) ii.
Exponential smoothing iii.
Controllable LTI systems (Chen, 1984)
Proof of Proposition 1.
We show each case separately.
We either provide a set of algebraic manipulations to obtain the desired model from a companion SSM, or alternatively invoke standard results from signal processing and system theory.
i.
We start with a standard ARMA ( p, q ) model
[Equation]
We consider two cases:
Case (1): Outputs y are a shifted (lag–1) version of the inputs u
[Equation]
where, without loss of generality, we have assumed that p > q for notational convenience.
The autoregressive system (16) is equivalent to
(cid:20)
A
B
C
D (cid:21) =
   . . .
(1 + ϕ 1 ) ( θ 1 + ϕ 2 ) . . .
θ d −θ d   .
in state-space form, with x ∈ R d and d = max( p, q ) .
Note that the state-space formulation is not unique.
Case (2): Outputs y are ”shaped noise”.
The ARMA(p,q) formulation (classically) defines inputs u as white noise samples 1 , ∀ k : p ( u k ) is a normal distribution with mean zero and some variance.
In this case, we can decompose the output as follows:
[Equation]
such that y k
= y ar k
+ y ma k .
The resulting state-space models are: (cid:20)
A ar
B ar
C ar
D ar (cid:21) =
   . . .
ϕϕ. . .
ϕ p −ϕ p 
 .
and (cid:20)
A ma
B ma
C ma
D ma (cid:21) =
   . . .
θθ. . .
θ q −θ q   .
Note that A ar ∈ R p × p , A ma ∈ R q × q .
More generally, our method can represent any ARMA process as the sum of two S PACE T IME heads : one taking as input the time series itself, and one the driving signal u .
ARIMA ARIMA processes are ARMA ( p, q ) applied to differenced time series.
For example, first-order differencing y k
= u k − u k − 1 .
Differencing corresponds to high–pass filtering of the signal y , and can be thus be realized via a convolution (Strang and Nguyen, 1996).
Any digital filter that can be expressed as a difference equation admits a state–space representation in companion form (Oppenheim, 1999), and hence can be learned by S PACE T IME .
ii. Simple exponential smoothing (SES) (Brown, 1959)
[Equation]
is an AR process with a parametrization involving a single scalar 0 < α < 1 and can thus be represented in companion form as shown above.
iii.
Let ( A , B , C ) be any controllable linear system.
Controllability corresponds to invertibility of the Krylov matrix (Chen, 1984, Thm 6.1, p145)
[Equation]
From rank ( K ) =
d , it follows that there exists a a ∈ R d
[Equation]
Thus
[Equation]
where G =
( S f − ae ⊤ d − 1 ) is a companion matrix.
[Equation]
Therefore G is similar to A .
We can then construct a companion form state space ( G , B , C , D ) from A using the relation above.
Proposition 2.
No class of continuous-time LSSL SSMs can represent the noiseless AR ( p ) process.
Proof of Proposition 2.
Recall from Sec.
3.1.1 that a noiseless AR ( p ) process is defined by
[Equation]
with coefficients ϕ 1 , . . .
, ϕ p .
This is represented by the SSM
[Equation]
when S ∈ R p × p is the shift matrix, B ∈ R p × 1 is the first basis vector e 1 , C ∈ R 1 × p is a vector of coefficients ϕ 1 , . . .
, ϕ p , and D = 0 , i.e. ,
[Equation]
We prove by contradiction that a continuous-time LSSL SSM cannot represent such a process.
Consider the following solutions to a continuous-time system and a system (18), both in autonomous form
[Equation]
It follows
[Equation]
we have reached a contradiction by (Culver, 1966, Theorem 1), as S is singular by definition and thus its matrix logarithm does not exist.
B. E FFICIENCY R ESULTS
We first prove that Algorithm 1 yields the correct output filter F y .
We then analyze its time complexity, showing that it takes time O ( ℓ log ℓ + d log d ) for sequence length ℓ and state dimension d .
Theorem 1.
Algorithm 1 returns the filter F y
=
( CB , . . .
, CA ℓ − 1 B ) .
Proof.
We follow the outline of the proof in Section 3.3.
Instead of computing F y directly, we compute its spectrum (its discrete Fourier transform):
[Equation]
where ω = exp( − 2 πi/ℓ ) is the ℓ -th root of unity.
This reduces to computing the quadratic form of the resolvent ( I − A ω m ) − 1 on the roots of unity (the powers of ω ).
Since A is a companion matrix, we can write A as a shift matrix plus a rankmatrix, A = S + ae Td , where e d is the d -th basis vector [0 , . . .
, 0 , 1] and the shift matrix S is:
S =   0 0 . . .
1  .
Thus Woodbury’s matrix identity (i.e., Sherman–Morrison formula) yields:
[Equation]
This is the resolvent of the shift matrix ( I − ω m S ) − 1 , with a rank-1 correction.
Hence
[Equation]
We now need to derive how to compute the quadratic form of a resolvent of the shift matrix efficiently.
Fortunately the resolvent of the shift matrix has a very special structure that closely relates to the Fourier transform.
We show analytically that:
[Equation]
   . . .
[Equation] [Equation]
[Equation] [Equation]
  .
It is easy to verify by multiplying this matrix with I − ω m S to see if we obtain the identity matrix.
Recall that multiplying with S on the left just shifts all the columns down by one index.
Therefore:    . . .
ω m . . .
ω 2 m ω m . . .
ω ( d − 1) m ω
( d − 2) m . . .
ω m   ( I − ω m S ) =
   . . .
ω m . . .
ω 2 m ω m . . .
ω ( d − 1) m ω
( d − 2) m . . .
ω m 
 − ω m S    . . .
ω m . . .
ω 2 m ω m . . .
ω ( d − 1) m ω
( d − 2) m . . .
ω m 
 =  
    . . .
ω m . . .
ω 2 m ω m . . .
ω ( d − 1) m ω
( d − 2) m . . .
ω m 
    − ω m
      . . .
0ω m . . .
ω ( d − 2) m ω ( d − 3) m . . .
1     =    . . .
ω m . . .
ω 2 m ω m . . .
ω ( d − 1) m ω
( d − 2) m . . .
ω m 
 −    . . .
ω m . . .
ω 2 m ω m . . .
ω ( d − 1) m ω
( d − 2) m . . .
ω 
 = I .
Thus the resolvent of the shift matrix indeed has the form of a lower-triangular matrix containing the roots of unity.
Now that we have the analytic formula of the resolvent, we can derive its quadratic form, given some vectors u, v ∈ R d .
Substituting in, we have
[Equation]
Grouping terms by powers of ω , we see that we want to compute u 1 v 1 + u 2 v 2 + · · · + u d v d , then u 2 v 1 + u 3 v 2 + · · · + u d v d − 1 , and so on.
The term corresponding to ω km is exactly the k -th element of the linear convolution u ∗ v .
Define q = u ∗ v , then u T ( I − ω m S ) − 1 v is just the Fourier transform of u ∗ v .
To deal with the case where d > ℓ , we note that the powers of roots of unity will repeat, so we just need to extend the output of u ∗ v to be multiples of ℓ , then split them into chunk of size ℓ , then sum them up and take the length- ℓ Fourier transform.
This is exactly the procedure quad( u, v ) defined in Algorithm 1.
Once we have derived the quadratic form of the resolvent ( I − ω m S ) − 1 , simply plugging it into the
Woodbury’s matrix identity (Equation (22)) yields Algorithm 1.
We analyze the algorithm’s complexity.
Theorem 2.
Algorithm 1 has time complexity O
( ℓ log ℓ + d log d ) for sequence length ℓ and state dimension d .
Proof.
We see that computing the quadratic form of the resolvent ( I − ω m S ) − 1 involves a linear convolution of size d and a Fourier transform of size ℓ .
The linear convolution can be done by performing an FFT of size 2 d on both inputs, multiply them pointwise, then take the inverse FFT of size 2 d .
This has time complexity O
( d log d ) .
The Fourier transform of size ℓ has time complexity
O ( ℓ log ℓ ) .
The whole algorithm needs to compute four such quadratic form, hence it takes time O ( ℓ log ℓ + d log d ) .
Remark.
We see that the algorithm easily extends to the case where the matrix A is a companion matrix plus low-rank matrix (of some rank k ).
We can write A as the sum of the shift matrix and a rank- ( k + 1) matrix (since A itself is the sum of a shift matrix and a rank-1 matrix).
Using the same strategy, we can use the Woodbury’s matrix identity for the rank- ( k +1) case.
The running time will then scale as O ( k ( ℓ log ℓ + d log d )) .
B. C OMPANION M ATRIX S TABILITY Normalizing companion parameters for bounded gradients
Proposition 3 (Bounded S PACE T IME Gradients) .
Given s , the norm of the gradient of a S PACET IME layer is bounded for all k < s if
[Equation]
Proof.
Without loss of generality, we assume x 0 = 0 .
Since the solution at time s is
[Equation]
we compute the gradient w.r.t u k as
[Equation]
The largest eigenvalue of A max { eig
[Equation]
Corollary of Gershgorin (Hirst and Macey, 1997, Theorem 1) =
using (cid:88)
[Equation]
is 1 , which implies convergence of the operator
CA s − k − 1 B .
Thus, the gradients are bounded.
We use the proposition above to ensure gradient boundedness in S PACE T IME layers by normalizing a every forward pass.
C E XPERIMENT D ETAILS C. I NFORMER F ORECASTING
Dataset details .
In Table 1, we evaluate all methods with datasets and horizon tasks from the Informer benchmark (Zhou et al., 2021).
We use the datasets and horizons evaluated on in recent works (Wu et al., 2021;
Zhou et al., 2022b;a; Zeng et al., 2022), which evaluate on electricity transformer temperature time series (ETTh1, ETTh2, ETTm1, ETTm2) with forecasting horizons { 96, 192, 336, 720 } .
We extend this comparison in Appendix D.2 to all datasets and forecasting horizons in Zhou et al.
(2021), which also consider weather and electricity (ECL) time series data.
Training details.
We train S PACE T IME on all datasets for 50 epochs using AdamW optimizer (Loshchilov and Hutter, 2017), cosine scheduling, and early stopping based on best validation standardized MSE.
We performed a grid search over number of SSMs { 64, 128 } and weight decay { 0 , 0 . 0001 } .
Like prior forecasting works, we treat the input lag sequence as a hyperparameter, and train to predict each forecasting horizon with either 336 or 720 time-step-long input sequences for all datasets and horizons.
For all datasets, we use a 3-layer S PACE T IME network with 128 SSMs per layer.
We train with learning rate 0 .
01 , weight decay 0 . 0001 , batch size 32 , and dropout 0 .
Hardware details.
All experiments were run on a single NVIDIA Tesla P100 GPU.
C. M ONASH F ORECASTING
The Monash Time Series Forecasting Repository (Godahewa et al., 2021) provides an extensive benchmark suite for time series forecasting models, with over 30 datasets (including various configurations) spanning finance, traffic, weather and medical domains.
We compare S PACE T IME against 13 baselines provided by the Monash benchmark: simple exponential smoothing (SES) (Gardner Jr, 1985), Theta (Assimakopoulos and Nikolopoulos, 2000), TBATS (De Livera et al., 2011),
ETS (Winters, 1960), DHR–ARIMA (Hyndman and Athanasopoulos, 2018), Pooled Regression (PR) (Trapero et al., 2015), CatBoost (Dorogush et al., 2018), FFNN, DeepAR (Salinas et al., 2020),
N-BEATS Oreshkin et al.
(2019), WaveNet (Oord et al., 2016), vanilla Transformer (Vaswani et al., 2017).
A complete list of the datasets considered and baselines, including test results (average
RMSE across 3 seeded runs) is available in Table 20.
Training details.
We optimize S PACE T IME on all datasets using Adam optimizer for 40 epochs with a linear learning rate warmup phase of 20 epochs and cosine decay.
We initialize learning rate at 0 .
001 , reach 0 .
004 after warmup, and decay to 0 .
We do not use weight decay or dropout.
We perform a grid search over number of layers { 3, 4, 5, 6 } , number of SSMs per layer { 8, 16, 32, 64, 128 } and number of channels (width of the model) { 1, 4, 8, 16 } .
Hyperparameter tuning is performed for each dataset.
We pick the model based on best validation RMSE performance.
Hardware details.
All experiments were run on a single NVIDIA GeForce RTX 3090 GPU.
C. T IME S ERIES C LASSIFICATION ECG
classification (motivation and dataset description).
Electrocardiograms (ECG) are commonly used as one of the first examination tools for assessing and diagnosing cardiovascular diseases, which are a major cause of mortality around the world (Amini et al., 2021).
However, ECG interpretation remains a challenging task for cardiologists and general practitioners (Jablonover et al., 2014; Cook et al., 2020).
Incorrect interpretation of ECG can result in misdiagnosis and delayed treatment, which can be potentially life-threatening in critical situations such as emergency rooms, where an accurate interpretation is needed quickly.
To mitigate these challenges, deep learning approaches are increasingly being applied to interpret
ECGs.
These approaches have been used for predicting the ECG rhythm class (Hannun et al., 2019), detecting atrial fibrillation (Attia et al., 2019b), rare cardiac diseases like cardiac amyloidosis (Goto et al., 2021), and a variety of other abnormalities (Attia et al., 2019a; Siontis et al., 2021).
Deep learning approaches have shown preliminary promise in matching the performance of cardiologists and emergency residents in triaging ECGs, which would permit accurate interpretations in settings where specialists may not be present (Ribeiro et al., 2020; Hannun et al., 2019).
We use the publicly available PTB-XL dataset (Wagner et al., 2020a;b; Goldberger et al., 2000), which contains 21 , 837 12 -lead ECG recordings of 10 seconds each obtained from 18 , 885 patients.
Each ECG recording is annotated by up to two cardiologists with one or more of the 71 ECG statements (labels).
These ECG statements conform to the SCP-ECG standard (Secretary, 2009).
Each statement belongs to one or more of the following three categories – diagnostic, form, and rhythm statements.
The diagnostic statements are further organised in a hierarchy containing 5 superclasses and 24 subclasses.
This provides six sets of annotations for the ECG statements based on the different categories and granularities: all (all ECG statements), diagnostic (only diagnostic statements including both subclass and superclass statements), diagnostic subclass (only diagnostic subclass statements), diagnostic superclass (only diagnostic superclass statements), form (only form statements), and rhythm (only rhythm statements).
These six sets of annotations form different prediction tasks which are referred to as all, diag, sub-diag, super-diag, form, and rhythm respectively.
The diagnostic superclass task is multi-class classification, and the other tasks are multi-label classification.
ECG classification training details.
To tune S PACE T IME and S4, we performed a grid search over the learning rate { 0 . 01 , 0 .
001 } , model dropout { 0 . 1 , 0 . 2 } , number of SSMs per layer { 128 , 256 } , and number of layers { 4 , 6 } , and chose the parameters that resulted in highest validation AUROC.
The SSM state dimension was fixed to 64 , with gated linear units as the non-linearity between stacked layers.
We additionally apply layer normalization.
We use a cosine learning rate scheduler, with a warmup period of 5 epochs.
We train all models for 100 epochs.
Speech Commands training details.
To train S PACE T IME , we use the same hyperparameters used by S4: a learning rate of 0 .
01 with a plateau scheduler with patience 20 , dropout of 0 . 1 , 128 SSMs per layer, 6 layers, batch normalization, trained for 200 epochs.
Hardware details.
For both ECG and Speech Commands, all experiments were run on a single
NVIDIA Tesla A100 Ampere 40 GB GPU.
D E XTENDED EXPERIMENTAL RESULTS D. E XPRESSIVITY ON DIGITAL FILTERS
We experimentally verify whether S PACE T IME can approximate the input–output map of digital filter admitting a state–space representation, with improved generalization over baseline models given test inputs of unseen frequencies.
We generate a dataset of 1028 sinusoidal signals of length
[Equation]
where ω ∈ [2 , 40] (cid:83) [50 , 100] in the training set and ω ∈ (40 , 50) in the test set.
The outputs are obtained by filtering x , i.e., y = F ( x ) where F is in the family of digital filters.
We introduce common various sequence-to-sequence layers or models as baselines: the original S diagonal plus low–rank (Gu et al., 2021a), a single-layer LSTM, a single 1d convolution (Conv1d), a dense linear layer (NLinear), a single self–attention layer.
All models are trained for 800 epochs with batch size 256 , learning rate 10 − 3 and Adam.
We repeat this experiment for digital filters of different orders (Oppenheim, 1999).
The results are shown in Figure 8.
S PACE T IME learns to match the frequency response of the target filter, producing the correct output for inputs at test frequencies.
[Table]
D. I NFORMER F ORECASTING
Univariate long horizon forecasts with Informer splits.
Beyond the ETT datasets and horizons evaluated on in Table 7, we also compare S PACE T IME to alternative time series methods on the complete datasets and horizons used in the original Informer paper (Zhou et al., 2021).
We compare against recent architectures which similarly evaluate on these settings, including ETSFormer (Woo et al., 2022), SCINet (Liu et al., 2021), and Yformer (Madhusudhanan et al., 2021), and other comparison methods found in the Informer paper, such as Reformer (Kitaev et al., 2020) and ARIMA.
S PACE T IME obtains best results on 20 out of 25 settings, the most of any method.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table]
Multivariate signals.
We additionally compare the performance of S PACE T IME to state-of-the-art comparison methods on ETT multivariate settings.
We focus on horizon length 720, the longest evaluated in prior works.
In Table 8, we find S PACE T IME is competitive with NLinear, which achieves best performance among compparison methods.
S PACE T IME also notably outperforming
S4 by large margins, supporting the companion matrix representation once more.
D. M ONASH F ORECASTING
We report the results across all datasets in Table 20.
We also investigate the performance of models by aggregating datasets based on common characteristics.
Concretely, we generate sets of tasksbased on the following properties:
• Large dataset: the dataset contains more than 2000 effective training samples.
• Long context: the models are provided a context of length greater than 20 as input.
• Long horizon: and the models are asked to forecast longer than 20 steps in the future.
Figure 6 shows the average x/ 13 model ranking in terms of test RMSE across splits.
We contextualize S PACE T IME results with best classical and deep learning methods (TBATS and DeepAR).
S PACE T IME relative performance is noticeably higher when context and forecasting horizons are longer, and when a larger number of samples is provided during training.
[Figure]
D. ECG C LASSIFICATION
In addition to our results table in the main paper
[Table]
we also provide the mean and standard deviations of the two models we ran in house (S PACE T IME and S4) in Table
[Table] [Table]
D. E FFICIENCY R ESULTS
We additionally empirically validate that S PACE T IME trains in near-linear time with horizon sequence length.
We also use synthetic data, scaling horizon from 1 − 1000 .
[Figure]
D. S PACE T IME A BLATIONS
To better understand how the proposed S PACE T IME SSMs lead to the improved empirical performance, we include ablations on the individual closed-loop forecasting SSM (Section 3.2) and preprocessing SSMs (Section 3.1.3).
D.6. C LOSED - LOOP F ORECASTING SSM
To study how the closed-loop SSM improves long horizon forecasting accuracy, we remove the closed-loop SSM component in our default S PACE T IME forecasting architecture ( c.f. , Appendix D.7, and compare the default S PACE T IME with one without any closed-loop SSMs on Informer forecasting tasks.
For models without closed-loop SSMs, we replace the last layer with the standard “openloop” SSM framework in Section 3.1.2), and keep all other layers the same.
Finally, for baseline comparison against another SSM without the closed-loop component, we compare against S4.
In Table 10, we report standardized MSE on Informer ETT datasets.
Adding the closed-loop SSM consistently improves forecasting accuracy, on average lowering relative MSE by 33.2%.
Meanwhile, even without the closed-loop SSM, S PACE T IME outperforms S4, again suggesting that the companion matrix parameterization is beneficial for autoregressive time series forecasting.
[Table]
D.6. P REPROCESSING SSM
To study how the preprocessing SSM improves long horizon forecasting accuracy, we next compare how S PACE T IME performs with and without the weight-initializing preprocessing SSMs introduced in Section 3.1.3.
We compare the default S PACE T IME architecture (Table 12 with (1) replacing the preprocessing SSMs with randomly initialized default companion SSMs, and (2) removing the preprocessing SSMs altogether.
For the former, we preserve the number of layers, but now train the first-layer SSM weights.
For the latter, there is one-less layer, but the same number of trainable parameters (as we fix and freeze the weights for each preprocessing SSM).
In Table 11, we report standardized MSE on Informer ETT datasets.
We find fixing the first layer
SSMs of a S PACE T IME network to preprocessing SSMs consistently improves forecasting performance, achieving 4.55% lower MSE on average than the ablation with just trainable companion matrices.
Including the preprocessing layer also improves MSE by 9.26% on average compared to removing the layer altogether.
These results suggest that preprocessing SSMs are beneficial for time series forecasting, e.g. , by performing classic time series modeling techniques on input data.
Unlike other approaches, S PACE T IME is able to flexibly and naturally incorporate these operations into its network layers via simple weight initializations of the same general companion SSM structure.
[Table]
D. S PACE T IME A RCHITECTURES
We provide the specific S PACE T IME architecture configurations used for forecasting and classification tasks.
Each configuration follows the general architecture presented in Section 3.1 and Figure 2, and consists of repeated Multi-SSM S PACE T
IME layers.
We first provide additional details on specific instantiations of the companion SSMs we use in our models, e.g. , how we instantiate preprocessing SSMs to recover specific techniques (Section 3.1.3).
We then include the layer-specific details of the number and type of SSM used in each network.
D.7. S PECIFIC SSM PARAMETERIZATIONS
In Section 3.1.1, we described the general form of the companion SSM used in this work.
By default, for any individual SSM we learn the a column in A and the vectors B , C as trainable parameters in a neural net module.
We refer to these SSMs specifically as companion SSMs .
In addition, as discussed in Sections 3.1.1 and 3.1.3, we can also fix a , B , or C to specific values to recover useful operations when computing the SSM outputs.
We describe specific instantiations of the companion SSM used in our models below (with dimensionality referring to one SSM).
Shift SSM.
We fix the a vector in the companion state matrix A ∈ R d × d to the 0 vector ∈ R d , such that A is the shift matrix (see Eq. 21 for an example).
This is a generalization of a 1-D “sliding window” convolution with fixed kernel size equal to SSM state dimension d .
To see how, note that if B is also fixed to the first basis vector e 1 ∈ R d × 1 , then this exactly recovers a 1-D convolution with kernel determined by C .
Differencing SSM.
As a specific version of the preprocessing SSM discussed in Section 3.1.3, we fix a = 0 , B = e 1 , and set C to recover various order differencing when computing the SSM, i.e. ,
C =
[Equation]
differencing
[Equation]
i.e
[Equation] [Equation]
identity
[Equation]
In this work, we only use the above 0, 1st, 2nd, or 3rd-order differencing instantiations.
With multiple differencing SSMs in a multi-SSM S PACE T IME layer, we initialize differencing SSMs by running through the orders repeatedly in sequence.
For example, given five differencing SSMs, the first four SSMs perform 0, 1st, 2nd, and 3rd-order differencing respectively, while the fifth performs 0-order differencing again.
Moving Average Residual (MA residual) SSM.
As another version of the preprocessing SSM, we can fix a = 0 , B = e 1 , and set C such that the SSM outputs sample residuals from a moving average applied over the input sequence.
For an n -order moving average, we compute outputs with
C specified as
[Equation]
For each MA residual SSM, we randomly initialize the order by uniform-randomly sampling an integer in the range [4 , d ] , where d is again the state-space dimension size (recall C ∈ R 1 × d ).
We pick 4 as a heuristic which was not finetuned; we leave additional optimization here for further work.
D.7. T ASK - SPECIFIC S PACE T IME A RCHITECTURES
Here, we provide layer-level details on the S PACE T IME networks used in this work.
For each task, we describe number of layers, number of SSMs per layer, state-space dimension (fixed for all SSMs in a network), and which SSMs are used in each layer.
Expanding on this last detail, as previously discussed in Section 3.1.2, in each S PACE T IME layer we can specify multiple SSMs in each layer, computing their outputs in parallel to produce a multidimensional output that is fed as the input to the next S PACE T IME layer.
The “types” of SSMs do not all have to be the same per layer, and we list the type (companion, shift, differencing, MA residual) and closed-loop designation (standard, closed-loop) of the SSMs in each layer below.
For an additional visual overview of a S PACE T IME network, please refer back to Figure 2.
[Table]
Forecasting: Informer and Monash.
We describe the architecture in Table 12.
We treat the first
S PACE T IME layer as “preprocessing” layer, which performs differencing and moving average residual operations on the input sequence.
We treat the last S PACE T IME layer as a “forecasting” layer, which autoregressively outputs future horizon predictions given the second-to-last layer’s outputs as an input sequence.
Classification: ECG.
We describe the architectures for each ECG classification task in Tables 13– 18.
For all models, we use state-space dimension d = 64 .
As described in the experiments, for classification we compute logits with a mean pooling over the output sequence, where pooling is computed over the sequence length.
Classification: Speech Audio.
We describe the architecture for the Speech Audio task in Table 19.
We use state-space dimension d = 1024 .
As described in the experiments, for classification we compute logits with a mean pooling over the output sequence, where pooling is computed over the sequence length.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Figure]
filtering
[Figure]
test
[Figure]
T a b l e 2 0 :
M on a s h f o r eca s ti ng .
W e r e po r t t e s t R
M S E o f S
P A C
E T
I
M E f
o r eac
h
d a t a s e t ( b e s t r e s
u lt s e l ec t e d v i a v a li d a ti on
R M
S E , a v e r a g
e o fr un s ) .
D a t a s e t
S P A C
E T
I M
E
S E S
T h
e t a
T B A
T S
ET S ( DH
R -)
A
R I M
A
P R
C a t B oo s t
D ee p
A R
N - B
E A T
S
W a v e
N e t
T r a n s f o r m e r
M
Y ea r l y . .1 . . . . . . . . . .
M
Q u a r t e r l y . . . . . . . . . . . .
M
M on t h l y . . . . . . . . . . . .
M
Y ea r l y . . . . . . . . . . . .
M
Q u a r t e r l y 676 .670 .567 .653 .598 .650 .605 .698 .606 .582 .606 .819 .
M
M on t h l y 897 .893 .
M
O t h e r 265 .309 .242 .217 .224 .220 .262 .349 .277 .248 .277 .271 .
M
Q u a r t e r l y 718 .732 .673 .672 .674 .710 .711 .714 .700 .684 .697 .739 .
M
M on t h l y .755 .683 .743 .705 .702 .720 .734 .740 .705 .787 .902 .
M
W ee k l y 348 .412 .405 .356 .
M
D a il y 183 .209 .210 .208 .230 .212 .213 .263 .343 .221 .220 .233 .
M
H ou r l y 255 . . .469 . .
T ou r
i s m Y ea r l y . .
T ou r
i s m Q u a r t e r l y . . . . . . . . . . . .
T ou r
i s m M on t h l y . . . . . . . . . . . .
P e d e s t r i a n 69 .228 .228 .261 .278 .
W ea t h e r 2 .2 .3 .2 .3 .3 .
NN
W ee k l y 16 .
S o l a r m i n 7 .7 .7 .10 .7 .
S o l a r
W ee k l y . . . .
E l ec t r i c it y H ou r l y 475 . .
E l ec t r i c it
y W ee k l y . . . . . . . . . . . .
F r e dM D . . . . . . . . . . . .
T r a f fi c H ou r l y 0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .0 .
T r a f fi
c W ee k l y 1 .1 .1 .1 .1 .1 .1 .1 .1 .1 .1 .1 .
H o s p it a l 40 .
C ov i d 490 .403 .370 .113 .102 .100 .394 .607 .230 .186 . .480 .
S a ug ee n
U S
B i r t
h s 630 .
S un s po t 3 .5 .5 .3 .5 .
C a r
P a r t s 0 .0 .0 .0 .0 .0 .0 .0 .0 .1 .0 .0 .
V e h
i c l e T r i p s 30 .36 .37 .25 .37 .35 .31 .27 .26 .