C ROSSFORMER : T RANSFORMER U TILIZING C ROSSD IMENSION D EPENDENCY FOR M ULTIVARIATE T IME S ERIES F ORECASTING
Recently many deep models have been proposed for multivariate time series (MTS) forecasting.
In particular, Transformer-based models have shown great potential because they can capture long-term dependency.
However, existing Transformerbased models mainly focus on modeling the temporal dependency (cross-time dependency) yet often omit the dependency among different variables (crossdimension dependency), which is critical for MTS forecasting.
To ﬁll the gap, we propose Crossformer, a Transformer-based model utilizing cross-dimension dependency for MTS forecasting.
In Crossformer, the input MTS is embedded into a 2D vector array through the Dimension-Segment-Wise (DSW) embedding to preserve time and dimension information.
Then the Two-Stage Attention (TSA) layer is proposed to efﬁciently capture the cross-time and cross-dimension dependency.
Utilizing DSW embedding and TSA layer, Crossformer establishes a Hierarchical Encoder-Decoder (HED) to use the information at different scales for the ﬁnal forecasting.
Extensive experimental results on six real-world datasets show the effectiveness of Crossformer against previous state-of-the-arts.
I NTRODUCTION
Multivariate time series (MTS) are time series with multiple dimensions, where each dimension represents a speciﬁc univariate time series (e g. a climate feature of weather).
MTS forecasting aims to forecast the future value of MTS using their historical values.
MTS forecasting beneﬁts the decision-making of downstream tasks and is widely used in many ﬁelds including weather (Angryk et al., 2020), energy (Demirel et al., 2012), ﬁnance (Patton, 2013), etc.
With the development of deep learning, many models have been proposed and achieved superior performances in MTS forecasting (Lea et al., 2017; Qin et al., 2017;
Flunkert et al., 2017; Rangapuram et al., 2018; Li et al., 2019a; Wu et al., 2020; Li et al., 2021).
Among them, the recent Transformer-based models (Li et al., 2019b; Zhou et al., 2021; Wu et al., 2021a; Liu et al., 2021a; Zhou et al., 2022; Chen et al., 2022) show great potential thanks to their ability to capture long-term temporal dependency (cross-time dependency).
Besides cross-time dependency, the cross-dimension dependency is also critical for MTS forecasting, i.e. for a speciﬁc dimension, information from associated series in other dimensions may improve prediction.
For example, when predicting future temperature, not only the historical temperature, but also historical wind speed helps to forecast.
Some previous neural models explicitly capture the cross-dimension dependency, i.e. preserving the information of dimensions in the latent feature space and using convolution neural network (CNN) (Lai et al., 2018) or graph neural network (GNN) (Wu et al., 2020; Cao et al., 2020) to capture their dependency.
However, recent Transformer-based models only implicitly utilize this dependency by embedding.
In general, Transformer-based models embed data points in all dimensions at the same time step into a feature vector and try to capture dependency among different time steps (like Fig.
1 (b)).
In this way, cross-time dependency is well captured, but cross-dimension dependency is not, which may limit their forecasting capability.
To ﬁll the gap, we propose Crossformer, a Transformer-based model that explicitly utilizes crossdimension dependency for MTS forecasting.
Speciﬁcally, we devise Dimension-Segment-Wise (DSW) embedding to process the historical time series.
In DSW embedding, the series in each dimension is ﬁrst partitioned into segments and then embedded into feature vectors.
The output of DSW embedding is a 2D vector array where the two axes correspond to time and dimension.
Then we propose the Two-Stage-Attention (TSA) layer to efﬁciently capture the cross-time and cross-dimension dependency among the 2D vector array.
Using DSW embedding and TSA layer,
Crossformer establishes a Hierarchical Encoder-Decoder (HED) for forecasting.
In HED, each layer corresponds to a scale.
The encoder’s upper layer merges adjacent segments output by the lower layer to capture the dependency at a coarser scale.
Decoder layers generate predictions at different scales and add them up as the ﬁnal prediction.
The contributions of this paper are:
1) We dive into the existing Transformer-based models for MTS forecasting and ﬁgure out that the cross-dimension dependency is not well utilized: these models simply embed data points of all dimensions at a speciﬁc time step into a single vector and focus on capturing the cross-time dependency among different time steps.
Without adequate and explicit mining and utilization of cross-dimension dependency, their forecasting capability is empirically shown limited.
We develop Crossformer, a Transformer model utilizing cross-dimension dependency for MTS forecasting.
This is one of the few transformer models (perhaps the ﬁrst to our best knowledge) that explicitly explores and utilizes cross-dimension dependency for MTS forecasting.
3) Extensive experimental results on six real-world benchmarks show the effectiveness of our Crossformer against previous state-of-the-arts.
Speciﬁcally, Crossformer ranks top-1 among themodels for comparison on 36 out of the 58 settings of varying prediction lengths and metrics and ranks top-2 on 51 settings.
R ELATED W ORKS
Multivariate Time Series Forecasting.
MTS forecasting models can be roughly divided into statistical and neural models.
Vector auto-regressive (VAR) model (Kilian & L ˜ Atkepohl, 2017) and Vector auto-regressive moving average (VARMA) are typical statistical models, which assume linear cross-dimension and cross-time dependency.
With the development of deep learning, many neural models have been proposed and often empirically show better performance than statistical ones.
TCN (Lea et al., 2017) and DeepAR (Flunkert et al., 2017) treat the MTS data as a sequence of vectors and use CNN/RNN to capture the temporal dependency.
LSTnet (Lai et al., 2018) employs
CNN to capture cross-dimension dependency and RNN for cross-time dependency.
Another category of works use graph neural networks (GNNs) to capture the cross-dimension dependency explicitly for forecasting (Li et al., 2018; Yu et al., 2018; Cao et al., 2020; Wu et al., 2020).
For example,
MTGNN (Wu et al., 2020) uses temporal convolution and graph convolution layers to capture crosstime and cross-dimension dependency.
These neural models capture the cross-time dependency through CNN or RNN, which have difﬁculty in modeling long-term dependency.
Transformers for MTS Forecasting.
Transformers (Vaswani et al., 2017) have achieved success in natural language processing (NLP)
(Devlin et al., 2019), vision (CV) (Dosovitskiy et al., 2021) and speech processing (Dong et al., 2018).
Recently, many Transformer-based models have been proposed for MTS forecasting and show great potential (Li et al., 2019b; Zhou et al., 2021;
Wu et al., 2021a; Liu et al., 2021a;
Zhou et al., 2022; Du et al., 2022).
LogTrans (Li et al., 2019b) proposes the LogSparse attention that reduces the computation complexity of Transformer from O ( L 2 ) to
O (cid:0) L (log L ) 2 (cid:1) 1 .
Informer (Zhou et al., 2021) utilizes the sparsity of attention score through KL divergence estimation and proposes ProbSparse self-attention which achieves O ( L log L ) complexity.
Autoformer (Wu et al., 2021a) introduces a decomposition architecture with an Auto-Correlation mechanism to Transformer, which also achieves the O ( L log L ) complexity.
Pyraformer (Liu et al., 2021a) introduces a pyramidal attention module that summarizes features at different resolutions and models the temporal dependencies of different ranges with the complexity of O ( L ) .
FEDformer (Zhou et al., 2022) proposes that time series have a sparse representation in frequency domain and develop a frequency enhanced Transformer with the O ( L ) complexity.
Preformer (Du et al., 2022) divides the embedded feature vector sequence into segments and utilizes segment-wise correlation-based attention [Figure] for forecasting.
These models mainly focus on reducing the complexity of dependency modeling, but the dimension dependency which is critical for MTS
Vision Transformers.
Transformer is initially applied to NLP for sequence modeling, recent works apply transformer to CV tasks to process images (Dosovitskiy et al., 2021;
Touvron et al., 2021; Liu et al., 2021b; Chen et al., 2021; Han et al., 2021).
These works achieve state-of-the-art performance on various tasks in CV and inspire our work.
ViT (Dosovitskiy et al., 2021) is one of the pioneers of vision transformers.
The basic idea of ViT is to split an image into non-overlapping medium-sized patches, then it rearranges these patches into a sequence to be input to the Transformer.
The idea of partitioning images into patches inspires our DSW embedding where MTS is split into dimensionwise segments.
Swin Transformer (Liu et al., 2021b) performs local attention within a window to reduce the complexity and builds hierarchical feature maps by merging image patches.
Readers can refer to the recent survey (Han et al., 2022) for comprehensive study on vision transformers.
M ETHODOLOGY
In multivariate time series forecasting, one aims to predict the future value of time series x T +1: T + τ ∈ R τ × D given the history x 1: T ∈ R T × D , where τ , T is the number of time steps in the future and past, respectively 2 .
D > 1 is the number of dimensions.
A natural assumption is that these D series are associated (e.g. climate features of weather), which helps to improve the forecasting accuracy.
To utilize the cross-dimension dependency, in Section 3.1, we embed the MTS using
Dimension-Segment-Wise (DSW) embedding.
In Section 3.2, we propose a Two-Stage Attention (TSA) layer to efﬁciently capture the dependency among the embedded segments.
In Section 3.3, using DSW embedding and TSA layer, we construct a hierarchical encoder-decoder (HED) to utilize information at different scales for ﬁnal forecasting.
3. D IMENSION -S EGMENT -W ISE E MBEDDING
To motivate our approach, we ﬁrst analyze the embedding methods of the previous Transformer-based models for MTS forecasting (Zhou et al., 2021; Wu et al., 2021a; Liu et al., 2021a;
Zhou et al., 2022).
As shown in Fig.
1 (b), existing methods embed data points at the same time step into a vector: x t → h t , x t ∈ R D , h t ∈ R d model , where x t represents all the data points in D dimensions at step t .
In this way, the input x 1: T is embedded into T vectors { h 1 , h 2 , . .
, h T } .
Then the dependency among the T vectors is captured for forecasting.
Therefore, previous Transformer-based models mainly capture cross-time dependency, while the cross-dimension dependency is not explicitly captured during embedding, which limits their forecasting capability.
Transformer was originally developed for NLP (Vaswani et al., 2017), where each embedded vector represents an informative word.
For MTS, a single value at a step alone provides little information.
While it forms informative pattern with nearby values in time domain.
Fig.
1 (a) shows a typical attention score map of original Transformer for MTS forecasting.
We can see that attention values have a tendency to segment, i.e. close data points have similar attention weights.
Based on the above two points, we argue that an embedded vector should represent a series segment of single dimension (Fig.
1 (c)), rather than the values of all dimensions at single step (Fig.
1 (b)).
To this end, we propose Dimension-Segment-Wise
(DSW)
embedding where the points in each dimension are divided into segments of length L seg and then embedded:
[Equation]
where x ( s ) i,d ∈ R L seg is the i -th segment in dimension d with length L seg .
For convenience, we assume that T, τ are divisible by L seg 3 .
Then each segment is embedded into a vector using linear projection added with a position embedding:
[Equation]
where E ∈ R d model ×
L seg denotes the learnable projection matrix, and E ( pos ) i,d ∈ R d model denotes the learnable position embedding for position ( i, d ) .
After embedding, we obtain a 2D vector array
H = (cid:110) h i,d | [Equation] ≤ i ≤ TL seg , 1 ≤ d ≤ D (cid:111) , where each h i,d represents a univariate time series segment.
The idea of segmentation is also used in Du et al.
(2022), which splits the embedded 1D vector sequence into segments to compute the Segment-Correlation in order to enhance locality and reduce computation complexity.
However, like other Transformers for MTS forecasting, it does not explicitly capture cross-dimension dependency.
3. T WO -S TAGE A TTENTION L AYER
For the obtained 2D array H , one can ﬂatten it into a 1D sequence so that it can be input to a canonical Transformer like ViT (Dosovitskiy et al., 2021) does in vision.
While we have speciﬁc considerations: 1) Different from images where the axes of height and width are interchangeable, the axes of time and dimension for MTS have different meanings and thus should be treated differently.
2) Directly applying self-attention on 2D array will cause the complexity of O
( D 2 T 2 L 2 seg ) , which is unaffordable for large D .
Therefore, we propose the Two-Stage Attention (TSA) Layer to capture cross-time and cross-dimension dependency among the 2D vector array, as sketched in Fig.
2 (a).
Cross-Time Stage Given a 2D array Z ∈ R L × D × d model as the input of the TSA Layer, where L and D are the number of segments and dimensions, respectively.
Z here can be the output of DSW embedding or lower TSA layers.
For convenience, in the following, we use Z i, : to denote the vectors of all dimensions at time step i , Z : ,d for those of all time steps in dimension d .
In the cross-time stage, we directly apply multi-head self-attention (MSA) to each dimension: [Equation]
[Equation]
where 1 ≤ d ≤ D and LayerNorm denotes layer normalization as widely adopted in Vaswani et al.
(2017); Dosovitskiy et al.
; Zhou et al.
(2021), MLP denotes a multi-layer (two in this paper) feedforward network, MSA ( Q , K , V ) denotes the multi-head self-attention (Vaswani et al., 2017) layer where Q , K , V serve as queries, keys and values. All dimensions ( 1 ≤ d ≤ D ) share the same MSA layer. ˆ Z time , Z time denotes the output of the MSA and MLP.
The computation complexity of cross-time stage is O ( DL 2 ) .
After this stage, the dependency among time segments in the same dimension is captured in Z time . Then Z time becomes the input of Cross-Dimension Stage to capture cross-dimension dependency. [Figure]
𝒁𝒁 𝒁𝒁 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 𝒀𝒀 ( 𝒁𝒁 𝑑𝑑𝑡𝑡m ) 𝐷𝐷 𝐿𝐿 (a) ……
𝒁𝒁 𝑡𝑡 ,: 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 𝒁𝒁 𝑡𝑡 ,: 𝑑𝑑𝑡𝑡𝑡𝑡 𝐷𝐷 ……
MSA 𝑑𝑑𝑡𝑡𝑡𝑡 ( 𝒁𝒁 𝑡𝑡 ,: 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 , 𝒁𝒁 𝑡𝑡 ,: 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 , 𝒁𝒁 𝑡𝑡 ,: 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 ) (b) …… ……
𝒁𝒁 𝑡𝑡 ,: 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 𝑩𝑩 𝑡𝑡 ,: ( 𝑐𝑐 =3)
𝒁𝒁 𝑡𝑡 ,: 𝑑𝑑𝑡𝑡𝑡𝑡
MSA 1𝑑𝑑𝑡𝑡𝑡𝑡 ( 𝑹𝑹 𝑡𝑡 ,: , 𝒁𝒁 𝑡𝑡 ,: 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 , 𝒁𝒁 𝑡𝑡 ,: 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 )
MSA 2𝑑𝑑𝑡𝑡𝑡𝑡 ( 𝒁𝒁 𝑡𝑡 ,: 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 , 𝑩𝑩 𝑡𝑡 ,: , 𝑩𝑩 𝑡𝑡 ,: ) 𝐷𝐷 (c)
Cross-Dimension Stage We can use a large L seg for long sequence in DSW Embedding to reduce the number of segments L in cross-time stage.
While in Cross-Dimension Stage, we can not partition dimensions and directly apply MSA will cause the complexity of O
( D 2 ) (as shown in Fig.
2 (b)), which is unaffordable for datasets with large D .
Instead, we propose the router mechanism for potentially large D .
As shown in Fig.
2 (c), we set a small ﬁxed number ( c << D ) of learnable vectors for each time step i as routers.
These routers ﬁrst aggregate messages from all dimensions by using routers as query in MSA and vectors of all dimensions as key and value.
Then routers distribute the received messages among dimensions by using vectors of dimensions as query and aggregated messages as key and value.
In this way, the all-to-all connection among D dimensions are built:
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
where R ∈ R
L × c × d model ( c is a constant) is the learnable vector array serving as routers.
B ∈ R L ×
c × d model is the aggregated messages from all dimensions.
Z dim denotes output of the router mechanism.
All time steps ( 1 ≤ i ≤ L ) share the same MSA dim, MSA dim.
ˆ Z dim , Z dim denote output of skip connection and MLP respectively.
The router mechanism reduce the complexity from O ( D 2 L ) to O ( DL ) .
Adding up Eq.
3 and Eq.
4, we model the two stages as:
[Equation]
where Z , Y ∈ R L × D × d model denotes the input and output vector array of TSA layer, respectively.
Note that the overall computation complexity of the TSA layer is O ( DL 2 + DL )
=
O ( DL 2 ) .
After the Cross-Time and Cross-Dimension Stages, every two segments (i.e. Z i 1 ,d 1 , Z i 2 ,d 2 ) in Z are connected, as such both cross-time and cross-dimension dependencies are captured in Y .
3. H IERARCHICAL E NCODER -D ECODER
Hierarchical structures are widely used in Transformers for MTS forecasting to capture information at different scales (Zhou et al., 2021;
Liu et al., 2021a).
In this section, we use the proposed DSW embedding, TSA layer and segment merging to construct a Hierarchical Encoder-Decoder (HED).
As shown in Fig.
3, the upper layer utilizes information at a coarser scale for forecasting.
Forecasting values at different scales are added to output the ﬁnal result.
Encoder
In each layer of the encoder (except the ﬁrst layer), every two adjacent vectors in time domain are merged to obtain the representation at a coarser level.
Then a TSA layer is applied to capture dependency at this scale.
This process is modeled as Z enc,l = Encoder ( Z enc,l − 1 ) :
[Equation]
6) where H denotes the 2D array obtained by DSW embedding; Z enc,l denotes the output of the l -th encoder layer; M ∈ R d model × 2 d model denotes a learnable matrix for segment merging; [ · ] denotes the concatenation operation; L l − 1 denotes the number of segments in each dimension in layer l − 1 , if it is not divisible by 2, we pad Z enc,l − 1 to the proper length; ˆ Z enc,l denotes the array after segment merging in the i -th layer.
Suppose there are N layers in the encoder, we use
Z enc, 0 , Z enc, 1 , . .
, Z enc,N , ( Z enc, 0 = H ) to represent the N + 1 outputs of the encoder.
The complexity of each encoder layer is O
( D T 2 L 2 seg ) .
Decoder Obtaining the N + 1 feature arrays output by the encoder, we use N + 1 layers (indexed by 0 , 1 , . .
. , N ) in decoder for forecasting.
Layer l takes the l -th encoded array as input, then outputs a decoded 2D array of layer l .
This process is summarized as Z dec,l = Decoder ( Z dec,l − 1 , Z enc,l ) :
[Equation]
where E ( dec ) ∈ R τ Lseg × D × d model denotes the learnable position embedding for decoder.
˜
Z dec,l is the output of TSA.
The MSA layer takes ˜ Z dec,l : ,d as query and Z enc,l : ,d as the key and value to build the connection between encoder and decoder.
The output of MSA is denoted as Z dec,l : ,d . ˆ Z dec,l , Z dec,l denote the output of skip connection and MLP respectively.
We use Z dec, 0 , Z enc, 1 , . . .
, Z dec,N to represent the decoder output.
The complexity of each decoder layer is O (cid:16) D τ ( T + τ ) L 2 seg (cid:17) .
Linear projection is applied to each layer’s output to yield the prediction of this layer.
Layer predictions are summed to make the ﬁnal prediction (for l = 0 , . .
. , N ): for l = 0 , . .
, N : x ( s ) ,l i,d
[Equation]
l Z dec,li,d
[Equation]
[Equation]
where W l ∈ R L seg × d model is a learnable matrix to project a vector to a time series segment.
x ( s ) ,l i,d ∈ R L seg denotes the i -th segment in dimension d of the prediction.
All the segments in layer [Table] l are rearranged to get the layer prediction x pred,lT +1: T + τ .
Predictions of all the layers are summed to obtain the ﬁnal forecasting x predT +1: T + τ .
E XPERIMENTS 4. P ROTOCOLS
Datasets We conduct experiments on six real-world datasets following Zhou et al.
; Wu et al.
(2021a). 1) ETTh1 (Electricity Transformer Temperature-hourly), 2) ETTm1 (Electricity
Transformer Temperature-minutely)
, 3) WTH (Weather), 4) ECL (Electricity Consuming Load), 5)
ILI (Inﬂuenza-Like Illness), 6) Trafﬁc .
The train/val/test splits for the ﬁrst four datasets are same as
Zhou et al.
(2021), the last two are split by the ratio of 0.7:0.1:0.2 following Wu et al. (2021a).
Baselines We use the following popular models for MTS forecasting as baselines: 1) LSTMa (Bahdanau et al., 2015), 2) LSTnet
(Lai et al., 2018), 3) MTGNN (Wu et al., 2020), and recent
Transformer-based models for MTS forecasting: 4) Transformer (Vaswani et al., 2017), 5)
Informer (Zhou et al., 2021), 6) Autoformer (Wu et al., 2021a), 7) Pyraformer (Liu et al., 2021a) and 8) FEDformer
(Zhou et al., 2022).
Setup We use the same setting as in Zhou et al.
(2021): train/val/test sets are zero-mean normalized with the mean and std of training set.
On each dataset, we evaluate the performance over the changing future window size τ .
For each τ , the past window size T is regarded as a hyper-parameter to search which is a common protocol in recent MTS transformer literature (Zhou et al., 2021; Liu et al., 2021a).
We roll the whole set with stride = 1 to generate different input-output pairs.
The Mean Square Error (MSE) and Mean Absolute Error (MAE) are used as evaluation metrics.
All experiments are repeated for 5 times and the mean of the metrics reported.
Our Crossformer only utilize the past series to forecast the future, while baseline models use additional covariates such as hour-of-the-day.
Details about datasets, baselines, implementation, hyper-parameters are shown in Appendix A.
4. M AIN R ESULTS
As shown in Table 1, Crossformer shows leading performance on most datasets, as well as on different prediction length settings, with the 36 top-1 and 51 top-2 cases out of 58 in total.
It is worth noting that, perhaps due to the explicit use of cross-dimension dependency via GNN, MTGNN outperforms many Transformer-based baselines.
While MTGNN has been rarely compared in existing [Table] [Figure] transformers for MTS forecasting literatures.
FEDformer and Autoformer outperform our model on
ILI.
We conjecture this is because the size of dataset ILI is small and these two models introduce the prior knowledge of sequence decomposition into the network structure which makes them perform well when the data is limited.
Crossformer still outperforms other baselines on this dataset.
4. A BLATION S TUDY
In our approach, there are three components: DSW embedding, TSA layer and HED.
We perform ablation study on the ETTh1 dataset in line with Zhou et al.
Liu et al.
(2021a).
We use
Transformer as the baseline and DSW+TSA+HED to denote Crossformer without ablation.
Three ablation versions are compared: 1) DSW 2) DSW+TSA 3) DSW+HED .
We analyze the results shown in Table 2. 1) DSW performs better than Transformer on most settings.
The only difference between DSW and Transformer is the embedding method, which indicates the usefulness of DSW embedding and the importance of cross-dimension dependency.
2) TSA constantly improves the forecasting accuracy.
This suggests that it is reasonable to treat time and dimension differently.
Moreover, TSA makes it possible to use Crossformer on datasets where the number of dimensions is large (e.g. D = 862 for dataset Trafﬁc).
Comparing DSW+HED with
DSW, HED decreases the forecasting accuracy when prediction length is short but increases it for long term prediction.
The possible reason is that information at different scales is helpful to long term prediction.
Combining DSW, TSA and HED, our Crossformer yields best results on all settings.
4. E FFECT OF H YPER -P ARAMETERS
We evaluate the effect of two hyper-parameters: segment length ( L seg in Eq. 1) and number of routers in TSA ( c in Cross-Dimension Stage of TSA) on the ETTh1 dataset.
Segment Length:
In Fig. 4(a), we prolong the segment length from 4 to 24 and evaluate MSE with different prediction windows.
For short-term forecasting ( τ = 24 , 48 ), smaller segment yields relevantly better results, but the prediction accuracy is stable.
For long-term forecasting ( τ ≥ 168 ), prolonging the segment length from 4 to 24 causes the MSE to decrease.
This indicates that long segments should be used for long-term forecasting.
We further prolong the segment length to 48 for τ = 336 , 720 , the MSE is slightly larger than that of 24.
The possible reason is that 24 hours exactly matches the daily period of this dataset, while 48 is too coarse to capture ﬁne-grained information.
Number of Routers in TSA
Layer: Number of Routers c controls the information bandwidth among all dimensions.
As Fig. 4(b) shows, the performance of Crossformer is stable w.r.t to c for τ ≤ 336 .
For τ = 720 , the MSE is large when c = 3 but decreases and stabilizes when c ≥ 5 .
In pratice, we set c = 10 to balance the prediction accuracy and computation efﬁciency.
4. C OMPUTATIONAL E FFICIENCY A NALYSIS
[Table]
The theoretical complexity per layer of
Transformer-based models is compared in
Table 3.
The complexity of Crossformer encoder is quadratic w.r.t T .
However, for long-term prediction where large L seq is used, the coefﬁcient 1 L 2 seq term can significantly reduce its practical complexity.
We evaluate the memory occupation of these models on ETTh1.
4 We set the prediction window τ = 336 and prolong input length T .
For Crossformer, L seg is set to 24, which is the best value for τ ≥(see Fig.
4 (a)).
The result in Fig.
4 (c) shows that Crossformer achieves the best efﬁciency among the ﬁve methods within the tested length range.
Theoretically, Informer, Autoformer and FEDformer are more efﬁcient when T approaches inﬁnity.
In practice, Crossformer performs better when T is not extremely large (e.g. T ≤ 10 4 ).
We also evaluate the memory occupation w.r.t the number of dimensions D .
For baseline models where cross-dimension dependency is not modeled explicitly, D has little effect.
Therefore, we compare Crossformer with its ablation versions in Section 4.3.
We also evaluate the TSA layers that directly use MSA in Cross-Dimension Stage without the Router mechanism, denoted as TSA(w/o Router).
Fig. 4 (d) shows that Crossformer without TSA layer (DSW and DSW+HED) has quadratic complexity w.r.t D .
TSA(w/o Router) helps to reduce complexity and the Router mechanism further makes the complexity linear, so that Crossformer can process data with D = 300 .
Moreover, HED can slightly reduce the memory cost and we analyze this is because there are less vectors in upper layers after segment merging (see Fig. 3).
Besides memory occupation, the actual running time evaluation is shown in Appendix B.6.
C ONCLUSIONS AND F UTURE W ORK
We have proposed Crossformer, a Transformer-based model utilizing cross-dimension dependency for multivariate time-series (MTS) forecasting.
Speciﬁcally, the Dimension-Segment-Wise (DSW) embedding embeds the input data into a 2D vector array to preserve the information of both time and dimension.
The Two-Stage-Attention (TSA) layer is devised to capture the cross-time and crossdimension dependency of the embedded array.
Using DSW embedding and TSA layer, a Hierarchical Encoder-Decoder (HED) is devised to utilize the information at different scales.
Experimental results on six real-world datasets show its effectiveness over previous state-of-the-arts. We analyzed the limitations of our work and brieﬂy discuss some directions for future research: 1) In
Cross-Dimension Stage, we build a simple full connection among dimensions, which may introduce noise on high-dimensional datasets.
Recent sparse and efﬁcient Graph Transformers (Wu et al., 2022) can beneﬁt our TSA layer on this problem.
A concurrent work (Zeng et al., 2023) which was accepted after the submission of this work received our attention.
It questions the effectiveness of Transformers for MTS forecasting and proposes DLinear that outperforms all Transformers including our Crossformer on three of the six datasets (details are in Appendix B.2). It argues the main reason is that MSA in Transformer is permutation-invariant. Therefore, enhancing the ordering preserving capability of Transformers is a promising direction to overcome this shortcoming .
Considering datasets used in MTS analysis are much smaller and simpler than those used in vision and texts, besides new models, large datasets with various patterns are also needed for future research.
A D ETAILS OF E XPERIMENTS A. B ENCHMARKING D ATASETS
We conduct experiments on the following six real-world datasets following Zhou et al.
; Wu et al.
(2021a): 1) ETTh1 (Electricity Transformer Temperature-hourly) contains 7 indicators of an electricity transformer in two years, including oil temperature, useful load, etc.
Data points are recorded every hour and train/val/test is 12/4/4 months.
2) ETTm1 (Electricity Transformer Temperature-minutely) contains the same indicators as ETTh but data points are recorded every 15 miniutes.
Train/val/test split is same as ETTh1. 3) WTH (Weather) contains 12 meteorological indicators in U.S. in 4 years, including visibility, wind speed, etc.
Train/val/test is 28/10/10 months.
4) ECL (Electricity Consuming Load) contains hourly electricity consumption (in Kwh) ofclients in two years.
Train/val/test is 15/3/4 months.
5) ILI (Inﬂuenza-Like Illness) contains 7 weekly recorded indicators of patients data from Centers for Disease Control and Prevention of the United States between between 2002 and 2021.
The ratio of train/validation/test splits is 0.7:0.1:0.2.
Trafﬁc contains hourly road occupancy rates measured by 862 sensors on San Francisco Bay area freeways in 2 years.
The ratio of train/validation/test splits is 0.7:0.1:0.2.
The train/val/test splits for ETTh1, ETTm1, WTH, ECL are same as Zhou et al.
(2021), for ILI and Trafﬁc are same as Wu et al. (2021a).
The ﬁrst four datasets are publicly available at https://github.com/zhouhaoyi/
Informer2020 and the last two are publicly available at https://github.com/thuml/
Autoformer .
A. B ASELINE M ETHODS
We brieﬂy describe the selected baselines:
1) LSTMa (Bahdanau et al., 2015) treats the input MTS as a sequence of multi-dimensional vectors.
It builds an encoder-decoder using RNN and automatically aligns target future steps with their relevant past.
2) LSTnet (Lai et al., 2018) uses CNN to extract cross-dimension dependency and short term crosstime dependency.
The long-term cross-time dependency is captured through RNN.
The source code is available at https://github.com/laiguokun/LSTNet . 3)
MTGNN (Wu et al., 2020) explicitly utilizes cross-dimension dependency using GNN.
A graph learning layer learns a graph structure where each node represents one dimension in MTS.
Then graph convolution modules are interleaved with temporal convolution modules to explicitly capture cross-dimension and cross-time dependency respectively.
The source code is available at https://github.com/nnzhan/MTGNN .
4) Transformer is closed to the original Transformer (Vaswani et al., 2017) that uses self-attention mechanism to capture cross-time dependency.
The Informer-style one-step generative decoder is used for forecasting, therefore this is denoted as Informer † in Informer (Zhou et al., 2021).
Informer (Zhou et al., 2021) is a Transformer-based model using the ProbSparse self-attention to capture cross-time dependency for forecasting.
The source code of Transformer and Informer is available at https://github.com/zhouhaoyi/Informer2020 .
6) Autoformer (Wu et al., 2021a) is a Transformer-based model using decomposition architecture with Auto-Correlation mechanism to capture cross-time dependency for forecasting.
The source code is available at https://github.com/thuml/Autoformer . 7) Pyraformer (Liu et al., 2021a) is a Transformer-based model learning multi-resolution representation of the time series by the pyramidal attention module to capture cross-time dependency for forecasting.
The source code is available at https:
A. H YPER - PARAMETER S ELECTION AND I MPLEMENTATION D ETAILS A.3. M AIN E XPERIMENTS
For the main experiments, we use the Crossformer with 3 encoder layers.
The number of routers in TSA layer c is set to 10.
For dataset ETTh1, ETTm1, WTH and ILI, dimension of hidden state d model is set to 256, the head number of multi-head attention is set to 4; For dataset ECL and Trafﬁc, dimension of hidden state d model is set to 64, the head number of multi-head attention is set to 2.
The segment length L seg is chosen from { 6 , 12 , 24 } via grid search.
We use MSE as loss function and batch size is set to 32.
Adam optimizer is used for training and the initial learning rate is chosen from { 5e-3, 1e-3, 5e-4, 1e-4, 5e-5, 1e-5 } via grid search.
The total number of epochs is 20.
If the validation loss does not decreases within three epochs, the training process will stop early.
For baseline models, if the original papers conduct experiments on the dataset we use, the hyperparameters (except input length T ) recommended in the original papers are used, including the number of layers, dimension of hidden states, etc.
Otherwise, the hyper-parameters are chosen through grid search using the validation set.
Following Zhou et al.
(2021), on datasets ETTh1, WTH, ECL and Trafﬁc, for different prediction length τ , the input length T is chosen from { 24 , 48 , 96 , 168 , 336 , 720 } ; on ETTm1, the input length is chosen from { 24 , 48 , 96 , 192 , 288 , 672 } ; on ILI, the input length is chosen from { 24 , 36 , 48 , 60 } .
All models including Crossformer and baselines are implemented in PyTorch and trained on a single
NVIDIA Quadro RTX 8000 GPU with 48GB memory.
A.3. E FFICIENCY A NALYSIS
To evaluate the computational efﬁciency w.r.t the input length T in Figure 4(c) of the main paper, we align the hyper-parameters of all Transformer-based models as follows: prediction length τ is set to 336, number of encoder layers is set to 2, dimension of hidden state d model is set to 256, the head number of multi-head attention is set to 4.
To evaluate the computational efﬁciency w.r.t the number of dimensions D in Figure 4(d) of the main paper, we align the hyper-parameters of ablation versions of Crossformer as follows as: both input length T and prediction length τ are set to 336, number of encoder layers is set to 3, d model is set to 64, the head number of multi-head attention is set to 2.
Experiments in the computational efﬁciency analysis section are conducted on a single NVIDIA
GeForce RTX 2080Ti GPU with 11GB memory.
A. D ETAILS OF A BLATION V ERSIONS OF C ROSSFORMER
We describe the models we used in ablation study below
: 1) DSW represents Crossformer without TSA and HED.
The input is embedded by DSW embedding and ﬂatten into a 1D sequence to be input to the original Transformer.
The only difference between this model and the Transformer is the embedding method.
2) DSW+TSA represents Crossformer without HED.
Compared with Crossformer, the encoder does not use segment merging to capture dependency at different scales.
The decoder takes the ﬁnal output of encoder (i.e. Z enc,N ) as input instead of using encoder’s output at each scale.
3) DSW+HED represents Crossformer without TSA.
In each encoder layer and decoder layer, the 2D vector array is ﬂatten into a 1D sequence to be input to the original self-attention layer for dependency capture.
B E XTRA E XPERIMENTAL R ESULTS B. S HOWCASES OF M AIN R ESULTS
Figure 5 shows the forecasting cases of three dimensions of the ETTm1 dataset with prediction length τ = 288 .
For dimension “HUFL”, all the ﬁve models capture the periodic pattern, but Crossformer is
[Figure]
E D
[Figure]
o
[Figure]
m
[Figure]
C r o ss f o r m e r
[Figure]
the closest to the ground truth.
For “HULL”, Pyraformer fails to capture the periodic pattern from the noisy data.
For “LUFL” where the data has no clear periodic pattern, MTGNN, FEDformer and
Crossformer capture its trend and show signiﬁcantly better results than the other two models.
Figure 6 shows the forecasting cases of three dimensions of the WTH dataset with prediction length τ = 336 .
For dimension “DBT”, all the ﬁve models capture the periodic pattern.
For “DPT”,
Autoformer and FEDformer fails to capture increasing trend of the data.
For “WD”, all models capture the periodic pattern from the noisy data, and the cruves output by MTGNN and Crossformer are sharper than the other three models.
B. C OMPARISON WITH E XTRA M ETHODS
{+We further compare with two additional concurrent methods which were either not peerreviewed (Grigsby et al., 2022) or were accepted after the submission of this work (Zeng et al., 
+}
): 1) STformer (Grigsby et al., 2022), a Transformer-based model that directly ﬂattens the multivariate time-series x 1: T ∈ R T × D into a 1D sequence to be input to Transformers; 2) DLinear
(Zeng et al., 2023), a simple linear model with seasonal-trend decomposition that challenges
Transformer-based models for MTS forecasting.
Results are shown in Table 4 and LSTMa and
LSTnet are omitted as they are not competitive with other models.
The basic idea of STformer is similar to our Crossformer: both of them extend the 1-D attention to 2D.
The explicit utilization of cross-dimension dependency makes STformer competitive with previous
Transformer-based models on ETTh1, ETTm1 and WTH, especially for short-term prediction.
However, STformer directly ﬂattens the raw 2-D time series into a 1-D sequence to be input to the
[Figure]
Transformer.
This straightforward method does not distinguish the time and dimension axes and is computationally inefﬁcient.
Therefore, besides the good performance for short-term prediction,
STformer has difﬁculty in long-term prediction and encounters the out-of-memory (OOM) problem on high-dimensional datasets (ECL and Trafﬁc).
While Crossformer uses the DSW embedding to capture local dependency and reduce the complexity.
The TSA layer with the router mechanism is devised to deal with the heterogeneity of time and dimension axis and further improve efﬁciency.
DLinear is on par with our Crossformer on ETTh1 and ETTm1 ( τ ≤ 96 ); has similar performance with FEDformer on ILI; performs worse than Crossformer on WTH; outperforms all Transformerbased models including our Crossformer on ETTm1 ( τ ≥ 288 ), ECL and Trafﬁc.
Considering its simplicity, the performance is impressive.
Based on the results, we analyze the limitations of
Crossformer and propose some directions to improve it in the future:
In Cross-Dimension Stage of TSA layer
, we simply
build an all-to-all connection among D dimensions with the router mechanism.
Besides capturing the cross-dimension dependency, this full connection also introduces noise, especially for high-dimensional dataset.
We think high-dimensional data has the sparse property: each dimension is only relevant to a small fraction of all dimensions.
Therefore, utilizing the sparsity to reduce noise and improve the computation efﬁciency of the TSA layer could be a promising direction.
2) Authors of DLinear (Zeng et al., 2023) argue that the Transformer-based models have difﬁculty in preserving ordering information because the attention mechanism is permutation-invariant and the absolute position embedding injected into the model is not enough for time series forecasting, which is an order-sensitive task.
Although Yun et al.
(2020) theoretically proves that Transformers with
[Table]
trainable positional embedding are universal approximators of sequence-to-sequence functions, the ordering information still needs to be enhanced in practice.
We think that relative position encoding in texts (Ke et al., 2021; Dufter et al., 2022) and vision (Wu et al., 2021b) could be useful for ordering information enhancement.
The sizes of datasets used for time series forecasting are much smaller than those for texts and vision, and the patterns in time series datasets are also simpler.
Considering vision transformers surpass inductive bias and achieves excellent results compared to CNNs after pre-trained on large amounts of data (Dosovitskiy et al., 2021), Transformers for time series may also require large-size datasets with various patterns to exploit their full potential.
As quoted from the paper, authors mentioned that DLinear “does not model correlations among variates”.
Therefore, incorporating cross-dimension dependency into DLinear to further improve prediction accuracy is also a promising direction.
Moreover, our DSW embedding to enhance locality and HED to capture dependency at different scales can also be potentially useful to further inspire and enhance DLinear.
B. A BLATION S TUDY OF THE R OUTER M ECHANISM
The ablation study of the three main components of Crossformer is shown in Sec. 4.3.
In this section, we conduct an ablation study of the router mechanism, a sub-module in TSA layer, and evaluate its impact on prediction accuracy.
It should be noticed that the router mechanism is mainly proposed to reduce the computation complexity when D is large.
Results are shown in Table 5.
Adding TSA(w/o
Router) constantly improves the prediction accuracy of DSW and DSW+HED, showing the necessity of capturing cross-time and cross-dimension dependency in two different stages.
For short term prediction ( τ ≤ 168 ), the performances of TSA(w/o Router) and TSA are similar, no matter whether
HED is used or not.
For long term prediction ( τ ≥ 336 ), the router mechanism slightly improves the prediction accuracy.
The possible reason is that we set separate routers for each time step, which helps capture long-term dependency that varies over time.
[Table]
K e y D i m e n s i o n
Q u e r y D
{+i m # 0 6 12 18 24 30 36 42 48 54 60 66 72 78 84 90 96 102 108 114 120 126 132 138 144 150 156 162 168 Key Time Step 
+}
{+K e y D i m e n s i o n 0 6 12 18 24 30 36 42 
+}
{+168 Key Time Step 
+}
K e y D i m e n s i o n
Q u e r y D
i m # 0 6 12 18 24 30 36 42 48 54 60 66 72 78 84 90 96 102 108 114 120 126 132 138 144 150 156 162 168 Key Time
[Figure]
K e y D i m e n s i o n
[Figure]
{+108 114 120 126 132 138 144 150 156 162 168 Key Time Step 
+}
K e y D i m e n s i o n [Figure] B. D EPENDENCY V ISUALIZATION As the attention scores computed by Crossformer are abstract and hard to visualize, we visualize scores computed by the ablation version, DSW, in Figure 7.
In addition to cross-time dependency that other Transformer models can compute, Crossformer also provides information about crossdimension dependency.
As shown in Figure 7, when predicting Dim #1, the model focus on both
Dim #1 and #3.
When predicting Dim #5, instead of focus on Dim #5 itself, more attention is paid to
Dim #4.
B. H IERARCHICAL P REDICTION P ATTERN V ISUALIZATION
Figure 8 shows the hierarchical prediction patterns output by our HED.
The top prediction layer,
Layer 3, captures the low frequency general trend and periodic pattern of the future value.
By adding predictions at ﬁner scales, ﬁner high frequency patterns are added and the prediction get closer to the ground truth curve.
B. R UNNING T IME E FFICIENCY A NALYSIS
In the main paper, we show the memory occupation w.r.t input length T and number of dimensions
D .
Here we evaluate the running time.
Figure 9 (a) shows the running time per batch of Crossformer [Figure] [Figure] and other Transformer-based models w.r.t input length T .
FEDformer is much slower than other
Transformer-based models.
Crossformer achieves the best computation speed among the ﬁve methods within the tested length range.
Figure 9 (b) shows the running time per batch of Crossformer and its ablation versions w.r.t the number of dimensions D .
Crossformers without TSA layer (DSW and DSW+HED) are faster when
D is small ( D ≤ 30 ).
However, they have difﬁculty processing high-dimensional MTS due to the quadratic complexity w.r.t D .
Indeed, for a single NVIDIA GeForce RTX 2080Ti GPU with 11GB memory, DSW and DSW+HED encounters the out-of-memory (OOM) problem when D > 50 .
Moreover, TSA(w/o Router) encounter the OOM problem when D > 200 .
C D ISCUSSION ON THE S ELECTION OF H YPER - PARAMETERS
We recommend to ﬁrst determine the segment length L seg , as it is related to both the model performance and computation efﬁciency.
The general idea is to use small L seg for short-term prediction and large L seg for long-term prediction.
Some priors about the data also help to select L seg .
For example, if the hourly sampled data has a daily period, it is better to set L seg = 24 .
[-108 114 120 126 132 138 144 150 156 162 168 Key Time Step 
 K e y D i m e n s i o n 0 6 12 18 24 30 36 42 
 168 Key Time Step 
 K e y D i m e n s i o n-]Next, we select the number
[Table]
[-K e y D i m e n s i o n 0 6 12 18 24 30 36 42 
-]
[-168 Key Time Step 
-]
of layers for encoder and decoder N .
[-Step 
-]
Crossformer with larger N can utilize information of more scales, but also requires more computing resources.
The number routers in TSA layer c can be set to 5 or 10 to balance the prediction accuracy and computation efﬁciency.
Finally, dimension of hidden states d model and head number of multi-head attention can be determined based on the available computing resources.
D S UPPLEMENTARY D ESIGN TO C ROSSFORMER D. H ANDLING I NDIVISIBLE L ENGTH
In the main paper, we assume that the input length T and prediction length τ are divisible by segment length L seg .
In this section, we use padding mechanism to handle cases where the assumption is not satisﬁed.
If T is not divisible by L seg , we have ( k 1 − 1)
L seg < T < k 1 L seg for some k 1 .
We pad k 1 L seg − T duplicated x 1 in front of x 1: T to get x (cid:48) 1: T :
[Equation]
(9) where [ , ] denotes the concatenation operation.
x (cid:48) 1: T ∈ R k
[Equation] L [Equation] can be input to the encoder of
Crossformer.
If τ is not divisible by L seg , we have ( k 2 − 1)
L seg < τ < k 2 L seg for some k 2 .
We set the learnable position embedding for decoder as E ( dec ) ∈ R k 2 × D × d model and input it to the decoder to get an output in shape of R k 2 L seg × D .
Then the ﬁrst τ steps of the output is used as x predT +1: T
+ τ .
We conduct experiment on ETTm1 dataset to evaluate the effect of indivisible length.
Results in
Table 6 show that with padding mechanism, indivisible length does not degrade model performance, for both short-term prediction and long-term prediction.
D. I NCORPORATING C OVARIATES
In the main text, we only use historical series x 1: T to forecast the future x T +1: T
+ τ .
In this section, we try to incorporate covariates c 1: T + τ into Crossformer.
We use a straightforward method: ﬁrst embed the covariates into point-wise vectors { d 1 , d 2 , . . .
, d T + τ } like previous Transformer-based models do (Zhou et al., 2021; Wu et al., 2021a; Liu et al., 2021a).
Then, merge the point-wise vectors into segment-wise vectors using learnable linear combination.
Finally, add the segment-wise vectors to each dimension of the 2D vector array obtained by DSW embedding:
[Equation]
where → denotes embedding method for point-wise covariates.
α j , 1 ≤ j ≤ L seg denotes learnable factors for linear combination.
d ( s ) i denotes the segment-wise covariate
[Table]
embedded vector with covariate information for the i -th segment in dimension d , where h i,d is the embedded vector obtained from DSW embedding in the main text.
The processing for the input of the decoder is similar, the segment-wise covariate embedding is added to the position embedding for decoder, i.e. E ( dec ) .
We conduct experiments on ETTh1 dataset to evaluate the effect of covariates.
Hour-of-the-day, dayof-the-week, day-of-the-month and day-of-the-year are used as covariates.
Results in Table 7 show that incorporating covariates does not improve the performance of Crossformer.
The possible reason is this straightforward embedding method does not cooperate well with Crossformer.
Incorporating covariates into Crossformer to further improve prediction accuracy is still an open problem.