C ROSSFORMER : T RANSFORMER U TILIZING C ROSSD IMENSION D EPENDENCY FOR M ULTIVARIATE T IME S ERIES F ORECASTING
Anonymous authors
Paper under double-blind review
A BSTRACT
Multivariate time series (MTS) forecasting applies to many practical scenarios.
Recently many deep models have been proposed for MTS forecasting.
In particular,
Transformer-based models have shown great potential because they can capture long-term dependency.
However, existing Transformer-based models mainly focus on modeling the temporal dependency (cross-time dependency) yet often omit the dependency among different variables (cross-dimension dependency), which is critical for MTS forecasting.
To ﬁll the gap, we propose Crossformer, a
Transformer-based model utilizing cross-dimension dependency for MTS forecasting.
In Crossformer, the input MTS is embedded into a 2D vector array through the Dimension-Segment-Wise (DSW) embedding to preserve time and dimension information.
Then the Two-Stage Attention (TSA) layer is proposed to efﬁciently capture the cross-time and cross-dimension dependency.
Utilizing DSW embedding and TSA layer, Crossformer establishes a Hierarchical Encoder-Decoder (HED) to use the information at different scales for the ﬁnal forecasting.
Extensive experimental results on six real-world datasets show the effectiveness of Crossformer against state-of-the-arts.
I NTRODUCTION
Multivariate time series (MTS) are time series with multiple dimensions, where each dimension represents a speciﬁc univariate time series (e.g. a climate feature of weather).
MTS forecasting aims to forecast the future value of MTS using their historical values.
MTS forecasting beneﬁts the decision-making of downstream tasks and is widely used in many ﬁelds including weather (Angryk et al., 2020), energy (Demirel et al., 2012), ﬁnance (Patton, 2013), etc.
With the development of deep learning, many models have been proposed and achieved superior performances in MTS forecasting (Lea et al., 2017; Qin et al., 2017;
Flunkert et al., 2017; Li et al., 2018;
Rangapuram et al., 2018; Wu et al., 2020).
Among them, the recent Transformer-based models (Li et al., 2019; Zhou et al., 2021; Wu et al., 2021; Liu et al., 2021a; Du et al., 2022; Zhou et al., 2022) show great potential thanks to their ability to capture long-term temporal dependency (cross-time dependency).
Besides cross-time dependency, the cross-dimension dependency is also critical for MTS forecasting, i.e. for a speciﬁc dimension, information from associated series in other dimensions may improve prediction.
For example, when predicting future temperature, not only the historical temperature, but also historical wind speed helps to forecast.
Some previous neural models explicitly capture the cross-dimension dependency, i.e. preserving the information of dimensions in the latent feature space and using convolution neural network (CNN) (Lai et al., 2018) or graph neural network (GNN) (Wu et al., 2020; Cao et al., 2020) to capture their dependency.
However, recent Transformer-based models only implicitly utilize this dependency by embedding.
In general, Transformer-based models embed data points in all dimensions at the same time step into a feature vector and try to capture dependency among different time steps (like Fig.
1 (b)).
In this way, cross-time dependency is well captured, but cross-dimension dependency is not well exploited and utilized, which may limit their forecasting capability.
To ﬁll the gap, we propose Crossformer, a Transformer-based model that explicitly utilizes crossdimension dependency for MTS forecasting.
Speciﬁcally, we devise Dimension-Segment-Wise (DSW) embedding to process the historical time series.
In DSW embedding, the series in each dimension is ﬁrst partitioned into segments and then embedded into feature vectors.
The output of DSW embedding is a 2D vector array where the two axes correspond to time and dimension.
Then we propose the Two-Stage-Attention (TSA) layer to efﬁciently capture the cross-time and cross-dimension dependency among the 2D vector array.
Using DSW embedding and TSA layer,
Crossformer establishes a Hierarchical Encoder-Decoder (HED) for forecasting.
In HED, each layer corresponds to a scale.
The encoder’s upper layer merges adjacent segments output by the lower layer to capture the dependency at a coarser scale.
Decoder layers generate predictions at different scales and add them up as the ﬁnal prediction.
The contributions of this paper are:
1) We dive into the existing Transformer-based models for MTS forecasting and ﬁgure out that the cross-dimension dependency is not well utilized: these models simply embed data points of all dimensions at a speciﬁc time step into a single vector and focus on capturing the cross-time dependency among different time steps.
Without adequate and explicit mining and utilization of cross-dimension dependency
their
forecasting
capability
is
empirically shown
limited
We develop Crossformer, a Transformer model utilizing cross-dimension dependency for MTS
forecasting
This is one of the
few
transformer models
(perhaps the
ﬁrst to
our best
knowledge
) that
explicitly explores and utilizes cross-dimension dependency for MTS forecasting.
Extensive experimental results on six real-world benchmarks show the effectiveness of our
Crossformer against previous state-of-the-arts.
Speciﬁcally, Crossformer ranks
top-
1 among themodels for comparison on 36 out of the 58 settings of varying prediction lengths and metrics and ranks top-2 on 51 settings.
Source code and detailed conﬁgurations will be made publicly available.
R ELATED W ORKS 2. M ULTIVARIATE T IME S ERIES F ORECASTING
MTS forecasting models can be roughly divided into statistical and neural models.
Vector autoregressive (VAR) model (Kilian & L ˜ Atkepohl, 2017) is a typical statistical model, which assumes linear cross-dimension and cross-time dependency.
Vector auto-regressive moving average (VARMA) model (Spliid, 1983) extends VAR by modeling the error term using linear combination.
With the development of deep learning, many neural models have been proposed and often empirically show better performance than statistical ones.
TCN (Lea et al., 2017) and DeepAR (Flunkert et al., 2017) treat the MTS data as a sequence of vectors and use CNN or RNN respectively to capture the temporal dependency.
LSTnet (Lai et al., 2018) employs CNN to capture cross-dimension dependency and
RNN to capture cross-time dependency for MTS forecasting.
Another category of works use graph neural networks (GNNs) to capture the cross-dimension dependency explicitly for forecasting (Li et al., 2018; Yu et al., 2018; Cao et al., 2020; Wu et al., 2020).
For example, MTGNN (Wu et al., 2020) uses a graph learning layer to construct a graph where each node represents one dimension in
MTS.
Then temporal convolution and graph convolution layers are used to capture cross-time and cross-dimension dependency.
These neural models capture the cross-time dependency through CNN or RNN, which have difﬁculty in modeling long-term dependency.
2. T RANSFORMERS FOR MTS F ORECASTING
Transformers (Vaswani et al., 2017) have achieved success in natural language processing (NLP)
(Devlin et al., 2019), vision (CV) (Dosovitskiy et al., 2021) and speech processing (Dong et al., 2018)
thanks to long-term dependency capture.
Recently, many Transformer-based models have been proposed for MTS forecasting and show great potential (Li et al., 2019;
Zhou et al., 2021; Wu et al., 2021; Liu et al., 2021a;
Zhou et al., 2022; Du et al., 2022).
LogTrans (Li et al., 2019) introduces convolution to Transformer to enhance locality and proposes the LogSparse attention that reduces the computation complexity of Transformer from O ( L 2 ) to O (cid:0) L (log L ) 2 (cid:1)
1 Informer
(Zhou et al., 2021) utilizes the sparsity of attention score through KL divergence estimation and proposes ProbSparse self-attention which achieves O ( L log L ) complexity.
Autoformer (Wu et al., 2021) introduces a decomposition architecture with an Auto-Correlation mechanism to Transformer, which also achieves the O ( L log L ) complexity.
Pyraformer (Liu et al., 2021a) introduces a pyramidal attention module that summarizes features at different resolutions and models the temporal dependencies of different ranges with the complexity of O ( L ) .
FEDformer (Zhou et al., 2022) proposes that time series have a sparse representation in frequency domain and develop a frequency enhanced Transformer with the
O ( L ) complexity.
Preformer (Du et al., 2022) divides the embedded feature vector sequence into segments and utilizes segment-wise correlation-based attention for forecasting.
These models mainly focus on reducing the complexity of cross-time dependency modeling, but omits the cross-dimension dependency which is critical for MTS forecasting.
2. V ISION T RANSFORMERS
Transformer is initially applied to NLP for sequence modeling, recent works apply transformer to CV tasks to process images (Dosovitskiy et al., 2021;
Touvron et al., 2021; Liu et al., 2021b; Chen et al., 2021; Han et al., 2021).
These works achieve state-of-the-art performance on various tasks in CV and inspire our work.
ViT (Dosovitskiy et al., 2021) is one of the pioneers of vision transformers.
The basic idea of ViT is to split an image into non-overlapping medium-sized patches, then it rearranges these patches into a sequence to be input to the Transformer.
The idea of partitioning images into patches inspires our DSW embedding where MTS is split into dimesion-wise segments.
Swin
Transformer (Liu et al., 2021b) performs local attention within a window to reduce the complexity and builds hierarchical feature maps by merging image patches.
Hierarchical structures are also widely used in Transformers for MTS forecasting.
Readers can refer to the recent survey (Han et al., 2022) for comprehensive study on vision transformers.
M ETHODOLOGY
In multivariate time series forecasting, one aims to predict the future value of time series x T +1: T + τ ∈ R τ × D given the history x 1: T ∈ R T × D , where τ , T is the number of time steps in the future and past, respectively 2 .
D > 1 is the number of dimensions.
A natural assumption is that these D series are associated (e.g. climate features of weather), which helps to improve the forecasting accuracy.
To utilize the cross-dimension dependency, in Section 3.1, we embed the multivariate time series using
Dimension-Segment-Wise (DSW) embedding.
In Section 3.2, we propose a Two-Stage Attention (TSA) layer to efﬁciently capture the dependency among the embedded segments.
In Section 3.3, using DSW embedding and TSA layer, we construct a hierarchical encoder-decoder (HED) to utilize information at different scales for ﬁnal forecasting.
3. D IMENSION -S EGMENT -W ISE E MBEDDING
To motivate our approach, we ﬁrst analyze the embedding methods of the previous Transformer-based models for MTS forecasting (Zhou et al., 2021; Wu et al., 2021; Liu et al., 2021a;
Zhou et al., 2022).
As shown in Fig.
1 (b), existing methods embed data points at the same time step into a vector: x t → h t , x t ∈ R D , h t ∈ R d model , where x t represents all the data points in D dimensions at step t .
In this way, the input x 1: T is embedded into T vectors { h 1 , h 2 , . .
, h T } .
Then the dependency among the T vectors is captured for forecasting.
Therefore, previous Transformer-based models mainly capture cross-time dependency, while the cross-dimension dependency is not explicitly captured during embedding, which limits their forecasting capability.
Transformer was originally developed for NLP (Vaswani et al., 2017), where each embedded vector represents an informative word.
For MTS, a single value at a step alone provides little information.
While it forms informative pattern with nearby values in time domain.
Fig.
1 (a) shows a typical attention score map of original Transformer for MTS forecasting.
We can see that attention values have a tendency to segment, i.e. close data points have similar attention weights.
Based on the above two points, we argue that an embedded vector should represent a series segment of single dimension (Fig.
1 (c)), rather than the values of all dimensions at single step (Fig.
1 (b)).
To this end, we propose Dimension-Segment-Wise
(DSW)
embedding where the points in each dimension are divided into segments of length L seg and then embedded:
[Equation]
[Figure]
Q u e r y T i m e S t e p
[Figure]
[Figure] [Figure]
[Figure] [Figure]
where x ( s ) i,d ∈ R L seg is the i -th segment in dimension d with length L seg .
For convenience, we assume that T, τ are divisible by L seg 3 .
Then each segment is embedded into a vector using linear projection added with a position embedding:
[Equation]
where E ∈ R d model ×
L seg denotes the learnable projection matrix, and E ( pos ) i,d ∈ R d model denotes the learnable position embedding for position ( i, d ) .
After embedding, we obtain a 2D vector array
H = (cid:110) h i,d | 1 ≤ i ≤ TL seg , 1 ≤ d ≤ D (cid:111) , where each h i,d represents a univariate time series segment.
The idea of segmentation is also used in Du et al.
(2022), which splits the embedded 1D vector sequence into segments to compute the Segment-Correlation in order to enhance locality and reduce computation complexity.
However, like other Transformers for MTS forecasting, it does not explicitly capture cross-dimension dependency.
3. T WO -S TAGE A TTENTION L AYER
For the obtained 2D array H , a simple treatment is to ﬂatten it into a 1D sequence so that it can be input to a canonical Transformer like ViT (Dosovitskiy et al., 2021) does in vision.
While there are two factors for consideration: 1) Different from images where the axes of height and width are interchangeable, the axes of time and dimension for MTS have different meanings and thus should be treated differently.
2) Directly applying self-attention on the 2D array will cause the complexity of O
( D 2 T 2 L 2 seg ) , which is unaffordable for large D .
Therefore, we propose the Two-Stage Attention (TSA) Layer to capture cross-time and cross-dimension dependency among the 2D vector array, as sketched in Fig.
2 (a).
Cross-Time Stage Given a 2D array Z ∈ R L × D × d model as the input of the TSA Layer, where L and D are the number of segments and dimensions, respectively.
Z here can be the output of DSW embedding or lower TSA layers.
For convenience, in the following, we use Z i, : to denote the vectors of all dimensions at time step i , Z : ,d for those of all time steps in dimension d .
In the cross-time stage, we directly apply multi-head self-attention (MSA) to each dimension:
[Equation]
(3) where 1 ≤ d ≤ D and LayerNorm denotes layer normalization as widely adopted in Vaswani et al.
(2017); Dosovitskiy et al.
; Zhou et al.
(2021), MLP denotes a multi-layer (two in this paper)
[Figure]
𝒁𝒁 𝒁𝒁 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 𝒀𝒀 ( 𝒁𝒁 𝑑𝑑𝑡𝑡m ) 𝐷𝐷 𝐿𝐿 (a) ……
𝒁𝒁 𝑡𝑡 ,: 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 𝒁𝒁 𝑡𝑡 ,: 𝑑𝑑𝑡𝑡𝑡𝑡 𝐷𝐷 ……
MSA 𝑑𝑑𝑡𝑡𝑡𝑡 ( 𝒁𝒁 𝑡𝑡 ,: 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 , 𝒁𝒁 𝑡𝑡 ,: 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 , 𝒁𝒁 𝑡𝑡 ,: 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 ) (b) …… ……
𝒁𝒁 𝑡𝑡 ,: 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 𝑩𝑩 𝑡𝑡 ,: ( 𝑐𝑐 =3)
𝒁𝒁 𝑡𝑡 ,: 𝑑𝑑𝑡𝑡𝑡𝑡
MSA 1𝑑𝑑𝑡𝑡𝑡𝑡 ( 𝑹𝑹 𝑡𝑡 ,: , 𝒁𝒁 𝑡𝑡 ,: 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 , 𝒁𝒁 𝑡𝑡 ,: 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 )
MSA 2𝑑𝑑𝑡𝑡𝑡𝑡 ( 𝒁𝒁 𝑡𝑡 ,: 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 , 𝑩𝑩 𝑡𝑡 ,: , 𝑩𝑩 𝑡𝑡 ,: ) 𝐷𝐷 (c) feedforward network, MSA ( Q , K , V ) denotes the multi-head self-attention (Vaswani et al., 2017) layer where Q , K , V serve as queries, keys and values.
All dimensions ( 1 ≤ d ≤ D ) share the same
MSA layer.
ˆ Z time , Z time denotes the output of the MSA and MLP.
The computation complexity of cross-time stage is O ( DL 2 ) .
After this stage, the dependency among time segments in the same dimension is captured in Z time .
Then Z time becomes the input of cross-dimension stage to capture cross-dimension dependency.
Cross-Dimension Stage We can use a large L seg for long sequence in DSW Embedding to reduce the number of segments L in cross-time stage.
While in cross-dimension stage, we can not partition dimensions and directly apply MSA will cause the complexity of O
( D 2 ) (as shown in Fig.
2 (b)), which is unaffordable for datasets with large D .
Instead, we propose the router mechanism for potentially large D .
As shown in Fig.
2 (c), we set a small ﬁxed number ( c << D ) of learnable vectors for each time step i as routers.
These routers ﬁrst aggregate messages from all dimensions by using routers as query in MSA and vectors of all dimensions as key and value.
Then routers distribute the received messages among dimensions by using vectors of dimensions as query and aggregated messages as key and value.
In this way, the all-to-all connection among D dimensions are built:
[Equation]
where R ∈ R
L × c × d model ( c is a constant) is the learnable vector array serving as routers.
B ∈
R L ×
c × d model is the aggregated messages from all dimensions.
Z dim denotes output of the router mechanism.
All time steps ( 1 ≤ i ≤ L ) share the same MSA dim, MSA dim.
ˆ Z dim , Z dim denote output of skip connection and MLP respectively.
The router mechanism reduce the complexity of from O ( D 2 L ) to O ( DL ) .
Adding up Eq.
3 and Eq.
4, we model the two stages as:
[Equation]
where Z , Y ∈ R L × D × d model denotes the input and output vector array of TSA layer, respectively.
Note that the overall computation complexity of the TSA layer is O ( DL 2 + DL )
=
O ( DL 2 ) .
After the cross-time and cross-dimension stages, every two segments (i.e. Z i 1 ,d 1 , Z i 2 ,d 2 ) in Z are connected, as such both cross-time and cross-dimension dependencies are captured in Y .
[Figure]
Encoder layer Decoder layer
Hiddenvectors 𝒙𝒙 1 : 𝑇𝑇 𝒁𝒁 𝑡𝑡𝑒𝑒𝑒𝑒 , 0 𝒁𝒁 𝑡𝑡𝑒𝑒𝑒𝑒 , 1 𝒁𝒁 𝑡𝑡𝑒𝑒𝑒𝑒 ,𝒁𝒁 𝑡𝑡𝑒𝑒𝑒𝑒 ,𝒁𝒁 𝑑𝑑𝑡𝑡𝑒𝑒 ,𝒁𝒁 𝑑𝑑𝑡𝑡𝑒𝑒 ,𝒁𝒁 𝑑𝑑𝑡𝑡𝑒𝑒 ,𝒁𝒁 𝑑𝑑𝑡𝑡𝑒𝑒 ,
DSW embedding 𝑬𝑬 ( 𝑑𝑑𝑡𝑡𝑒𝑒 )
Linear projection 𝒙𝒙 𝑇𝑇+1 :
𝑇𝑇+𝜏𝜏 𝑝𝑝𝑝𝑝𝑡𝑡𝑑𝑑
Hierarchical structures are widely used in Transformers for MTS forecasting to capture information at different scales (Zhou et al., 2021;
Liu et al., 2021a).
In this section, we use the proposed DSW embedding, TSA layer and segment merging to construct a Hierarchical EncoderDecoder (HED).
As shown in Fig.
3, the upper layer utilizes information at a coarser scale for forecasting.
Forecasting values at different scales are added to output the
Encoder
In each layer of the encoder (except the ﬁrst layer), every two adjacent vectors in time domain are merged to obtain the representation at a coarser level.
Then a TSA layer is applied to capture dependency at this scale.
This process is modeled as Z enc,l = Encoder ( Z enc,l − 1 ) :
[Equation]
where H denotes the 2D array obtained by DSW embedding; Z enc,l denotes the output of the l -th encoder layer; M ∈ R d model × 2 d model denotes a learnable matrix for segment merging; [ · ] denotes the concatenation operation; L l − 1 denotes the number of segments in each dimension in layer l − 1 , if it is not divisible by 2, we pad Z enc,l − 1 to the proper length; ˆ Z enc,l denotes the array after segment merging in the i -th layer.
Suppose there are N layers in the encoder, we use
Z enc, 0 , Z enc, 1 , . .
, Z enc,N , ( Z enc, 0 = H ) to represent the N + 1 outputs of the encoder.
The complexity of each encoder layer is O
( D T 2 L 2 seg ) .
Decoder Obtaining the N + 1 feature arrays output by the encoder, we use N + 1 layers (indexed by 0 , 1 , . .
. , N ) in decoder for forecasting.
Layer l takes the l -th encoded array as input, then outputs a decoded 2D array of layer l .
This process is summarized as Z dec,l = Decoder ( Z dec,l − 1 , Z enc,l ) :
[Equation] [Equation]
where E ( dec ) ∈ R τLseg × D × d model denotes the learnable position embedding for decoder.
˜
Z dec,l denotes the output of TSA.
The MSA layer takes ˜ Z dec,l : ,d as query and Z enc,l : ,d as the key and value to build the connection between encoder and decoder.
The output of MSA layer is denoted as Z dec,l : ,d . ˆ Z dec,l , Z dec,l denote output of skip connection and MLP respectively.
We use
Z dec, 0 , Z enc, 1 , . . .
, Z dec,N to represent the output of decoder.
The complexity of each decoder layer is O (cid:16) D τ ( T + τ ) L 2 seg (cid:17) .
Linear projection is applied to each layer’s output to yield the prediction of this layer.
Layer predictions are summed to make the ﬁnal prediction (for l = 0 , . .
. , N ): for l = 0 , . .
, N : x ( s ) ,l i,d = W l Z dec,li,d x pred,lT +1: T
+ τ
[Equation]
(cid:26) [Equation]
[Equation]
l
[Equation]
(cid:27)
[Equation]
(8) where W l ∈ R L seg × d model is a learnable matrix to project a vector to a time series segment.
x ( s ) ,l i,d ∈ R L seg denotes the i -th segment in dimension d of the prediction.
All the segments in layer l are rearranged to get the layer prediction x pred,lT +1: T + τ .
Predictions of all the layers are summed to obtain the ﬁnal forecasting x predT +1: T + τ .
E XPERIMENTS 4. P ROTOCOLS
Datasets We conduct experiments on six real-world datasets following Zhou et al.
; Wu et al.
1) ETTh1 (Electricity Transformer Temperature-hourly), 2) ETTm1 (Electricity
Transformer Temperature-minutely)
, 3) WTH (Weather), 4) ECL (Electricity Consuming Load), 5)
ILI (Inﬂuenza-Like Illness), 6) Trafﬁc .
The train/val/test splits for the ﬁrst four datasets are same as
Zhou et al.
(2021), the last two are split by the ratio of 0.7:0.1:0.2 following Wu et al.
Baselines We use the following popular models for MTS forecasting as baselines: 1) LSTMa (Bahdanau et al., 2015), 2) LSTnet
(Lai et al., 2018), 3) MTGNN (Wu et al., 2020), and recent
Transformer-based models for MTS forecasting: 4) Transformer (Vaswani et al., 2017), 5)
Informer (Zhou et al., 2021), 6) Autoformer (Wu et al., 2021), 7) Pyraformer (Liu et al., 2021a) and 8) FEDformer
(Zhou et al., 2022).
Setup We use the same setting as in Zhou et al.
(2021): train/val/test sets are zero-mean normalized with the mean and std of training set.
On each dataset, we evaluate the performance over the changing future window size τ .
For each τ , the corresponding past window size T is regarded as a hyperparameter to search which is a common protocol in recent MTS transformer literature (Zhou et al., 2021; Liu et al., 2021a).
We roll the whole set with stride = 1 to generate different input-output pairs.
The Mean Square Error (MSE) and Mean Absolute Error (MAE) are used as evaluation metrics.
All experiments are repeated for 5 times and the mean of the metrics reported.
Our Crossformer only utilize the past series x 1: T to forecast the future x T +1: T + τ , while baseline models use additional covariates such as hour-of-the-day.
Details about datasets, basline models, implementation, hyperparameter selection are shown in Section A of the Appendix.
4. M AIN R ESULTS
As shown in Table 1, Crossformer shows leading performance on most datasets, as well as on different prediction length settings, with the 36 top-1 and 51 top-2 cases out of 58 in total.
It is worth noting that, perhaps due to the explicit use of cross-dimension dependency via GNN, MTGNN outperforms many Transformer-based baselines.
While MTGNN has been rarely compared in existing transformers for MTS forecasting literatures.
FEDformer and Autoformer outperform our model on
ILI.
We conjecture this is because the size of dataset ILI is small and these two models introduce the prior knowledge of sequence decomposition into the network structure which makes them perform well when the data is limited.
Crossformer still outperforms other baselines on this dataset.
4. A BLATION S TUDY
In our approach, there are three components: DSW embedding, TSA layer and HED.
We perform ablation study on the ETTh1 dataset in line with Zhou et al.
Liu et al.
(2021a).
We use
Transformer as the baseline and DSW+TSA+HED to denote Crossformer without ablation.
Three ablation versions are compared:
[Equation]
We analyze the results shown in Table 2. 1) DSW performs better than Transformer on most settings.
The only difference between DSW and Transformer is the embedding method, which indicates
[Table]
[Figure] (d [Figure] the usefulness of DSW embedding and the importance of cross-dimension dependency.
2) TSA constantly improves the forecasting accuracy.
This suggests that it is reasonable to treat time and dimension differently.
Moreover, TSA makes it possible to use Crossformer on datasets where the number of dimensions is large (e.g. D = 862 for dataset Trafﬁc).
Comparing DSW+HED with
DSW, HED decreases the forecasting accuracy when prediction length is short but increases it for long term prediction.
The possible reason is that information at different scales is helpful to long term prediction.
Combining DSW, TSA and HED, our Crossformer yields best results on all settings.
4. E FFECT OF H YPER -P ARAMETERS
We evaluate the effect of two hyper-parameters: segment length ( L seg in Eq. 1) and number of routers in TSA ( c in Cross-Dimension Stage of TSA) on the ETTh1 dataset.
Segment Length:
In Fig. 4(a), we prolong the segment length from 4 to 24 and evaluate MSE with different prediction windows.
For short-term forecasting ( τ = 24 , 48 ), smaller segment yields relevantly better results, but the prediction accuracy is stable.
For long-term forecasting ( τ ≥ 168 ), prolonging the segment length from 4 to 24 causes the MSE to decrease.
This indicates that long segments should be used for long-term forecasting.
We further prolong the segment length to 48 for τ = 336 , 720 , the MSE is slightly larger than that of 24.
The possible reason is that 24 hours exactly matches the daily period of [Table] this dataset, while 48 is too coarse to capture ﬁne-grained information.
Number of Routers in TSA
Layer: Number of Routers c controls the information bandwidth among all dimensions.
As Fig. 4(b) shows, the performance of Crossformer is stable w.r.t to c for τ ≤ 336 .
For τ = 720 , the MSE is large when c = 3 but decreases and stabilizes when c ≥ 5 .
In pratice, we set c = 10 to balance the prediction accuracy and computation efﬁciency.
4. C OMPUTATIONAL E FFICIENCY A NALYSIS
- tion window
[Table]
The theoretical complexity per layer of
Transformer-based models is compared in
Table 3.
The complexity of Crossformer encoder is quadratic w.r.t T .
However, for long-term prediction where large L seq is used, the coefﬁcient 1 L 2 seq term can significantly reduce its practical complexity.
We evaluate the memory occupation of these models on ETTh1.
4 We set the prediction window τ = 336 and prolong input length T .
For Crossformer, L seg is set to 24, which is the best value for τ ≥ 168 (see
Fig.
4 (a)).
The result in Fig.
4 (c) shows that Crossformer achieves the best efﬁciency among the ﬁve methods within the tested length range.
Theoretically, Informer, Autoformer and FEDformer are more efﬁcient when T approaches inﬁnity.
In practice, Crossformer performs better when T is not extremely large (e.g. T ≤ 10 4 ).
We also evaluate the memory occupation w.r.t number of dimensions D .
For baseline models where cross-dimension dependency is not modeled explicitly, D has little effect.
Therefore, we compare
Crossformer with its ablation versions in Section 4.3.
We also evaluate TSA layers that directly use
MSA in Cross-Dimension Stage without the Router mechanism, denoted as TSA(w/o Router).
Fig.(d) shows that Crossformer without TSA layer (DSW and DSW+HED) has quadratic complexity w.r.t D .
TSA(w/o Router) helps to reduce complexity and the Router mechanism further makes the complexity linear, so that Crossformer can process data with D = 300 .
Moreover, HED can slightly reduce the memory cost and we analyze this is because there are less vectors in upper layers after segment merging (see Fig. 3).
Besides memory occupation, the actual running time evaluation is shown in Section B.4 of the Appendix.
C ONCLUSION
We have proposed Crossformer, a Transformer-based model utilizing cross-dimension dependency for multivariate time series forecasting.
Speciﬁcally, the Dimension-Segment-Wise (DSW) embedding embeds the input data into a 2D vector array to preserve the information of both time and dimension.
The Two-Stage-Attention (TSA) layer is devised to capture the cross-time and cross-dimension dependency of the embedded array.
Using DSW embedding and TSA layer, a Hierarchical EncoderDecoder (HED) is devised to utilize the information at different scales.
Extensive experimental results on six real-world datasets demonstrate the effectiveness of Crossformer against state-of-the-arts.
A D ETAILS OF E XPERIMENTS A. B ENCHMARKING D ATASETS
We conduct experiments on the following six real-world datasets following Zhou et al.
; Wu et al.
1) ETTh1 (Electricity Transformer Temperature-hourly) contains 7 indicators of an electricity transformer in two years, including oil temperature, useful load, etc.
Data points are recorded every hour and train/val/test is 12/4/4 months.
2) ETTm1 (Electricity Transformer Temperature-minutely) contains the same indicators as ETTh but data points are recorded every 15 miniutes.
Train/val/test split is same as ETTh1. 3) WTH (Weather) contains 12 meteorological indicators in U.S. in 4 years, including visibility, wind speed, etc.
Train/val/test is 28/10/10 months.
4) ECL (Electricity Consuming Load) contains hourly electricity consumption (in Kwh) ofclients in two years.
Train/val/test is 15/3/4 months.
5) ILI (Inﬂuenza-Like Illness) contains 7 weekly recorded indicators of patients data from Centers for Disease Control and Prevention of the United States between between 2002 and 2021.
The ratio of train/validation/test splits is 0.7:0.1:0.2.
Trafﬁc contains hourly road occupancy rates measured by 862 sensors on San Francisco Bay area freeways in 2 years.
The ratio of train/validation/test splits is 0.7:0.1:0.2.
The train/val/test splits for ETTh1, ETTm1, WTH, ECL are same as Zhou et al.
(2021), for ILI and Trafﬁc are same as Wu et al.
The ﬁrst four datasets are publicly available at https://github.com/zhouhaoyi/
Informer2020 and the last two are publicly available at https://github.com/thuml/
Autoformer .
A. B ASELINE M ETHODS
We brieﬂy describe the selected baselines:
1) LSTMa (Bahdanau et al., 2015) treats the input MTS as a sequence of multi-dimensional vectors.
It builds an encoder-decoder using RNN and automatically aligns target future steps with their relevant past.
2) LSTnet (Lai et al., 2018) uses CNN to extract cross-dimension dependency and short term crosstime dependency.
The long-term cross-time dependency is captured through RNN.
The source code is available at https://github.com/laiguokun/LSTNet . 3)
MTGNN (Wu et al., 2020) explicitly utilizes cross-dimension dependency using GNN.
A graph learning layer learns a graph structure where each node represents one dimension in MTS.
Then graph convolution modules are interleaved with temporal convolution modules to explicitly capture cross-dimension and cross-time dependency respectively.
The source code is available at https://github.com/nnzhan/MTGNN .
4) Transformer is closed to the original Transformer (Vaswani et al., 2017) that uses self-attention mechanism to capture cross-time dependency.
The Informer-style one-step generative decoder is used for forecasting, therefore this is denoted as Informer † in Informer (Zhou et al., 2021).
Informer (Zhou et al., 2021) is a Transformer-based model using the ProbSparse self-attention to capture cross-time dependency for forecasting.
The source code of Transformer and Informer is available at https://github.com/zhouhaoyi/Informer2020 .
6) Autoformer (Wu et al., 2021) is a Transformer-based model using decomposition architecture with Auto-Correlation mechanism to capture cross-time dependency for forecasting.
The source code is available at https://github.com/thuml/Autoformer . 7) Pyraformer (Liu et al., 2021a) is a Transformer-based model learning multi-resolution representation of the time series by the pyramidal attention module to capture cross-time dependency for forecasting.
The source code is available at https://
A. H YPER - PARAMETER S ELECTION AND I MPLEMENTATION D ETAILS A.3. M AIN E XPERIMENTS
For the main experiments, we use the Crossformer with 3 encoder layers.
The number of routers in TSA layer c is set to 10.
For dataset ETTh1, ETTm1, WTH and ILI, dimension of hidden state d model is set to 256, the head number of multi-head attention is set to 4; For dataset ECL and Trafﬁc, dimension of hidden state d model is set to 64, the head number of multi-head attention is set to 2.
The segment length L seg is chosen from { 6 , 12 , 24 } via grid search.
We use MSE as loss function and batch size is set to 32.
Adam optimizer is used for training and the initial learning rate is chosen from { 1e-3, 5e-4, 1e-4, 5e-5, 1e-5 } via grid search.
The total number of epochs is 10.
If the validation loss does not decreases within three epochs, the training process will stop early.
For baseline models, if the original papers conduct experiments on the dataset we use, the hyperparameters (except input length T ) recommended in the original papers are used, including the number of layers, dimension of hidden states, etc.
Otherwise, the hyper-parameters are chosen through grid search using the validation set.
Following Zhou et al.
(2021), on datasets ETTh1, WTH, ECL and Trafﬁc, for different prediction length τ , the input length T is chosen from { 24 , 48 , 96 , 168 , 336 , 720 } ; on ETTm1, the input length is chosen from { 24 , 48 , 96 , 192 , 288 , 672 } ; on ILI, the input length is chosen from { 24 , 36 , 48 , 60 } .
All models including Crossformer and baselines are implemented in PyTorch and trained on a single
NVIDIA Quadro RTX 8000 GPU with 48GB memory.
A.3. E FFICIENCY A NALYSIS
To evaluate the computational efﬁciency w.r.t the input length T in Figure 4(c) of the main paper, we align the hyper-parameters of all Transformer-based models as follows: prediction length τ is set to 336, number of encoder layers is set to 2, dimension of hidden state d model is set to 256, the head number of multi-head attention is set to 4.
To evaluate the computational efﬁciency w.r.t the number of dimensions D in Figure 4(d) of the main paper, we align the hyper-parameters of ablation versions of Crossformer as follows as: both input length T and prediction length τ are set to 336, number of encoder layers is set to 3, d model is set to 64, the head number of multi-head attention is set to 2.
Experiments in the computational efﬁciency analysis section are conducted on a single NVIDIA
GeForce RTX 2080Ti GPU with 11GB memory.
A. D ETAILS OF A BLATION V ERSIONS OF C ROSSFORMER
We describe the models we used in ablation study below:
1) DSW represents Crossformer without TSA and HED.
The input is embedded by DSW embedding and ﬂatten into a 1D sequence to be input to the original Transformer.
The only difference between this model and the Transformer is the embedding method.
2) DSW+TSA represents Crossformer without HED.
Compared with Crossformer, the encoder does not use segment merging to capture dependency at different scales.
The decoder takes the ﬁnal output of encoder (i
e. Z enc,N ) as input instead of using encoder’s output at each scale.
3) DSW+HED represents Crossformer without TSA.
In each encoder layer and decoder layer, the 2D vector array is ﬂatten into a 1D sequence to be input to the original self-attention layer for dependency capture.
B E XTRA E XPERIMENTAL R ESULTS B. S HOWCASES OF M AIN R ESULTS
Figure 5 shows the forecasting cases of three dimensions of the ETTm1 dataset with prediction length τ = 288 .
For dimension “HUFL”, all the ﬁve models capture the periodic pattern, but Crossformer is
[Figure]
D
[Figure]
m
[Figure]
C r o ss
[Figure]
o r m e
[Figure]
the closest to the ground truth.
For “HULL”, Pyraformer fails to capture the periodic pattern from the noisy data.
For “LUFL” where the data has no clear periodic pattern, MTGNN, FEDformer and
Crossformer capture its trend and show signiﬁcantly better results than the other two models.
Figure 6 shows the forecasting cases of three dimensions of the WTH dataset with prediction length τ = 336 .
For dimension “DBT”, all the ﬁve models capture the periodic pattern.
For “DPT”,
Autoformer and FEDformer fails to capture increasing trend of the data.
For “WD”, all models capture the periodic pattern from the noisy data, and the cruves output by MTGNN and Crossformer are sharper than the other three models.
B. D EPENDENCY V ISUALIZATION
As the attention scores computed by Crossformer are abstract and hard to visualize, we visualize scores computed by the ablation version, DSW, in Figure 7.
In addition to cross-time dependency that other Transformer models can compute, Crossformer also provides information about crossdimension dependency.
As shown in Figure 7, when predicting Dim #1, the model focus on both
Dim #1 and #3.
When predicting Dim #5, instead of focus on Dim #5 itself, more attention is paid to
Dim #4.
B.
[Figure]
Figure 8 shows the hierarchical prediction patterns output by our HED.
The top prediction layer,
Layer 3, captures the low frequency general trend and periodic pattern of the future value.
By adding [Figure] predictions at ﬁner scales, ﬁner high frequency patterns are added and the prediction get closer to the ground truth curve. B. R UNNING T IME E FFICIENCY A NALYSIS
In the main paper, we show the memory occupation w.r.t input length T and number of dimensions
D .
Here we evaluate the running time.
Figure 9 (a) shows the running time per batch of Crossformer and other Transformer-based models w.r.t input length T .
FEDformer is much slower than other
Transformer-based models.
Crossformer achieves the best computation speed among the ﬁve methods within the tested length range.
Figure 9 (b) shows the running time per batch of Crossformer and its ablation versions w.r.t the number of dimensions D .
Crossformers without TSA layer (DSW and DSW+HED) are faster when
D is small ( D ≤ 30 ).
However, they have difﬁculty processing high-dimensional MTS due to the quadratic complexity w.r.t D .
Indeed, for a single NVIDIA GeForce RTX 2080Ti GPU with 11GB memory, DSW and DSW+HED encounters the out-of-memory (OOM) problem when D > 50 .
Moreover, TSA(w/o Router) encounter the OOM problem when D > 200 .
C D ISCUSSION ON THE S ELECTION OF H YPER - PARAMETERS
We recommend to ﬁrst determine the segment length L seg , as it is related to both the model performance and computation efﬁciency.
The general idea is to use small L seg for short-term prediction and large L seg for long-term prediction.
Some priors about the data also help to select L seg .
For example,
Q u
[Figure]
y D
i
[Figure] [Figure]
108 114 120 126 132 138 144 150 156 162 168 Key Time Step
K e y D i m e n s i o n 0 6 12 18 24 30 36 42
168 Key Time Step
K e y D i m e n s i o n
Q
[Figure]
108 114 120 126 132 138 144 150 156 162 168 Key Time
[Figure]
K e y D i m e n s i o n 0 6 12 18 24 30 36 42
168 Key Time Step
K e y D i m e n s i o n
[Figure]
[Figure] [Figure]
[Figure] [Figure]
168 Key
[Figure]
Step
[Figure]
K e y D i m e n s i o n
[Figure]
mechanism, while the y axis represents dimensions.
Brighter colors represent higher weights.
[Figure]
if the hourly sampled data has a daily period, it is better to set L seg = 24 .
Next, we select the number of layers for encoder and decoder N .
Crossformer with larger N can utilize information of more scales, but also requires more computing resources.
The number of routers in TSA layer c can be set to 5 or 10 to balance the prediction accuracy and computation efﬁciency.
Finally, dimension of hidden states d model and head number of multi-head attention can be determined based on the available computing resources.
[Figure]
[Table]
D S UPPLEMENTARY D ESIGN TO C ROSSFORMER D. H ANDLING I NDIVISIBLE L ENGTH
In the main paper we assume that the input length T and prediction length τ are divisible by segment length L seg .
In this section, we use padding mechanism to handle cases where the assumption is not satisﬁed.
If T is not divisible by L seg , we have ( k 1 − 1)
L seg < T < k 1 L seg for some k 1 .
We pad k 1 L seg − T duplicated x 1 in front of x 1: T to get x (cid:48) 1: T :
[Equation]
where [ , ] denotes the concatenation operation.
x (cid:48) 1:
T ∈ R k 1 L seg × D can be input to the encoder of
Crossformer.
If τ is not divisible by L seg , we have ( k 2 − 1)
L seg < τ < k 2 L seg for some k 2 .
We set the learnable position embedding for decoder as E ( dec ) ∈ R k 2 × D × d model and input it to the decoder to get an output in shape of R k 2 L seg × D .
Then the ﬁrst τ steps of the output is used as x predT +1: T
+ τ .
We conduct experiment on ETTm1 dataset to evaluate the effect of indivisible length.
Results in
Table 4 show that with padding mechanism, indivisible length does not degrade model performance, for both short-term prediction and long-term prediction.
D. I NCORPORATING C OVARIATES
In the main text, we only use historical series x 1: T to forecast the future x T +1: T
+ τ .
In this section, we try to incorporate covariates c 1: T + τ into Crossformer.
We use a straightforward method: ﬁrst embed the covariates into point-wise vectors { d 1 , d 2 , . . .
, d T + τ } like previous Transformer-based models do (Zhou et al., 2021; Wu et al., 2021; Liu et al., 2021a).
Then, merge the point-wise vectors into segment-wise vectors using learnable linear combination.
Finally, add the segment-wise vectors to each dimension of the 2D vector array obtained by DSW embedding:
[Equation]
where → denotes embedding method for point-wise covariates.
α j , 1 ≤ j ≤ L seg denotes learnable factors for linear combination.
d ( s ) i denotes the segment-wise covariate embedding.
h covi,d denotes the embedded vector with covariate information for the i -th segment in dimension d , where h i,d is the embedded vector obtained from DSW embedding in the main text.
The processing for the input of the decoder is similar, the segment-wise covariate embedding is added to the position embedding for decoder, i.e. E ( dec ) .
[Table]
We conduct experiments on ETTh1 dataset to evaluate the effect of covariates.
Hour-of-the-day, dayof-the-week, day-of-the-month and day-of-the-year are used as covariates.
Results in Table 5 show that incorporating covariates does not improve the performance of Crossformer.
The possible reason is this straightforward embedding method does not cooperate well with Crossformer.
Incorporating covariates into Crossformer to further improve prediction accuracy is still an open problem.