AutoST: Towards the Universal Modeling of Spatio-temporal Sequences
Abstract
The analysis of spatio-temporal sequences plays an important role in many realworld applications, demanding a high model capacity to capture the interdependence among spatial and temporal dimensions.
Previous studies provided separated network design in three categories: spatial ﬁrst, temporal ﬁrst, and spatio-temporal synchronous.
However, the manually-designed heterogeneous models can hardly meet the spatio-temporal dependency capturing priority for various tasks.
To address this, we proposed a universal modeling framework with three distinctive characteristics:
(i) Attention-based network backbone, including S2T Layer (spatial ﬁrst),
T2S Layer (temporal ﬁrst), and STS Layer (spatio-temporal synchronous).
(ii)
The universal modeling framework, named UniST, with a uniﬁed architecture that enables ﬂexible modeling priorities with the proposed three different modules.
(iii)
An automatic search strategy, named AutoST, automatically searches the optimal spatio-temporal modeling priority by network architecture search.
Extensive experiments on ﬁve real-world datasets demonstrate that UniST with any single type of our three proposed modules can achieve state-of-the-art performance.
Furthermore, AutoST can achieve overwhelming performance with UniST.
Introduction
Modeling and predicting the future of spatio-temporal (ST) sequences based on past observations has been extensively studied and has been successfully applied in many ﬁelds, such as road trafﬁc [13], medical diagnosis [28] and meteorological research [24].
Traditional statistical methods typically require input sequence satisfying certain assumptions, which limits its ability in capturing the complex spatial-temporal dependency.
Then, recurrent neural network (RNN) methods [15] leverage the universal approximation property to build separated network branches to model dependency and make predictions with fusion gate blocks from the stacking branches.
The intrinsic gradient ﬂow in the back-propagation training process [7] may bring the ST dependency into incorrespondence with the
S p a ti a l
Temporal Spatial-first task task2 task
Temporal Temporal-first task task2 task
Temporal Spatio-temporal sync.
task task3 task
Temporal
Ours task task task3 S p a ti a l
Temporal
Ideal task task task
[Figure]
[Figure]
network branches’ conﬁguration, especially for the deeper network [11].
Recently, the
Transformer-based models show larger modeling capacity in both spatial and temporal modeling [21, 31, 30], which motivates us to ﬁnd a universal way to capture the ST dependency in a universal framework simultaneously.
As shown in Fig.1, the ST dependency individually exists in sequences: spatial correspondences, temporal correspondences, and spatiotemporal correspondences.
Take the road trafﬁc
forecasting as an example, previous research fall into three typical paradigms next road junction has strong tions to previous intersections
However, the distribution of ST dependencies varies and depends on the forecasting task and corresponding datasets.
They are mixed in a compound way when modeling ST sequences, and the three tasks in Fig.1(b) are the representative ones.
What makes it worse is that, the prevalent modeling methods show anisotropic tendency to capture the ST dependency.
If we use the spatial-ﬁrst models on the three tasks in Fig.2(b), the task 1’s states are highly inﬂuenced by the surrounding information, and the periodic pattern is the underlying factors, which makes the spatial-ﬁrst model ﬁts it properly.
We can compare the model ability (red lines) with the ideal one in Fig.2(a), this kind of model will be insufﬁcient for task 2 and task 3.
Similarly, suppose we use the temporal-ﬁrst models on the three tasks in Fig.2(c).
In that case, the model ability only matches task 3, where the periodic pattern decides the states other than the spatial information.
The previous analysis also applies to the spatio-temporal synchronous situation, where the states are mainly inﬂuenced by the complex associations across the spatial and temporal, like semantic relationships.
In this paper, we aim to propose a universal model that alleviates the the modeling gap on different tasks.
The contributions are: 1) The ﬁrst to raise and address the modeling order proplem in spatio-temperal forecasting tasks by proposing a universal modeling framework UniST and an automatic structure search strategy AutoST. 2) Proposing 3 replacable and uniﬁed attention-based modeling units named
S2T, T2S and STS, which model spatio-time sequence with three different priorities: spatial ﬁrst, temperal ﬁrst and spatio-temperal synchronous.
3) Extensive experiments on 5 datasets andsequence forcasting tasks demonstrate that only using our three modeling units (S2T, T2S, and
STS) outperforms the baseline methods, and our framework together with AutoST achieves the new state-of-the-art performance.
Related Work
Existing spatio-temporal forecasting methods can be roughly grouped into three categories: spatialﬁrst, temporal-ﬁrst, and spatio-temporal synchronous methods.
Spatial-ﬁrst : STG2Seq [1] uses stacking GCN layers to capture the entire inputs sequence, where each GCN layer operates on a limited historical time window, and the ﬁnal results are concatenated together to make forecasting.
In the view of this paper, it belongs to spatial-ﬁrst modeling.
STGCN [26] propose the blocks that contains two temporal gated convolution layers with one spatial graph convolution layer in the middle, which starts from a convolution-based temporal layer.
DCRNN [13] is proposed to forecast trafﬁc ﬂow using diffusion convolution and recurrent units to capture spatial and temporal information successively.
Temporal-ﬁrst : Graph WaveNet [23] built the basic modeling layers with two gated temporal convolution modules at the beginning and followed by a graph convolution module, which models from temporal to spatial.
GSTNet [5] builds several layers of spatial-temporal blocks to produce the forecasting, which is consists of a multi-resolution temporal module followed by a global correlated spatial module.
Spatio-temporal synchronous : STSGCN [19] construct a spatiotemporal synchronous extraction module composed of graph convolutional networks.
STFGNN [12] modeling spatio-temporal correlations simultaneously by fusing a dilated convolutional neural network with a gating mechanism and a spatio-temporal fusion graph module.
ST-ResNet [27] using convolution on a sequence of image-like 2D matrices to model spatio-temporal at the same time.
ASTGCN [6] proposed a spatial-temporal convolution that simultaneously captures the spatial patterns and temporal features.
Preliminary 3.1 Spatio-temporal Sequence Forecasting
Spatio-temporal sequence forecasting (STSF) is to predict the future sequence of spatio-temporal inputs based on the historical observations.
Speciﬁcally, given a graph G =
( V, E, A ) , where V and
E are the node set and edge set, and N is the number of nodes, A 2 R N ⇥ N is the adjacency matrix of G .
If v i , v j 2 V and ( v i , v j ) 2 E , A ij = 1 , otherwise A ij = 0 .
X
=
{ X 1 , X 2 , . . .
, X T } is a ST sequence of T time steps, where X 2 R T ⇥ N ⇥ C .
The snapshot at time step t is denoted as x t 2 R N ⇥ C , where C is the feature dimension of a node.
Then the ST sequence forecasting problem can be deﬁned as: given S time steps historical observations of input graph G , the goal is to predict the future sequence of the features on each node with a learning function f : ⇥ [Equation] t , G ] f ! X
[Equation]
t + [Equation] ):( t + P ) , where X ( t  S ): t and X ( t +1):( t + P ) are the ST sequence with length S and P respectively. 3.2 Network Architecture Search
Network (neural) architecture search (NAS) are automated methods for generating and optimizing neural networks.
A representative gradient-based approach is DARTS [16], which is the foundation of our proposed training framework.
DARTS aims to search optimal directed edge connections on a directed acyclic graph with predeﬁned computing cells as nodes.
The result connections of node j is denoted as x ( j ) =
P i<
j o ( i,j )  x ( i )  , where o ( i,j ) is an operator, e.g. layers in a model, represented by a directed edge from node i to node j. DARTS proposes a method to relax the discrete searching space to be continuous, and uses bi-level optimization to learn a differentiable objective on the joint optimization problem of both network architecture and model weights.
The objective function is: min ↵ L val ( w ⇤ ( ↵ ) , ↵ ) s.t.
w ⇤ ( ↵ ) =
argmin w L train ( w, ↵ ) , where ↵ is the architecture, and w is the model weights.
In Section 4.4, we improve the design of the directed acyclic graph of the search architecture, and the two-stage optimization of the architecture parameters.
Methods
In this section, we ﬁrstly introduce two basic modeling units: the time series linear self-attention, and the high order mix graph convolution.
Then we proposed three layers as different network backbones, and we build a universal modeling framework based on the tree “atomic” layers.
Next, we propose an automatic searching strategy for spatio-temporal information fusion, which aimed for the optimal order of spatio-temporal modeling on various downstream tasks.
A
Q
K
V
HighOrderMixGC
X LX L
HighOrderMixGC
LinearSelf-Attention
Add&Norm
FeedForward
Add&Norm
A
Q
K
V
X LX L
HighOrderMixGC
LinearSelf-Attention
Add&Norm
FeedForward
Add&Norm
A
Q K
V
X LHighOrderMixGC Linear Self-Attention
Add&Norm
FeedForward
Add&Norm
X L
S2T Layer
T2S Layer
STS Layer
4.1 Spatial / Temporal Modeling Unit 4.1.1 Time Series Linear Self-Attention
Self-attention mechanism [21] has been widely used in nature language processing, computer vision, and time series forecasting, which is deﬁned as: Attention( Q , K , V ) =
V 0 =
Softmax( QK > / p d ) V , where Q = XW Q , K = XW K , V = XW V , and the projection matrix [Equation] [Equation]
However, the original self-attention suffers from high computational and memory cost.
Because the dot product computation of Q and K leads to O ( N 2 ) time and space complexity.
[9] proposed linear self-attention, which represents the similarity function of Q and K in the self-attention by a kernel function: V {+
+}
i =
( Q i ) T
P
Nj =1  ( K j ) V Tj /  ( Q i ) T
P
Nj =1  ( K j ) . Such that for each query
Q i , the two terms P [Equation] Nj =1  ( K j ) are the same and reused for efﬁcient computing.
Following [30], we use the technique of linear self-attention in representing time series features.
4.1.2 High-order Mix Graph Convolution
To acquire better spatial information representation, we propose a high-order mix graph convolutional operation for spatial information mixing and feature extraction of the original inputs, it is deﬁned as: [Equation] order
[Equation]
where order denotes the total order of the graph convolution operations, i.e., to consider orderhop neighbor relationship of each node.
In this paper, we deﬁne the 1 st -order mix convolutional operation by combining the 1 st -order ChebNet [10] and the Adaptive Diffusion Convolution [23]:
[Equation]
ˆ A adp
[Equation]
adp , where ˆ A = D  1 / 2 ˜ AD  1 / 2 is a normalized adjacency matrix with self-loop.
ChebNet focuses on 1st-order neighbor information, while AdapDC focuses on multi-hop information.
˜
A is deﬁned as ˜ A
=
A + I , where D ii =
P j ˜
A ij , I is an identity matrix.
P f =
A rowsum( A ) ,
P b =
A > rowsum( A > ) refers to a forward and backward state transition matrix, respectively.
ˆ A adp is an adaptive matrix for complementary spatial state information, which is calculated by two learnable node embedding matrices E 1 , E 2 2 R N ⇥ C [20] as ˆ A adp = Softmax(ReLU( E 1 E > 2 )) .
4.2 Uniﬁed Spatio-temporal Modeling Backbone
In order to solve the problem of spatio-temporal dependency distribution differences in the modeling procedure, we ﬁrst propose three novel modules: S2T Layer , T2S Layer , STS Layer , that are suitable for three typical spatio-temporal dependencies: spatial-ﬁrst, temporal-ﬁrst, spatio-temporal synchronous, respectively.
We design all these three modeling module to have the same dimension of inputs and outputs.
This provides a solid foundation for our later ﬂexible and universal modeling.
4.2.1 Spatial-ﬁrst Modeling Layer
The spatial-ﬁrst sequence modeling method, S2T Layer , models from spatial to temporal.
The spatial information between the nodes on the graph is ﬁrst characterized on a single slice.
After that, node information at different times is exchanged along the time dimension, whose spatial information has been shared with its neighbors.
As shown in Fig.3(a), S2T Layer ﬁrst uses two high-order mix graph convolution deﬁned in Eq.(1) to process the input spatio-temporal sequences X L  1 to obtain two sequence representations with mixed spatial information.
Then the key K and value V of the input of the subsequent self-attention are obtained by a transformation using the parameter matrix W K , W V , respectively, while the query Q is obtained by transforming the original input ST sequence using the parameter matrix as follows:
Q =
X L  1 W Q , K = HighOrder 1 ( X L  1 , A , order ) W K , V =
HighOrder 2 ( X L  1 , A , order ) W V .
Then the original ST sequence and the new sequence with mixed spatial information are processed using a multi-head linear self-attention, from which it learns temporal dependencies and exchanges information at different time slices to obtain further representations of the ST sequence: Z =
Attention( Q , K , V ) .
[Equation]
The output is concatenated with the initial input once for residuals and processed a layer normalization, followed by a two-layer fully connected network for further ST representation learning.
This network is applied separately and identically to each point-in-time position in the ST sequence, thus maintaining the continuous transfer of position-encoded information.
Finally, the resulting
ST sequence representation is again connected to the initial input with one residual and layer normalization to obtain the output X L = Norm(max(0 , Norm( Z + X L  1 ) W 1 + b 1 ) W 2 + b 2 ) .
4.2.2 Temporal-ﬁrst Modeling Layer
This module is designed to model the ST sequence from temporal to spatial, named T2S Layer .
Different from spatial-ﬁrst modeling, at the beginning, the original inputs are projected into Q , K , V by three weight matrices as Q =
X L  1 W Q , K =
X L  1 W K , and V =
X L  1 W V .
The projection results are used to calculate temporal representations at ﬁrst, using the time series linear self-attention in Eq.(2).
Then the temporal representation on each node are send to the high-order mix graph convolution, together with the adjacency matrix, to fusion the temporal information from every neighbors.
Z {+
+} =
HighOrder( Z , A , order ) .
Finally, the output representations are executed with the feed forward and layer normalization operations the same way as the S2T Layer .
4.2.3 Spatial-temporal Synchronous Layer
This module named STS Layer , which aims to model the spatial and temporal information simultaneously.
Different from the former two modules, the inputs are directly used to calculate spatial and temporal representations at the same time.
For temporal modeling part, it still project the original inputs into Q , K , V , and execute a linear self-attention operation for a temporal representation.
For the spatial part, it accepts the original inputs and the spatial information and uses high-order mix graph convolution operation to construct spatial representation.
Temporal Z 1 = Attention( Q , K , V ) , Spatial
Z 2 =
HighOrder( X L  1 , A , order ) .
The outputs are concatenated together as: Z 0 = concat[ Z 1 , Z 2 ] .
Then it is executed with the following operations and output as X L similar with the former two modules.
4.3 Universal Modeling Framework
EmbeddingLayer EmbeddingLayer
TokenEncoding
TemporalEncoding
PositionalEncoding
SpatialEncoding
STExtractor STExtractor
Add
FeedForward
ReLU
FeedForward
ReLU
Outputs
S2TLayer
T2SLayer
STSLayer
TokenEmbedding
TemporalEmbedding
PositionalEmbedding
SpatialEmbedding
D ec o d er
E n c o d er
Inputs
Targeting to the ST sequence forecasting task we propose a uniﬁed ST sequence modeling framework (UniST) with the proposed uniﬁed modeling backbones in Fig.4, which follows the encoderdecoder architecture.
It uses a uniﬁed architecture with interchangeable and replaceable mode units.
4.3.1 Spatio-temporal Embedding Layer
Since the Transformer model solely relies on the self-attention for global alignments, the positional embedding [21] and extra embeddings [31] are needed to capture spatio-temporal dependency.
Then we introduce four types of embeddings E P ,
E V , E S , E T in Appendix A.
Fusion embedding .
The four embeddings are summed together as the ﬁnal embedding added to the inputs: E F = E P + E V + E S + E T , note that the shape of token embedding E V 2 R T ⇥ N ⇥ d , while other embeddings’ shape are E P 2 R 1 ⇥ 1 ⇥ d ,
E S 2 R 1 ⇥ N ⇥ d , E T 2 R T ⇥ 1 ⇥ d .
When calculating the summation, they will be replicated and expanded with broadcast on the respective missing dimensions.
4.3.2 Encoder
The encoder of UniST consists of multiple Spatio-Temporal Extractors ( STE( · ) ), which can be arbitrarily chosen from { T2S Layer , S2T Layer , STS Layer }.
All extractors are connected end to end, i.e., the output of the previous one is the input of the next one.
To acquire a more diversity representation, the outputs of each extractor are added to form the ﬁnal output of the encoder.
Let the outputs of the embedding layer be X 0 , the encoder is computed as: X en =
[-=
-]P
Li =1 STE i ( X 0 ) , where
L refers to the number of spatio-temporal extractors.
4.3.3 Decoder
The decoder accepts the output of encoder, i.e., L outputs from L spatio-temporal extractors.
They are ﬁrstly added as a uniﬁed spatio-temporal representation.
Then the results are through two times of ReLU activation and Linear projection, and produce the ﬁnal sequence forecasting result.
Denote X ` as the output of extractor ` , we have the calculation of decoder as: ¯ Y = X de =
Linear(ReLU(Linear(ReLU( P ` X ` )))) .
4.4 Automated Search for UniST
With the proposed uniﬁed ST sequence modeling framework UniST, it still suffers from the potential wrong network conﬁguration problem, where we build an arbitrary modeling order with the replaceable model units { T2S Layer , S2T Layer , STS Layer }.
Considering the various downstream tasks, how can we build a universal model with an optimal conﬁguration?
Here we propose the
Automated Spatio-Temporal modeling approach (AutoST), which learns the optimal combinatorial order that suits the spatio-temporal dependency of the current task.
We designed two schemes for layer combination.
In this section, we ﬁrst deﬁne the basic searching unit of AutoST, then we introduce two designs of AutoST with different searching schemes.
4.4.1 AutoST Cell
The basic searching unit in AutoST is the network cell.
Here we deﬁne its structure and computing process.
Deﬁnition 1.
AutoST Cell.
Let G =
( V , E ) be a direct acyclic graph (DAG), V denotes the node set, each node refers to a representation comes from the outputs of a computation layer.
The representation on node i is deﬁned as H i 2 R T ⇥ |V| ⇥ d , where T is the length of spatio-temporal seqence, d is the
AutoSTCell
E n c o d e r
H
S2TLayer
T2SLayer
STSLayer H
H out
Add
H i
H
H out
S2TLayer T2SLayer STSLayer
H i
H j
H out
AutoSTCell
E n c o d e r
H
H
H out
H H
H
S2TLayer
T2SLayer STSLayer
Identity Zero
STSLayer (
AutoSTCell
E n c o d er
H
S2TLayer
T2SLayer
STSLayer H
H out
Add
H i
S2TLayer T2SLayer STSLayer
H i
H
AutoSTCell
E n c o d er
H
H
H out
H H
H
S2TLayer
T2SLayer STSLayer
Identity Zero
STSLayer (b) AutoST 2 : the hybrid assembling scheme.
feature dimension.
The input of each AutoST Cell is denoted as H 0 , and the output of each cell is the summation of all nodes, i.e., all interval representations: H out = P |V| i H i .
On graph G , the directed edge ( i, j ) from node i to node j stands for a mixture of all candidate modeling modules
O = {T2S Layer, S2T Layer, STS Layer}, and it is represented as o ( i,j ) .
So that the representation between node j and other nodes can be written as: H j = P i<
j o ( i,j ) H i .
On each directed edge, there exist a set of weight parameters ↵ ( i,j ) =
{ ↵ ( i,j ) o | o 2 O } , which indicates the probability of the corresponding modeling module should be retained.
Every weight parameters of the candidate modeling module is calculated as: H j =
P i< j
P o
[Equation]
O
[Equation]
o )
P o 02 O exp
[Equation]
,j )
o
[Equation]
o  H i  .
4.4.2 Sequential Stacking Search
AutoSTCell
E n c o d er
H
S2TLayer
T2SLayer
STSLayer H
H out
Add
H i
H j
H out
S2TLayer T2SLayer STSLayer
H i
H j
H out
AutoSTCell E n c o d er
H
H
H out
H H
H
S2TLayer
T2SLayer
STSLayer
Identity
Zero
STSLayer
The searching goal is to choose the optimal one from the candidate modules between the given computation node H i and H j .
Based on the proposed UniTS, we propose two searching schemes to ﬁnd better combination of the modeling layers.
The proposed searching schemes are lossless replacements for the encoder of UniTS.
We simply replace the encoder’ spatiotemporal extractor with the AutoST Cell.
The ﬁrst one is multi-layer sequential stacking searching scheme, with which the whole model is named AutoST 1 .
As illustrated in Fig.5(a), it has simple structure within each AutoST Cell, while has more complicated stacking structure between cells.
Each cell holds a DAG with three nodes, one input node H 0 , one output node H out and an intermediate node H 1 .
And two directed edges are pre-deﬁned between the former two nodes’ output and the output node: ( H 0 , H out ) and ( H 1 , H out ) .
The searching space of this scheme is shown as the directed red dotted line.
The red dashed box is the search candidate set, including { T2S Layer , S2T Layer , STS Layer }.
We conduct two gradient based network architecture search methods in the experiments, i.e., DARTS [16], PAS [22].
After searching, the cell essentially becomes one of the three model units.
This scheme allows multiple stacking of cells, therefore, the new encoder of the whole model will become the sequential stacking between different modeling layers.
4.4.3 Hybrid Assembling Search
The another searching scheme is called hybrid assembling searching.
The structure is similar with the sequential stacking searching.
However, in this scheme, the encoder only consists of a single AutoST
Cell.
The cells are not stacked layer-by-layer, it will conduct searching on a more complicated DAG on a single AutoST Cell.
In this searching scheme, the DAG of AutoST Cell is shown as Fig.5(b).
There are multiple nodes in the cell, generally, it will be set as 4-7 nodes in the experiments.
The pre-deﬁned connections are
[Table]
each node’s output to the output node.
The multiple directed red dotted lines show the searching space of this scheme.
The candidate set is expanded with two operations: Identity and Zero .
Identity means no modeling module in this edge, i.e., build directly connection with no operation.
Zero means to set all the data passing through it to zero, i.e., no connection is made.
This scheme makes the entire
DAG form a complex and deep network structure.
Through the complex structure design inside the cell, multiple spatio-temporal modeling modules are combined to ﬁt the spatio-temporal dependency distribution in a target task.
Experiments
This section empirically evaluates the effectiveness of UniST and AutoST models with shortterm, medium-term, and long-term ST sequence forecasting tasks on ﬁve real-world datasets.
The experimental settings are described in Appendix Section
B. The code is available at https: //github.com/shuaibuaa/autost2022 .
5.1 Main Results
Table 1 summarizes the ST sequence forecasting results.
UniST S , UniST T , UniST ST stands for our
UniST framework with three same stacking layers of S2T Layer , T2S Layer , STS Layer , respectively.
The results in bold font in Table 1 show that our proposed UniST outperforms all baseline methods and achieves State-of-the-Art on all 9 tasks of 5 datasets, with all three proposed layers.
This demonstrates our proposed uniﬁed spatio-temporal modeling layers and the uniﬁed forecasting framework are more expressive than traditional methods.
Speciﬁcally, compared with GMAN, which is also based on self-attention mechanism, our methods achieve at most 18.41%, 15.31%, 13.82% MAE decreases on the short-term, medium-term, and long-term forecasting task, respectively.
Compared with the most advanced method STFGNN, our methods gain 12.06%, 10.61%, 9.43% MAE decreases on the short, medium, long-term forecasting tasks.
From the last two columns of Table 1, we can see that both of AutoST 1 and AutoST 2 beat all other methods on every metric of every task, including our proposed UniST.
Recall that the difference between AutoST and UniST is the encoder part, UniST uses all same layers from { S2T Layer , T2S
Layer , STS Layer }, while AutoST aims to search for a better combination and connection using the three types of layers.
This demonstrates that combining and integrating modules with different spatio-temporal modeling abilities can better deal with uncertainty spatio-temporal dependency and more comprehensively model spatio-temporal sequences.
At the same time, AutoST 2 beats AutoSTon every task.
It shows that using a single-layer but complex internal connection searching method for these tasks is more effective.
Moreover, we can see that the best methods among all baselines are STFGNN, the only method in spatio-temporal synchronous modeling.
Comparing GMAN and Graph WaveNet, the two representative methods in spatial-ﬁrst modeling and temporal-ﬁrst modeling, respectively, we can see that although they beat other baselines, while our proposed Uni- series model shows a better performance.
From this perspective, the proposed method matches the design.
5.2 Ablation Study
In AutoST, we design to use gradient-based network architecture search methods to optimize the connections of the modeling modules, and we choose DARTS and PAS as the NAS methods.
At the same time, we also conduct experiments to compare the method of random search and AutoSTG [17], which also uses the network architecture search technology for ST sequence prediction.
The results are shown in Table 2.
We can conclude that our proposed AutoST can outperform AutoSTG with all searching methods.
That is because AutoSTG uses more ﬁne-grained temporal convolution and graph convolution structure as the candidates, while AutoST’s candidate layers are designed to model the three different spatio-temporal dependencies, which can make better use of the modeling ability of the three layers on different temporal and spatial relations.
It reduces the searching space and the search time simultaneously so that AutoST can make better efﬁciency and accuracy.
[Table]
AutoST 1 -R 6.
3.65 AutoST 2 -R 6.
AutoST 1 -D 6.
3.72 AutoST 2 -D 6.
AutoST 1 -P 6.
3.52 AutoST 2 -P 6. (a) The search results of AutoST 1 . AutoSTCell E n c o d er
H
S2TLayer
T2SLayer STSLayer H H out Add
AutoSTCell
E n c o d er
H
H
H out
H
H H S2TLayer T2SLayer STSLayer Identity
Zero
S2TLayer
H H
H out
T2SLayer H
H
H out
S2TLayer H
H
H out
STSLayer H
H
H out
STSLayer
H H
H out
H
T2SLayer STSLayer STSLayer
S2TLayer T2SLayer
T2SLayer
H
AutoSTCell
E n c o d er
H
S2TLayer
T2SLayer
STSLayer H
H out
Add
AutoSTCell
E n c o d er
H
H
H out
H H
H
S2TLayer T2SLayer
STSLayer Identity
Zero
S2TLayer
H H
H out
T2SLayer H
H
H out
S2TLayer H
H
H out
STSLayer H
H
H out
STSLayer
H H
H out
H
T2SLayer STSLayer STSLayer
S2TLayer T2SLayer
T2SLayer
H
5.3 Result Visualization
Forecasting Visualization: We randomly selected one day from the PEMS-BAY dataset and compared our methods’ forecasting results and baseline methods in a visualization way.
Typical results are shown in Fig.8, the time span is selected from 13:00 to 19:00, which can represent the most typical scene of the rush hour from afternoon to evening.
From the forecasting lines compared to the ground truth, we can see that our proposed methods outperform the two typical baseline methods mostly when forecasting smooth trafﬁc and trafﬁc jam.
The bar lies on the bottom of the ﬁgures is the forecasting error of each method.
We can ﬁnd that all methods encountered accuracy drop around 14:00 and 17:30.
That is because the status of the road is changed dramatically.
However, our methods are more stable with the changes and can quickly be adapted to the new road state.
For the complete data visualization of the day, please refer to Fig.9 in the appendix.
Learned Architecture Visualization: The learned architectures of the AutoST 1 and AutoST 2 on
PEMS-08 dataset are shown in Fig.7.
We can ﬁnd that although UniST T has a better effect when modeling with single type modeling module, the search result of AutoST 1 show that the STS Layer occupies a larger number, and the model achieves better performance than UniST T .
This demonstrates that combining and stacking multiple spatio-temporal dependency modeling methods reasonably can better ﬁt the real spatio-temporal dependencies.
In addition, we can ﬁnd that AutoST 2 obtains a complex connections between the four computation nodes in a single cell.
And this learned architecture helps AutoST 2 achieve state-of-the-art on this task.
Although we cannot yet explain why stacking the modules leads to better results, we can see a potentially broad range of applications for uniﬁed architecture searching in this way.
Conclusion
In this work, we illustrated the existence of the modeling gap problem, especially the modeling order, in the spatio-temporal analysis.
Moreover, we build three different layers, namely S2T, T2S, and STS, as new network modeling backbones.
Then, an automatic searching strategy is proposed to search the optimal modeling priority automatically.
Extensive experiments on ﬁve real-world datasets show the overwhelming performance over SOTA baselines.
Acknowledgments
This work was supported by grants from the Natural Science Foundation of China (U20B2053, 62202029) and Foshan HKUST Projects (FSUST21-FYTRI01A, FSUST21-FYTRI02A).
Thanks for computing infrastructure provided by Beijing Advanced Innovation Center for Big Data and Brain Computing.
This work was also sponsored by CAAI-Huawei MindSpore Open Fund.
We gratefully acknowledge the support of MindSpore, CANN (Compute Architecture for Neural Networks) and
Ascend AI Processor used for this research.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] Due to the space limits, the limitations are in the supplemental material.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The code, data and instructions to reproduce the experimental results are provided in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] These details are in the Appendix Section B. (c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] The details are in Section B.1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] Our code are included in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] We only used public datasets in this paper.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A]
To the best of our knowledge, the trafﬁc datasets used in our paper do not have any personally identiﬁable information or offensive content
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
No human subjects are involved.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A]
No human subjects are involved.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
No human subjects are involved.