AutoST: Towards the Universal Modeling of Spatio-temporal Sequences
Anonymous Author(s)
AfﬁliationAddress email
Abstract
The analysis of spatio-temporal sequences plays an important role in many realworld applications, demanding a high model capacity to capture the interdependence among spatial and temporal dimensions.
Previous studies provided separatednetwork design in three categories: spatial ﬁrst, temporal ﬁrst, and spatio-temporalsynchronous.
However, the manually-designed heterogeneous models can hardlymeet the spatio-temporal dependency capturing priority for various tasks.
Toaddress this, we proposed a universal modeling framework with three distinctivecharacteristics: (i) Attention-based network backbone, including S2T Layer (spatial ﬁrst), T2S Layer (temporal ﬁrst), and STS Layer (spatio-temporal synchronous). (ii)
The universal modeling framework, named UniST, with a uniﬁed architecturethat enables ﬂexible modeling priorities with the proposed three different modules.
(iii)
An automatic search strategy, named AutoST, automatically searches the optimal spatio-temporal modeling priority by network architecture search.
Ex- tensive experiments on ﬁve real-world datasets demonstrate that UniST with anysingle type of our three proposed modules can achieve state-of-the-art performance.
Furthermore, AutoST can achieve overwhelming performance with UniST.
Introduction
S p a ti a l
Temporal task task task
Time [Figure]
Modeling and predicting the future of spatiotemporal (ST) sequences based on past observations has been extensively studied and has beensuccessfully applied in many ﬁelds, such as roadtrafﬁc [13], medical diagnosis [28], and meteorological research [24].
Traditional statisticalmethods typically require input sequence satisfying certain assumptions, which limits its ability in capturing the complex spatial-temporaldependency.
Then, recurrent neural network(RNN) methods [15] leverage the universal approximation property to build separated networkbranches to model dependency and make predictions with fusion gate blocks from the stacking branches.
The intrinsic gradient ﬂow in theback-propagation training process [7] may bring the ST dependency into incorrespondence with thenetwork branches’ conﬁguration, especially for the deeper network [11].
Recently, the Transformerbased models show larger modeling capacity in both spatial and temporal modeling [21, 31, 30],which motivates us to ﬁnd a universal way to capture the ST dependency in a universal frameworksimultaneously.
S p a ti a l
Temporal Spatial-first task task2 task
Temporal Temporal-first task task2 task
Temporal Spatio-temporal sync.
task task3 task
Temporal
Ours task task task3 S p a ti a l
Temporal
Ideal task task task
[Figure]
As shown in Fig.1, the ST dependency individually exists in sequences: spatial correspondences,temporal correspondences, and spatio-temporal correspondences.
Take the road trafﬁc forecasting asan example, previous research fall into three typical paradigms.
(a) Spatial-ﬁrst modeling [1, 26, 13]:
Predicting the trafﬁc for next road junction, which has strong connections to previous intersections,stops, and surrounding trafﬁc.
(b) Temporal-ﬁrst modeling [23, 5]: Predicting the trafﬁc of ahigh school on Friday afternoon, which shows strong periodic relationships with the school’s timesschedule.
(c) Spatio-temporal synchronous modeling [19, 12, 27, 6]:
Analyzing the city-widetrafﬁc, which is tightly entangled in both spatial and temporal.
During the sequences modeling, themain problem is to align the network design with the natural spatio-temporal distribution.
However, the distribution of ST dependencies varies and depends on the forecasting task and corresponding datasets.
They are mixed in a compound way when modeling ST sequences, and the threetasks in Fig.1(b) are the representative ones.
What makes it worse is that, the prevalent modelingmethods show anisotropic tendency to capture the ST dependency.
If we use the spatial-ﬁrst modelson the three tasks in Fig.2(b), the task 1’s states are highly inﬂuenced by the surrounding information,and the periodic pattern is the underlying factors, which makes the spatial-ﬁrst model ﬁts it properly.
We can compare the model ability (red lines) with the ideal one in Fig.2(a), this kind of modelwill be insufﬁcient for task 2 and task 3.
Similarly, suppose we use the temporal-ﬁrst models onthe three tasks in Fig.2(c).
In that case, the model ability only matches task 3, where the periodicpattern decides the states other than the spatial information.
The previous analysis also applies tothe spatio-temporal synchronous situation, where the states are mainly inﬂuenced by the complexassociations across the spatial and temporal, like semantic relationships.
In this paper, we aim topropose a universal model that alleviates the the modeling gap on different tasks.
The contributions are: 1)
The ﬁrst to raise and address the modeling order proplem in spatio-temperalforecasting tasks by proposing a universal modeling framework UniST and an automatic structuresearch strategy AutoST.
2) Proposing 3 replacable and uniﬁed attention-based modeling units named
S2T, T2S and STS, which model spatio-time sequence with three different priorities: spatial ﬁrst,temperal ﬁrst and spatio-temperal synchronous. 3) Extensive experiments on 5 datasets and 3sequence forcasting tasks demonstrate that only using our three modeling units (S2T, T2S, and STS) outperforms the baseline methods, and our framework together with AutoST achieves the newstate-of-the-art performance.  Related Work
Existing spatio-temporal forecasting methods can be roughly grouped into three categories: spatialﬁrst, temporal-ﬁrst, and spatio-temporal synchronous methods. Spatial-ﬁrst :
STG2Seq [1] usesstacking GCN layers to capture the entire inputs sequence, where each GCN layer operates on alimited historical time window, and the ﬁnal results are concatenated together to make forecasting.
In the view of this paper, it belongs to spatial-ﬁrst modeling.
STGCN [26] propose the blocks thatcontains two temporal gated convolution layers with one spatial graph convolution layer in the middle,which starts from a convolution-based temporal layer.
DCRNN [13] is proposed to forecast trafﬁcﬂow using diffusion convolution and recurrent units to capture spatial and temporal informationsuccessively.
Temporal-ﬁrst : Graph WaveNet [23] built the basic modeling layers with two gatedtemporal convolution modules at the beginning and followed by a graph convolution module, whichmodels from temporal to spatial.
GSTNet [5] builds several layers of spatial-temporal blocks toproduce the forecasting, which is consists of a multi-resolution temporal module followed by a
A
Q
K
V
HighOrderMixGC
X LX L
HighOrderMixGC
LinearSelf-Attention
Add&Norm
FeedForward
Add&Norm
A
Q
K
V
X LX L
HighOrderMixGC
LinearSelf-Attention
Add&Norm
FeedForward
Add&Norm
A
Q K
V
X LHighOrderMixGC Linear Self-Attention
Add&Norm
FeedForward
Add&Norm
X L
S2T Layer
T2S Layer
STS Layer global correlated spatial module. Spatio-temporal synchronous : STSGCN [19] construct a spatiotemporal synchronous extraction module composed of graph convolutional networks. STFGNN[12] modeling spatio-temporal correlations simultaneously by fusing a dilated convolutional neuralnetwork with a gating mechanism and a spatio-temporal fusion graph module. ST-ResNet [27]using convolution on a sequence of image-like 2D matrices to model spatio-temporal at the sametime.
ASTGCN [6] proposed a spatial-temporal convolution that simultaneously captures the spatialpatterns and temporal features.
Preliminary3.1 Spatio-temporal Sequence Forecasting
Spatio-temporal sequence forecasting (STSF) is to predict the future sequence of spatio-temporalinputs based on the historical observations.
Speciﬁcally, given a graph G =
( V, E, A ) , where V and
E are the node set and edge set, and N is the number of nodes, A 2 R N ⇥ N is the adjacency matrixof G .
If v i , v j 2 V and ( v i , v j ) 2 E , A ij = 1 , otherwise A ij = 0 .
X
=
{ X 1 , X 2 , . . .
, X T } is a STsequence of T time steps, where X 2 R T ⇥ N ⇥ C .
The snapshot at time step t is denoted as x t 2 R N ⇥ C, where C is the feature dimension of a node.
Then the ST sequence forecasting problem can bedeﬁned as: given S time steps historical observations of input graph G , the goal is to predict the futuresequence of the features on each node with a learning function f : ⇥ X ( t  S ): t , G ] f ! X ( t +1):( t + P ) , 97 where X ( t  S ): t and X ( t +1):( t + P ) are the ST sequence with length S and P respectively.
3.2 Network Architecture Search
Network (neural) architecture search (NAS) are automated methods for generating and optimizingneural networks.
A representative gradient-based approach is DARTS [16], which is the foundationof our proposed training framework.
DARTS aims to search optimal directed edge connections on adirected acyclic graph with predeﬁned computing cells as nodes.
The result connections of node j isdenoted as x ( j ) =
P i<
j o ( i,j )  x ( i )  , where o ( i,j ) is an operator, e.g. layers in a model, representedby a directed edge from node i to node j. DARTS proposes a method to relax the discrete searchingspace to be continuous, and uses bi-level optimization to learn a differentiable objective on the jointoptimization problem of both network architecture and model weights.
The objective function is:min ↵ L val ( w ⇤ ( ↵ ) , ↵ ) s.t.
w ⇤ ( ↵ ) =
argmin w L train ( w, ↵ ) , where ↵ is the architecture, and w isthe model weights.
In Section 4.4, we improve the design of the directed acyclic graph of the searcharchitecture, and the two-stage optimization of the architecture parameters.
Methods
In this section, we ﬁrstly introduce two basic modeling units: the time series linear self-attention, andthe high order mix graph convolution.
Then we proposed three layers as different network backbones,and we build a universal modeling framework based on the tree “atomic” layers.
Next, we propose anautomatic searching strategy for spatio-temporal information fusion, which aimed for the optimalorder of spatio-temporal modeling on various downstream tasks.
4.1 Spatial / Temporal Modeling Unit
4.1.1 Time Series Linear Self-Attention
Self-attention mechanism [21] has been widely used in nature language processing, computervision, and time series forecasting, which is deﬁned as: Attention( Q , K , V ) =
V 0 =
Softmax( QK > / p d ) V , where Q = XW Q , K = XW K , V = XW V , and the projection ma- [Equation]
However, the original self-attention suffers from high computational and memory cost.
Because thedot product computation of Q and K leads to O ( N 2 ) time and space complexity.
[9] proposed linearself-attention, which represents the similarity function of Q and K in the self-attention by a kernelfunction:
V 0 i =  ( Q i ) T P Nj =1  ( K j ) V Tj /  ( Q i ) T
P
Nj =1  ( K j ) . Such that for each query Q i ,the two terms
P Nj =1  ( K j ) V j and P Nj =1  ( K j ) are the same and reused for efﬁcient computing.
Following [30], we use the technique of linear self-attention in representing time series features.
4.1.2 High-order Mix Graph Convolution
To acquire better spatial information representation, we propose a high-order mix graph convolutionaloperation for spatial information mixing and feature extraction of the original inputs, it is deﬁned as:
HighOrder( X , A , order )
[Equation]
where order denotes the total order of the graph convolution operations, i.e., to consider orderhop neighbor relationship of each node.
In this paper, we deﬁne the 1 st -order mix convolutionaloperation by combining the 1 st -order ChebNet [10] and the Adaptive Diffusion Convolution [23]: MixGC( X , A ) =
ChebNet( X , A ) + AdapDC( X , A )
=
ˆ AXW g + P f XW f + P b XW b +ˆ A adp XW adp , where ˆ A = D  1 / 2 ˜ AD  1 / 2 is a normalized adjacency matrix with self-loop.
ChebNetfocuses on 1st-order neighbor information, while AdapDC focuses on multi-hop information.
˜
Ais deﬁned as ˜ A
=
A + I , where D ii =
P j ˜
A ij , I is an identity matrix.
P f =
A rowsum( A ) ,
P b =
A > rowsum( A > ) refers to a forward and backward state transition matrix, respectively.
ˆ A adp is anadaptive matrix for complementary spatial state information, which is calculated by two learnablenode embedding matrices E 1 , E 2 2 R N ⇥ C [20] as ˆ A adp = Softmax(ReLU( E 1 E > 2 )) .
4.2 Uniﬁed Spatio-temporal Modeling Backbone
In order to solve the problem of spatio-temporal dependency distribution differences in the modelingprocedure, we ﬁrst propose three novel modules: S2T Layer , T2S Layer , STS Layer , that aresuitable for three typical spatio-temporal dependencies: spatial-ﬁrst, temporal-ﬁrst, spatio-temporalsynchronous, respectively.
We design all these three modeling module to have the same dimension ofinputs and outputs.
This provides a solid foundation for our later ﬂexible and universal modeling.
Spatial -ﬁrst Modeling Layer
The spatial-ﬁrst sequence modeling method, S2T Layer , models from spatial to temporal.
The spatialinformation between the nodes on the graph is ﬁrst characterized on a single slice.
After that, node information at different times is exchanged along the time dimension, whose spatial information hasbeen shared with its neighbors.
As shown in Fig.3(a), S2T Layer ﬁrst uses two high-order mix graph convolution deﬁned in Eq.(1) toprocess the input spatio-temporal sequences
X L  1 to obtain two sequence representations with mixedspatial information.
Then the key K and value V of the input of the subsequent self-attention areobtained by a transformation using the parameter matrix W K , W V , respectively, while the query Q isobtained by transforming the original input ST sequence using the parameter matrix as follows: Q =
X L  1 W Q , K = HighOrder 1 ( X L  1 , A , order ) W K , V =
HighOrder 2 ( X L  1 , A , order ) W V .
Then the original ST sequence and the new sequence with mixed spatial information are processedusing a multi-head linear self-attention, from which it learns temporal dependencies and exchangesinformation at different time slices to obtain further representations of the ST sequence:
[Equation]
The output is concatenated with the initial input once for residuals and processed a layer normalization,followed by a two-layer fully connected network for further ST representation learning.
Thisnetwork is applied separately and identically to each point-in-time position in the ST sequence,thus maintaining the continuous transfer of position-encoded information.
Finally, the resulting
ST sequence representation is again connected to the initial input with one residual and layernormalization to obtain the output X L = Norm(max(0 , Norm( Z + X L  1 ) W 1 + b 1 ) W 2 + b 2 ) .
4.2.2 Temporal-ﬁrst Modeling Layer
This module is designed to model the ST sequence from temporal to spatial, named T2S Layer .
Different from spatial-ﬁrst modeling, at the beginning, the original inputs are projected into Q , K , Vby three weight matrices as Q =
X L  1 W Q , K =
X L  1 W K , and V =
X L  1 W V .
The projection 171 results are used to calculate temporal representations at ﬁrst, using the time series linear self-attentionin Eq.(2).
Then the temporal representation on each node are send to the high-order mix graphconvolution, together with the adjacency matrix, to fusion the temporal information from everyneighbors.
Z {+
+} =
HighOrder( Z , A , order ) .
Finally, the output representations are executed with thefeed forward and layer normalization operations the same way as the S2T Layer .
4.2.3 Spatial-temporal Synchronous Layer
EmbeddingLayer EmbeddingLayer
TokenEncoding
TemporalEncoding
PositionalEncoding
SpatialEncoding
STExtractor STExtractor
Add
FeedForward ReLU FeedForward
ReLU
Outputs
S2TLayer T2SLayer STSLayer TokenEmbedding
TemporalEmbedding
PositionalEmbedding SpatialEmbedding
[Figure]
For temporal modeling part, it still project theoriginal inputs into Q , K , V , and execute a linearself-attention operation for a temporal representation.
For the spatial part, it accepts the original inputs and the spatial information and useshigh-order mix graph convolution operation toconstruct spatial representation.
Temporal
Z 1 =
Attention( Q , K , V ) , Spatial Z 2 =
HighOrder( X L  1 , A , order ) .
The outputs areconcatenated together as: Z 0 = concat[ Z 1 , Z 2 ] .
Then it is executed with the following operationsand output as X L similar with the former twomodules.
4.3 Universal Modeling Framework
Targeting to the ST sequence forecasting task, we propose a uniﬁed ST sequence modeling framework(UniST) with the proposed uniﬁed modeling backbones in Fig.4, which follows the encoder-decoderarchitecture.
It uses a uniﬁed architecture with interchangeable and replaceable mode units.
AutoSTCell
E n c o d e r
H
S2TLayer
T2SLayer
STSLayer H
H out
Add
H i
H
H
S2TLayer T2SLayer STSLayer
H i
H j
H out
AutoSTCell
E n c o d e r
H
H
H out
H H
H
S2TLayer
T2SLayer STSLayer
Identity Zero
STSLayer
AutoSTCell
E n c o d er
H
S2TLayer
T2SLayer
STSLayer H
H out
Add
H i
S2TLayer T2SLayer STSLayer
H i
H
AutoSTCell
E n c o d er
H
H
H out
H H
H
S2TLayer
T2SLayer STSLayer
Identity Zero
STSLayer (b) AutoST 2 : the hybrid assembling scheme.
4.3.1 Spatio-temporal Embedding Layer
Since the Transformer model solely relies on the self-attention for global alignments, the positionalembedding [21] and extra embeddings [31] are needed to capture spatio-temporal dependency. Thenwe introduce four types of embeddings E P , E V , E S , E T in Appendix A. Fusion embedding . The four embeddings are summed together as the ﬁnal embedding added to theinputs: E F = E P + E V + E S + E T , note that the shape of token embedding E V 2 R T ⇥ N ⇥ d , whileother embeddings’ shape are E P 2 R 1 ⇥ 1 ⇥ d , E S 2 R 1 ⇥ N ⇥ d , E T 2 R T ⇥ 1 ⇥ d .
When calculatingthe summation, they will be replicated and expanded with broadcast on the respective missingdimensions.
The encoder of UniST consists of multiple Spatio-Temporal Extractors ( STE( · ) ), which can bearbitrarily chosen from { T2S Layer , S2T Layer , STS Layer }.
All extractors are connected end toend, i.e., the output of the previous one is the input of the next one.
To acquire a more diversityrepresentation, the outputs of each extractor are added to form the ﬁnal output of the encoder.
Let theoutputs of the embedding layer be X 0 , the encoder is computed as: X en = P Li =1 STE i ( X 0 ) , where
L refers to the number of spatio-temporal extractors.
4.3.3 Decoder
The decoder accepts the output of encoder, i.e., L outputs from L spatio-temporal extractors.
Theyare ﬁrstly added as a uniﬁed spatio-temporal representation.
Then the results are through twotimes of ReLU activation and Linear projection, and produce the ﬁnal sequence forecasting result.
Denote X ` as the output of extractor ` , we have the calculation of decoder as: ¯ Y = X de =
Linear(ReLU(Linear(ReLU( P ` X ` )))) .
4.4 Automated Search for UniST
AutoSTCell
E n c o d er
H
S2TLayer T2SLayer
STSLayer H
H out
Add
H i
H j
H out
S2TLayer T2SLayer STSLayer
H i
H j
H out
AutoSTCell E n c o d er
H H
H out
H H
H
S2TLayer
T2SLayer
STSLayer
Identity
Zero
STSLayer
The searching goal is to choose the optimal one from the candidate modules between the given computation node H i and H j .
With the proposed uniﬁed ST sequence modelingframework UniST, it still suffers from the potential wrong network conﬁguration problem, wherewe build an arbitrary modeling order with the replaceable model units { T2S Layer , S2T Layer ,
STS Layer }.
Considering the various downstreamtasks, how can we build a universal model with anoptimal conﬁguration?
Here we propose the Automated Spatio-Temporalmodeling approach (AutoST), which learns theoptimal combinatorial order that suits the spatiotemporal dependency of the current task.
We de- signed two schemes for layer combination.
In this section, we ﬁrst deﬁne the basic searching unit of
AutoST, then we introduce two designs of AutoST with different searching schemes.
4.4.1 AutoST Cell
The basic searching unit in AutoST is the network cell.
Here we deﬁne its structure and computingprocess.
Deﬁnition 1.
AutoST Cell.
Let G =
( V , E ) be a direct acyclic graph (DAG), V denotes the node set, 240 each node refers to a representation comes from the outputs of a computation layer.
The representationon node i is deﬁned as H i 2 R T ⇥ |V| ⇥ d , where T is the length of spatio-temporal seqence, d is thefeature dimension.
The input of each AutoST Cell is denoted as H 0 , and the output of each cell isthe summation of all nodes, i.e., all interval representations: H out = P |V| i H i .
On graph G , the 244 directed edge ( i, j ) from node i to node j stands for a mixture of all candidate modeling modules
O = {T2S Layer, S2T Layer, STS Layer}, and it is represented as o ( i,j ) .
So that the representationbetween node j and other nodes can be written as: H j = P i<
j o ( i,j ) H i .
On each directed edge,there exist a set of weight parameters ↵ ( i,j ) =
{ ↵ ( i,j ) o | o 2 O } , which indicates the probability of 248 the corresponding modeling module should be retained.
Every weight parameters of the candidatemodeling module is calculated as: H j =
[Equation]
4.4.2 Sequential Stacking Search
Based on the proposed UniTS, we propose two searching schemes to ﬁnd better combination ofthe modeling layers.
The proposed searching schemes are lossless replacements for the encoder of
UniTS.
We simply replace the encoder’ spatio-temporal extractor with the AutoST Cell.
The ﬁrst one is multi-layer sequential stacking searching scheme, with which the whole model isnamed AutoST 1 .
As illustrated in Fig.5(a), it has simple structure within each AutoST Cell, whilehas more complicated stacking structure between cells.
Each cell holds a DAG with three nodes, oneinput node H 0 , one output node H out and an intermediate node H 1 .
And two directed edges arepre-deﬁned between the former two nodes’ output and the output node: ( H 0 , H out ) and ( H 1 , H out ) .
The searching space of this scheme is shown as the directed red dotted line.
The red dashed box isthe search candidate set, including { T2S Layer , S2T Layer , STS Layer }.
We conduct two gradientbased network architecture search methods in the experiments, i.e., DARTS [16], PAS [22].
Aftersearching, the cell essentially becomes one of the three model units.
This scheme allows multiplestacking of cells, therefore, the new encoder of the whole model will become the sequential stackingbetween different modeling layers.
4.4.3 Hybrid Assembling Search
The another searching scheme is called hybrid assembling searching.
The structure is similar with thesequential stacking searching.
However, in this scheme, the encoder only consists of a single AutoST
Cell.
The cells are not stacked layer-by-layer, it will conduct searching on a more complicated DAGon a single AutoST Cell.
In this searching scheme, the DAG of AutoST Cell is shown as Fig.5(b).
There are multiple nodes inthe cell, generally, it will be set as 4-7 nodes in the experiments.
The pre-deﬁned connections areeach node’s output to the output node.
The multiple directed red dotted lines show the searchingspace of this scheme.
The candidate set is expanded with two operations: Identity and Zero .
Identitymeans no modeling module in this edge, i.e., build directly connection with no operation.
Zero meansto set all the data passing through it to zero, i.e., no connection is made.
This scheme makes the entire
DAG form a complex and deep network structure.
Through the complex structure design inside thecell, multiple spatio-temporal modeling modules are combined to ﬁt the spatio-temporal dependencydistribution in a target task.
Experiments
This section empirically evaluates the effectiveness of UniST and AutoST models with short-term,medium-term, and long-term ST sequence forecasting tasks on ﬁve real-world datasets. [Table]
5.1 Main Results
Table 1 summarizes the ST sequence forecasting results.
UniST S , UniST T , UniST ST stands for our UniST framework with three same stacking layers of S2T Layer , T2S Layer , STS Layer , respectively.
The results in bold font in Table 1 show that our proposed UniST outperforms all baseline methods andachieves State-of-the-Art on all 9 tasks of 5 datasets, with all three proposed layers.
This demonstratesour proposed uniﬁed spatio-temporal modeling layers and the uniﬁed forecasting framework are moreexpressive than traditional methods.
Speciﬁcally, compared with GMAN, which is also based onself-attention mechanism, our methods achieve at most 18.41%, 15.31%, 13.82% MAE decreases onthe short-term, medium-term, and long-term forecasting task, respectively.
Compared with the mostadvanced method STFGNN, our methods gain 12.06%, 10.61%, 9.43% MAE decreases on the short,medium, long-term forecasting tasks.
From the last two columns of Table 1, we can see that both of AutoST 1 and AutoST 2 beat all othermethods on every metric of every task, including our proposed UniST.
Recall that the differencebetween AutoST and UniST is the encoder part, UniST uses all same layers from { S2T Layer , T2S
Layer , STS Layer }, while AutoST aims to search for a better combination and connection usingthe three types of layers.
This demonstrates that combining and integrating modules with differentspatio-temporal modeling abilities can better deal with uncertainty spatio-temporal dependency andmore comprehensively model spatio-temporal sequences.
At the same time, AutoST 2 beats AutoST 1on
every task.
It shows that using a single-layer but complex internal connection searching methodfor these tasks is more effective.
Moreover, we can see that the best methods among all baselines are STFGNN, the only method inspatio-temporal synchronous modeling.
Comparing GMAN and Graph WaveNet, the two representative methods in spatial-ﬁrst modeling and temporal-ﬁrst modeling, respectively, we can see thatalthough they beat other baselines, while our proposed Uni- series model shows a better performance.
From this perspective, the proposed method matches the design.
5.2 Ablation Study
In AutoST, we design to use gradient-based network architecture search methods to optimize theconnections of the modeling modules, and we choose DARTS and PAS as the NAS methods.
At thesame time, we also conduct experiments to compare the method of random search and AutoSTG [17],which also uses the network architecture search technology for ST sequence prediction.
The resultsare shown in Table 2.
We can conclude that our proposed AutoST can outperform AutoSTG withall searching methods.
That is because AutoSTG uses more ﬁne-grained temporal convolution andgraph convolution structure as the candidates, while AutoST’s candidate layers are designed to model the three different spatio-temporal dependencies, which can make better use of the modeling ability [Table] of the three layers on different temporal and spatial relations.
It reduces the searching space and the [Table] search time simultaneously so that AutoST can make better efﬁciency and accuracy.
[Table]
(a) The search results of AutoST 1 .
AutoSTCell
E n c o d er
H
S2TLayer
T2SLayer
STSLayer H
H out
Add
AutoSTCell
E n c o d er
H
H
H out
H
H
H
S2TLayer T2SLayer STSLayer Identity Zero S2TLayer
H H H out
T2SLayer H
H H out S2TLayer H H
H out STSLayer H H H out
STSLayer
H
H
H out
H
T2SLayer STSLayer STSLayer
S2TLayer T2SLayer T2SLayer
H(b
The
AutoSTCell
E n c o d er
H S2TLayer T2SLayer STSLayer H
H out
Add
AutoSTCell
E n c o d er H H H out
H H H
S2TLayer T2SLayer STSLayer Identity Zero
S2TLayer
H H H out
T2SLayer H
H
H out
S2TLayer H H H out
STSLayer H
H H out
STSLayer
H
H
H out
H
T2SLayer STSLayer STSLayer
S2TLayer T2SLayer
T2SLayer
H
5.3 Result Visualization
Forecasting Visualization: We randomly selected one day from the PEMS-BAY dataset and compared our methods’ forecasting results and baseline methods in a visualization way.
Typical resultsare shown in Fig.8, the time span is selected from 13:00 to 19:00, which can represent the mosttypical scene of the rush hour from afternoon to evening.
From the forecasting lines compared to theground truth, we can see that our proposed methods outperform the two typical baseline methodsmostly when forecasting smooth trafﬁc and trafﬁc jam.
The bar lies on the bottom of the ﬁguresis the forecasting error of each method.
We can ﬁnd that all methods encountered accuracy droparound 14:00 and 17:30.
That is because the status of the road is changed dramatically.
However, ourmethods are more stable with the changes and can quickly be adapted to the new road state.
For thecomplete data visualization of the day, please refer to Fig.9 in the appendix.
Learned Architecture Visualization: The learned architectures of the AutoST 1 and AutoST 2 on
PEMS-08 dataset are shown in Fig.7.
We can ﬁnd that although UniST T has a better effect whenmodeling with single type modeling module, the search result of AutoST 1 show that the STS Layeroccupies a larger number, and the model achieves better performance than UniST T .
This demonstratesthat combining and stacking multiple spatio-temporal dependency modeling methods reasonably canbetter ﬁt the real spatio-temporal dependencies.
In addition, we can ﬁnd that AutoST 2 obtains a complex connections between the four computationnodes in a single cell.
And this learned architecture helps AutoST 2 achieve state-of-the-art on thistask.
Although we cannot yet explain why stacking the modules leads to better results, we can see apotentially broad range of applications for uniﬁed architecture searching in this way.
Conclusion
In this work we illustrated the existence of the modeling gap problem, especially the modeling order,in the spatio-temporal analysis.
Moreover, we build three different layers, namely S2T, T2S and STS,as new network modeling backbones.
Then, an automatic searching strategy is proposed to search theoptimal modeling priority automatically.
Extensive experiments on ﬁve real-world datasets show theoverwhelming performance over SOTA baselines.