AutoST: Towards the Universal Modeling of Spatio-temporal Sequences
Anonymous Author(s)
AffiliationAddress email
Abstract
The analysis of spatio-temporal sequences plays an important role in many realworld applications, demanding a high model capacity to capture the interdependence among spatial and temporal dimensions.
Previous studies provided separatednetwork design in three categories: spatial first, temporal first, and spatio-temporalsynchronous.
However, the manually-designed heterogeneous models can hardlymeet the spatio-temporal dependency capturing priority for various tasks.
Toaddress this, we proposed a universal modeling framework with three distinctivecharacteristics: (i) Attention-based network backbone, including S2T Layer (spatialfirst), T2S Layer (temporal first), and STS Layer (spatio-temporal synchronous).(ii)
The universal modeling framework, named UniST, with a unified architecturethat enables flexible modeling priorities with the proposed three different modules.
(iii)
An automatic search strategy, named AutoST, automatically searchesthe optimal spatio-temporal modeling priority by network architecture search.
Extensive experiments on five real-world datasets demonstrate that UniST with anysingle type of our three proposed modules can achieve state-of-the-art performance.
Furthermore, AutoST can achieve overwhelming performance with UniST.
Introduction
[Figure]
S p a ti a l
Temporal task task task
Time (b) (a)
Modeling and predicting the future of spatiotemporal (ST) sequences based on past observations has been extensively studied and has beensuccessfully applied in many fields, such as roadtraffic [13], medical diagnosis [28], and meteorological research [24].
Traditional statisticalmethods typically require input sequence satisfying certain assumptions, which limits its ability in capturing the complex spatial-temporaldependency.
Then, recurrent neural network(RNN) methods [15] leverage the universal approximation property to build separated networkbranches to model dependency and make predictions with fusion gate blocks from the stacking branches.
The intrinsic gradient flow in theback-propagation training process [7] may bring the ST dependency into incorrespondence with thenetwork branches’ configuration, especially for the deeper network [11].
Recently, the Transformerbased models show larger modeling capacity in both spatial and temporal modeling [21, 31, 30],which motivates us to find a universal way to capture the ST dependency in a universal frameworksimultaneously.
S p a ti a l
Temporal Spatial-first task task2 task
Temporal Temporal-first task task2 task
Temporal Spatio-temporal sync.
task task3 task
Temporal
Ours task task task3 S p a ti a l
Temporal
Ideal task task task
[Figure]
As shown in Fig.1, the ST dependency individually exists in sequences: spatial correspondences,temporal correspondences, and spatio-temporal correspondences.
Take the road traffic forecasting asan example, previous research fall into three typical paradigms.
(a) Spatial-first modeling [1, 26, 13]:
Predicting the traffic for next road junction, which has strong connections to previous intersections,stops, and surrounding traffic.
(b) Temporal-first modeling [23, 5]: Predicting the traffic of ahigh school on Friday afternoon, which shows strong periodic relationships with the school’s timesschedule.
(c) Spatio-temporal synchronous modeling [19, 12, 27, 6]:
Analyzing the city-widetraffic, which is tightly entangled in both spatial and temporal.
During the sequences modeling, themain problem is to align the network design with the natural spatio-temporal distribution.
However, the distribution of ST dependencies varies and depends on the forecasting task and corresponding datasets.
They are mixed in a compound way when modeling ST sequences, and the threetasks in Fig.1(b) are the representative ones.
What makes it worse is that, the prevalent modelingmethods show anisotropic tendency to capture the ST dependency.
If we use the spatial-first modelson the three tasks in Fig.2(b), the task 1’s states are highly influenced by the surrounding information,and the periodic pattern is the underlying factors, which makes the spatial-first model fits it properly.
We can compare the model ability (red lines) with the ideal one in Fig.2(a), this kind of modelwill be insufficient for task 2 and task 3.
Similarly, suppose we use the temporal-first models onthe three tasks in Fig.2(c).
In that case, the model ability only matches task 3, where the periodicpattern decides the states other than the spatial information.
The previous analysis also applies tothe spatio-temporal synchronous situation, where the states are mainly influenced by the complexassociations across the spatial and temporal, like semantic relationships.
In this paper, we aim topropose a universal model that alleviates the the modeling gap on different tasks.
Related Work
Existing spatio-temporal forecasting methods can be roughly grouped into three categories: spatialfirst, temporal-first, and spatio-temporal synchronous methods.
Spatial-first : STG2Seq [1] usesstacking GCN layers to capture the entire inputs sequence, where each GCN layer operates on alimited historical time window, and the final results are concatenated together to make forecasting.
In the view of this paper, it belongs to spatial-first modeling.
STGCN [26] propose the blocks thatcontains two temporal gated convolution layers with one spatial graph convolution layer in the middle,which starts from a convolution-based temporal layer.
DCRNN [13] is proposed to forecast trafficflow using diffusion convolution and recurrent units to capture spatial and temporal informationsuccessively.
Temporal-first : Graph WaveNet [23] built the basic modeling layers with two gatedtemporal convolution modules at the beginning and followed by a graph convolution module, whichmodels from temporal to spatial.
GSTNet [5] builds several layers of spatial-temporal blocks toproduce the forecasting, which is consists of a multi-resolution temporal module followed by aglobal correlated spatial module.
Spatio-temporal synchronous : STSGCN [19] construct a spatiotemporal synchronous extraction module composed of graph convolutional networks.
STFGNN[12] modeling spatio-temporal correlations simultaneously by fusing a dilated convolutional neuralnetwork with a gating mechanism and a spatio-temporal fusion graph module.
ST-ResNet [27]using convolution on a sequence of image-like 2D matrices to model spatio-temporal at the sametime.
ASTGCN [6] proposed a spatial-temporal convolution that simultaneously captures the spatialpatterns and temporal features.
A
Q
K
V
HighOrderMixGC
X LX L
HighOrderMixGC
LinearSelf-Attention
Add&Norm
FeedForward
Add&Norm
A
Q
K
V
X LX L
HighOrderMixGC
LinearSelf-Attention
Add&Norm
FeedForward
Add&Norm
A
Q K
V
X LHighOrderMixGC Linear Self-Attention
Add&Norm
FeedForward
Add&Norm
X L
S2T Layer
T2S Layer
STS Layer
Preliminary3.1 Spatio-temporal Sequence Forecasting
Spatio-temporal sequence forecasting (STSF) is to predict the future sequence of spatio-temporalinputs based on the historical observations.
Specifically, given a graph G =
( V, E, A ) , where V and
E are the node set and edge set, and N is the number of nodes, A ∈ R N × N is the adjacency matrixof G .
If v i , v j ∈ V and ( v i , v j ) ∈ E , A ij = 1 , otherwise A ij = 0 .
X
=
{ X 1 , X 2 , . . .
, X T } is a STsequence of T time steps, where X ∈ R T × N × C .
The snapshot at time step t is denoted as x t ∈ R N × C, where C is the feature dimension of a node.
Then the ST sequence forecasting problem can bedefined as: given S time steps historical observations of input graph G , the goal is to predict the futuresequence of the features on each node with a learning function f : (cid:2) X ( t − S ): t , G ] f → X ( t +1):( t + P ) ,where X ( t − S ): t and X ( t +1):( t + P ) are the ST sequence with length S and P respectively.
3.2 Network Architecture Search
Network (neural) architecture search (NAS) is automated methods for generating and optimizingneural networks.
A representative gradient-based approach is DARTS [16], which is the foundationof our proposed trainning framework.
DARTS aims to search optimal directed edge connectionson a directed acyclic graph with predefined computing cells as nodes.
The result connections ofnode j is denoted as x ( j )
= (cid:80) i<
j o ( i,j ) (cid:0) x ( i ) (cid:1) . DARTS proposes a method to relax the discretesearching space to be continuous, and uses bi-level optimization to learn a differentiable objectiveon the joint optimization problem of both network architecture and model weights.
The objectivefunction is: min α L val ( w ∗ ( α ) , α ) s.t.
w ∗ ( α ) =
argmin w L train ( w, α )
In Section 4.4, we improvethe design of the directed acyclic graph of the search architecture, and the two-stage optimization ofthe architecture parameters.
Methods
In this section, we firstly introduce two basic modeling units: the time series linear self-attention, andthe high order mix graph convolution.
Then we proposed three layers as different network backbones,and we build a universal modeling framework based on the tree “atomic” layers.
Next, we propose anautomatic searching strategy for spatio-temporal information fusion, which aimed for the optimalorder of spatio-temporal modeling on various downstream tasks.
4.1 Spatial / Temporal Modeling Unit
4.1.1 Time Series Linear Self-Attention
Self-attention mechanism [21] has been widely used in nature language processing, computervision, and time series forecasting, which is defined as: Attention( Q , K , V ) =
V ′ =
Softmax( QK ⊤ / √ d ) V , where Q = XW Q , K = XW K , V = XW V , and the projection matrix W Q ∈ R C × D , W K ∈ R C × D , W V ∈ R C × D .
However, the original self-attention suffers from high computational and memory cost.
Because thedot product computation of Q and K leads to O ( N 2 ) time and space complexity.
[9] proposed linearself-attention, which represents the similarity function of Q and K in the self-attention by a kernelfunction: V ′ i =
ϕ ( Q i ) T (cid:80) Nj =1 ϕ ( K j ) V
Tj /ϕ ( Q i ) T (cid:80) Nj =1 ϕ ( K j ) .
Such that for each query Q i ,the two terms (cid:80) Nj =1 ϕ ( K j ) V j and (cid:80) Nj =1 ϕ ( K j ) are the same and reused for efficient computing.
Following [30], we use the technique of linear self-attention in representing time series features.
4.1.2 High-order Mix Graph Convolution
To acquire better spatial information representation, we propose a high-order mix graph convolutionaloperation for spatial information mixing and feature extraction of the original inputs, it is defined as:
HighOrder( X , A , order )
[Equation]
where order denotes the total order of the graph convolution operations, i.e., to consider order -hopneighbor relationship of each node.
In this paper, we define the 1 st -order mix convolutional operationas the combine of 1 st -order ChebNet [10] and the Adaptive Diffusion Convolution proposed by[23]: MixGC( X , A ) =
ChebNet( X , A )+AdapDC( X , A )
=
ˆ AXW g + P f XW f + P b XW b +ˆ A adp XW adp , where ˆ A = D − 1 / 2 ˜ AD − 1 / 2 is a normalized adjacency matrix with self-loop.
˜
Ais defined as ˜ A
=
A + I , where D ii =
(cid:80)
j ˜
A ij , I is an identity matrix.
P f =
A rowsum( A ) ,
P b =
A ⊤ rowsum( A ⊤ ) refers to a forward and backward state transition matrix, respectively.
ˆ A adp is anadaptive matrix for complementary spatial state information, which is calculated by two learnablenode embedding matrices E 1 , E 2 ∈ R N × C [20] as ˆ A adp = Softmax(ReLU( E 1 E ⊤ 2 )) .
4.2 Unified Spatio-temporal Modeling Backbone
In order to solve the problem of spatio-temporal dependency distribution differences in the modelingprocedure, we first propose three novel modules: S2T Layer , T2S Layer , STS Layer , that aresuitable for three typical spatio-temporal dependencies: spatial-first, temporal-first, spatio-temporalsynchronous, respectively.
And we can notice that, all these three modeling module has the exactlysame inputs and outputs.
This provides a solid foundation for our later flexible and universal modeling.
4.2.1 Spatial-first Modeling Layer
The spatial-first sequence modeling method, S2T Layer , models from spatial to temporal.
The spatialinformation between the nodes on the graph is first characterized on a single slice.
After that, nodeinformation at different times is exchanged along the time dimension, whose spatial information hasbeen shared with its neighbors.
As shown in Fig.3(a), S2T Layer first uses two high-order mix graph convolution defined in Eq.(1) toprocess the input spatio-temporal sequences
X L − 1 to obtain two sequence representations with mixedspatial information.
Then the key K and value V of the input of the subsequent self-attention areobtained by a transformation using the parameter matrix W K , W V , respectively, while the query Q isobtained by transforming the original input ST sequence using the parameter matrix as follows: Q =
X L − 1 W Q , K = HighOrder 1 ( X L − 1 , A , order ) W K , V =
HighOrder 2 ( X L − 1 , A , order ) W V .
Then the original ST sequence and the new sequence with mixed spatial information are processedusing a multi-head linear self-attention, from which it learns temporal dependencies and exchanges information at different time slices to obtain further representations of the ST sequence:
Z = Attention( Q , K , V ) .
The output is concatenated with the initial input once for residuals and processed a layer normalization,followed by a two-layer fully connected network for further ST representation learning.
Thisnetwork is applied separately and identically to each point-in-time position in the ST sequence,thus maintaining the continuous transfer of position-encoded information.
Finally, the resulting
ST sequence representation is again connected to the initial input with one residual and layernormalization to obtain the output X L = Norm(max(0 , Norm( Z + X L − 1 ) W 1 + b 1 ) W 2 + b 2 ) .
4.2.2 Temporal-first Modeling Layer
This module is designed to model the ST sequence from temporal to spatial, named T2S Layer .
Different from spatial-first modeling, at the beginning, the original inputs are projected into Q , K , Vby three weight matrices as Q =
X L − 1 W Q , K =
X L − 1 W K , and V =
X L − 1 W V .
The projectionresults are used to calculate temporal representations at first, using the time series linear self-attentionin Eq.(2).
Then the temporal representation on each node are send to the high-order mix graphconvolution, together with the adjacency matrix, to fusion the temporal information from everyneighbors.
Z [-
-]′ =
HighOrder( Z , A , order ) .
Finally, the output representations are executed with thefeed forward and layer normalization operations the same way as the S2T Layer .
4.2.3 Spatial-temporal Synchronous Layer
EmbeddingLayer EmbeddingLayer
TokenEncoding
TemporalEncoding
PositionalEncoding
SpatialEncoding
STExtractor STExtractor
Add
[Figure]
ReLU
[Figure]
framework, starting with an embedding layer and followed by an encoder and a decoder.
The embedding layer generates the sum of four embeddings.
The encoder consists of multiple spatio-temporal extractors, including S2T, T2S or STS Layer.
This module named STS Layer , which aims tomodel the spatial and temporal information simultaneously.
Different from the former two modules,the inputs are directly used to calculate spatialand temporal representations at the same time.
For temporal modeling part, it still project theoriginal inputs into Q , K , V , and execute a linearself-attention operation for a temporal representation.
For the spatial part, it accepts the original inputs and the spatial information and useshigh-order mix graph convolution operation toconstruct spatial representation.
Temporal ←
Z 1 =
Attention( Q , K , V ) , Spatial ← Z 2 =
HighOrder( X L − 1 , A , order ) .
The outputs areconcatenated together as: Z ′ = concat[ Z 1 , Z 2 ] .
Then it is executed with the following operationsand output as X L similar with the former twomodules.
4.3 Universal Modeling Framework
Targeting to the ST sequence forecasting task, we propose a unified ST sequence modeling framework(UniST) with the proposed unified modeling backbones in Fig.4, which follows the encoder-decoderarchitecture.
It uses a unified architecture with interchangeable and replaceable mode units.
4.3.1 Spatio-temporal Embedding Layer
Since the Transformer model solely relies on the self-attention for global alignments, the positionalembedding [21] and extra embeddings [31] are needed to capture spatio-temporal dependency. Thenwe introduce four types of embeddings E P , E V , E S , E T in Appendix A. Fusion embedding . The four embeddings are summed together as the final embedding added to theinputs: E F = E P + E V + E S + E T , note that the shape of token embedding E V ∈ R T × N × d whileother embeddings’ shape
E P ∈ R 1 × 1 × d
E S ∈ R 1 × N × d
E T ∈ R T × 1 × d calculatingthe summation, they will be replicated and expanded with broadcast on the respective missingdimensions.
AutoSTCell
E n c o d e r
H
S2TLayer
T2SLayer
STSLayer H
H out
Add
H i
H
H
S2TLayer T2SLayer STSLayer
H i
H j
H out
AutoSTCell
E n c o d e r
H
H
H out
H H
H
S2TLayer
T2SLayer STSLayer
Identity Zero
STSLayer
AutoSTCell
E n c o d er
H
S2TLayer
T2SLayer
STSLayer H
H out
Add
H i
S2TLayer T2SLayer STSLayer
H i
H
AutoSTCell
E n c o d er
H
H
H out
H H
H
S2TLayer
T2SLayer STSLayer
Identity Zero
STSLayer (b) AutoST 2 : the hybrid assembling scheme.
4.3.2 Encoder
The encoder of UniST consists of multiple Spatio-Temporal Extractors ( STE( · ) ), which can be arbitrarily chosen from { T2S Layer , S2T Layer , STS Layer }.
All extractors are connected end to end, i.e.,the output of the previous one is the input of the next one.
To acquire a more diversity representation,the outputs of each extractor are added to form the final output of the encoder.
Let the outputs of theembedding layer be X 0 , the encoder is computed as: X en = STE( · · · STE(STE( X 0 ))) .
4.3.3 Decoder
The decoder accepts the output of encoder, i.e., L outputs from L spatio-temporal extractors.
Theyare firstly added as a unified spatio-temporal representation.
Then the results are through twotimes of ReLU activation and Linear projection, and produce the final sequence forecasting result.
Denote X ℓ as the output of extractor ℓ , we have the calculation of decoder as: ¯ Y = X de =
Linear(ReLU(Linear(ReLU( (cid:80) ℓ X ℓ )))) .
4.4 Automated Search for UniST
AutoSTCell
E n [Figure] d er
H
[Figure]
STSLayer
[Figure] out
Add
H i
H j
H out
[Figure] STSLayer
H i
[Figure]
out
AutoSTCell E n c o [Figure] er
H H
[Figure]
H out
H H
[Figure]
H
S2TLayer
T2SLayer
STSLayer
Identity
Zero
STSLayer
The searching goal is to choose the optimal one from the candidate modules between the given computation node H i and H j .
With the proposed unified ST sequence modelingframework UniST, it still suffers from the potential wrong network configuration problem, wherewe build an arbitrary modeling order with the replaceable model units { T2S Layer , S2T Layer ,
STS Layer }.
Considering the various downstreamtasks, how can we build a universal model with anoptimal configuration?
Here we propose the Automated Spatio-Temporalmodeling approach (AutoST), which learns theoptimal combinatorial order that suits the spatiotemporal dependency of the current task.
We designed two schemes for layer combination.
In this section, we first define the basic searching unit of
AutoST, then we introduce two designs of AutoST with different searching schemes.
4.4.1 AutoST Cell
The basic searching unit in AutoST is the network cell.
Here we define its structure and computingprocess.
Definition 1.
AutoST Cell.
Let G =
( V , E ) be a direct acyclic graph (DAG), V denotes the node set,each node refers to a representation comes from the outputs of a computation layer.
The representationon node i is defined as H i ∈ R T ×|V|× d , where T is the length of spatio-temporal seqence, d is thefeature dimension.
The input of each AutoST Cell is denoted as H 0 , and the output of each cell isthe summation of all nodes, i.e., all interval representations: H out = (cid:80) |V| i H i .
On graph G , thedirected edge ( i, j ) from node i to node j stands for a mixture of all candidate modeling modules
O = {T2S Layer, S2T Layer, STS Layer}, and it is represented as o ( i,j ) .
So that the representationbetween node j and other nodes can be written as: H j = (cid:80) i<
j o ( i,j ) H i .
On each directed edge,there exist a set of weight parameters α ( i,j ) =
{ α ( i,j ) o | o ∈ O } , which indicates the probability ofthe corresponding modeling module should be retained.
Every weight parameters of the candidatemodeling module is calculated as: H j = (cid:80) i< j (cid:80) o ∈ O [Equation]
[Equation]
4.4.2 Sequential Stacking Search
Based on the proposed UniTS, we propose two searching schemes to find better combination ofthe modeling layers.
The proposed searching schemes are lossless replacements for the encoder of
UniTS.
We simply replace the encoder’ spatio-temporal extractor with the AutoST Cell.
The first one is multi-layer sequential stacking searching scheme, with which the whole model isnamed AutoST 1 .
As illustrated in Fig.5(a), it has simple structure within each AutoST Cell, whilehas more complicated stacking structure between cells.
Each cell holds a DAG with three nodes, oneinput node H 0 , one output node H out and an intermediate node H 1 .
And two directed edges arepre-defined between the former two nodes’ output and the output node: ( H 0 , H out ) and ( H 1 , H out ) .
The searching space of this scheme is shown as the directed red dotted line.
The red dashed box isthe search candidate set, including { T2S Layer , S2T Layer , STS Layer }.
We conduct two gradientbased network architecture search methods in the experiments, i.e., DARTS [16], PAS [22].
Aftersearching, the cell essentially becomes one of the three model units.
This scheme allows multiplestacking of cells, therefore, the new encoder of the whole model will become the sequential stackingbetween different modeling layers.
4.4.3 Hybrid Assembling Search
The another searching scheme is called hybrid assembling searching.
The structure is similar with thesequential stacking searching.
However, in this scheme, the encoder only consists of a single AutoST
Cell.
The cells are not stacked layer-by-layer, it will conduct searching on a more complicated DAGon a single AutoST Cell.
In this searching scheme, the DAG of AutoST Cell is shown as Fig.5(b).
There are multiple nodes inthe cell, generally, it will be set as 4-7 nodes in the experiments.
The pre-defined connections areeach node’s output to the output node.
The multiple directed red dotted lines show the searchingspace of this scheme.
The candidate set is expanded with two operations: Identity and Zero .
Identitymeans no modeling module in this edge, i.e., build directly connection with no operation.
Zero meansto set all the data passing through it to zero, i.e., no connection is made.
This scheme makes the entire
DAG form a complex and deep network structure.
Through the complex structure design inside thecell, multiple spatio-temporal modeling modules are combined to fit the spatio-temporal dependencydistribution in a target task.
Experiments
This section empirically evaluates the effectiveness of UniST and AutoST models with short-term,medium-term, and long-term ST sequence forecasting tasks on five real-world datasets.
5.1 Main Results
We summarize the ST sequence forecasting results in Table 1.
UniST S , UniST T , UniST ST stands forour proposed UniTS framework with three same stacking layers of S2T Layer , T2S Layer , STS Layer ,respectively.
The best results of all manually designed models on each task are in bold black.
Thebest results of all methods are marked in bold and underlined.
The results in bold font in Table 1 show that our proposed UniST outperforms all baseline methods andachieves State-of-the-Art on all 9 tasks of 5 datasets, with all three proposed layers.
This demonstratesour proposed unified spatio-temporal modeling layers and the unified forecasting framework are moreexpressive than traditional methods.
Specifically, compared with GMAN, which is also based onself-attention mechanism, our methods achieve at most 18.41%, 15.31%, 13.82% MAE decreases on [Table] the short-term, medium-term, and long-term forecasting task, respectively.
Compared with the mostadvanced method STFGNN, our methods gain 12.06%, 10.61%, 9.43% MAE decreases on the short,medium, long-term forecasting tasks.
From the last two columns of Table 1, we can see that both of AutoST 1 and AutoST 2 beat all othermethods on every metric of every task, including our proposed UniST.
Recall that the differencebetween AutoST and UniST is the encoder part, UniST uses all same layers from { S2T Layer , T2S
Layer , STS Layer }, while AutoST aims to search for a better combination and connection usingthe three types of layers.
This demonstrates that combining and integrating modules with differentspatio-temporal modeling abilities can better deal with uncertainty spatio-temporal dependency andmore comprehensively model spatio-temporal sequences.
At the same time, AutoST 2 beats AutoST 1on
every task.
It shows that using a single-layer but complex internal connection searching methodfor these tasks is more effective.
Moreover, we can see that the best methods among all baselines are STFGNN, the only method inspatio-temporal synchronous modeling.
Comparing GMAN and Graph WaveNet, the two representative methods in spatial-first modeling and temporal-first modeling, respectively, we can see thatalthough they beat other baselines, while our proposed Uni- series model shows a better performance.
From this perspective, the proposed method matches the design.
5.2 Ablation Study
In AutoST, we design to use gradient-based network architecture search methods to optimize theconnections of the modeling modules, and we choose DARTS and PAS as the NAS methods.
At thesame time, we also conduct experiments to compare the method of random search and AutoSTG [17],which also uses the network architecture search technology for ST sequence prediction.
The resultsare shown in Table 2.
We can conclude that our proposed AutoST can outperform AutoSTG withall searching methods.
That is because AutoSTG uses more fine-grained temporal convolution andgraph convolution structure as the candidates, while AutoST’s candidate layers are designed to modelthe three different spatio-temporal dependencies, which can make better use of the modeling abilityof the three layers on different temporal and spatial relations.
It reduces the searching space and thesearch time simultaneously so that AutoST can make better efficiency and accuracy.
[Table]
AutoSTCell
E n c o d er
H
S2TLayer
T2SLayer
STSLayer H
H out
Add
AutoSTCell
E n c o d er
H
H
H out
H H
H
S2TLayer T2SLayer
STSLayer Identity
Zero
[Table]
[Table]
AutoSTCell
E n c
d
er
H
S2TLayer
T2SLayer
STSLayer H
H out
Add
AutoSTCell
E n c o d er
H
H
H out
H H
H
S2TLayer T2SLayer
STSLayer Identity
[Figure]
STSLayer STSLayer
S2TLayer T2SLayer
T2SLayer
H
[Figure]
5.3 Result Visualization
[Figure] Forecasting Visualization: We randomly selected one day from the PEMS-BAY dataset and com- pared our methods’ forecasting results and baseline methods in a visualization way.
Typical resultsare shown in Fig.8, the time span is selected from 13:00 to 19:00, which can represent the mosttypical scene of the rush hour from afternoon to evening.
From the forecasting lines compared to theground truth, we can see that our proposed methods outperform the two typical baseline methodsmostly when forecasting smooth traffic and traffic jam.
The bar lies on the bottom of the figuresis the forecasting error of each method.
We can find that all methods encountered accuracy droparound 14:00 and 17:30.
That is because the status of the road is changed dramatically.
However, ourmethods are more stable with the changes and can quickly be adapted to the new road state.
For thecomplete data visualization of the day, please refer to Fig.9 in the appendix.
Learned Architecture Visualization: The learned architectures of the AutoST 1 and AutoST 2 on
PEMS-08 dataset are shown in Fig.7.
We can find that although UniST T has a better effect whenmodeling with single type modeling module, the search result of AutoST 1 show that the STS Layeroccupies a larger number, and the model achieves better performance than UniST T .
This demonstratesthat combining and stacking multiple spatio-temporal dependency modeling methods reasonably canbetter fit the real spatio-temporal dependencies.
In addition, we can find that AutoST 2 obtains a complex connections between the four computationnodes in a single cell.
And this learned architecture helps AutoST 2 achieve state-of-the-art on thistask.
Although we cannot yet explain why stacking the modules leads to better results, we can see apotentially broad range of applications for unified architecture searching in this way.
Conclusion
In this work, we illustrated the existence of the modeling gap problem, especially the modeling order,in the spatio-temporal analysis.
Moreover, we build three different layers, namely S2T, T2S, and STS,as new network modeling backbones.
Then, an automatic searching strategy is proposed to search theoptimal modeling priority automatically.
Extensive experiments on five real-world datasets show theoverwhelming performance over SOTA baselines. Checklist
For all authors...(
a) Do the main
claims made
in
the abstract and introduction
accurately reflect
the paper’scontributions and
scope?
[Yes
(b) Did you describe the limitations of your work?
[Yes]
Due to the space
limits
, the
limitations
are in the supplemental
material
(c) Did you discuss any potential negative societal impacts of your work?
[No] To the best
of our knowledge, there is no potential
negative societal impacts.(d) review guidelines ensured paper conforms tothem?
Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
There is notheoretical result in the paper.
(b) Did you include complete proofs of all theoretical results?
[N/A]
There is no theoretical
result in the paper.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] The code, dataand instructions to reproduce the experimental results are provided in the supplemental
material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] These details are in the Appendix Section B.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[No] Due to the space limits in Table 1, we did not report errorbars.
(d) Did you include the total amount of compute and the type of resources used (e
g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] The details are in Section B.1.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]Our code are included in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] We only used public datasets in this paper.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] To the best of our knowledge, the traffic datasets
used in our paper do not have any personally identifiable information or offensive
content.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
No human subjects are involved.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
No human subjects are involved.
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
No human subjects are involved.