T RAJ GRU-A TTENTION -ODE: N OVEL S PATIOTEMPORAL P REDICTIVE M ODELS
Anonymous authors
Paper under double-blind review A BSTRACT
To perform the long-term spatiotemporal sequence prediction (SSP) task with irregular time sampling assumptions, we build the sequence-to-sequence models based on the Trajectory Gated Recurrent Unit (TrajGRU) network and our proposed deep learning modules.
First, we design a novel attention mechanism, namely Motion-based Attention (MA), and insert it into the TrajGRU network to create the TrajGRU-Attention model.
In particular, the TrajGRU-Attention model can alleviate the impact of the vanishing gradient, which leads to the blurry effect in the long-term predictions and handle both regularly sampled and irregularly sampled time series.
Second, leveraging the advances in Neural Ordinary
Differential Equation (NODE) technique
, we proposed the TrajGRU-AttentionODE model, which can be applied in continuous-time applications.
To evaluate the performance of the proposed models, we select three available spatiotemporal datasets based on the complexity level, including the MovingMNIST, MovingMNIST++, and KTH Action.
Our models outperform the state-of-the-art NODE model and generate better results than the standard TrajGRU model for SSP tasks with different circumstances of time sampling.
I NTRODUCTION
Spatiotemporal Sequence Predictive (SSP) learning has been an essential research topic and plays a critical role in many real-world applications, attracting widespread attention in recent years.
In real life, spatiotemporal data often occur around us whenever data are collected across space and time (Wang et al., 2020).
Overall, SSP learning refers to the approaches used for studying and processing these data for practical tasks ranging from many daily practical applications (e.g., motion prediction (Wu et al., 2017), traffic flow prediction (Luo et al., 2019)) to many global-scale applications (e.g., precipitation and weather hazard
nowcasting (Shi et al., 2015; 2017)).
With SSP learning tasks, researchers have studied and proposed various machine learning algorithms and advanced deep learning models to capture the spatial and temporal properties (Fang et al., 2021).
Typically, there are two types of time sampling in temporal analysis: regular time sampling (i.e., the time interval is assumed to be constant) and irregular time sampling (i.e., the time interval is considered arbitrary).
Although some sequence models usually apply a standard time sampling variable and use the concept of frame rate to sample the data, there are numerous scenarios in which this assumption is not practical.
First, not all spatiotemporal datasets follow this sampling type; some research areas are interested in data recorded at arbitrary time steps, such as medical, economics, and climate modelling (Foley, 2010; Kanakidou et al., 2005).
Besides discrete series, some spatiotemporal data in continuous space are applied for continuous-time physical models (Zhong et al., 2019), biology applications (Pinckaers & Litjens, 2019) and telecommunications (Huai et al., 2022).
Second, an observed time series can lose data points at some time because of hardware errors, noise, and outside effects on machines.
Furthermore, handling time series at arbitrary time steps can help systems be flexibly used for numerous applications and save time and resources.
Recent studies proposed approaches to solve complicated SSP tasks with fixed frame rates, and these models are designed to address the vanishing gradient issue (Shi et al., 2015; 2017; Wang et al., 2017; Su et al., 2020).
Other works used the concept of irregular time sampling in continuous-time video generation and prediction (Chen et al., 2018;
Yildiz et al., 2019; De Brouwer et al., 2019).
However, those continuous-time models, as described in Section 2, experimented with relatively simple datasets; thus, they are not practical for real-world applications with complicated datasets.
Based on those observations, we propose two novel sequence-to-sequence predictive models which can work effectively with the overall SSP task with different time sampling assumptions:
TrajGRUAttention and TrajGRU-Attention-ODE.
In the two models, we presented a novel attention module, namely Motion-based Attention (MA), which can be flexibly dropped into a Recurrent Neural Network (RNN) to boost memory ability and help the model handle irregularly sampled time series.
In the TrajGRU-Attention-ODE, besides the MA modules, we design a novel RNN, namely Trajectory Gated Recurrent Unit integrating Ordinary Differential Equation techniques (TrajGRU-ODE), which can perform continuous-time spatiotemporal prediction tasks.
According to extensive experiments on various video datasets, the proposed models can generate predictions properly with high accuracy.
Furthermore, the proposed models outperform state-of-the-art ODE-based approaches for
SSP tasks with different circumstances of time sampling.
R ELATED W ORK
Convolutional Recurrent Neural Networks (ConvRNNs).
ConvRNNs are proposed based on
Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs).
Convolutional
Long-short Term Memory (ConvLSTM)
(Shi et al., 2015), and Convolutional Gated Recurrent Unit (ConvGRU) (Ballas et al., 2015; Siam et al., 2017) are two prior variants in this group.
Based on the fundamental idea of the learnable spatial transformer module, Shi et al.
(2017) proposed the
Trajectory Gated Recurrent Unit (TrajGRU) model that can learn the location-variant structure for the recurrent connection and capture the spatiotemporal corrections effectively.
Wang et al.
(2017; 2018; 2022) proposed PredRNN models with novel ConvRNN layers for the SSP task.
Essentially, the PredRNN cell is the extended version of the ConvLSTM unit containing dual memories: the temporal and spatial memory states.
As a result, the recurrent layer can store more knowledge of the observed series, and the system can alleviate the vanishing gradient problem.
In addition, the ConvLSTM and ConvGRU models are considered first-order Markovian models because their memory cell takes only one previous hidden state at a time to generate the spatiotemporal representation state (Soltani & Jiang, 2016).
This causes an intrinsic difficulty in learning spatiotemporal correlations in the long-term forecasting task.
Therefore, Su et al.
(2020) extended the ConvLSTM to a higherorder network that can combine several previous states and feed them into the memory mechanism at a time step, namely Convolutional Tensor-Train Long Short-Term Memory (Conv-TT-LSTM).
Neural Ordinary Differential Equations (NODEs).
For the SSP task, most of the recent sequence models follow the regular time sampling assumption.
Therefore, the learning ability of these models is limited to specific datasets with a constant time gap between adjacent data frames.
As a result, these models cannot perform irregularly sampled or continuous-time prediction tasks.
A temporary solution to this problem is to preprocess the sequences by removing some intermediate data points of irregularly sampled sequences.
However, this will significantly impact the training and testing processes of the whole system because some data are ignored, especially with sparse datasets.
Consequently, the preprocessing step is not optimal for the given task.
In that situation, NODEs have been proposed as promising methods for irregularly sampled and continuous-time prediction problems.
The group of ODE-based models is a new family of deep neural network models (Chen et al., 2018; 2020).
In the SSP field, the evolution of a phenomenon can be represented by differential equations, and these equations can be solved by mathematical methods.
Based on this observation,
ODE-based models have been designed, and specific neural networks have been proposed to implement the differential equations describing the changing processes of representation states in the latent field.
For instance, Rubanova et al.
(2019) proposed a generative latent function continuoustime model (Latent-ODE), considered one of the first NODEs implemented for time series prediction tasks.
Yildiz et al.
(2019) proposed Deep generative second-order ODEs with Bayesian neural networks (ODE2VAE) for autoencoding irregularly sampled time series.
Although the ODE-based models above can handle irregularly sampled data and learn the continuous-time state dynamics, their performances were investigated with relatively simple datasets, such as image rotation and bouncing ball (Rubanova et al., 2019;
Yildiz et al., 2019; De Brouwer et al., 2019).
Park et al.
(2021) proposed a continuous-time video generation model using the ODE technique, namely Vid-ODE, which shows some promising performances with real-world videos.
Vid-ODE essentially combined the standard ConvGRU network, NODE techniques, and the pixel-level composition technique.
M ODELS 3. P ROBLEM S TATEMENT
With the given spatiotemporal sequence input, the main objective of the SSP task is to forecast the output sequence at future time steps.
To do that, sequence-to-sequence models are typically implemented with an encoding-decoding structure.
Simply put, the encoder receives, processes the input sequence, and sends practical information to the decoder, generating output predictions.
In this work, the main SSP problem formulation can be expressed as
[Equation]
where X ∈ R S × C × H × W indicates the examined sequence, consisting of S data frames at time steps { t 1 , t 2 , ..., t S } ; X input ∈ R N × C × H × W ,
X target ∈
R K × C × H × W are the input and target subsequences, respectively (i.e., S = N + K );
X prediction ∈
R K × C × H × W presents the output prediction of the model; f encoder , f decoder represent the encoder and the decoder of the model, respectively; f loss indicates the main loss function, as illustrated in Appendix A.2.
3. E NCODING - DECODING S TRUCTURE
In this work, a model architecture is developed from the stacked style.
In the stacked ConvLSTM architecture (Shi et al., 2015), multiple recurrent layers are stacked: the lowest-level layer is the bottom layer, and the highest-level one is the top layer.
During the recurrent computation of a specific layer, each recurrent neural unit’s output hidden state is sent to the higher-level layer’s neural unit.
Therefore, the neural unit of the top layer at a specific time step might forget some practical information coming from the neural unit of the bottom layer at the previous time step when the system uses a high number of stacked layers (Wang et al., 2017).
Consequently, the final representation state of each recurrent layer in the encoder cannot contain sufficient knowledge of a long-range series; thus, the decoder cannot learn the spatiotemporal features effectively.
Based on the above observations, we design a novel architecture combining the stacked style, zigzag connection, and reverse layers at the decoder (see Figure 1).
To alleviate the vanishing gradient effect, we take the idea of the spatiotemporal memory flow of Wang et al.
(2017) to design a zigzag connection, which creates a shortcut to communicate between the top and bottom recurrent layers.
This zigzag shortcut can boost the model’s memory ability by increasing the network’s depth (i.e., the connection between units of top and bottom layers allows the model to generate more representations between two timestamps).
Furthermore, this architecture enables the model to use more stacked intermediate layers.
Consequently, it improves the system’s overall performance.
At the encoder, the operations of neural units when applying the zigzag connection can be formulated as
[Equation]
and vice versa at the decoder, the information flows are reversed so that we have the following equations for the zigzag connection
[Equation]
Here, Z zz ( t i ) indicates the update gate at the time step t i , which decides how much information of the zigzag state H zz ( t i ) will be obtained; W upsampling and W downsampling refer to the learnable parameters of the up-sampling and down-sampling layers, respectively; W h , W zz refer to the learnable parameters of the convolutional layers applied to extract the features of the hidden states.
Consequently, the new state H newbottom ( t i ) at the encoder and H newtop ( t i ) at the decoder can be obtained.
This work divides the model building blocks into two groups: the central and connection building blocks.
In the group of connection blocks, besides up-sampling and down-sampling blocks, we present MA modules: Motion-based Encoding Attention (MEA) and Motion-based Decoding Attention (MDA) modules used in the TrajGRU-Attention model.
In the TrajGRU-Attention-ODE, the MDA modules are not used, and TrajGRU blocks are replaced with the TrajGRU-ODE blocks.
3. T RAJ GRU-A TTENTION
Typically, the current representation state can be computed using the current data input and the previous hidden state at each timestamp when using the traditional recurrent computation.
Since the updating computation at each neural cell takes into account the short-term dependencies of two adjacent data frames, the long-term dependencies between the current input and all previous series are not effectively captured (Vaswani et al., 2017).
This issue leads to the vanishing gradient problem reducing the reconstruction quality, especially with long-range examined sequences.
In the time domain, the evolution and changing processes over time of the spatiotemporal phenomena intrinsically involve the motion characteristics, which are mathematically described in terms of position, velocity, acceleration, and time variables.
Therefore, we leverage the characteristic motion analysis as the key idea to build the MA module.
The main objective of this mechanism is to help the system look back on prior knowledge and pay more attention to valuable parts based on the motion characteristics of the observed hidden states and input frames.
Moreover, by learning the representation states of the motions over time, the system is capable of handling data points sampled at arbitrary given timestamps.
In this work, the MA module is built as a connection building block which can be flexibly inserted into different RNNs.
According to the encoding-decoding model structure, there are also two attention mechanisms: the MEA and MDE.
These mechanisms create the parallelization computation inside the recurrent operation of central blocks (e.g., TrajGRU).
Motion-based Encoding Attention (MEA) mechanism.
From the side of the encoder, the MEA mechanism is proposed for performing the parallelization computations on the input sequence.
The entire step-by-step of applying the MEA is described as follows.
First, at the time step t Q of the input sequence, we derive the formulas used to compute additional states representing the motion characteristics.
By assuming the set of input frames { X ( t i ) } i ≤ Q,i ∈ N (i.e., the set length is Q ) as the position information, we can define the set of velocity and acceleration states as follows
[Equation]
In Equation 4, f V , f A indicate distinct networks: each includes a convolutional layer with a kernel size of (1 × 1) and an average pooling layer.
Thus, the sets { V ( t i ) , A ( t i ) } i ≤ Q,i ∈ N can be represented by tensors V t Q and A t Q , respectively, with the same shape of ( Q × B × C × 1) (where Q denotes the set length, B denotes the batch size, C denotes the number of channels of the state).
The set of time steps T t Q is reshaped from ( Q × B × 1) to ( Q × B × C × 1) by replicating the values.
Second, based on the computations of Luong et al.
, the formulation computing the correlation between the current input data X ( t Q ) with its previous frames { X ( t i ) } i ≤ Q,i ∈ N , is written as
[Equation]
where W a 1 , W a 2 represent the learnable weights of convolutional layers; Avg denotes the average pooling layer.
In Equation 5; the term W a 1 ∗ X ( t Q ) is added to each element in the set of tensors
W a 2 ∗ { X ( t i ) } i ≤ Q,i ∈ N ; then they are stacked together.
Consequently, four tensors V t Q , A t Q , T t Q ,
C t Q are obtained with the same shape of ( Q × B × C × 1) , presenting four keys of the attention.
In particular, those keys are expected to help the model make a decision about how much attention to pay to each input data frame.
Furthermore, since those keys contain information on the changing rates (i.e., the evolution) regardless of regular or irregular sampling assumptions, the model can learn and recognize spatiotemporal representations at specified time steps.
In the next step, based on those keys, we compute the alignment scores, which indicate the collection of weights to assign to each element in the set of input
[Equation]
where f combine represents the linear layer with the number of input channels being (4 × C ) and the number of output channels being ( C ) .
Finally, we assign this score S t Q to the set of input frames { X ( t i ) } i ≤ Q,i ∈ N to generate the attention state and the new input at time step t Q
[Equation]
where ” ◦ ” denotes the Hadamard product, ” ∗ ” denotes the convolutional operator; W 1 × 1 represents the learnable weights of the convolutional layers with a kernel size of (1 × 1) .
Consequently, the new input intrinsically obtains the knowledge of data frames stretching from the beginning timestamp tto the current instant t Q .
Therefore, the MEA helps the encoder possesses a long-term memory and an ability to learn the irregularly sampled time series.
Motion-based Decoding Attention (MDA) mechanism.
From the perspective of the decoder,
MDA is proposed for making parallel computations on the collection of hidden state sequences instead of the input sequences in the case of the MEA.
As described previously, the encoder’s recurrent layers take the attention states generated by implementing the MEA on the input series to create the output hidden form for each timestamp.
Since the decoder uses the representation states from the encoder to make predictions, the same attention mechanism is applied to the set of hidden states in the decoder.
Therefore, the entire attention operations are straightforward, and the information flows are distributed following a common recurrent computation.
Overall, a MA block has a set of learnable parameters coming from five convolutional layers (i.e., f V , f A , W a 1 , W a 2 , W 1 × 1 ) and one linear layer (i.e., f combine ).
The details of MA module settings for the entire model are illustrated in Appendix A.3.
Finally, the TrajGRU-Attention model is constructed by combining the TrajGRU networks and the MA module setting (see Figure 1).
3. T RAJ GRU-A TTENTION -ODE
In the TrajGRU-Attention-ODE structure, the TrajGRU and MDA blocks are replaced by the
TrajGRU-ODE network.
This neural network implements an internal loop to perform the recurrent computation like many conventional RNNs.
However, instead of just using the pair of the input
X ( t i ) at the time step t i and the previous hidden state H ( t i − 1 ) , a new intermediate state D ( t i ) is defined, namely the difference representation; then, it is inserted into the recurrent computation alongside this pair.
In particular, this state is computed using the couple of latent continuous-time states { H ode ( t i − 1 ) , H ode ( t i ) } which is generated by the ODE solver component.
Hence, this state can emphasize the distinctness and the evolution of data from time step t i − 1 to t i .
Eventually, the formulation to compute D ( t i ) can be rewritten as follows
[Equation]
where W odecombine denotes the convolutional layer with a kernel size of 1 × 1 .
More specifically, we will examine a simple task to generate T unseen data frames at T random timestamps using the given initial knowledge.
First, the integrated ODE solver module is defined by a neural network that played a role in the ODE function and a numerical integration method (Chen et al., 2018).
In this work, an ODE function is represented by a sequence of convolutional and tanh activation layers.
Then, this ODE solver will take the initial state to generate the set of continuous-time hidden states { H ode ( t 0 ) , ..., H ode ( t T ) } in the latent space at the expected timestamps from t 0 to t T .
Although those states obtain the continuous-time dynamics, the ODE function (i.e., the ODE neural network) of the ODE solver module is insufficient to capture important spatial features and reconstruct the resulting outcomes with high sharpness simultaneously, especially when the sequence data contain different complicated spatiotemporal properties.
Therefore, these latent continuoustime states are incorporated with the hidden states generated by the extended TrajGRU layer to represent learnable features more effectively.
According to Equation 8, the state D ( t i ) at time step t i , allows the system to foresee how the examined state can evolve at this time step.
After that, the extended TrajGRU will combine this state with other knowledge processed by the gated mechanism to reconstruct the resulting output.
Therefore, the TrajGRU-ODE block (see Figure 2) can simultaneously learn the sequence’s continuous-time and spatiotemporal dynamics.
As a result, this technique can be applied to handle the irregularly sampled time series and perform continuous-time prediction tasks.
The detailed operation formulations of the TrajGRU-ODE unit can be written as follows
[Equation]
t i −
[Equation]
(cid:32)
W xz ∗ X ( t i )
+
L (cid:88) l =
W hz,l ∗ warp ( H ( t i − 1 ) , U ( t i
[Equation]
l ) , V ( t i , l )) + B z (cid:33) ,
R ( t i ) =
σ (cid:32)
W xr ∗ X ( t i )
+
L (cid:88) l =
W hr,l ∗ warp ( H ( t i − 1 ) , U ( t i , l ) , V ( t i , l )) + B r (cid:33) , ˜ H ( t i ) = tanh (cid:32)
W xh ∗ X ( t i )
+ R ( t i ) ◦
L (cid:88) l =
W hh,l ∗ warp ( H ( t i − 1 ) , U ( t i , l ) , V ( t i , l )) + B h (cid:33) ,
H TrajGRU ( t i ) =
Z ( t i ) ◦ H
( t i − 1 )
+ (1 − Z
( t i )) ◦
˜ H ( t i ) ,
Z ode ( t i ) =
σ ( W hzode ∗ H TrajGRU ( t i )
+ W dzode ∗ D ( t i ) + B zode ) ,
R ode ( t i ) =
σ ( W hrode ∗ H TrajGRU ( t i ) + W drode ∗ D ( t i ) + B rode ) , ˜ H ode ( t i )
=
tanh ( W hhode ∗ H TrajGRU ( t i )
+ R ode ( t i ) ◦
( W dhode ∗ D ( t i ))
+ B hode ) ,
H ( t i ) =
Z ode ( t i ) ◦ H TrajGRU ( t i )
[Equation]
From the perspective of the TrajGRU operation, X ( t i ) ∈ R C in × H in × W in , H ( t i − 1 ) ∈ R C
h
× H h
× W h indicate the input data at the time step t i and the previous hidden state, respectively; γ represents the neural network generating the continuous flow fields U ( t i ) , V ( t i ) with L different connection links in the transition; warp represents a function used to combine the sets of flow fields and the hidden state to generate an intermediate state with approximated location information; Z ( t i ) , R ( t i ) indicate the update, reset gates, respectively.
˜ H ( t i ) represents the memory state, which is modified according to the reset gate and information flows at each time step;
H TrajGRU ( t i ) represents the output hidden state of the standard TrajGRU unit.
In terms of the extended gates, the states
H TrajGRU ( t i ) and D ( t i ) will be the input data and the hidden state, respectively; Z ode ( t i ) , R ode ( t i ) denote the additional update and reset gates; ˜ H ode ( t i ) represents the new memory state; eventually, the final output state H ( t i ) contains both continuous-time and spatiotemporal features.
The system has collections of learnable and trainable parameters for the optimizing process: W k , B k .
Finally, “ ∗ ”, as previously, denotes the convolution operator; “ ◦ ” denotes the Hadamard product; σ refers to the sigmoid function; tanh refers to the tanh function.
Overall, a TrajGRU-ODE network includes a TrajGRU network (described in Appendix A.1), three extended gates represented by six convolutional layers, an ODE network represented by two pairs of convolutional and tanh layers, and other activation layers.
From the model structure and architecture perspective, the TrajGRU-Attention-ODE is designed in the same way as the TrajGRU-Attention (see Figure 1).
The main difference between these two models is that the TrajGRU-ODE blocks replace the combination of TrajGRU blocks and MDA modules as the central blocks (i.e., central layers) in the decoder.
Since the TrajGRU-ODE blocks themselves can generate continuous hidden states and reconstruct the outcomes at the expected time steps, the MEA modules are implemented at the encoder to generate the initial representation states.
E XPERIMENTS 4. I MPLEMENTATION
This section implements the TrajGRU-Attention and TrajGRU-Attention-ODE models with three video datasets: the MovingMNIST, MovingMNIST++, and KTH Action.
The detailed configuration of the proposed models is presented in Appendix A.5.
For the training process, we use the combination of Mean Square Error (MSE), Mean Absolute Error (MAE), and Structural Similarity
Index Measure (SSIM), as the loss function, as presented in Appendix A.2.
To compare the performance of the proposed model, we implement the ConvGRU, TrajGRU, and Vid-ODE as the baseline models whose configurations are described in Appendix A.1.
4. D ATASETS
For data preprocessing, each examined sequence is divided into two subsequences (i.e., the input and the target), and then the sampling methods (i.e., regular and irregular sampling) are applied to each subsequence.
The detailed information on sampling assumptions is presented in Appendix A.4.
MovingMNIST.
Essentially, the MovingMNIST illustrates the movements of the handwriting
MNIST digits in 64 × 64 frames.
For data configuration, all data frames consist of two random
MNIST digits; the input is ten frames long, the target is ten frames long, and the time interval between adjacent frames depends on the sampling assumption.
The training set containssequences, the validation set contains 2000 sequences, and the testing set contains 5000 sequences.
MovingMNIST++.
Compared to the MovingMNIST dataset, each frame contains three MNIST digits, and more complicated motion patterns for the MNIST digits are added to all frames, such as random rotations, scale changes, and illumination changes (Shi et al., 2017).
However, similar to the previous dataset, the same preprocessing and sampling methods are used to generate the input and target sequences for the training, validation, and testing sets.
KTH Action.
KTH Action is a video database describing six types of human actions: walking, jogging, running, boxing, hand waving, and clapping (Schuldt et al., 2004).
For data preprocessing, we use 255 videos for training and 144 videos for validating and testing.
Similar to the case of the
MovingMNIST, the input and target lengths are ten, and the two time-sampling methods are also applied for the SSP task.
4. R ESULTS AND C OMMENTS
To test the models’ performance, we use four evaluation metrics: MSE, MAE, SSIM, and PeakSignal-to-Noise Ratio (PSNR) (Hore & Ziou, 2010), as described in Table 1.
In addition, some prediction results using the MovingMNIST are shown in Figures 3, 4, and prediction results using the KTH Action are shown in Figures 5, 6.
Based on the experiments, the results show that the two proposed models achieve better performance than the baseline models.
In particular, the TrajGRU-Attention and TrajGRU-Attention outperform the Vid-ODE for the irregularly sampled time series assumptions.
Moreover, regularly sampled sequence prediction results demonstrate that the MA modules can alleviate the blurry effect caused by the vanishing gradient issue.
Overall, the TrajGRU-Attention yields the best performance in the regular sampling assumption, and the TrajGRU-Attention-ODE achieves the best results when the data is collected at arbitrary timestamps.
Additional comparisons are provided in Appendix A.6.
Datasets
Model
MSE ( ↓ )
MAE( ↓ )
SSIM ( ↑ )
PSNR( ↑ ) × 10× 10×
MovingMNIST
ConvGRU 80 .178 .92 .21 .(regular sampling)
TrajGRU 58 .141.
Vid-ODE 319 .507 .70 .15 .
TrajGRU-Attention 39.
TrajGRU-Attention-ODE 41 .
MovingMNIST
Vid-ODE 356 .
.14 .(irregular sampling)
TrajGRU-Attention 77.
TrajGRU-Attention-ODE 91 .
MovingMNIST++
ConvGRU 107 .298 .
.(regular sampling)
TrajGRU 92 .262 .78 .20 .
Vid-ODE 369 .494 .72 .14 .
TrajGRU-Attention 78.
TrajGRU-Attention-ODE 90 .
MovingMNIST++
Vid-ODE 378 .500 .61 .14 .(irregular sampling)
TrajGRU-Attention 112 .307 .75 .19 .
TrajGRU-Attention-ODE 111.
KTH Action
TrajGRU 22 .183 .87 .27 .(regular sampling)
Vid
[Table]
C ONCLUSION
This paper proposes two sequence-to-sequence models named TrajGRU-Attention and TrajGRUAttention-ODE that can handle irregularly sampled spatiotemporal data.
Overall, the proposed models can perform the overall SSP task with different datasets and outperform the baseline models.
Despite some promising results of our models, there is still room for further improvement in our system.
Furthermore, the combination of attention mechanisms and NODE is very promising since the TrajGRU-Attention-ODE can perform the SSP task effectively in some instances.
E THICS S TATEMENT
We acknowledge that all co-authors of this work have read and committed to adhering to the ICLR
Code of Ethics
R EPRODUCIBILITY S TATEMENT
We strive to ensure the reproducibility of the experiment results.
The entire implementation details are described in Appendix A.1, Appendix A.2, Appendix A.3, Appendix A.5 and Appendix A.
Finally, the source code will be made publicly available
A A PPENDIX A. B ASELINE M ODEL C ONFIGURATION
In this work, three models are implemented as the baseline models: the ConvGRU, TrajGRU
(Shi et al., 2017), and Vid-ODE (Park et al., 2021).
As mentioned earlier, both regular and irregular time-sampling assumptions will be examined for the datasets; thus, ConvGRU and TrajGRU models cannot be used in some cases.
For a fair comparison, we use some common hyperparameters of the ConvGRU, TrajGRU and the proposed models.
With the Vid-ODE, we use the model structure, optimizer, and loss function, described in the Vid-ODE paper ((Park et al., 2021)). We also implemented this model based on their original code.
For the training, validation, and testing processes, we use some hyperparameters like the number of epochs, the number of iterations, and the initial learning rate, similar to the proposed models’ hyperparameters.
Our simulation results and the learning curves in Appendix A.6 show that the number of epochs used is sufficient.
In addition, the configurations of the ConvGRU and TrajGRU models are illustrated in Table 2, and the Vid-ODE model parameters are described in Table 3.
ConvGRU.
The main formulas of the ConvGRU network can be expressed as
[Equation]
where X ( t i ) and
TrajGRU.
The main formulas of the TrajGRU network can be expressed as
[Equation]
[Equation] where γ represents the neural network generating the continuous flow fields U ( t i ) , V ( t i ) with L different connection links in the transition; warp represents a function used to combine the sets of flow fields and the hidden state to generate an intermediate state with approximated location information;
Z ( t i ) , R ( t i ) represent the update gate and reset gate, respectively, regulating the information flows; ˜
H ( t i ) represents the memory state storing new information of the recent input data; ” ∗ ” indicates convolution operator and ” ◦ ” denotes Hadamard product; ” σ ” ” refers to the Sigmoid activation function, ” tanh ” refers the tanh activation function.
For learnable parameters, a TrajGRU cell consists of six convolutional layers ( W xz , W hz , W xr , W hr , W xh , W hh ) and a spatial transformer module γ containing three convolutional layers.
Figure 7 illustrates the overall operations of the ConvGRU and TrajGRU networks, and Figure 8 presents the common structure for the two models.
Vid-ODE.
Vid-ODE essentially combined the standard ConvGRU network, NODE techniques, and the pixel-level composition technique (Park et al., 2021).
From the structure perspective, this model used an encoding-decoding structure for the video generation task.
The encoder is constructed by the ConvGRU layer and a specified ODE neural network which helps capture the irregular sampling intervals; the decoder is constructed by an ODE solver and the pixel-level composition technique. A. T RAINING P ROCESS
Loss function. Generally, the Mean Absolute Error (MAE) and Mean Square Error (MSE) are the commonly used loss functions for sequence prediction tasks. However, many prior studies have mentioned the drawbacks when using these error functions for the training process. Since they compute the global errors between the output and ground truth during the process, the system might ignore some local errors.
Consequently, the sharpness of the resulting images decreased significantly, especially with long-term sequence prediction problems. Therefore, some related research proposed other loss functions instead of using only a simple MAE or MSE (Tran & Song, 2019). Therefore, we propose the combination of the MAE, MSE, and the Structural Similarity Index Measure (SSIM) as the loss function to guide the training process (Hore & Ziou, 2010). With the SSIM metric, we can somewhat alleviate the blurry effect by modelling the distortions in the loss function.
The main formula of the proposed loss function can be derived as
[Equation] TrajGRU ConvGRU Common hyperparameters 3 down-sampling blocks (encoder): (Conv( C dataset × 64 × 3 × 3 ) ,leakyRELU) (Conv( 64 × 96 × 3 × 3 ) ,leakyRELU) (Conv( 96 × 128 × 3 × 3 ) ,leakyRELU) 4 up-sampling blocks (decoder): (Deconv( 128 × 96 × 4 × 4 ) ,leakyRELU) (Deconv( 96 × 64 × 4 × 4 ) ,leakyRELU) (Deconv( 64 × 32 × 3 × 3 ) ,leakyRELU)
Conv( 32 × 1 × 1 × C dataset )
Central building blocks
Encoder : Encoder :
TrajGRU ( C = (64 , 64) , K = 3 , K γ = 1 , L = 9)
ConvGRU ( C = (64 , 64) , K = 3)
TrajGRU ( C = (96 , 96) , K = 3 , K γ = 1 , L = 9)
ConvGRU ( C = (96 , 96) , K = 3)
TrajGRU ( C = (128 , 128) , K = 3 , K γ = 1 , L = 9)
ConvGRU ( C = (128 , 128) , K = 3)
Decoder : Decoder :
TrajGRU ( C = (64 , 64) , K = 3 , K γ = 1 , L = 9)
ConvGRU ( C = (64 , 64) , K = 3)
TrajGRU ( C = (96 , 96) , K = 3 , K γ = 1 , L = 9)
ConvGRU ( C = (96 , 96) , K = 3)
TrajGRU ( C = (128 , 128) , K = 3 , K γ = 1 , L = 9)
ConvGRU ( C = (128 , 128) , K = 3)
Total number of parameters 4859181 (MovingMNIST), 3695553 (MovingMNIST), 4860367 (KTH Action).
3696739 (KTH Action).
Here, X target , X prediction represent the target and output prediction sequences, respectively; a = 1 , b = 1 , c = 0 .
05 denote the weights of the three elements of the loss function.
The weights of the MSE and MAE are equal and more significant than the SSIM because MSE and MAE are supported to be the main elements manipulating the global errors, and the SSIM is the supportive factor impacting the local errors.
The comparison between different loss functions is shown in Table
Based on Table 4, our proposed loss function helps the model generate the results with the best evaluation metrics, so we choose this loss function as the standard function to train the models (i.e., ConvGRU, TrajGRU, ConvGRU-Attention, ConvGRU-Attention-ODE, TrajGRU-Attention, and TrajGRU-Attention-ODE).
Optimizer.
This work uses the Adam optimizer to train all the models because of its stability and adaptive features (Kingma & Ba, 2014).
In addition, the training processes of all models used the early stopping mechanism to obtain the final results. The initial learning rate is 10 − 4 , which is decayed by 0 . 99 after every epoch. For all models, we set the maximum number of epochs to 25 .
In addition, the early stopping mechanism is applied to obtain the final results and the corresponding set of trainable parameters (Prechelt, 1998).
A. M OTION - BASED A TTENTION S ETTING
In the TrajGRU-Attention model, the MEA blocks are used at the encoder, and the MDE blocks are used at the decoder to handle the irregularly sampled time series.
Moreover, the model can deal with long-range sequence prediction with these attention mechanisms.
However, if the MEA and MDE modules are integrated into all central blocks of the model, there will be considerable computational complexity with a massive set of trainable parameters.
Therefore, we just insert those attention modules on specific layers to balance the overall performance and the complexity.
Encoder: 4 down-sampling blocks: (Conv( 64 × 64 × 3 × 3 ), BatchNorm, RELU), (Conv( 64 × 128 × 4 × 4 ), BatchNorm, RELU), (Conv( 128 × 256 × 4 × 4 ), BatchNorm, RELU), (Conv( 256 × 512 × 4 × 4 ), BatchNorm, RELU).
ConvGRU ( C = (512 , 512) , K = 3)
ODE network: (Conv( 512 × 128 × 3 × 3 ), Tanh), (Conv( 256 × 256 × 3 × 3 ), Tanh), (Conv( 256 × 256 × 3 × 3 ), Tanh), (Conv( 256 × 256 × 3 × 3 ), Tanh), (Conv( 256 × 256 × 3 × 3 ), Tanh).
ODE method: Euler.
Decoder:
ODE network: (Conv( 512 × 128 × 3 × 3 ), Tanh), (Conv( 256 × 256 × 3 × 3 ), Tanh), (Conv( 256 × 256 × 3 × 3 ), Tanh), (Conv( 256 × 256 × 3 × 3 ), Tanh), (Conv( 256 × 256 × 3 × 3 ), Tanh).
4 up-sampling blocks: (Bilinear(scale=2), Conv( 512 × 256 × 3 × 3 ),BatchNorm,RELU), (Bilinear(scale=2), Conv( 256 × 128 × 3 × 3 ),BatchNorm,RELU), (Bilinear(scale=2), Conv( 128 × 64 × 3 × 3 ),BatchNorm,RELU), (Bilinear(scale=2), Conv( 64 × 32 × 3 × 3 ),BatchNorm,RELU), Conv( 32 × C dataset × 3 × 3 ).
ODE method: Euler.
GAN network:
Generator: Encoder and Decoder.
Discriminator: (Conv( 1 × 64 × 4 × 4 ), InstanceNorm, leakyRELU), (Conv( 64 × 128 × 4 × 4 ), InstanceNorm, leakyRELU), (Conv( 128 × 256 × 4 × 4 ), InstanceNorm, leakyRELU), (Conv( 256 × 512 × 4 × 4 ), InstanceNorm, leakyRELU),
Conv( 512 × 64 × 4 × 4 ).
Total number of parameters: 60060164 (MovingMNIST), 60067078 (KTH Action).
[Table]
To choose the optimal attention mechanism settings, the TrajGRU-Attention model is tuned using the MovingMNIST data in the scenario where the numbers of attention modules vary and the other hyperparameters are constant.
In addition, the training process is described in Appendix A.2.
Table 5 illustrates the resulting evaluation metrics of the TrajGRU-Attention model with different attention settings when training and evaluating with the MovingMNIST dataset.
For instance, when the encoder uses the attention module at its central layer 3 and the decoder uses the attention module at its central layer 3 , we abbreviate this setting as “en-a3-de-a3”.
Based on Table 5, the setting “en-a23-de-a21” shows the best results in all four metrics, so we chose this setting to design the
TrajGRU-Attention and TrajGRU-Attention-ODE models, as presented in Appendix A.5.
A. T IME S AMPLING M ETHODS .
For the overall SSP task, each series is divided into two subsequences (i.e., the input and the target) and then the sampling methods are applied to each subsequence.
We define two collections of timestamps representing the regular and irregular sampling assumptions, as shown in Table 6.
[Table] [Table]
In Table 6, the set size represents the number of distinct collections of time steps, and the input and target time intervals denote the minimum and maximum time steps for each sampling method.
In the experiments shown in Section 4, the input and target sequences have the same length of 10 .
A. P ROPOSED M ODEL C ONFIGURATION
Table 7 describes the two proposed models’ overall configurations and important hyperparameters.
Generally, we use the TrajGRU network containing the convolutional layers that extract the image features of each data frame, as illustrated in Figures 2 and 7.
In addition, for the hierarchical feature extraction, we insert a resampling block (i [Table] e., an upsampling or a downsampling block) between two adjacent central blocks (i.e., central neural layers) for the SSP task to help the system capture spatial features with different levels.
We build the up-sampling block using convolutional and LeakyRELU activation layers in the proposed models.
In contrast, the down-sampling block is constructed by a pair of deconvolutional and LeakyRELU activation layers.
As such, the shape of the output states can be easily adjusted by defining the hyper-parameters of the convolutional and deconvolutional layers (e.g., the kernel size, number of input and output channels, padding, stride, dilation).
TrajGRU-Attention
TrajGRU-Attention-ODE
Common hyperparameters 3 down-sampling blocks (encoder): (Conv( C dataset × 64 × 3 × 3 ) ,leakyRELU) (Conv( 64 × 96 × 3 × 3 ) ,leakyRELU) (Conv( 96 × 128 × 3 × 3 ) ,leakyRELU)
Zigzag connection (encoder): (Deconv( 128 × 64 × 4 × 4 ), Conv( 64 × 64 × 3 × 3 ) ,
Conv( 64 × 64 × 3 × 3 ))
4 up-sampling blocks (decoder): (Deconv( 128 × 96 × 4 × 4 ) ,leakyRELU) (Deconv( 96 × 64 × 4 × 4 ) ,leakyRELU) (Deconv( 64 × 32 × 3 × 3 ) ,leakyRELU)
Conv( 32 × 1 × 1 × C dataset )
Zigzag connection (decoder): (Conv( 64 × 128 × 4 × 4 ), Conv( 128 × 128 × 3 × 3 ) ,
Conv( 128 × 128 × 3 × 3 ))
Central building blocks
Encoder :
TrajGRU( C = (64 , 64) , K = 3 , K γ = 1 , L = 9 ) TrajGRU( C = (64 , 64) , K = 3 , K γ = 1 , L = 9 )
TrajGRU( C = (96 , 96) , K = 3 , K γ = 1 , L = 9 ) TrajGRU( C = (96 , 96) , K = 3 , K γ = 1 , L = 9 )
TrajGRU( C = (128 , 128) , K = 3 , K γ = 1 , L = 9 ) TrajGRU( C = (128 , 128) , K = 3 , K γ = 1 , L = 9 )
Decoder :
TrajGRU( C = (64 , 64) , K = 3 , K γ = 1 , L = 9 )
TrajGRU-ODE( C = (64 , 64) , K = 3 , K γ = 1 ,
L = 9 , n ode = 2 , K ode = 3 , method ode = Euler )
TrajGRU( C = (96 , 96) , K = 3 , K γ = 1 , L = 9 )
TrajGRU-ODE( C = (96 , 96) , K = 3 , K γ
=
L = 9 , n ode = 2 , K ode = 3 , method ode = Euler )
TrajGRU( C = (128 , 128) , K = 3 , K γ = 1 , L = 9 )
TrajGRU-ODE( C = (128 , 128) , K = 3 , K γ
=
L = 9 , n ode = 2 , K ode = 3 , method ode = Euler )
Attention blocks
Encoder :
MEA ( 4 conv ( C = (96 , 96) , K = 1) ;
MEA ( 4 conv ( C = (96 , 96) , K = 1) ; 1 linear ( C = (384 , 96))
1 linear ( C = (384 , 96))
MEA ( 4 conv ( C = (128 , 128) , K = 1) ;
MEA ( 4 conv ( C = (128 , 128) , K = 1) ; 1 linear ( C = (512 , 128))
1 linear ( C = (512 , 128))
Decoder :
MDA ( 4 conv ( C = (96 , 96) , K = 1) ; 1 linear ( C = (384 , 96))
MDA ( 4 conv ( C = (64 , 64) , K = 1) ; 1 linear ( C = (256 , 64))
Total number of parameters 7258509 (MovingMNIST) 8844685 (MovingMNIST) 7259695 (KTH Action) 8845871 (KTH Action)
A. A DDITIONAL C OMPARISON B ETWEEN M ODELS
As shown in Table 1, the Vid-ODE can not perform the SSP task properly in some scenarios, especially with the MovingMNIST and MovingMNIST++.
The reasons leading to the limitations of
Vid-ODE’s performance are the time sampling assumptions and the properties of the datasets (i.e., the MovingMNIST and MovingMNIST++).
In the Vid-ODE paper, the authors used the concept of frame rate that can be fixed in the encoding and decoding parts (e.g., the encoder’s frame rate is 1 , and the decoder’s frame rate is 0 . 5 ).
Regarding the model structure, the composition mask and image difference techniques are ineffective when the time gap between adjacent frames is large and variable.
Furthermore, the Vid-ODE decoder uses only an ODE solver component as the central layer to generate hidden states; thus, these states may not contain enough spatial features to reconstruct the output.
On the other hand, the proposed models use multiple TrajGRU-ODE networks (the combination of TrajGRU networks and ODE solvers) at the TrajGRU-Attention-ODE model, and groups of TrajGRU networks and Motion-based Attention blocks at the TrajGRU-Attention model.
For a fair comparison, the Vid-ODE and proposed models are implemented with the KTH Action dataset for the SSP task where the input sequence has 5 frames, and the target sequence has 5 frames which is the setting used in Vid-ODE paper (Park et al.
Finally, the evaluation metrics are displayed in Table 8, and the prediction results are shown in Figure 9.
As shown in Table 8, the Vid-ODE model performs more effectively with short-range prediction; however, the proposed models generate better prediction results, especially with the irregular time sampling assumption.
To clarify the improvement of the proposed modules (i.e., the MA and TrajGRU-ODE network), we compare the proposed models (i.e., TrajGRU-Attention and TrajGRU-Attention-ODE) with the
ConvGRU-Attention and ConvGRU-Attention-ODE models.
Basically, the ConvGRU-Attention and ConvGRU-Attention-ODE use the ConvGRU networks illustrated in Table 2 as the central building blocks instead of the TrajGRU networks.
These models are trained, validated, and tested with the MovingMNIST and MovingMNIST++ datasets.
Finally, the evaluation metrics are displayed in
Table 9, and the prediction results are shown in Figure 10.
As shown in Tables 9 and 1, the MA modules can improve the performance of the ConvRNNs (e.g.,
ConvGRU and TrajGRU) and the combinations of the ConvRNNs and ODE modules can allow the models to perform the SSP task with data frames at arbitrary time steps.
Compared to the ConvGRUAttention and ConvGRU-Attention-ODE, the proposed models can achieve better prediction results.
[Table] [Table]
282 .76 .20 .(regular sampling)
ConvGRU-Attention-ODE 110 .301 .75 .19 .
TrajGRU
[Table]
Attention 78.
TrajGRU
[Table]
Attention
[Table]
ODE 90 .260 .78 .20 .
MovingMNIST++
ConvGRU
[Table]
Attention 134 .340 .72 .18 .(irregular sampling)
[Table]
ODE 139 .351 .71 .18 .
[Table]
[Table]
[Table]
[Table]
[Table]
In addition, the learning curves of the models used in Section 4, are shown in Figures 11, 12, 13, 14.
They can evaluate models’ learning ability.
Overall, the TrajGRU-Attention and TrajGRUAttention-ODE models achieve better performance during the training and validation processes.