From: ICLR.cc/2023/Conference/Paper3336/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper3336/Reviewer_anii
title: Thanks for the response
comment: Thanks for the response!

After adjusting the placement of Figure 2 in the original manuscript, the language description of the architecture is much clearer now. However, I would still suggest the authors explain the architecture together with Figure 1 (in the current manuscript) which is visually helpful. 

One more thing is about the background of the work. Though there is one paragraph dedicated to discussing Neural ODE, it is still not clear to me what exactly Neural ODE does. Probably, something similar to the background section in Latent-ODE[1] would help. 

[1] Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for irregularly-sampled time series. Advances in neural information processing systems, 32, 2019.

From: ICLR.cc/2023/Conference/Paper3336/Authors
title: A Gentle Reminder
comment: Dear Reviewer jDYn

First, we would like to thank you very much for your helpful comments.

Second, we wonder if our answers can satisfy your questions and comments. Please let us know if you have any further comments. We will try to address them (before the rebuttal deadline ends on Dec 12).   

Thanks!

The Authors

From: ICLR.cc/2023/Conference/Paper3336/Authors
title: A Gentle Reminder
comment: Dear Reviewer anii

First, we would like to thank you very much for your helpful comments.

Second, we wonder if our answers can satisfy your questions and comments. Please let us know if you have any further comments. We will try to address them (before the rebuttal deadline ends on Dec 12).   

Thanks!

The Authors

From: ICLR.cc/2023/Conference/Paper3336/Authors
title: A Gentle Reminder
comment: Dear Reviewer Jeez

First, we would like to thank you very much for your helpful comments.

Second, we wonder if our answers can satisfy your questions and comments. Please let us know if you have any further comments. We will try to address them (before the rebuttal deadline ends on Dec 12).   

Thanks!

The Authors

From: ICLR.cc/2023/Conference/Paper3336/Authors
title: A gentle reminder
comment: Dear Reviewer aiiz,

First, we would like to thank you very much for your helpful comments.

Second, we wonder if our answers can satisfy your questions and comments. Please let us know if you have any further comments. We will try to address them (before the rebuttal deadline ends on Dec 12).   

Thanks!

The Authors

From: ICLR.cc/2023/Conference/Paper3336/Authors
title: Response to Review anii
comment: First, we would like to thank you for your reviews.
Second, we will try to clarify some issues you mentioned by adjusting and providing more related information.
1) Some terms are not clear
- “Neural unit”: Generally, a recurrent network contains multiple neural units, and the number of units is equal to the number of data points of the examined sequence.
- “Top layer”: With the stacked architecture, multiple recurrent networks are stacked, and each recurrent network is a layer with a specific level. The highest-level layer is the top layer and the lowest-level layer is the bottom layer.
- “zigzag connection”: At the encoder, it is the connection between the neural unit of the bottom at the current timestamp with the neural unit of the top layer at the next timestamp. At the decoder, it is the connection between the neural unit of the top at the current timestamp with the neural unit of the bottom layer at the next timestamp.
-  We removed the term “left-right approach” and replaced it by the term “recurrent computation.”. Generally, RNNs use recurrent computation to generate hidden states at each time step, and it works as an internal loop. 
2) Some expressions are not clear
- We changed the expression ”these latent continuous-time states are incorporated with the hidden states generated by the extended TrajGRU layer to get the work done more effectively” to “these latent continuous-time states are incorporated with the hidden states generated by the extended TrajGRU layer to represent learnable features more effectively.”. In the TrajGRU-ODE network, there are two main parts: the TrajGRU network and the ODE part. First, at each time step t(i), the TrajGRU network processes the input data X(i) and previous hidden state H(i-1) to generate the current hidden state, called “TrajGRU state” H_TrajGRU(i). This state contains spatiotemporal features of the current hidden state H(i). At the same time, the ODE solver at the ODE part generates ODE states H_ODE(i-1), and H_ODE(I), representing the continuous-time dynamics. Finally, H_TrajGRU(i), H_ODE(i-1), and H_ODE(i) are processed using the gated technique to generate the hidden state H(i). Therefore, the TrajGRU-ODE network can learn the sequence's continuous-time and spatiotemporal dynamics simultaneously.  
- We changed the expression “the learnable ODE function of the ODE solver module is not sufficient to capture practical features and reconstruct the resulting outcomes with high sharpness simultaneously” to “the ODE function (i.e., the ODE neural network) of the ODE solver module is insufficient to capture important spatial features and reconstruct the resulting outcomes with high sharpness simultaneously, especially when the sequence data contain different complicated spatiotemporal properties.”. In this expression, “important spatial features” are contained in the ODE states generated by the ODE neural network.
3) Ablation study
- We do further ablation study on the ConvGRU-ODE and ConvGRU-Attention-ODE models with the MovingMNIST and MovingMNIST++ datasets. Generally, a ConvGRU-ODE network combines a ConvGRU network and an ODE component. This information is added in Appendix Additional Comparison Between Models, Table 9, pages 18-19.
=> We have some observations: the MA modules can improve the performance of the ConvRNNs (e.g., ConvGRU and TrajGRU), and the combinations of the ConvRNNs and ODE modules can allow the models to perform the SSP task with data frames at arbitrary time steps. Compared to the ConvGRU-Attention and ConvGRU-Attention-ODE, the proposed models can achieve better prediction results.

Once again, we would like to thank the reviewer for providing valuable feedback to improve the work. Please let us know if you have any further comments.

From: ICLR.cc/2023/Conference/Paper3336/Reviewer_jDYn
title: Thank you for your response
comment: Thank you for your response. Please add those clarifications to the revision.

From: ICLR.cc/2023/Conference/Paper3336/Authors
title: Response to Reviewer Jeez
comment: First, we would like to thank the reviewer for the helpful comments. 
Second, we will try to clarify the issues raised by the reviewer by modifying the paper and providing more information.

1) Ablation studies
- With regard to the lack of detailed ablative studies, we have designed the ConvGRU, Attention, and ODE components. We are currently implementing the ConvGRU-ODE and ConvGRU-Attention-ODE with the MovingMNIST dataset. The training, validation, and testing time took about 1 to 2 days to finish.  We will then compare them with our proposed models.
=> A table comparing the results of four models (ConvGRU-Attention, ConvGRU-Attention-ODE, TrajGRU-Attention, TrajGRU-Attention-ODE) will be added to Appendix Proposed Model Configuration.

2) The Gradient Highway Unit (GHU) in PredRNN++ can also alleviate the impact of vanishing gradients. Does the MA achieve better effectiveness than the GHU? Since both the GHU and MA are designed to mitigate the vanishing gradient, it's necessary to compare these two modules concisely.
- We have read that The Gradient Highway Unit (GHU) in PredRNN++ can also alleviate the impact of vanishing gradients. We will implement the PredRNN++ and compare this model and the proposed techniques in future work. Unfortunately, we cannot make the comparison because of the time limitation. Still, one important benefit of our Motion-based Attention mechanisms is that they can handle irregularly sampled time series.

3) This proposed model has not validated its effectiveness on more complex datasets like Human3.6M or Bair, while only doing experiments on simple digit datasets or KTH. On the other hand, the proposed model did not compare with novel state-of-the-art long-term spatiotemporal prediction models like LMC[1]. The LMC achieves competitive performance even predicting the next 100 frames. This paper did not show the effectiveness of extreme long-term performance.
- In this work, we choose three datasets: MovingMNIST, MovingMNIST++, and KTH Action, according to the ascending order of complexity level. Indeed, real-world video datasets like Penn Action, CAM5, Human3.6M, or Bair have more complicated spatial features than the MovingMNIST, MovingMNIST++, and KTH Action. However, in terms of the temporal references and the correlation between the spatial and temporal properties, irregularly sampled MovingMNIST++ sequences can contain some complicated features and properties because they contain the rotation, illusion, and scaling effects, and their changing rates are also relatively fast. We also plan to use the proposed methods with more complicated datasets like weather radar data (e.g., HKO-7 and TAASRAD19). This work focuses on the spatiotemporal sequence prediction task with different time sampling methods (regular and irregular time sampling), so we did not investigate the extreme long-term performance. With two subsequences and two time-sampling assumptions, we have analyzed all four scenarios with the proposed models, and we just included the results of two scenarios in this paper. In future works, we will try to implement the proposed methods with long-term sequences and compare them with state-of-the-art long-term spatiotemporal prediction models like LMC.

4) It seems that the ODE mechanism can be applied to other RNN-based models. Since the paper only uses the existing GRU-based model trajGRU as the backbone, to strengthen the novelty of the paper, I suggest that the author should do further ablation study on different backbones.
- We will do further ablation study on the ConvGRU-ODE and ConvGRU-Attention-ODE with the MovingMNIST and MovingMNIST++ datasets. 
=> A table comparing the results of four models (ConvGRU-Attention, ConvGRU-Attention-ODE, TrajGRU-Attention, TrajGRU-Attention-ODE) will be added to Appendix Proposed Model Configuration.

Once again, we would like to thank the reviewer for providing valuable feedback to improve the work. Please let us know if you have any further comments.

From: ICLR.cc/2023/Conference/Paper3336/Authors
title: Response to Reviewer aiiz
comment: First, we would like to thank the reviewer for the helpful comments. 
Second, we will try to clarify the issues raised by the reviewer by modifying the paper and providing more information.

1) While the authors demonstrate superior performance, it is hard to conclude the reason for the superiority. Specifically, the authors lack detailed ablative studies. 
- We are currently implementing the ConvGRU-ODE and ConvGRU-Attention-ODE with the MovingMNIST, MovingMNIST++ datasets and two time-sampling assumptions (regular and irregular sampling assumptions). The training, validation and testing times take about 2 to 3 days. We will then compare them with our proposed models. 
=> A comparison the results of four models (ConvGRU-Attention, ConvGRU-Attention-ODE, TrajGRU-Attention, TrajGRU-Attention-ODE) will be added to Appendix Proposed Model Configuration.

2) Limitations of Vid-ODE: 
- The main reasons leading to the limitations of Vid-ODE’s performance are the time sampling assumptions and the properties of the datasets (MovingMNIST and MovingMNIST++). In the Vid-ODE paper, the authors used the concept of frame rate that can be fixed in the encoding and decoding parts (e.g., the encoder’s frame rate is 1, and the decoder’s frame rate is 0.5). However, compared to the KTH Action and Penn Action datasets, the time gaps of the MovingMNIST and MovingMNIST++ are larger, and they are arbitrary. 
- In terms of the model structure, the composition mask and image difference techniques are not effective when the time gap between adjacent frames is large and variable. Furthermore, the Vid-ODE decoder uses only an ODE solver component as the central layer to generate hidden states; thus, these states may not contain enough spatial features to reconstruct the output. On the other hand, our models use multiple TrajGRU-ODE networks (the combination of TrajGRU networks and ODE solvers) at the TrajGRU-Attention-ODE model, and groups of TrajGRU networks and Motion-based Attention blocks at the TrajGRU-Attention model.

3) The authors should justify why they do not consider the same experimental setting as Vid-ODE. Specifically, they conduct additional experiments on Penn Action and CAM5, which are real-world videos instead of synthetic video datasets.
- With regard to the experimental setting of Vid-ODE, we used the same model structure and implemented this model based on their source code. For the datasets, we selected the KTH-Action, also used in their paper to compare different models. Real-world video datasets like Penn Action and CAM5 have indeed more complicated spatial features than MovingMNIST, MovingMNIST++, and KTH Action. However, in terms of temporal references and correlation between the spatial and temporal properties, irregularly sampled MovingMNIST++ sequences can contain some complicated features and properties because they contain the rotation, illusion, and scaling effects, and their changing rates are also relatively fast. We also plan to implement the methods with more complicated datasets like weather radar data (e.g., HKO-7 and TAASRAD19).  
=> This discussion will be added to Appendix Baseline Model Configuration.

4) The authors mentioned that the baseline ViD-ODE is implemented by themselves. Is the reproduced performance similar to the ones reported by Park et al., 2021?
- The Vid-ODE model was implemented using the source code from the paper's authors. The performance of Vid-ODE we obtained is not similar to what  Park et al. reported because we used different time sampling assumptions and sequence lengths (10 frames for input and target sequence instead of 5 frames). 
=> To further clarify this issue, we will implement three models (Vid-ODE, TrajGRU-Attention, TrajGRU-Attention-ODE) in the scenario (the input sequence has 5 frames and target sequence has 5 frames) with the KTH Action dataset. 

5) The image feature extractions.
- Generally, we use the TrajGRU network containing the convolutional layers that extract the image features of each data frame, as illustrated in Figures 1 and 7. In addition, for the hierarchical feature extraction, we insert a resampling block between two adjacent central blocks (i.e., central neural layers) for the SSP task to help the system capture spatial features with different levels. We implement the up-sampling block using convolutional and LeakyRELU activation layers in the proposed models. In contrast, the down-sampling block consists of a pair of deconvolutional and LeakyRELU activation layers. Hence, the shape of the output states can be easily adjusted by defining the hyper-parameters of the convolutional and deconvolutional layers (e.g., the kernel size, number of input and output channels, padding, stride, dilation).
=> This discussion will be added to Appendix Proposed Model Configuration

Once again, we would like to thank the reviewer for providing valuable feedback to improve the work. Please let us know if you have any further comments.

From: ICLR.cc/2023/Conference/Paper3336/Authors
title: Response to Reviewer jDYn 
comment: First, we would like to thank the reviewer for the helpful comments. 

Second, we will try to clarify some issues you mentioned by adjusting and providing more related information.

1) The baselines are from their own implementation, so it is really hard to assess whether the claimed performance improvement is real. For example, in the Vid-ODE paper, Vid-ODE's PSNR on the KTH Action dataset is 28.19, which is higher than the numbers shown in this paper.

- The baseline models are implemented based on the Vid-ODE paper (Park et al., 2021) ‘s original source code using the same model structure, optimizer and loss function. For the training, validation, and testing processes, we use hyperparameters like the number of epochs, the number of iterations, and the initial learning rate, similar to the proposed models’ hyperparameters. Our simulation results and the learning curves in Appendix A.6 show that the number of epochs used is sufficient. 
- The main reasons leading to the limitations of Vid-ODE’s performance are the time sampling assumptions and the properties of the MovingMNIST and MovingMNIST++ datasets. In the Vid-ODE paper, the authors used the concepts of frame rate that can be fixed in the encoding and decoding parts (e.g., the encoder’s frame rate is 1, and the decoder’s frame rate is 0.5. Furthermore, the Vid-ODE decoder uses only an ODE solver component as the central layer to generate hidden states; thus, these states may not contain enough spatial features to reconstruct the output. On the other hand, our models use multiple TrajGRU-ODE networks (the combination of TrajGRU networks and ODE solvers) at the TrajGRU-Attention-ODE model, and groups of TrajGRU networks and Motion-based Attention blocks at the TrajGRU-Attention model. 
- In terms of the KTH Action dataset, the performance of Vid-ODE we obtained is not similar to what  Park et al. reported because we used different time sampling assumptions and different sequence lengths (10 frames for input and target sequence instead of 5 frames). 
=> To further clarify this issue, we will implement three models (Vid-ODE, TrajGRU-Attention, TrajGRU-Attention-ODE) in the scenario (the input sequence has 5 frames and the target sequence has 5 frames) with the KTH Action dataset. This discussion will be added to Appendix Additional Comparison Between Models.

2) Please adjust the placement of the figures. For example, Figure 2 is referenced on Page 3, but the figure is placed on Page 7.
- We will recheck the paper and adjust the placement of the figures. We will release our source code and put the link into the paper. 

Once again, we would like to thank the reviewer for providing valuable feedback to improve the work. Please let us know if you have any further comments.

From: ICLR.cc/2023/Conference/Paper3336/Reviewer_aiiz
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The authors propose sequence-to-sequence models based on the Trajectory Gated Recurrent Unit (TrajGRU) network for long-term spatiotemporal sequence prediction tasks with irregular time sampling assumptions. Specifically, they propose a Motion-based Attention module that alleviates the vanishing gradient's impact. Then, they utilize Neural Ordinary Differential Equation (NODE) technique for continuous time prediction tasks. They benchmark the proposed methods on MovingMNIST, MovingM- NIST++, and KTH Action. They perform superior comparisons with different baselines under regular and irregular time sampling.
strength_and_weaknesses: Strength
1. Spatiotemporal Sequence Predictive (SSP) learning has been an essential research topic and plays a critical role in many real-world applications. In this work, the authors study if there is a model that can handle two types of time sampling in temporal analysis, i.e., regular time sampling (i.e., the time interval is assumed to be constant) and irregular time sampling (i.e., the time interval is considered arbitrary). Having an SSP model that can handle the two sampling strategies is valuable.
2. The authors proposed two predictive models, i.e., TrajGRU- Attention and TrajGRU-Attention-ODE. Empirically, the proposed methods outperform different baselines under both regular and irregular time sampling on MovingMNIST, MovingM- NIST++, and KTH Action.

Weaknesses
1. While the authors demonstrate superior performance, it is hard to conclude the reason for the superiority. Specifically, the authors lack detailed ablative studies. For instance, what is the performance of ConvGRU + ODE with attention and TrajGRU + ODE with attention? In other words, there are many design choices suggested in this paper. The authors should conduct a detailed analysis to identify the reason for the superiority. The proposed method gives the reviewer the impression that it is a piece of highly-crafted machinery for specific SSP tasks.
2. The authors should provide detailed discussions on Vid-ODE. Specifically, what could the design limitations of Vid-ODE that lead to inferior performance on the proposed benchmark?
3. The authors should justify why they do not consider the same experimental setting as Vid-ODE. Specifically, they conduct additional experiments on Penn Action and CAM5, which are real-world videos instead of synthetic video datasets.
4. The authors mentioned that the baseline ViD-ODE is implemented by themselves. Is the reproduced performance similar to the ones reported by Park et al., 2021?
5. The authors did not specify the architecture used for image feature extractions.

clarity,_quality,_novelty_and_reproducibility: Overall, the reviewer agrees that the work tackles an essential problem for spatial-temporal prediction. However, several parts require further justifications by the authors to conclude the contributions of the work. 
summary_of_the_review: Spatiotemporal Sequence Predictive (SSP) learning has been an essential research topic and plays a critical role in many real-world applications. In this work, the authors study if there is a model that can handle two types of time sampling in temporal analysis, i.e., regular time sampling (i.e., the time interval is assumed to be constant) and irregular time sampling (i.e., the time interval is considered arbitrary). Having an SSP model that can handle the two sampling strategies is valuable.

However, several parts require further justifications by the authors to conclude the contributions of the work. In particular, a detailed comparison with Vid-ODE is missing. The reviewer suggests the authors provide the corresponding justifications to prove the importance of the work.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: Not applicable
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3336/Reviewer_Jeez
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper has proposed a seq-to-seq model based on the TrajGRU, including a novel attention mechanism Motion-based Attention (MA). This model also leverages the NODE technique into the TrajGRU-Attention model. This TrajGRU-Attention-ODE model can be applied in continuous-time applications and outperforms the SOTA NODE model for several tasks with different circumstances of time sampling.
strength_and_weaknesses: Strength:

- This paper has tried to leverage the NODE into the spatiotemporal predictive models to perform continuous-time spatiotemporal prediction tasks and achieved better performance among SOTA NODE models. 
- This proposes a novel Motion-based Attention to alleviate the impact of vanishing gradients.
- This paper conducts extensive experiments on various video datasets like Moving MNIST, Moving MNIST++, and KTH to prove that the proposed models can generate predictions properly with high accuracy.


Weakness:
- The author has claimed that the MA can be applied to different RNN models. However, It seems that the paper did not conduct any ablation study to show the effectiveness of MA applying to different RNNs. The author should apply the MA with different RNN backbones like PredRNN, ConvLSTM. 

- The Gradient Highway Unit (GHU) in PredRNN++ can also alleviate the impact of vanishing gradients. Does the MA achieve better effectiveness than the GHU? Since both the GHU and MA are designed to mitigate the vanishing gradient, it's necessary to compare these two modules concisely.

- This proposed model has not validated its effectiveness on more complex datasets like Human3.6M or Bair, while only doing experiments on simple digit datasets or KTH. On the other hand, the proposed model did not compare with novel state-of-the-art long-term spatiotemporal prediction models like LMC[1]. The LMC achieves competitive performance even predicting the next 100 frames. This paper did not show the effectiveness of extreme long-term performance.

- It seems that the ODE mechanism can be applied to other RNN-based models. Since the paper only uses the existing GRU-based model trajGRU as the backbone, to strengthen the novelty of the paper, I suggest that the author should do further ablation study on different backbones.

[1] Video prediction recalling long-term motion context via memory alignment learning, CVPR2021

- 


clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is need to improve on clarity.

Quality, originality: Combining with NODE is somewhat novel.
summary_of_the_review: This paper has proposed a novel seq-to-seq long-term spatiotemporal predictive model.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3336/Reviewer_anii
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper presents predictive models for long-term spatiotemporal sequence prediction. The model is built upon TrajGRU with two extra components, namely, Motion-based Attention and Neural ODE. The former is to alleviate the impact of the vanishing gradient, the latter enables the model to handle irregular sampling data. 
strength_and_weaknesses: Strength:
1. The motion-based attention module is flexible and can be integrated into other existing approaches.
2. The problem of prediction on irregularly sampled time series is important. 


Weaknesses:
1. The major weakness is clarity. The writing of the paper is confusing and probably prevents the reader from understanding the underlying technical merit. Here are some examples:
    1. The paper needs to be more self-contained. While the approach is built upon TrajGRU, it is difficult for the reader (at least for me) to understand what exactly is ‘the neural unit of the top layer’, ‘horizontal’, ‘vertical’, ‘zigzag connection’, etc. without a reference. Simply from the language description, it is difficult to infer the intuition behind the design choice. A proper background section is required, at least, for people to fully appreciate the contribution of the paper. 
    2. The description of TrajGRU-Attention and TrajGRU-Attention-ODE suffers from the same issue. Some terms borrowed from previous works are confusing, e.g. left-right approach. 
    3. Certain expressions are also confusing, for example, 
        * ‘these latent continuous-time states are incorporated with the hidden states generated by the extended TrajGRU layer to get the work done more effectively', what does that mean by ‘get the work done’ here?
        * ‘the learnable ODE function of the ODE solver module is not sufficient to capture practical features and reconstruct the resulting outcomes with high sharpness simultaneously’ What does practical feature mean here?

Minor points:
1. It is better to use consistent punctuation, for example, the full stop sign at the end of captions.
2 ‘Therefore, the TrajGRU-ODE block (see Figure 1) can simultaneously learn the sequence’s continuous-time and spatiotemporal dynamics simultaneously.’, ‘simultaneously’ is duplicated. 

clarity,_quality,_novelty_and_reproducibility: Clarity: There is large room for the paper to improve on clarity. The paper in its current form is hardly self-contained. 

Quality, originality: Other than the writing, the paper looks fine except that using attention is not entirely new in sequence prediction.
summary_of_the_review: Given the current form of the paper, it is hard for me to have a clear understanding of the technical contribution. It can potentially be an impactful paper, however, it is difficult to judge the significance of the proposed method with the current manuscript.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3336/Reviewer_jDYn
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper presents the TrajGRU-Attention-ODE model, which is an extension to the traditional TrajGRU model, for spatiotemporal sequence prediction. The authors point out that the TrajGRU model has the drawback of not being able to capture long-range series. To address this problem, the authors proposed the TrajGRU-Attention model with Motion-based Encoding Attention (MEA) and Motion-based Decoding Attention (MDA). The TrajGRU-Attention-ODE model further extends the TrajGRU-Attention model by using Ordinary Differential Equation (ODE).

The authors evaluated the proposed TrajGRU-Attention and TrajGRU-Attention-ODE models on the MovingMNIST, MovingMNIST++, and KTH Action datasets. They implemented the TrajGRU and Vid-ODE baselines to compare against. Both quantitative and qualitative results show that their TrajGRU-Attention model predicts future images a lot more accurately than the state-of-the-art baselines.
strength_and_weaknesses: --- Strengths

- The proposed models are evaluated on public datasets. Both quantitative and qualitative results show that their TrajGRU-Attention model predicts future images a lot more accurately than the state-of-the-art baselines.


--- Weaknesses

- The baselines are from their own implementation, so it is really hard to assess whether the claimed performance improvement is real. For example, in the Vid-ODE paper, Vid-ODE's PSNR on the KTH Action dataset is 28.19, which is higher than the numbers shown in this paper.


--- Other comments

- Please adjust the placement of the figures. For example, Figure 2 is referenced on Page 3, but the figure is placed on Page 7.
clarity,_quality,_novelty_and_reproducibility: - The presentation is clear.

- The authors promised to release the source code.
summary_of_the_review: The proposed models are evaluated on public datasets and are shown to improve over the state-of-the-art baselines. However, those baselines are from their own implementation, so it is really hard to assess whether the claimed performance improvement is real. For example, in the Vid-ODE paper, Vid-ODE's PSNR on the KTH Action dataset is 28.19, which is higher than the numbers shown in this paper.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold