From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper18/Authors
title: Reply to review from Oct/23/18
comment: We thank the reviewer for the comments.

We agree that the goal of the paper is mainly to present our dataset, as well as discuss the community's interest in the spatio-temporal study of such real physical systems in general. As such, we deliberately chose not to emphasize the results of Machine Learning models on that problem, and just provided a single baseline model to show that prediction on this dataset is indeed technically feasible and that there is a room for improvements from that baseline in terms of accuracy. 

Best Regards,
The Authors

From: ~Ransalu_Senanayake1
comment: This paper presents a data set based on videos of real systems rather than simulations, for the purpose of training and testing machine learning models on dynamic systems. More specifically, the authors set up, perform, and record a double-pendulum experiment, then extract the positions at each time step. They then trained LSTM and MLP models on the sine and cosine values of the angles of the pendulum and used them to predict the values at the next time step.

1. The references [1,5,9,11] were interesting demonstrations of the power of the network models for "learning" the dynamics of low-dimensional, chaotic systems. The paper itself did teach the reviewer anything particularly new.
2. The reviewer did not find anything particularly exciting about the work, however there is clearly value to the community in generating the kind of high-quality data sets presented in the paper
3. Most papers in this area rely on simulated data sets to test and train their models, this paper was written to change that.
4. The best argument that can be made for the paper belonging within the workshop is that the work presents a valuable product on which to test the robustness of predictive models (and associated theory) on real systems. However, the work does not in itself present significant or new ideas in advancing the state of modeling and decision-making in the spatiotemporal domain.
5. Cross, Rod. "A double pendulum swing experiment: In search of a better bat." American journal of physics 73.4 (2005): 330-33. http://www.physics.usyd.edu.au/~cross/PUBLICATIONS/29.%20DPendulum.pdf
Levien, R.B. "Double pendulum: An experiment in chaos" https://www.researchgate.net/publication/241238661_Double_pendulum_An_experiment_in_chaos
Kelly, Daniel Gibson "Design, Fabrication, and Testing of a Driven Double Pendulum " http://csma31.csm.jmu.edu/physics/html/theses/Thesis-Kelly08.pdf
title: From: NIPS 2018 Workshop Spatiotemporal Paper18 AnonReviewer1

From: ~Alexis_Asseman1
comment: We thank the reviewer for the provided comments. We are glad to see such a positive feedback and recognition of the significance of prediction of chaotic systems. We agree with the reviewer that it is hard to find methods that are directly applicable, however, we weren't aware of that when we started to work on that project. As the dataset includes raw videos with extracted positions of datums (making it a spatio-temporal prediction task), a combination of CNNs as image encoder with LSTM for learning long-term predictions seemed to us to be a natural end-to-end solution. We realised that it won't be that easy right after extracting the ground truth positions and trying to apply LSTM in a regular time-series prediction manner.

We find the questions raised by the reviewer interesting, and somehow overlapping with questions that we were asking ourselves when collecting the sequences and thinking about/developing baselines. What ratio between the lenght of input and output sequences ("past-to-future" ratio) would be useful in real-world applications? Should the metric incorporate some kind of punishment for missing the more distant predictions? How should we process the dataset so it will be more/less challenging?

Summarizing, despite the dataset seems to relatively small, in our opinion it is well focused on a particular aspect that will become more important when learning systems will have to cope with natural environments, where inherently chaotic events will happen on regular basis. We decided to leave the dataset as it is (not divided into clips of a given lenghts etc.) as we think it will leave more options for its usage. We are looking forward for the interesting discussions on those subjects during the workshop.

Best ragards,
The authors
title: Reply to review from Oct/19/2018

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper18/AnonReviewer2
evaluation: "The dataset, which consists of images with ground truth collected from a real mechanical system (with environmental noise etc.), is the main contribution of this paper...  we strongly believe that learning from real-world data is crucial for applications such as autonomous driving vehicles or natural disaster forecasting, where models have to cope with unobservable variables and randomness."

The authors chose a double pendulum for their experiments partly on the grounds that it is chaotic and therefore more sensitive to environmental noise than a simulation of a similar system might be.  The paper describes the experimental setup, as well as a recurrent neural network (based on LSTMs) to analyze the angles of the pendulum arms as a time series.

This paper certainly qualifies as a time-series paper.  Since it is meant to present a dataset rather than any specific method, it is hard to imagine any particular time-series methods literature that might be relevant.  The known chaotic behavior of the system seems almost certain to break most time-series models, especially autocorrelation models.  The baseline LSTM model accumulates large errors as the time series progresses.

The authors bill the importance and novelty of the contribution in virtue of it being a "real-world" dataset.  It is almost certainly not a fair comparison to more complex real-world datasets containing feedback loops, such as those arising in relation to autonomous systems, or exhibiting scale-free behavior such as that seen in many natural phenomena.  However, the new dataset provides a benchmark for a specific type of behavior (chaos) that may be present in other real-world systems.  The benchmark is intriguing in part because the baseline description of the state of the system at any point in time is so simple -- making it as easy to test against as classic datasets such as MNIST, or the univariate time series data on the flow of the Nile. 

The most interesting question the paper raises in my mind is that of how the community will respond to it.  For example, in this case we know the system is chaotic, so that any prediction looking sufficiently far ahead will accumulate a catastrophically large prediction error.  Is RMSE really a fair metric of performance in this case, and if not, what is?  How will e.g. Bayesian time-series methods which give estimates of their own uncertainty perform here, and will the uncertainty estimates be honest in light of the system's long-term behavior?  Would our assessment of these things change if we did not know or expect chaotic behavior in the system?  It is hard to say what the impact will be, but perhaps that's all the more reason to discuss it.

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper18/AnonReviewer2
relevance: 5: very high
impact: 4: high
novelty: 5: very high