Paper:
Adaptive Skip Intervals: Temporal Abstraction for Recurrent Dynamical Models
We introduce a method which enables a recurrent dynamics model to be temporally abstract. Our approach, which we call Adaptive Skip Intervals (ASI), is based on the observation that in many sequential prediction tasks, the exact time at which events occur is irrelevant to the underlying objective. Moreover, in many situations, there exist prediction intervals which result in particularly easy-to-predict transitions. We show that there are prediction tasks for which we gain both computational efficiency and prediction accuracy by allowing the model to make predictions at a sampling rate which it can choose itself.

Review:
The authors propose a method to learn temporally abstract transition models for time series prediction/classification. Starting with the insight that for many tasks not all time steps need to be accurately predicted, they propose to predict the "best" inputs within a fixed horizon H instead. This is done by evaluating the reconstruction loss for all H future inputs, but back-propagate only on the smallest one, akin to pooling over losses. As a result, the method can learn a sequence of future inputs, with varying delays, that are easy to predict. The target label of a time series can be predicted based either on the last or the average predicted input. The authors evaluate their method on two complex tasks based on input-images to show it outperforms time series prediction with fixed step-sizes.  The idea to pool over the loss of future prediction is, to the best knowledge of the reviewer, quite original and appears highly relevant. It is important to distinguish the presented method from model learning for control, though, which have to be conditioned on control signals or actions. The paper is well written and the approach is clearly explained. Experiments are sufficiently complex and show the method's advantage clearly, although the reviewer was missing the comparison with another state-of-the-art algorithm as baseline. Note that the classification problem is somewhat artificially combined with the model-learning, which makes the presented method an unsupervised pre-processing step, rather than a combined classification method. It would be interesting to see if one could backprop the gradient of the classification problem somehow in the model-learning procedure. To this end, the authors are encouraged to try a RNN for classification. RNN yield gradients at each abstract time step, which could be propagated into the model-prediction.  The reviewer recommends to accept this paper. It is a well written and presents a simple, but interesting twist on time series prediction. The technique should only work in tasks where the labels depend almost exclusively on the final predicted states, but the basic idea may be applicable beyond that.   COMMENTS: l.144: this paragraph was hard to follow l.172+: "separately" twice in one sentence l.221: use the term "NN with seven convolutional layers" instead of "seven-layer fully-convolutional NN" to avoid confusion with seven fully-connected layers l.269: "proposes proposes" fig.7: plotting the best 3 out of 6 runs is a metric that can be easily manipulated and should be avoided fig.9+10: as you mention yourself, these plots are not informative fig.12: Figure 12 refers to itself and the end of the sentence is missing fig.1-12: showing the interquantile range of 3 samples is misleading. Use STD, SEM or minimum/maximum instead