Paper:
To flexibly and efficiently reason about temporal sequences, abstract representations that compactly represent the important information in the sequence are needed. One way of constructing such representations is by focusing on the important events in a sequence. In this paper, we propose a model that learns both to discover such key events (or keyframes) as well as to represent the sequence in terms of them. We do so using a hierarchical Keyframe-Inpainter (KEYIN) model that first generates keyframes and their temporal placement and then inpaints the sequences between keyframes. We propose a fully differentiable formulation for efficiently learning the keyframe placement. We show that KEYIN finds informative keyframes in several datasets with diverse dynamics. When evaluated on a planning task, KEYIN outperforms other recent proposals for learning hierarchical representations.

Review:
This paper introduces a variational objective to train a model which can jointly select keyframes of a video and generate the intervening frames to produce a resultant video. The model is provided an initial set of frames as context. At training time the model always learns to produce N*J frames, where N is the number of keyframes and J is a fixed number of frames to generate for each keyframe. The authors compare their method for selecting informative keyframes on a number of baselines and show an improvement over these baselines. The problem is interesting and well-motivated, but I have some concerns with the proposed approach and experiments. As such, I am a weak reject.
comments / questions:
- Equation 3 lacks context. Initially, when looking at the authors' objective it seems that the inner expectation should be taken with respect to the joint time indices for the current and next keyframe. Only later after equation 4 do they mention that they always predict a fixed number of frames J. 
- The need for normalizing over the first T timesteps in equation 4 seems quite messy. Is it guaranteed that all of the needed keyframes will actually be within the first T timesteps? How does this work in practice?
- Many important details of the inference procedure are relegated to the appendix. For example, there are no details for extracting which of the 60 keyframes that were trained for a sequence (due to the fixed length sequences) should be selected at test time. Looking at the appendix, it is clear that the approach requires an extensive planning algorithm at inference time, which seems like an important component.
- The authors prominently highlight that their method is fully differentiable, yet they train in two stages while freezing weights. Why isn't the model trained end-to-end? The stated reason for doing so is that this "simple" two-stage procedure improves optimization. What exactly happens if you don't do this two stage training process? Does it fail to learn? Some experimental numbers would be nice to see. 
- The authors do not compare their method to any strong keyframe prediction baselines. Considering there is existing work in keyframe prediction, it seems important to highlight the difference between other competing models, rather than relying on simple baselines. Why don't they use self-information/surprisal as a baseline i.e., by training an autoregressive model on the frames and then picking the N frames with the largest -log(p)? This is a metric that has been investigated frequently and has better interpretability than defining a new measure of surprise. Note that Kipf et al. (2019) uses this notion of surprisal as well.
- Sauer et al. (BMVC 2019) should likely be cited as it does very similar keyframe analysis. Also, as the ICML 2019 conference had already concluded by the ICLR submission deadline, is it really fair to state the work with Kipf et al. (2019) was conducted in parallel?
- Why does the model trained to learn a fixed number of timesteps for the intermediate frames? Did they investigate jointly predicting the indices for the current and next timesteps? It seems like it would greatly simplify their inference scheme if they did this. If they tried that approach and it failed, maybe that should be mentioned in the paper (with an explanation as to why it fails).
- In the literature review, when discussing hierarchical temporal structure, the authors state: "However, these models rely on autoregressive techniques for text generation and are not applicable to structured data, such as videos." Autoregressive techniques have been investigated in relation to videos; in fact, the authors later describe papers that have used autoregressive techniques for modeling videos.