Paper:
Learning to Decompose and Disentangle Representations for Video Prediction
Our goal is to predict future video frames given a sequence of input frames. Despite large amounts of video data, this remains a challenging task because of the highdimensionality of video frames. We address this challenge by proposing the Decompositional Disentangled Predictive Auto-Encoder (DDPAE), a framework that combines structured probabilistic models and deep networks to automatically (i) decompose the high-dimensional video that we aim to predict into components, and (ii) disentangle each component to have low-dimensional temporal dynamics that are easier to predict. Crucially, with an appropriately specified generative model of video frames, our DDPAE is able to learn both the latent decomposition and disentanglement without explicit supervision. For the Moving MNIST dataset, we show that DDPAE is able to recover the underlying components (individual digits) and disentanglement (appearance and location) as we intuitively would do. We further demonstrate that DDPAE can be applied to the Bouncing Balls dataset involving complex interactions between multiple objects to predict the video frame directly from the pixels and recover physical states without explicit supervision.

Review:
The paper addresses the problem of predicting future frames in videos from previously seen frames. This task has been gaining a lot on popularity due to it's applications in different areas [8, 22], but also because it aims at modeling the underlying video representations in terms of motion and appearance. The paper proposes a new Decompositional Disentangled Predictive Auto-Encoder (DDPAE), a framework that combines structured probabilistic models and deep networks to automatically (i) decompose the high-dimensional video that we aim to predict into components, and (ii) disentangle each component to have low-dimensional temporal dynamics that are easier to predict. The main novelty is the automatic decomposition of the video into components that are easier to predict, as well as the disentanglement of each component into static appearance and 2D motion.         Positive sides:       1. Well written and easy to read. The paper is well structured, the language maintains high standards and is easy to read.        2. Relevant and interesting task. The paper is tackling the problem of learning generative models of object dynamics and appearance in videos which is a challenging task with strong potential impact.        3. Novelty. Although the paper re-uses a lot of existing ideas and works, still the combination is novel and relevant to consider.        4. Experimental evaluation. The proposed model outperforms existing works with decomposition and disentanglement.         Negative sides:       1. Clarity. The paper is really hard to follow and fully understand. Curently the paper spends little time in explaining the relevant details of this work: how is the video decomposed? What is the initial starting point when the video starts? How do you set the number of components in the video? How do you learn the content part of the disentangled representation? What is the parameterization of the pose part? How do you predict the betas? Why using a seq2seq model? Is the order between components arbitrary? If not (suggested by Fig 2) then how do you choose the order at training and test time? How do you ensure that the components correspond to objects (digits and balls)? How does the inference work? It is hard to grasp such a comlex part from only 3 lines of text.        2. Experiments. The experiments are done only in a very controlled setup with small and toy datasets. While that is fine as a debugging step to understand the model, ultimately the paper should tackle real-world datasets and showcase potential usability in the real world. Most of the maade assumptions (constant latent part) is fine on these toy datasets, but it brakes in real life. It is easy to define a priori the number of components in these toy datasets, because it is very constrained. In the real world the objects have much more complex pose dynamics, especially non-rigid objects. The datasets considered have black backgrounds only, which is a significant downside. You might hope that the proposed model in the real world might allocate a single (or several) components to the background?    After reading the authors rebuttal, I remain with my decision to accept this paper. The rebuttal has been helpful to further understand the paper, which indicates that the paper is hard to thoroughly understand, however, the contribution is significant and this work will be definitely well accepted in the computer vision world.