Paper:
Recurrent Ladder Networks
We propose a recurrent extension of the Ladder networks [22] whose structure is motivated by the inference required in hierarchical latent variable models. We demonstrate that the recurrent Ladder is able to handle a wide variety of complex learning tasks that benefit from iterative inference and temporal modeling. The architecture shows close-to-optimal results on temporal modeling of video data, competitive results on music modeling, and improved perceptual grouping based on higher order abstractions, such as stochastic textures and motion cues. We present results for fully supervised, semi-supervised, and unsupervised tasks. The results suggest that the proposed architecture and principles are powerful tools for learning a hierarchy of abstractions, learning iterative inference and handling temporal information.

Review:
In this paper, the authors introduce an extension of ladder networks for temporal data. Similarly to ladder network, the model proposed in the paper consists in an encoder-decoder network, with "lateral" connections between the corresponding layers of the encoder and the decoder. Contrary to the original model, recurrent ladder networks also have connection between different time steps (as well as skip connection from the encoder to the next time step). These recurrent connections allows to model temporal data or to perform iterative inference (where more than one step of message passing is required). The proposed model is then evaluated on different tasks. The first one is a classification task of partially occluded and moving MNIST digit, where the authors show that the proposed model performs slightly better than baselines corresponding to the proposed models with various ablation. The second task is generative modeling of various piano pieces. Finally, the last experiment is a classification task of MNIST digits with Brodatz textures, making the classification very challenging.
While the ideas presented in this paper are natural and well founded, I believe that the current version of the papers has some issues. First, the proposed model is very succinctly described (in section 2), assuming that the reader is familier with ladder networks. This is not the case for me, and I had to look at previous papers to understand what is the model. I believe that the paper should be self-contained, which I do not think is the case for the submitted version. I also believe that some statement in the paper are a bit unclear or misleading: for example, I do not think that Fig. 1c corresponds to message passing with latent variables varying in time (I think that it corresponds to iterative inference). Indeed, in the case of a temporal latent variable model, there should also be messages from t to t-1. I also found the experimental section a bit weak: most of the experiments are performed on toy datasets (with the exception of the piano dataset, where the model is not state of the art).
Overall, I am a bit ambivalent about the paper. While the ideas presented are interesting, I think that the proposed model is a bit incremental compared to previous work, and that the experiments are a bit weak.
Since I am not familiar at all with this research area, I put a low confidence score for my review.