Paper:
LEARNING WHAT YOU CAN DO BEFORE DOING ANYTHING
Intelligent agents can learn to represent the action spaces of other agents simply by observing them act. Such representations help agents quickly learn to predict the effects of their own actions on the environment and to plan complex action sequences. In this work, we address the problem of learning an agent's action space purely from visual observation. We use stochastic video prediction to learn a latent variable that captures the scene's dynamics while being minimally sensitive to the scene's static content. We introduce a loss term that encourages the network to capture the composability of visual sequences and show that it leads to representations that disentangle the structure of actions. We call the full model with composable action representations Composable Learned Action Space Predictor (CLASP). We show the applicability of our method to synthetic settings and its potential to capture action spaces in complex, realistic visual settings. When used in a semi-supervised setting, our learned representations perform comparably to existing fully supervised methods on tasks such as action-conditioned video prediction and planning in the learned action space, while requiring orders of magnitude fewer action labels.

Review:
The paper proposes a Variational IB based approach to learn action representations directly from video of actions being taken. The basic goal of the work is to disentangle the dynamic parts of the scene in the video from the static parts and only capture those dynamic parts in the representation. Further, a key property of these learned representations is that they contain compositional structure of actions so as to their cumulative effects. The outcome of such a method is better efficiency of the subsequent learning methods while requiring lesser amount of action label videos. 
To achieve this, the authors start with a previously proposed video prediction model  that uses variational information bottleneck to learn minimal action representation. Next, this model is augmented with composability module where in latent samples across frames are composed into a single trajectory and is repeated in a iterative fashion and again the minimal representation for the composed action space is learned using IB based objective. The two objectives are learned in a joint fashion. Finally, they use a simple MLP based bijection to learn the correspondence between actions and their latent representations. Experiments are done on two datasets - reacher and BAIR - and evaluation is reported for action  conditioned video prediction and visual servoing.
- The paper is well written and provides adequate details to understand the flow of the material.
- The idea of learning disentangled representation is being adopted in many domains and hence this contribution is timely and very interesting to the community.
- The overall motivation of the paper to emulate how humans learn by looking at other's action is very well taken. Being able to learn from only videos is a nice property especially when the actual real world environment is not accessible.
- High Performance in terms of error and number of required action labeled videos demonstrates the effectiveness of the approach.
However, there are some concerns with the overall novelty and some technical details in the paper:
- It seems the key contribution of the paper is to add the L_comp part to the already available L_pred part in Denton and Fergus 2018. The trick use to compose the latent variables is not novel and considering that variational IB is also available, the paper lacks overall novelty. A better justification and exposition of novelty in this paper is required.
- Two simple MLP layers for bijection seems very adhoc. I am not able to see why such a simple bijection would be able to map the disentangled composed action representations to the actual actions. It seems it is working from the experiments but a better  analysis is required on how such a bijection is learned and if there are any specific properties of such bijection such that it will work only in some setting. Will the use of better network improve the learned bijection?
- While videos are available, Figures in the paper itself are highly unreadable. I understand the small figures in main paper but it should not be an issue to use full pages for the figure on appendix.
- Finally, it looks like one can learn the composed actions (Right + UP) representation while being not sensitive to static environment. If that is the case, does it work on the environment where except the dynamic part everything else is completely different? For example, it would be interesting to see if a model is trained where the only change in environment is a robot's hand moving in 4 direction while everything else remaining same. Now would this work, if the background scene is completely changed while keeping the same robot arm?