Paper:
Model-based reinforcement learning approaches have the promise of being sample efficient. Much of the progress in learning dynamics models in RL has been made by learning models via supervised learning. There is enough evidence that humans build a model of the environment, not only by observing the environment but also by interacting with the environment. Interaction with the environment allows humans to carry out experiments: taking actions that help uncover true causal relationships which can be used for building better dynamics models. Analogously, we would expect such interaction to be helpful for a learning agent while learning to model the environment dynamics. In this paper, we build upon this intuition, by using an auxiliary cost function to ensure consistency between what the agent observes (by acting in the real world) and what it imagines (by acting in the "learned" world). Our empirical analysis shows that the proposed approach helps to train powerful policies as well as better dynamics models.

Review:
-------------
Summary
-------------
The authors propose to train a policy while concurrently learning a dynamics model. In particular, the policy is updated using both the RL loss (rewards from the environment) and the "consistency constraint", which the authors introduce. This consistency constraint is a supervised learning signal, which compares trajectories in the environment with trajectories in the imagined world (produced with the dynamics model). 
---------------------
Main Feedback
---------------------
I feel like there might be some interesting ideas in this work, and the results suggest that this approach performs well. However, I had a difficult time understanding how exactly the method works, and what its advantages are. These are my main questions:
1) At the beginning of Section 4 the authors write "The learning agent has two pathways for improving its behaviour: (...) (ii) the open loop path, where it imagines taking actions and hallucinates the state transitions that could happen". Do you actually do this? This is not mentioned in anywhere. And as far as I understand, the reward function is not learned - hence there will be no training signal in the open loop path. Does the reward signal always come from the true environment?
2) Is the dynamics model used for anything else than action-selection during training? Planning? If not, I don't really understand the results and why this works at all (k=20 being better than k=5, for example).
3) Is the dynamics model pre-trained in any way? I find it surprising that the model-free method and the proposed method perform similar at the beginning (Figure 3). If the agent chooses its actions based on the state that is predicted by the dynamics model, this should throw off the learning of the policy at the beginning (when the dynamics model hasn't learned anything sensible yet).
-----------------------
Other Questions
-----------------------
4) How exactly does training without the consistency constraint look? Is this the same as k=1?
5) Could the authors comment on the evaluation protocol in the experimental section? Are the results averages over multiple runs? If so, it would help to see confidence intervals to make a fair assessment of the results. 
6) For the swimmer in Figure 2, the two lines (with consistency and without consistency) start at different initial returns, why is that so? If the same architecture and seed was used, shouldn't this be the same (or can you just not see it in the graph)?
---------
Clarity
---------
The title and introduction initially gave me a slightly wrong impression on what the paper is going to be about, and several things were not followed up on later in the paper.
Title:
8) "generative models" reminds of things like a VAE or GAN; however, I believe the authors mean "dynamics models" instead
9) "by interaction" is a bit vague as to what the contribution is (aren't policies and dynamic models in general trained by interacting with the environment?); the main idea of the paper is the consistency constraint
Abstract / Introduction:
10) The authors talk about humans carrying out "experiments via interaction" to help uncover "true causal relationships". This idea is not brought up again in the methods section, and I don't see evidence that with the proposed approach, the policy does targeted experiments to uncover causal relationships. It is not clear to me why this is the intuition that motivates the consistency constraint. 
11) As the authors state in the introduction, the hope of model-based RL is better sample complexity. This is usually achieved by using the model in some way, for example by planning several steps ahead when choosing the current action. Could the authors comment on where they would place their proposed method - how does it address sample complexity?
12) In the introduction, the authors discuss the problem of compounding errors. These must be a problem in the proposed method as well, especially as k grows. Could the authors comment on that? How come that the performance is so good for k=20?
13) The authors write that in most model-based approaches, the dynamics model is "learned with supervised learning techniques, i.e., just by observing the data" and not via interaction. There's two things I don't understand: (1) in the existing model-based approaches the authors refer to, the policy also interacts with the world to get the data to do supervised learning - what exactly is the difference? (2) The auxiliary loss "which explicitly seeks to match the generative behaviour to the observed behaviour" is just a supervised learning loss as well, so how is this different?
For me, it would help the readability and understanding of the paper if some concepts were introduced more formally.
14) In Section 2, it would help me to see a formal definition of the MDP and what exactly is optimised. The authors write "optimise a reward signal" and "maximise its expected reward", however I believe it should be the expected cumulative reward (i.e., return). 
15) The loss function for the dynamics model is not explicitly stated. From the text I assume that it is the mean squared error for the per-step loss, and a GAN loss for the trajectory-wise loss.
16) Could the authors explicitly state what the overall loss function is, and how the RL and supervised objective are combined? Is the dynamics model f trained only on the supervised loss, and the policy pi only on the RL loss?
17) In 2.3 the variable z_t is not formally introduced. What does it represent?
------------------------
Other Comments
------------------------
18) I find it problematic to use words such as "hallucination" and "imagination" when talking about learning algorithms. I would much prefer to see formal/factual language (like saying that the dynamics model is used to do make predictions / do planning, rather than that the agent is hallucinating). 
-- edit (19.11.) ---
- updated score to 5
- corrected summary