Paper:
Discovering and exploiting the causal structure in the environment is a crucial challenge for intelligent agents. Here we explore whether modern deep reinforcement learning can be used to train agents to perform causal reasoning. We adopt a meta-learning approach, where the agent learns a policy for conducting experiments via causal interventions, in order to support a subsequent task which rewards making accurate causal inferences. We also found the agent could make sophisticated counterfactual predictions, as well as learn to draw causal inferences from purely observational data. Though powerful formalisms for causal reasoning have been developed, applying them in real-world domains can be difficult because fitting to large amounts of high dimensional data often requires making idealized assumptions. Our results suggest that causal reasoning in complex settings may benefit from powerful learning-based approaches. More generally, this work may offer new strategies for structured exploration in reinforcement learning, by providing agents with the ability to perform-and interpret-experiments.

Review:
Note: This review is coming in a bit late, already after one round of responses. So I write this with the benefit of having read the helpful previous exchange. 
I am generally positive about the paper and the broader project. The idea of showing that causal reasoning naturally emerges from certain decision-making tasks and that modern (meta-learning) RL agents can become attuned to causal structure of the world without being explicitly trained to answer causal questions is an attractive one. I also find much about the specific paper elegant and creative. Considering three grades of causal sophistication (from conditional probability to cause-effect reasoning to counterfactual prediction) seems like the right thing to do in this setting.
Despite these positive qualities, I was confused by many of the same issues as other reviewers, and I think the paper does need some more serious revisions. Some of these are simple matters of clarification as the authors acknowledge; others, however, require further substantive work. It sounds like the authors are committed to doing some of this work, and I would like to add one more vote of encouragement. While the paper may be slightly too preliminary for acceptance at this time, I am optimistic that a future version of this paper will be a wonderful contribution.
(*) The authors say at several points that the approach “did not require explicit knowledge of formal principles of causal inference.” But there seem to be a whole of lot of causal assumptions that are critically implicit in the setup. It would be good to understand this better. In particular, the different agents are hardwired to have access to different kinds of information. The interventional agent is provided with data that the conditional agent simply doesn’t get to see. Likewise, the counterfactual agent is provided with information about noise. Any sufficiently powerful learning system will realize that (and even how) the given information is relevant to the decision-making task at hand. A lot of the work (all of the work?) seems to be done by supplying the information that we know would be relevant.
(*) Previous reviewers have already made this point - I think it’s crucial - and it’s also related to the previous concern: It is not clear how difficult the tasks facing these agents actually are, nor is it clear that solving them genuinely requires causal understanding. What seems to be shown is that, by supplying information that’s critical for the task at hand, a sufficiently powerful learning agent is able to harness that information successfully. But how difficult is this task, and why does it require causal understanding? I do think that some of the work the authors did is quite helpful, e.g., dividing the test set between the easy and hard cases (orphan / parented, unconfounded / confounded). But I do not feel I have an adequate understanding of the task as seen, so to say, from the perspective of the agent. Specifically:
(*) I completely second the worry one of the reviewers raised about equivalence classes and symmetries. The test set should be chosen more deliberately - not randomly - to rule out deflationary explanations of the agents’ purported success. I’m happy to hear that the authors will be looking more into this and I would be interested to know how the results look.
(*) The “baselines” in this paper are often not baselines at all, but rather various optimal approaches to alternative formulations of the task. I feel we need more actual baselines in order to see how well the agents of interest are doing. I don’t know how to interpret phrases like “close to perfect” without a better understanding of how things look below perfection. 
As a concrete case of this, just like the other reviewers, I was initially quite confused about the passive agents and why they did better than the active agents. These are passive agents who actually get to make multiple observations, rather than baseline passive agents who choose interventions in a suboptimal way. I think it would be helpful to compare against an agent who makes the same number of observations but chooses them in a suboptimal (e.g., random) way. 
(*) In relation to the existing literature on causal induction, it’s telling that implementing a perfect MAP agent in this setting is even possible. This makes me worry further about how easy these tasks are (again, provided one has all of the relevant information about the task). But it also shows that comparison with existing causal inference methods is simply inappropriate here, since those methods are designed for realistic settings where MAP inference is far from possible. I think that’s fine, but I also think it should be clarified in the paper. The point is not (at least yet) that these methods are competitors to causal inference methods that do “require explicit knowledge of formal principles of causal inference,” but rather that we have a proof-of-concept that some elementary causal understanding may emerge from typical RL tasks when agents are faced with the right kinds of tasks and given access to the right kinds of data. That’s an interesting claim on its own. The penultimate paragraph in the paper (among other passages) seems to me quite misleading on this point.
(*) One very minor question I have is why actions were softmax selected even in the quiz phase. What were the softmax parameters? And would some of the agents not perform a bit better if they maximized?