Paper:
Randomized Experimental Design for Causal Graph Discovery
We examine the number of controlled experiments required to discover a causal graph. Hauser and Buhlmann [1] showed that the number of experiments required is logarithmic in the cardinality of maximum undirected clique in the essential graph. Their lower bounds, however, assume that the experiment designer cannot use randomization in selecting the experiments. We show that significant improvements are possible with the aid of randomization -in an adversarial (worst-case) setting, the designer can then recover the causal graph using at most O(log log n) experiments in expectation. This bound cannot be improved; we show it is tight for some causal graphs. We then show that in a non-adversarial (average-case) setting, even larger improvements are possible: if the causal graph is chosen uniformly at random under a Erdös-Rényi model then the expected number of experiments to discover the causal graph is constant. Finally, we present computer simulations to complement our theoretic results. Our work exploits a structural characterization of essential graphs by Andersson et al. [2]. Their characterization is based upon a set of orientation forcing operations. Our results show a distinction between which forcing operations are most important in worst-case and average-case settings.

Review:
UPDATE AFTER DISCUSSION ROUND WITH AUTHORS:thanks for accepting discussing your paper with us. Really liked reviewing it !Still unclear to me when you state "Specifically, this result show that having many large cliques is not an insurmountable barrier." I would have thought that it is ?! See K. S. Sesh Kumar and F. Bach. Convex Relaxations for Learning Bounded Treewidth Decomposable Graphs. Technical report, HAL 00763921, 2012. Proceedings of the International Conference on Machine Learning (ICML) on the subject. A discussion in relationship to tree-width (in a future version of your paper) would really be a plus ! My understanding is that ONE dense and large clique is already a big problem...But in my framework, it's about structure network recovery, not to have an orientation from a graph skeleton...Best wishes !%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%This paper proposes to give conditions on the needed number of experiments to reconstruct a '"causal" graph from (interventional) data (given the skeleton is known from observational data). It follows from work of others ([1]). It states that only $ \log \log n $ experiments ($ n $ being the numbe of variables or nodes) are needed to recover the full causal structure in a "very adverse" situation (optimality of the result claimed) and this could fall down to a constant sample size when the graph is thought to be Erdös (is it really useful ??).Section 1 is an introduction, Section 2 gives background on graph, causality and the potential to reconstruct from observations and single perturbations. Section 3 introduces multiple random experiments so as to improve the actually needed number of experiments to be ale to decipher the causal structure. Section 4 gives a stringer result in a more constrained situation where graphs are generated according to an Erdös-Renyi model. Section 5 presents some simulation resultsOriginal results are clearly presented and seem to be an interesting step forward in a very important field: deciphering true causal relationships in complex systems and not only co-occurrence relationships. The paper is ok to read but some slight improvements could be made in the writing? A careful reading and the authors could be trusted to achieve a better readability !Few comments: - is it ok to put references in the astract in NIPS policy ? And to use CR when writing it ? - l24: yes, I remember a presentation on Rau et al. BMC Syst. Biol. which also mentioned that there are much cleverer perturbation than single KO experiments ?! This makes sense and can be related to the graph structure at hand. - last sentence: rephrase, this is unclear. - l37: dependences or dependencies ? - in [article missing] seminal work. - l43: sentence could be turned better. - try to avoid footnote. And footnote #1 is not a very accurate statement ?! - Some work of Guyon and colleagues may be cited. - why is "our results" section 1.1 and no 1.2 follows ? - I assume the 'adversary' notion is clear for those familiar with the game framework of [2] but not for the entire NIPS readership ?! - l61 "pre-commit" ?? - how do you framework compare to that of Peters & Bühlmann, Biometrika 2014 ? - would you say that your result on Erdös-Renyi graph is surprising ? I would say NO, their structure is relatively poor as compared to simulated but slighlty richer classes of models. Not to speak about true graph which obviously fall in none of these simulated framework (Erdös-Renyi, scale-free, small world...). - Table 1: I really beg you to avoid such a table ot present your results, they are difficult to read. Not on slides please ??!! And give a caption which has a signification please ! - A wee conclusion/discussion might be a plus...