Paper:
Causal Inference through a Witness Protection Program
One of the most fundamental problems in causal inference is the estimation of a causal effect when variables are confounded. This is difficult in an observational study because one has no direct evidence that all confounders have been adjusted for. We introduce a novel approach for estimating causal effects that exploits observational conditional independencies to suggest "weak" paths in a unknown causal graph. The widely used faithfulness condition of Spirtes et al. is relaxed to allow for varying degrees of "path cancellations" that will imply conditional independencies but do not rule out the existence of confounding causal paths. The outcome is a posterior distribution over bounds on the average causal effect via a linear programming approach and Bayesian inference. We claim this approach should be used in regular practice to complement other default tools in observational studies.

Review:
The paper describes the so-called Witness Protection Program (WPP) algorithm to compute bounds on an inferred average causal effect (ACE) that takes possible violations of the standard faithfulness assumption in causal inference into account. It does so by introducing a set of relaxation parameters to constrain unfaithful interactions, and then uses a linear programming framework to solve for the resulting upper and lower bounds on the ACE for a given link by adjusting for an appropriate set of covariates, subject to these constraints.This problem currently receives a fair amount of attention, so the paper is both timely and interesting. I like the approach and ideas introduced in the paper: it is a principled attempt to explicitly quantify the impact of (weak) unfaithful interaction patterns, in combination with well-known mathematical tools to solve the corresponding maximization/minimization problems.Whether or not they found the right method to capture this impact remains to be seen. The resulting output is hard to interpret from a conceptual point of view: a distribution over bounds which are a function of 'unfaithful relaxations' relative to a given model ... but how likely these unfaithful instances are, and with it how relevant/realistic these bounds themselves are remains unclear. Many relaxations lead to wide bounds to the point where they can become less than useful. The method does not commit to any preferred point estimate within the interval ... even though people are likely to interpret it that way. Experimental evaluation is poor: the 'bias' evaluation metric is meaningless in comparing the WPP method with its competitors as they literally measure two completely different things. As a result it is unclear if the WPP output is a valuable addition to practitioners or merely a complicated distraction. Also the practical performance on the influenza data set is worryingly weak: ACE in (-0.23,0.64) implies next to no possible relevant conclusion ... although that could be the fairest assessment of them all.QualityThe approach introduced in the paper is technically sound, with many details explicated in the supplementary file (I did not check any details in there). Ideas are well introduced, although the overall context is lost on occasion. Important aspects like choosing the relaxation parameters are relegated to a future article, which makes it difficult to get a feel for potential issues with this part of the method. Experimental evaluation is not balanced or robust enough. ClarityDecently written (with exception of conclusion, see comments below). The algorithmic steps and subsequent modifications in terms of many subtly different parameters was quite hard to follow, in part due to lack of explanation of certain details of the method. (I think I understand what happens, but I am not sure I could fully implement it from just this description; supplement may just be sufficient).OriginalityAs far as I can tell the method is a novel combination of familiar techniques. It is closely related to, but sufficiently different from recent work on the same problem. As such it may stimulate a fruitful cross-fertilization of several approaches on the subject.SignificanceThe significance of the paper mainly lies in the ideas and direction it brings to the problem, showing how to incorporate other techniques in solving this highly challenging problem. In its current implementation it seems unlikely to represent the finished article for practitioners, although some of the ideas employed in the WPP algorithm could well end up playing an important role. Summary:Relevant and highly topical problem. Interesting method that will undoubtedly find an audience. Current application limited, but likely to extend to other, more general cases. Reasonably well written, though could be improved. Technically interesting solutions. Experimental evaluation is poor. Relevance and proper interpretation of output distribution over bounds remains unclear.----------------Other comments:p1/5: 'Witness Protection Program' - > great name for an algorithm p1/47: 'unstable' is not the right word (they are stable, but may not hold true) p2/74: 'cartoon representation'?p2/93-94: technically this also assumes causal sufficiency or knowledge W - > Xp3/127: explain that witness W is itself not needed in the computation of the ACE, only as a means to identify set Zp3/143: typo 'an useful' p3/144: 'gives greater control over violations of faithfulness' : too vague as I do not see the control aspect p4/164: might - > can;     idem: '(conditional)' - > remove or change to '(unconditional)' , as always W dep. U | X p4/172-180: very sudden transition in level of abstraction; hard to interpret for readers less familiar with the subject - > explain / give some intuition on what these parameters represent,p4/183: link to original introduction in (1),p4/185-189: again interpret the constraint parameters; (easily misread as weak interactions)p5/221: briefly describe the relation between the search part and the bounds computation,idem: explain how the identification of admissible sets Z for W depends on the relaxation parameters thetap5/239: 'Background ... used here.' - > sentence/purpose unclear: used how? important for the paper?p5/259: to be clear: you mean max. 10 free parameters to describe the entire set Z, not a set of 10 variables, right?p6/304: 'We know of no previous analytical bounds' - > This seems closely related to recent papers such as 'Estimating Causal Eff ects by Bounding Confounding' (Geiger, Janzing,  Sch olkopf, UAI2014)p6/305: as a last-minute reviewer I have not checked the supplementary material in detailp6/319-332; I like the method but I doubt the practical efficacy of the back-substitution process in refining bounds;  seems rare to be able to exploit such constraint violations, (can you give some idea of how effective this step is in section 5?)p7/350: 'more than one bound' - > this is due to different possible admissible sets for different witnesses right?p7/357: 'comparison is not straightforward' - > no: the subsequent trivial optimal example shows that comparison based on the bias metric is meaningless.p7/360: 'only about 30% of the interval (-1,1)' - > that still covers (0,0.6) which implies 'anywhere between a strong causal link and no causal link' ...p7/369: '5000 points' - > typical (medical) trials often have a lot less data available: worried about the impact on the size of the bounds for say 500 records,p8/384: it is completely impossible to gauge meaningful information from Table 1 on the benefit of using WPP over the standard point estimate using faithfulness.p8/391: 'WPP is quite stable' - > seems like a nice way of saying that the bounds derived by WPP are so loose that they don't even depend on whether the problem is solvable or not p8/406-8: 'This does not mean ... within the bounds.' - > this is a crucial statement for interpretation: many casual readers will read the bounds as a confidence interval around the best guess value. It also highlights the difficulty in using the WPP results in practice: what do the bounds we derive actually mean? p8/421+: 'risky Bayesian approaches', 'WPP bounds keep inference more honest' - > emotive language that suggests a biased evaluation and/or erroneous interpretation of resultsidem: 'providing a compromise' - > WPP is not a compromise but focusses on an additional aspect of the overall inference problem,idem: 'and purely theory-driven analyses that refuse to look at competing models' - > I have no idea what you refer to in this statement, but it definitely sounds unduly dismissivep9: typo ref 9: inuenza