Paper:
Graphical Models for Inference with Missing Data
We address the problem of recoverability i.e. deciding whether there exists a consistent estimator of a given relation Q, when data are missing not at random. We employ a formal representation called 'Missingness Graphs' to explicitly portray the causal mechanisms responsible for missingness and to encode dependencies between these mechanisms and the variables being measured. Using this representation, we derive conditions that the graph should satisfy to ensure recoverability and devise algorithms to detect the presence of these conditions in the graph.

Review:
Review Summary: This paper presents a graphical models 
      approach to reasoning about when probabilities can be computed in the 
      presence of missing data. The approach hinges on definitions of MCAR and 
      MAR in terms of conditional independencies between data and response 
      indicator variables. The MAR definition is considerably stronger than the 
      standard definition studied by Little and Rubin, which encodes the minimal 
      assumption required to obtain unbiased estimates while ignoring the 
      missing data mechanism. The current work is certainly novel and 
      interesting. It appears to be technically correct given the initial 
      definitions. However, the main component the paper is missing is a 
      discussion of the impact of changing the definitions. Detailed comments 
      are provided below. Novelty/Originality: The work appears to be 
      novel. The fact that the authors are able to provide a treatment of the 
      missing data taxonomy using graphical models is interesting and stems from 
      their MCAR/MAR definitions based on independence between random variables. 
      The standard definition of MAR is a statement of symmetry within the 
      distribution P(R|X) that depends on the actual values of R and X. It is 
      not a statement of independence between random variables, so Bayesian 
      networks can not be used to describe it. Technical Correctness: 
      Given the initial MCAR/MAR/MNAR definitions, the remaining technical 
      development appears to be correct. The authors show that they can apply 
      their framework to reason effectively about what queries are recoverable, 
      even in some fairly complex situations. The main issue with the 
      paper is the definitions themselves. Defining MAR at the random variable 
      level means that data sets that would be considered MAR by Little and 
      Rubin could correspond to graphs considered MNAR here. This creates some 
      confusion with the claims in the paper. The authors repeatedly 
      make claims like "Though it is widely believed that relations in MNAR data 
      sets are not recoverable, [we] demonstrate otherwise" (lines 229-230). 
      This is confusing since the majority of work uses the standard definition 
      of MNAR from Little and Rubin, which differ from the definitions used 
      here. Are the authors saying that under *their* definition of MNAR, it is 
      widely believed that queries are not recoverable? Additionally, 
      the authors should clarify what the relationship is between the the 
      recoverable MNAR cases under their framework, and how data from the 
      corresponding models would be labeled by Little and Rubin's definitions. 
      For example, is it the case that all of the MNAR graphs presented here 
      that admit recoverable queries would generate data that Little and Rubin 
      would consider MAR? Are any of the examples shown graphs that are NAMR 
      under the authors' definition, but admit recoverable queries, and would 
      also generate data considered MNAR by Little and Rubin? 
      Significance/Impact: The significance of the work is difficult to 
      judge without the clarifications requested above. There could be some 
      interest in these alternative definitions and the example applications of 
      the framework are quite nice. On the other hand, the limitations have not 
      been described by the authors at all. An additional issue with the 
      paper is the question of where the structures that are reasoned about come 
      from? Given a novel domain, coming up with a model over the primary 
      variables can be challenging. The additional complexity to soliciting a 
      joint data/response model would be quite onerous. Given possibly NMAR 
      data, what are the prospects for learning these model structures? Given 
      two candidate structures, one in which a given query is recoverable and 
      another where it is not, can the framework itself provide any guidance on 
      which structure to choose? Presentation/Clarity: The paper is well 
      written although quite dense. Focusing the development around specific 
      examples does help, but the examples themselves become quite complex. 
      * The name of the model is introduced twice on page 3 (lines 80, 
      115, 135). * In equation 1, what does the value m stand for? It is 
      not introduced prior to this point in the paper. I assume it stands for 
      "missing"? * I would consider a more politically correct example 
      on line 149.