Paper:
Causal Inference and Mechanism Clustering of A Mixture of Additive Noise Models
The inference of the causal relationship between a pair of observed variables is a fundamental problem in science, and most existing approaches are based on one single causal model. In practice, however, observations are often collected from multiple sources with heterogeneous causal models due to certain uncontrollable factors, which renders causal analysis results obtained by a single model skeptical. In this paper, we generalize the Additive Noise Model (ANM) to a mixture model, which consists of a finite number of ANMs, and provide the condition of its causal identifiability. To conduct model estimation, we propose Gaussian Process Partially Observable Model (GPPOM), and incorporate independence enforcement into it to learn latent parameter associated with each observation. Causal inference and clustering according to the underlying generating mechanisms of the mixture model are addressed in this work. Experiments on synthetic and real data demonstrate the effectiveness of our proposed approach.

Review:
This paper proposes an approach to estimate causal relationships between two variables X and Y when properties of the mechanism changes across the dataset. The authors propose and extension of the non-linear additive noise model [Hoyer et al. 2009] to the case of a mixture of a finite number of non-linear additive noise models, coined Additive Noise Model- Mixture Model (ANM-MM). The authors propose a theoretical identifiability result based on the proof of [Hoyer et al. 2009], then provide an estimation algorithm based on Gaussian Process Partially Observable Models (GPPOM), introduced as a generalization of Gaussian Process Latent Variable Models (GPLVM). Comparison of the approach to baseline for causal inference and clustering are provided on real and simulated data.  The problem addressed in this paper is definitively interesting. While some of the experimental results are promising, theoretical and empirical provide a limited understanding of the approach, which is rather complex, and in particular of its strength and limitations. Main concerns follow. Theoretical results:  It is unclear to me why the theoretical identifiability result of Theorem 1 is not a very simple corollary of [Hoyer et al., 2009]: assume both directions admit an ANM-MM, then there is a pair of function parameters values (\theta,\omega) corresponding to each direction, that happen simultaneously with non-zero probability. Then conditioning on this event, we get an ANM for both directions and can conclude that this cannot happen in a generic case, based on [Hoyer et al., 2009]. Am I missing something? At the same time, this theoretical result does not offer guaranties for finite samples. Given the complexity of the model, I can imagine several things that can go wrong when applying the model: - When the number of clusters gets too large, overfitting of the models (due to lack of samples) may reduce the information in additive noise that can be used for causal inference. How can we assess (theoretically or experimentally) and possibly handle this issue? This is partially shown in Fig. 3a), but the study restrict itself to the case of a number of clusters inferior or equal to ground truth. On the other hand, there are baseline methods that do not assume a mixture of mechanisms (such as LiNGAM or PNL) that seem quite robust to variations in the mechanism. - In the particular context of ANM, it would be good that the authors provide a toy example showing that mixing mechanisms can lead to misleading results. Intuitively, not capturing the variation in the mechanism with lead to an increase in additive noise. Is there any toy example where one can prove ANM would fail? Experimental results Regarding simulation, one simple baseline should use clustering combined with ANM inference (all “standard” causal inference approaches can handle a single mechanism). Moreover, clustering (such as K-means) in the PCA domain seems a fair and reasonable baseline that is likely to perform better than in the original space. Given the complexity of the algorithm, it seems difficult to choose the hyperparameters for a given dataset (this includes number of clusters and parameter of independence). How to we choose them to reproduce the results on the Tuebingen dataset? Also it is unclear whether clustering of mechanism is necessary in most of these datasets, so I wonder if it would not be worth focusing on more datasets where the mechanism variability is known. The FOEN data seems to be an interesting example, and I would suggest to put part of supplemental Fig.5 in main text.