Paper: Optimal Kernel Choice for Score Function-based Causal Discovery (ICLR-2024)

Reviewer comment (Novelty):
The difference from the existing Gaussian-process-based framework is unclear, as the idea of optimizing kernel hyper parameters in the context of Bayesian network structure learning has already been developed more than 20 years ago.

Author response:
**A1.** I believe there might be some misunderstanding here. The contribution of our method does not lie in using Gaussian processes to optimize kernel hyperparameters. 
Our contribution builds upon Huang et al.'s method.
Huang et al. utilize the characterization of general independence relationships with conditional covariance operators and propose using the regression model for causal discovery with the regression form $\phi(X; \sigma_x) = f(PA; \theta) + \epsilon$. 
In Huang et al.'s model, **the parameter $\sigma_x$ in $\phi(\cdot)$ is pre-defined**. 
It assumes that the pre-defined feature $\phi(X)$ is generated from a non-linear projection $f(PA_X)$ with an independent noise $\epsilon$ added, which is hard to satisfy with such an manually selected $\phi(\cdot)$ and may not fully capture the characteristics of the data in many cases.
While in our method, we extended the regression model above with trainable parameters of $\phi(\cdot)$ rather than the fixed one.

By introducing more learnable parameters, our model becomes more flexible and can more accurately uncover relationships within the data. Additionally, due to the introduction of extra parameters, we propose using the joint distribution's marginal likelihood as the score for parameter learning and causal discovery, which is different from Huang's conditional likelihood based ones. 
To the best of our knowledge, there is currently no other approaches using the same model and score function for causal discovery.

And our proposed score function in Eq.13 is not the same with that of [1]. 
**Our score has an extra term (last term in Eq.13), which reflects the difference of relationship modeling mention above.**
And this last term arises from transforming the joint distribution of dependent and independent variables into the joint distribution of the independent variables and the estimated mixed noise. 
It results from simplifying the Jacobian matrix obtained from using the change-of-variables theorem.

Reviewer comment (Writing):
In particular, the notations around Eq. (1) are unclear. Noise $\varepsilon$ is introduced without definition. It is unclear whether it is a function in RKHS, i.e., $\varepsilon \in \mathcal{H}_X$, and if so, it should be stated clearly. The definition of $k_x$ is unclear: $x$ is a realization of random variable $X$? The relationship with kernel feature mapping $\phi_X$ is not explained—does $\phi_X(x) = k_x$? Is it defined in the same way as in Section 3.1?

Author response:
**A2.** When we introduce an RKHS space $H_\mathcal{X}$ on $\mathcal{X}$, it comes with the definition of a continuous feature mapping $\phi_\mathcal{X}$ and a measurable positive-definite kernel function $k_\mathcal{X} : \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$. 
Since $\phi_X$ is in an infinite-dimensional space without a proper probability measure. 
For practical purposes, given an observation $x$ of $X$, we project $\phi_\mathcal{X}(x)$ into its empirical feature map, which is $\mathbf{k}_x$.

Reviewer comment (Theory):
The connection between Eqs. (3) and (4) is unclear. It seems the authors express $\phi_i(x)$ as the i-th element of a (possibly) infinite-dimensional feature vector $\phi_X(x)$, where $\phi_X \in \mathcal{H}_X$. However, they also say that $g(x) = \phi_i(x)$ belongs to the same RKHS $\mathcal{H}_X$. This needs clarification, especially regarding the definitions of indices $i$ and $j$.

Author response:
**A3.** We are sorry for your confusion. 
The Eq.3 is derived from Theorem 7 in [1], which is the theoretical foundation that allows us to transform the independence relations test into a regression model selection problem. 
Eq.4 is the corresponding regression model, where a continuous feature mapping $\phi(X)$ is applied to the dependent variable $X$. 
Due to the properties of characteristic kernels, $\phi(X)$ can be decomposed into an infinite linear combination of basis functions $[\phi_1, \cdots, \phi_i, \cdots]$, and each $\phi_i \in {\mathcal{H_X}}$ is orthogonal to each other. 
We associate each $\phi_i$ with the function $g$ in Eq.3, leading to the conclusion that Eq.5 holds for any given $\phi \in {\mathcal{H_X}}$. 
Thus, based on Eq.5, we can further transform the conditional independence test into a parameter selection problem for the regression model defined in Eq.4.

Reviewer comment (Writing):
In Eq. (6), there is no definition or description of parameter $\theta$. The type of parameters considered by the authors should be specified.

Author response:
**A4.** In the later part, we introduced Gaussian processes to mitigate model overfitting. By parameterizing $f$, we aim to reduce the number of parameters in $f$, representing the parameters $\theta$ as optimizable parameters in the Gaussian process.

Reviewer comment (Theory):
There is a need for a more in-depth exploration of why joint marginal likelihood successfully avoids overfitting—potentially due to the marginalization of $f$—as this would contribute to a clearer understanding of the method’s advantages.

Author response:
**A1.** What we want to highlight here is that **using the likelihood of the joint distribution instead of conditional distribution is not to mitigate overfitting**. Given that our model introduces additional learnable parameters (kernel function parameters imposed on the dependent variable), we choose to employ the likelihood of the joint distribution **to constrain these parameters and learn a non-trivial relationship**. 
As mentioned in the last paragraph of Section 3.2, if we were to use conditional likelihood for learning these extra parameters, the model would directly fail to capture real relationships but project all samples into a very small range of the feature space with excessively high scores, which is uninformative for causal discovery purposes.
This is the reason why we must use the joint distribution—it fundamentally stems from the differences in the assumed models.
Building on this, to alleviate model overfitting, we utilized Gaussian processes to parameterize the projection function $f$, aiming to enhance the effectiveness of causal discovery. 
The corresponding score also takes the form of joint marginal likelihood.

Reviewer comment (Theory):
A term-by-term comparison between the equations of joint marginal likelihood and conditional likelihood score functions could provide valuable insights into the sources of their respective strengths and weaknesses.

Author response:
**A3.** It is not the same with the conditional likelihood. If it is a conditional distribution, in Equation (11), only the first term will be present, and there won't be the term $E[\log | det J_i|]$. That is, $p(x|pa, f, \sigma_x) = p(f(pa) + \widetilde{{\pmb \epsilon}}| pa, f, \sigma_x) = p(\widetilde{\epsilon})$.

Reviewer comment (Experiments):
In terms of empirical validation, the paper focuses on demonstrating the learning of simple bandwidths, but consideration of more complex kernels, such as deep kernels, would be beneficial to underscore the method’s versatility and general applicability.

Author response:
**A2.** Your valuable insights are appreciated. Due to the fact that our approach is based on the characterization of conditional independence with conditional covariance operators[1], it requires the use of the characteristic kernel. The use of more complex deep kernels, etc., will need further exploration.

Reviewer comment (Novelty):
The main contribution of the paper seems to be the introduction of a trainable kernel hyper-parameter (instead of previous work that estimates it heuristically from data) and an extended objective that minimizes the MI between parents of a node and noise variables. This seems too incremental as a conference contribution and in my (the reviewer's) opinion is of insufficient originality.

Author response:
The choice of the kernel is a crucial step in kernel-based methods, as it directly impacts the accuracy of the results. Unlike previous approaches, which often heuristically select kernel parameters manually, our method introduces an optimization-based approach for adaptive kernel parameter selection. Our approach dynamically considers the characteristics of the given data and selects appropriate kernel parameters accordingly. Introducing additional parameters poses a challenge for existing conditional likelihood-based methods to supervise model parameters. Therefore, we propose a supervised approach based on mutual information. Our experimental results demonstrate the stability improvement of our proposed method compared to existing approaches.

Reviewer comment (Novelty):
The paper misses several key references in the field, e.g., [2-4], but several others as well, e.g., [5-7].

Author response:
Our proposed method introduces a scoring approach that models relationships in a more generalized manner, focusing on the local associations between given potential independent and dependent variables. 
Based on our understanding, references [2-5] predominantly 
focus on searching for DAGs under the assumption of a relatively simple local relationship model.
While the mentioned references [6-7] appear to address extreme scenarios involving hard intervention targets or missing variables in the field of causal discovery. 
Given the extensive scope and diverse content of causal discovery, we have chosen to align our work with a narrower field relevant to our method.

Reviewer comment (Experiments):
This paper states the criticality of choosing the right kernel. However, it keeps the original kernel choice as Huang et al. 2018 ("More specifically, we utilize the widely-used Gaussian kernel throughout the paper" on page 5). The lack of in-depth discussions on different choices of kernels severely undermines the completeness of the discussion.

Author response:
**A1.** In this paper, we do not select kernel across different kernel families. What we do in this paper is to first give a specific characteristic kernel (such as Gaussian kernel used in the paper) and apply it to the dependent variable, and then optimize the parameters of the kernel (such as the width in the Gaussian kernel) to find the optimal parameters that best fit the data.

Reviewer comment (Reproducibility):
This paper seems to have reproduced Huang et al. 2018 incorrectly. The variance parameter of Marginal Likelihood of Huang et al. 2018 is actually trainable ("where the hyperparameter \hat{\sigma}^2_i, as the noise variance, is learned by maximizing the marginal likelihood with gradient methods", page 1555 of https://dl.acm.org/doi/pdf/10.1145/3219819.3220104), while the authors of this paper state that the width is chosen in a manual and pre-defined way in Huang et al. 2018 and reproduce it as fixed.

Author response:
**A2.** We believe there might be some misunderstanding here. What we want to highlight is that our method can optimize the parameters of the kernel function **imposed on the dependent variable, which is pre-defined heuristically and fixed in the existing paper**.  
Specifically, in Eq.6 of our paper (i.e., $\phi(X; \sigma_x) = f(PA; \theta) + \epsilon$), the kernel function parameter $\sigma_x$ involved in $\phi(X)$ imposed on $X$ is learnable in our approach, while it is fixed in Huang et al.'s method. Due to the introduction of additional learnable parameters, we extended the regression model in RKHS and accordingly use joint distribution likelihood $p(X, PA |\\bf{\sigma})$ to simultaneously optimize $\sigma_x$ and other learnable parameters (i.e., $\theta$ in $f$) in the model including the mentioned noise variance.

Reviewer comment (Theory):
The algorithm proposed by the authors is to maximize the marginal log-likelihood \log p(X = x, PA = pa | f, \sigma_x), while the corresponding term in Huang et al. 2018 is \log p(X = x | PA, f, \sigma_x). What is the difference in their dynamics that leads to potential performance differences needs more discussion?

Author response:
**A5.** The likelihood of the conditional distribution is insufficient to constrain the parameters of the kernel function applied to the dependent variable. From the conditional distribution, we can observe that $p(x|pa, f, \sigma_x) = p(f(pa) + \widetilde{{\mathbf \epsilon}}| pa, f, \sigma_x) = p(\widetilde{\epsilon})$, lacking the term $E[\log|\mathrm{det} J_i|]$ in Eq.11 of the paper. Consequently, it will fail to capture real relationships if we use $\log p(X = x | PA, f, \sigma_x)$ as the score function with trainable parameters in $\phi(X)$. Specifically, it will project all samples into a very small range of the feature space with excessively high scores, which is uninformative for causal discovery purposes. Therefore, we must use $\log p(X = x, PA = pa | f, \sigma_x)$ as our score function, driven by the differences in the assumed models.