Paper: Post-Nonlinear Causal Relationship with Finite Samples: A Maximal Correlation Perspective (ICLR-2024)

Reviewer comment (Theory):
There are extra assumptions to ensure the correctness of Lemma 4, such as "composition of PNL functions and un-mixing nonlinearities are linear" (provided in Proof Sec. F), which should be incorporated into the claim of Lemma 4.

Author response:
Lemma 4 is in the context of PNL learning and follows the same assumptions as Lemma 3. As we have assumed the **invertibility** of underlying functions $\bar{f}$ and $\bar{g}$ in Lemma 4, one can always find $\hat{f},\hat{g}$ such that " the composition of PNL functions and un-mixing nonlinearities are linear".
We have added more descriptions to make Lemma 4 complete.

Reviewer comment (Theory):
The contribution of this paper is a proposed learning framework for PNL with some correctness analysis. My main concern is whether the theoretical contribution is insufficient due to the identification bound of PNL has not improved. I am willing to raise my score if the authors can further illustrate the signification impact of proposed methods in the fields of causal discovery.

Author response:
We acknowledge that the identification result has not been improved. However, we clarify the identifiability results in the context of PNL learning in Lemma 4, which has not been discussed in the existing literature. Besides, the estimation algorithm development is also fairly important, since there is a lack of effective and efficient algorithms to recover the underlying nonlinearities. Our proposed method is easy to implement and has a benign optimization structure, which provides a practical and efficient way to learn the underlying nonlinearities. We hope this can bring new thoughts to both the machine learning and causal inference communities.

Reviewer comment (Experiments):
In the "NONLINEAR FUNCTION FITTING" experiment, there are only two group generation mechanisms are used. More results with more types of non-linear functions should be provided if possible.

Author response:
The two synthetic examples were presented for illustration, and one of them was used in Uemura & Shimizu (2020). Following your suggestions, we updated the supplement with more examples. Please find additional experiments in our anonymous code link.

Reviewer comment (Experiments):
Have you applied this method to real-world data?

Author response:
The gene data from the DREAM4 competition is designed to mimic real data. In addition, we also showed the causal discovery result with uncertainty quantification using a real dataset (Tuebingen) in Figure 10 in suppl.I.2.

Reviewer comment (Experiments):
The numerical experiments look very simple (at least to me), and hence not entirely convincing about the use of overparameterized neural networks.

Author response:
**W1: The use of over-parameterized neural network (NN)**.
We need to clarify that NN has been used in some previous PNL-based methods (e.g., MLP-PNL, AbPNL), but they are not used in our method. We observed the *over-fitting* and *optimization* issues when we tried to reproduce their results using NN, which motivates the benefits of using our proposed method based on the RFF. We did use different NN architectures (over-parameterized wide NN and a relatively smaller deep-narrow NN) to illustrate the issues of previous works, see supp. C, i.e., they both produce unsatisfying results. 

The reasons that we include the over-parameterized NN for comparison are as follows: 1) it works well in many practical applications, and 2) it has a nice loss landscape for optimization (no bad basin/spurious valley under mild assumptions [Sun et al. 2020]).

Reviewer comment (Theory):
When \( f_{1} \) is a linear function in some basis of \( X \), say \( \beta^{\top} \phi (X) \), the proposed model is reduced to the single-index model. For single-index models, the identifiability of the parameter \( \beta \) will be problematic, and often one assumes \( \beta \) to be on the unit sphere. Is there a similar concern when one expands the modeling of \( f_{1} \) to a purely nonlinear one?

Author response:
**W2: In analogy with the single-index model, is there any identification issue when expanding $f_1$ to a purely nonlinear one?**

Thanks for pointing out the relation to the single-index model. If we understood correctly, you are referring to the noiseless case, $y =m(x)= f_2(f_1(x))$, which is not discussed in our manuscript. When $f_1(x)= \beta^T\phi(x)$, the PNL model reduces to a single-index model as defined in [Lin and Kulasekera, 2007]. And $\beta$ is identifiable when it has a unit norm and $m$ is a non-constant continuous function.

 If $f_1$ is extended to a more flexible nonlinear function w.r.t the parameter $\theta$ (e.g., a NN parameterized by $\theta$), similar results **MAY NOT** hold, i.e., $m(x) = g(f(x))=h(q(x))$ but $\{g\neq h, f\neq q \}$. A counter-example (in scalar case)  is $m(x)= \sin^2(x)$. The nonlinear transformations can be $\{g(z) = (\alpha z)^2,f(x)=\frac{1}{\alpha} \sin(x) \}$ and $\{h(z)= sin^2(\beta z), q(x) = \frac{1}{\beta} x \}$, where $\alpha,\beta \neq 0$ are some scaling factors.

 However, in the PNL model in our paper, **there is an independent noise term**, and our Lemma 2 gives the corresponding identifiability result.

Please correct us if we misunderstood your question. We'll be happy to address further questions/comments.

Reviewer comment (Evaluation):
Why use maximal correlation? There are obviously many "nonparametric correlations" at our disposal, e.g. d-corr, maximal information coefficients (by Reshef et al.), and many others.

Author response:
**Q1: Why use maximal correlation? There are obviously many "nonparametric correlations" at our disposal, e.g. d-corr, maximal information coefficients (by Reshef et al.), and many others.**

Maximal Correlation is a natural choice for PNL learning, as the optimized transformations $f$ and $g$ are closely related to the underlying nonlinear functions $f_1,f_2^{-1}$ in the PNL model. However, other dependence measures do not involve such a transformation learning process. Another consideration is computational efficiency. Since we need to compute dependence during optimization, many known dependence measures (e.g., maximal information coefficient) are not suitable, see the time comparison in the RDC paper (Lopez-Paz et al., 2013).  For the independence test, however, other dependence measures could be considered as potential replacements.

Reviewer comment (Theory):
Is there any connection between the proposed method (or any other related method) and canonical correlation analysis (CCA)? From the formulation of the optimization problem alone, they look quite similar.

Author response:
**Q2: Any connection to CCA?**
Certainly, some components in our method (e.g., HGR and RDC) are strongly related to nonlinear CCA. One famous work along this research line is Kernelized CCA by Bach and Jordan (2002), where the connection to CCA was thoroughly discussed. The distinction between Kernelized CCA and RDC lies in the representations of $f$ and $g$: Kernelized CCA uses $K_1\alpha_1$ for $f(X)$ (representer theorem), while RDC uses the linear combination of RFFs, $\Phi(X)^T\alpha$.

Reviewer comment (Writing):
The writing can be improved greatly. Few assertions are vague (e.g. "HSIC can get easily stuck at "meaningless" local minima", IGCI cannot provide "transparent and interpretable transformations") and undefined in the main paper (e.g. randomized dependence coefficient is not defined clearly despite it being used in the proposed method).

Author response:
We refer to the "meaningful solutions" as those solutions that can reflect the underlying nonlinear transformations. As one can see in the last two rows in Figure 6, the learned solutions cannot reveal the underlying functions as shown in Figure. 4, and thus are less meaningful. We have added a short description in the main paper to make this clear.

The causal discovery method, IGCI, is not designed for PNL learning and cannot give explicit function transformations of $f_1,f_2$, thus cannot provide "transparent and interpretable transformations". We have rewritten this phrase to "explicit nonlinear transformations". 

For the convenience of the audience, we have added the formal definition of randomized dependence coefficient (RDC) in the updated version.

Reviewer comment (Evaluation):
The main contributions seem overstated. Among a host of different causal disvoery methods compared in Table 2, the proposed method MC-PNL is faster compared to only AB-PNL by 300x, while there exists a competitive method in IGCI that is 60x faster than the proposed method. Overall, the experimental results don't seem to give the impression that MC-PNL outperforms other benchmarks.

Author response:
The reported numerical gain is compared with the SOTA PNL-based method. Our focus lies on comparing with interpretable baselines, as other baselines are incapable of providing explicit transformations for $f_1$ and $f_2$.

Reviewer comment (Experiments):
While one of the disadvantages of the PNL algorithms is claimed to be the optimization issue, thus motivating the RFF parametrization and linear HSIC kernel, neither is the experimental performance of this variant discussed, nor is its theoretical properties. MC-PNL still uses a gradient-based algorithm for the universal kernel and banded loss regularizer.

Author response:
We did include the theoretical properties in the text under eq (14). We show that the BCD algorithm can converge to a critical point and each sub-problem is a quadratic program.
The experimental convergence results have been shown in Figure 3. Only for fine-tuning, the gradient-based optimization is needed to handle the universal kernel and banded loss regularizer.

Reviewer comment (Writing):
Is it supposed to be < -\delta in Line 3 in Algorithm 2?

Author response:
Yes, it should be `$<-\delta$` in Line 3 in Algorithm 2.

Reviewer comment (Experiments):
Is there any ablation study done to determine the parameters of the RFF?

Author response:
The parameters are empirically fixed as $k=30,s=2$ in our paper, and the corresponding model expressivity was checked on various continuous functions within a certain input range (note that $(X,Y)$ were standardized), see `syn_data_bivariate_causal_discovery
genPNL_check_expressive.ipynb` in the anonymous code repo.

From our experience, increasing the number of parameters $k$ slightly influences the ROC-AUC performance, however, brings a computational burden; tuning variance factor $s$ can bring a performance gain.  We report the AUC on one synthetic dataset in the following table for your reference.

| k   | s   | AUC   | Time     |
|-----|-----|-------|----------|
| 30  | 0.5 | 0.628 | 43.56105 |
| 60  | 0.5 | 0.64  | 80.27628 |
| 120 | 0.5 | 0.656 | 234.0929 |
| 30  | 2   | 0.692 | 46.71287 |
| 60  | 2   | 0.685 | 80.79978 |
| 120 | 2   | 0.691 | 236.0387 |
| 30  | 4   | 0.73  | 45.18569 |
| 60  | 4   | 0.725 | 80.4238  |
| 120 | 4   | 0.712 | 237.8293 |