Paper: Learning Counterfactually Invariant Predictors (ICLR-2024)

Reviewer comment (Theory):
The fairness example in section 4.3 is based on Fig. 1(e), where our goal is to learn a predictor $\hat{Y}$ that exhibits counterfactual invariance with respect to $\mathbf{A}$ within the context of $\mathbf{W}=\mathbf{X} \cup \mathbf{C}$. As per Theorem 3.2, our first step is to identify a valid adjustment set $\mathbf{S}$ for $(\mathbf{A} \cup \mathbf{W}, \mathbf{Y})$. However, in this case, the adjustment set $\mathbf{S}$ between $\mathbf{A} \cup \mathbf{W}$ and $\mathbf{Y}$ should be empty. Upon reviewing Appendix F.5 and Fig. 8, it seems that it is inappropriate to use a subset of $\mathbf{A} \cup \mathbf{W}$ as the adjustment set. A similar issue arises in the experiments conducted on synthetic and image datasets, in which, we have $\mathbf{W}$ encompassing all variables except for $\mathbf{Y}$ and $\mathbf{A}$. Besides, notice that, in the latter case, the authors state that 'We seek a predictor $\hat{Y}$ that is CI in the x-position with respect to $\textbf{all other observed variables}$', so $\mathbf{W}$ includes all the observational variables except $\mathbf{Y}$ and $\mathbf{A}$ as well.

Author response:
- **Valid adjustment set**: Theorem 3.2 does not necessitate as set $S$ the optimal adjustment set, but a valid one. In Appendix F.5, it is explained that the conditioning set includes the variables {race, nationality} that consist in the node $C$ of Fig.8. Even though this is a subset of $A \cup W$, the set $C$ represents a valid (not optimal) adjustment set for $(A \cup W, Y)$. Based on this, here selecting the adjustment set $S$ as $C$ is consistent with the theoretical results. Similarly, in the image and synthetic experiments a valid adjustment set is chosen in the conditioning.

Reviewer comment (Theory):
Clarification on the 'Injectivity' Assumption in Theorem 3.2. Fawkes & Evans (2023) shows that CI cannot be decided from the observational distribution unless strong assumptions are made. Theorem 3.2 provides a valid assumption: injectivity of $g$. I searched for the term ‘injectivity’ in the proof but cannot find it, so how does this assumption work here? Furthermore, while the authors claim that ‘Guaranteeing CI necessarily requires strong untestable additional assumptions, but we demonstrated that CIP performs well empirically even when these are violated’, they only provide an example of a specific violation scenario involving the absence of unobserved confounders. It is recommended that the authors conduct additional experiments to demonstrate the method's efficacy under more general violation conditions.

Author response:
- **Clarification on the “Injectivity” Assumption**: We use the set-theoretic notion of “injectivity”, and we are happy to further clarify this point. As mentioned in our previous reply, there are models where the injectivity notion is fulfilled, such as ANMs. In our proof, the injectivity assumption is used in Lemma A.6 to prove that Eq. (5) holds.

Reviewer comment (Experiments):
Scalability and Computational Complexity. The experimental section primarily employs simple causal graphs with a small number of nodes. It is essential to evaluate the scalability of the proposed Counterfactual Invariance Predictor (CIP) on causal graphs with a larger number of nodes, such as 20, 30 nodes. Additionally, it would be better to provide insights into the computational complexity associated with finding the adjustment set and estimating the Hilbert-Schmidt Conditional Independence Criterion (HSCIC).

Author response:
- **Scalability and Computational Complexity**: we are working on experiments with $20$ nodes. In a dataset with $n=1000$ data points, in the same structure as the first experimental setting shown in the paper (Fig. 1(d)), the average running time for an epoch without the regularization term is $0.003 s$ and $1.112 s$ with the regularization term, training with SGD with batch-size of $512$. By using smaller batch sizes, e.g. $n=128$, the extra computational cost can further decrease. From a theoretical perspective, the estimation of the HSCIC requires kernel ridge regression (see eq. (1) in our submission). In the high-dimensional image example, with a mini batch-size of $512$, the average running time for an epoch with the regularization term is $64.03 s$ and $34.01 s$ without. This operation generally requires  $O(n^3)$ and $O(n^2)$ memory, with n the size of the dataset. However, these bounds can be significantly improved by using, i.e., Fourier Features (see, i.e., [2,3]). By using Fourier features, the resulting approximate kernel ridge regression estimator can be computed in $O(ns^2)$ and $O(ns)$ memory. Here, s is a parameter determining the accuracy of the approximation. In practice, s can be set to be significantly smaller than the problem size, resulting in a dramatic speed-up. Other methods for efficient kernel computation include the popular Nystrom approximation [4,5], and Memory-Efficient Kernel Approximation (MEKA) [6].

Reviewer comment (Presentation):
Organisational Improvement. Section 2.2, "Related Work," requires more structured organization. Although the section discusses almost sure conditional independence, distributional conditional independence, and F-CI along with their relationships, it appears that they may not directly relate to the contributions of this work. Therefore, it would be advisable to omit this portion. Moreover, the paper should offer a concise and focused overview of the approaches used in prior research to attain counterfactually invariant predictors.

Author response:
- **Organisational Improvement**: We appreciate the reviewer's suggestion, we think that contrasting our approach with other interpretations of counterfactual invariance is crucial for situating our research within the existing body of literature. In Section 4, under the 'Baselines' paragraph, we present a comparison with various methods employed to achieve counterfactual invariant predictors.

Reviewer comment (Theory):
My main concern is on the usage of counterfactual invariance. Counterfactual refers to individual-level potential outcomes, rather than conditional or sub-populational levels. Hence, pursuing counterfactual outcome relies on prior SCM model or very sharp bounds, and estimating counterfactual outcome from observational is nearly possible even with the aid of AB tests. Hence, as your independence regularization only enforces tthe populational independence, how can your CIP achieves targets in the counterfactual level?

Author response:
We appreciate the concern regarding the application of counterfactual invariance. Our approach indeed focuses on translating counterfactual independence into observational distribution independence, acknowledging the challenges inherent in this transition. We emphasize in Section 2.2 that such a translation is non-trivial and necessitates specific assumptions about the graphical structure and the data generation process, as detailed in Theorem 3.2. This forms a crucial part of our theoretical framework, enabling a more feasible analysis of counterfactuals within observational data, however with necessary limitations and assumptions. Specifically our proof consists of the following steps: 
1. we first construct a causal graph for the pre and post-interventional random variables of the model;
2. we show that $S$ acts as a valid adjustment set for $(A∪W,Y)$ in the graph constructed in (1);
3. we then use the adjustment criterion to prove the desired result. 

Part (3) is not-trivial and it makes use of the assumptions on the DGP as in Theorem 3.2.

Reviewer comment (Novelty):
Such invariance learning is not new to me.

Author response:
Thank you for the observation regarding the novelty of invariance learning. While it is true that the concept of counterfactual invariance is not new, our contribution lies in achieving an established concept of counterfactual invariance ($\mathcal{D}$-CI) within a novel, model-agnostic framework. This is outlined in the Contribution subsection (page 3). We have also conducted a thorough review of related literature to position our work within the current academic literature. We would greatly appreciate any specific references or examples the reviewer might have that could further inform our understanding and discussion of this topic in the context of our work.

Reviewer comment (Theory):
Prior causal graph is restrictive for realistic applications.

Author response:
We agree that knowing the causal graph is a strong assumption. However, essentially all work on cause-effect estimation and causal invariance starts from this assumption. In fact, while the somewhat orthogonal field of causal discovery aims at inferring the causal graph from (observational and/or interventional) data, most other work on causality starts from assuming some background knowledge of the data-generating process, we are aware that this is indeed a potentially problematic assumption. Usually, one refers to “expert knowledge” for how to come up with the graph. Ultimately, dismissing our work on the grounds that the true causal graph is hard to know in practice, is essentially dismissing large parts of the work on causality in machine learning altogether. 

We thank the reviewer for the comments. In light of the above clarifications and our previous discussions, we kindly request the reviewer to reevaluate our work and consider increasing the score.