Paper: Counterfactual Fairness With the Human in the Loop (ICLR-2024)

Reviewer comment (Theory):
The motivation of response function \(r\): Are there any other function forms for strategic classification models in ML community? If so, what are the advantages and disadvantages of the assumed models?

Author response:
* A.3 There are other forms for strategic classification. For example, when a cost function is defined on an attribute $U_{i}$ as $c(u_{i}, u_{i}’)$ (which means how much the individual need to cost by changing the attribute), the response function of this attribute is the $\arg \max_{u_{i'}} \hat{Y}(u_{i}’) - c(u_{i}, u_{i'})$.  
 
   The advantage for the response function form we used in our paper is that it is simple yet can reflect a general kind of people’s response. It means that people try to take action to change their status along the direction that will change the prediction mostly. The disadvantage is that when the prediction model is not differentiable, the response function could be out of work.

Reviewer comment (Theory):
The form of distribution \(P(U)\): Does the proposed method require the knowledge about \(P(U)\)? There is no description about this in Theorem 3.1. I am not sure, but is the SCM in Eq. (5) deterministic? I believe that performing MCMC at line 2 in Alg. 1 requires the form of \(P(U)\). In practice, how can we obtain it? Although the real-world data experiments in Sec. 4.2 seem to consider the cases where it is available, it is usually unknown, isn't it?

Author response:
* B.2   Yes, we need the knowledge about the prior distribution $P(U)$. 
 
* B.3  The Eq.5 is deterministic meaning that $\alpha, \beta, \gamma$ and $w$ are fixed. However, we do not know the specific value of $U$ and it is a random variable.  
 
* B.4 Yes, when we used MCMC to get the conditional distribution of $U$, we need to know $P(U)$. It is a common requirement for counterfactual fairness. For the experiment, we used the same prior as the one used in the experiment of Kunser et al. 2017 .

Reviewer comment (Writing):
Example 2.2 is a very important example for illustrating the importance of long-term fairness. However, it seems a bit unclear. Please clearly illustrate what features \(A\) and \(X\) can be considered in this example. For instance, is a credit score included in \(X\)? The sentence connection between "Assume that ..." and "However, in the counterfactual world, ..." is unclear. Why is he/she not qualified? Please elaborate more.

Author response:
* C.1. $A$ is the sensitive attribute which determines different demographic groups.  For example, it can be the gender of the person. $X$ are the features, for example, the income and credit score. We are trying to predict whether the person will default or not, and the credit score it is also included in $X$. Note that credit score is not the same as default probability. 
 
* C.2 We can imagine a case, we can observe the income of a person. For simplicity, assume that the income dictates the qualification of an applicant. And there is a hidden attribute U represents how hard the person works. Both $U$ and $A$ can impact income. So, in the counterfactual world where the person has the same $U$ with different gender may have a different income. If that income is low, the counterfactual person would not be qualified. 
 
   Mathematically, $Y_{A\leftarrow a}(U)$ and $Y_{A\leftarrow a’}(U)$ do not have the same distribution necessarily. However, counterfactual fairness implies that the predictions $\hat{Y}_{A\leftarrow a}(U)$ and $\hat{Y}_{A\leftarrow a'}(U)$ should have the same distribution.

Reviewer comment (Theory):
Example 2.3 is not intuitive. The description in Footnote 2 is correct if the SCM is given. In usual, however, since the SCM is unknown, it is difficult to imagine how we can construct the prediction model only with unobservable inputs \(U\).

Author response:
* C.3 We assumed that SCM is known in our problem which is a common practice (e.g., Kusner et al. 2017)

Reviewer comment (Theory):
In Section 2.2, "the prediction \(\hat{Y}\) in the factual world should be the same as that in the counterfactual world ..." is not exactly correct because the counterfactual fairness is defined via marginalization over \(U\) and hence is probabilistically defined, as stated in the first equation in Sec. 2.2.

Author response:
* D.1 Sorry, you are correct. We meant the distribution of \hat{Y}. We have corrected it in the newly updated version.

Reviewer comment (Theory):
Please clearly state that Theorem 3.1 considers the causal graph structure in Figure 1 (b). In particular, there is no definition of \(U_1, ..., U_d\). Please clearly claim the assumption that these exogenous variables do not include unobserved confounders, i.e., \(X_i \leftarrow U_i \rightarrow X_j\) (\(i \neq j\)).

Author response:
* D.2 The assumption is implicitly included in Eq.5. We also state it explicitly in theorem 3.1.

Reviewer comment (Writing):
Based on the title, I thought it would be a work on counterfactual fairness involving a human (i.e., AI-Human interactions). It’s unclear to me how the human is in the loop in this work -- also, there’s a typo in the title: Counterfactual Fairness with a Human in the Loop. I suggest changing the title to avoid confusion, maybe something along the lines of “Lookahead Counterfactual Fairness” or “Counterfactual Fairness under Strategic Classification”.

Author response:
Yes, we use human in the loop to emphasize that a human is interacting with an AI model and responds to it through a response function $r$. 
 
We appreciate your opinion about the title. We are willing to change the title to Lookahead Counterfactual Fairness if the ICLR policy allows us to change the title.

Reviewer comment (Theory):
From (1), it seems causal sufficiency is assumed for the paper, no? Please clarify if so, and whether it’s a limitation to the problem formulation. Especially in this setting where the authors are considering time (though this somewhat unclear as well) and the risk of hidden confounders increases.

Author response:
Yes, causal sufficiency is required. Generally speaking, counterfactual variables are defined based on the complete causal structure. So, we need to measure all the hidden variables. Both counterfactual fairness (Kusner et al. 2017) and lookahead counterfactual fairness can be studied when the causal structure is known. There are various methods for finding a causal model, see e.g., Peters et al. (https://arxiv.org/pdf/1309.6779.pdf).

Reviewer comment (Theory):
In the motivating example (Example 2.2: starting from “Assume that this person…”), it’s unclear how there is a counterfactual world not captured by the causal graph used for CF. The counterfactual instance generated there to test for counterfactual fairness is specific to the SCM available or assumed (i.e., the worldview). That same counterfactual should be, in principle, the closest possible world to what is observed, no? How is this individual not qualified in the counterfactual world? Are you implying model bias?

Author response:
CF considers both factual and counterfactual worlds. However, it imposes constraint on the prediction without considering the consequence of the decision in factual and counterfactual worlds. We are not claiming that counterfactual fairness is a bad fairness measure and has limitations. Just we are saying LCF and CF have different goals.  In particular, we want to emphasize that while \hat{Y} is the same in the factual and counterfactual world under CF constraint, true Y and future Y’ may not be the same in both worlds. If Y is not the same in both worlds, and we want to make sure that Y’ will be the same in the factual and counterfactual world, then CF fairness constraint cannot handle this situation.  You can see more about why CF does not imply LCF in theorem A.1 we added in the updated version.

Reviewer comment (Theory):
I understand that (correct me if I’m wrong) this individual is unlabeled, but somehow we know that in his/her the factual world \(Y=1\) and in the counterfactual world \(Y^{CF}=0\), while the model being studied for counterfactual fairness gives \(\hat{Y}=\hat{Y}^{CF}=1\). This reasoning I understand, but it’s unclear how and from where this additional information is available. This is important because this ambiguity appears also in the formalization of LFC.

Author response:
In our assumption, the individual is labeled (or at least we can approximate the $Y$     with the causal graph). So, when the individual in the factual world and counterfactual world belongs to the different demographic group (which means different value for the node representing this attribute in the causal graph), it is possible to get $Y = 1$ and $Y^{CF} = 0$ under LCF.

Reviewer comment (Theory):
To me this is clear when looking at the operationalization of LFC, which is based on changing the latent variables \(U_i'\) (Section 3: response function for \(U_i’\)). Under counterfactual fairness, the \(U\)’s are often interpreted as inherent but unobserved skills to the individual (like aptitude). These are given and materialize through the observed variables (like SAT scores). These variables, though, are exogenous to the system. We can infer them using the abduction step, but how exactly is an individual basing his/her response on a variable that is out of his/her control?

Author response:
We want to emphasize that $U$ and $U'$ are not the same variables. $U'$ is the hidden variable after the individual receives a decision. On the other hand, U is the exogenous hidden variable (without any parents) before decision \hat{Y}. U and U' are completely separate variables. U’ can be impacted by $U$ and $\hat{Y}$. As you can see in our model, $U' = r (U, \hat{Y}) $.  
 
If you think of $U$ and $U'$ as two separate variables, then it is easy to see that counterfactuals are non-backtracking in this paper. You can interpret U as an inherent hidden variable (similar to what we have seen in literature) and $U'$ as the hidden variable but it is the effect of $U$ and $\hat{Y}$.

Reviewer comment (Theory):
The formulation in Section 3 seems like a backtracking counterfactual formulation where we imagine a setting were the individual can update the exogenous variables (see Sander et al.’s Backtracking Counterfactuals). The issue here then is that counterfactual fairness is built on non-backtracking counterfactuals, meaning once the exogenous variables are set, all interventions occur in the observable space. Therefore, I don’t understand the second term in the response function for \(U_i’\).

Author response:
If you think of $U$ and $U'$ as two separate variables, then it is easy to see that counterfactuals are non-backtracking in this paper. You can interpret U as an inherent hidden variable (similar to what we have seen in literature) and $U'$ as the hidden variable but it is the effect of $U$ and $\hat{Y}$.

Reviewer comment (Theory):
I’d argue this misconception is due to using the strategic classification framework: these strategic agents are not strategic in that they improve because of \(\hat{Y}\) (as in their given aptitude \(U_X\) changes and \(X\) increases) but are strategic in that they lie about improving so that \(\hat{Y}\) changes (the aptitude \(U_X\) cannot change, but they can lie about \(X\) to give the impression that it does). This type of agent doesn’t seem useful for what LFC aims to achieve.

Author response:
We respectfully ask the reviewer to look at the paper one more time. We strongly believe that if the reviewer reads the paper one more time and considers that $(U, X, Y) $ and $(U', X', Y') $ are two separate variables, the story of our paper would make more sense. You can also see the strategic classification does not cause any issue as $U$ and $U'$ belong to separate time and are different in nature. $U$ is exogenous and no variable is a cause for $U$. While $U'$ is not exogenous and other variables can be a cause for $U'$.

Reviewer comment (Theory):
In the Implementation part for 4.2, the authors argue “we assume that individuals adjust knowledge \(K\) in response to the prediction model \(\hat{Y}\)”. \(K\) here is measured through GPA, LSAT, and FYA (as shown in Figure 4(a)) and is exogenously given to the SCM. How exactly can \(K\) change under the current SCM?

Author response:
Given this explanation, I think our experiment in section 4.2 makes sense. We consider two separate variables $K$ and $K'$. In our setting, the decision $\hat{Y}$ cannot change $K$. $K$ does not change over time because $K$ is the knowledge at the current time. $K'$ is a separate variable and its value is different from $K$.

Reviewer comment (Theory):
Overall, it’s not clear what is the long-term fairness problem LFC addresses (as highlighted by the Law School Success example). What would help LFC is explicitly admitting \(\hat{Y}\) into its formulation. The authors claim that the model has a “downstream impact” on the future outcome \(Y’\). Doesn't this imply \(\hat{Y} \rightarrow Y’\)? Where is \(\hat{Y}\) in (4)? As it is setup, it states that \(X\) and \(A\) have (at least) some association to \(Y’\) but so does \(\hat{Y}\), no? Please clarify.

Author response:
We want to emphasize that LCF takes into account $\hat{Y}$. As we mentioned earlier, $Y'$ and $Y$ are two separate variables, and $\hat{Y}$ is a cause for $Y'$. Therefore, by imposing a constraint on $Y'$, we are imposing certain constraints on $\hat{Y}$ as well.  
 
The whole point of this paper is to make sure that $\hat{Y}$ is designed in a way that equation 4 is satisfied. Even though $\hat{Y}$ does not appear in equation 4, it is a cause for $Y'$. Therefore, we need to design $\hat{Y}$ very carefully to satisfy equation 4.

Reviewer comment (Novelty):
While the application problem is interesting, I think the extension of counterfactual fairness is somewhat incremental. For example, why not simply relabel $Y$ and treat $Y'$ as the actual response for the prediction task? The response function (3) and structural equation for $Y$ are fixed. The intermediate prediction of $Y$ is largely irrelevant, inconsequential since the fairness of $Y'$ is the deciding issue.

Author response:
### For weakness 2 and question 1:
 Note that LCF is not a minor extension of CF. We cannot simply rename $Y$ and treat $Y'$ as the new label. $Y'$ is a consequence of the decision $\hat{Y}$ and depends on both $Y$ and $\hat{Y}$. As discussed in our paper, satisfying LCF is much more challenging, and we cannot achieve LCF using the same approach as for CF. In the new version of our paper, we have also included Theorem A.1 in the appendix to demonstrate that a predictor satisfying CF does not necessarily satisfy LCF. 

Furthermore, we want to emphasize that imposing a fairness constraint on $\hat{Y}$ does not necessarily mitigate harm if we do not consider the future outcome/consequence of $\hat{Y}$. For example, in a loan approval problem, having the same selection rate among different demographic groups may seem like a plausible fairness constraint. However, it is possible that the majority of people in one demographic group are unable to repay the loan and default. Consequently, having the same selection rate across different demographic groups may adversely affect one demographic group. Therefore, we must consider the consequences of our decision. In our model, $Y'$ represents the future outcome/consequence of the predictor $\hat{Y}$.

Reviewer comment (Novelty):
The main text explores the case of linear causal models. The non-linear case would be more interesting for ICLR, and more generally applicable.

Author response:
### For weakness 3 and question 3:
We have already included Theorem 3.3 for the nonlinear causal model in our paper. Additionally, in the new version, we have added Theorem A.2 to demonstrate that our results can be extended to the nonlinear case.

Reviewer comment (Novelty):
Could the Example 3.2 be used to make a stronger case for why LCF cannot be considered a minor extension of CF (after renaming $Y$ and treating $Y'$ as the main outcome)?

Author response:
### For weakness 2 and question 1:
 Note that LCF is not a minor extension of CF. We cannot simply rename $Y$ and treat $Y'$ as the new label. $Y'$ is a consequence of the decision $\hat{Y}$ and depends on both $Y$ and $\hat{Y}$. As discussed in our paper, satisfying LCF is much more challenging, and we cannot achieve LCF using the same approach as for CF. In the new version of our paper, we have also included Theorem A.1 in the appendix to demonstrate that a predictor satisfying CF does not necessarily satisfy LCF. 

Furthermore, we want to emphasize that imposing a fairness constraint on $\hat{Y}$ does not necessarily mitigate harm if we do not consider the future outcome/consequence of $\hat{Y}$. For example, in a loan approval problem, having the same selection rate among different demographic groups may seem like a plausible fairness constraint. However, it is possible that the majority of people in one demographic group are unable to repay the loan and default. Consequently, having the same selection rate across different demographic groups may adversely affect one demographic group. Therefore, we must consider the consequences of our decision. In our model, $Y'$ represents the future outcome/consequence of the predictor $\hat{Y}$.

Reviewer comment (Theory):
Identification of counterfactuals: The authors may want to make it clear if they only consider deterministic counterfactual (each individual has deterministic noise U) or probabilistic counterfactual (each individual has a distribution of U). Note that without strong assumptions, it is often impossible to identify U, especially when U is probabilistic. As U is the input of the LCF predictor g, U has to be identifiable from observational data to make the whole proposed framework work in practice.

Author response:
We are considering the probabilistic counterfactual. In our model, we can only infer the distribution of $U$ conditioned on the observed features. We utilized $U$ as the input for our LCF predictor, similar to Kusner et al. 2017 (Section 4.1), for training a counterfactually fair predictor. We sample the value of $U$ from the conditional distribution and take it as the input. Under the distribution of $U$, our algorithm minimizes the expected error over $U$, $X$, and $Y$. For inference, we can also find the expectation of the output over $U$ (similar to Kusner et al. 2017).

Reviewer comment (Experiments):
Limited baselines and related work: The authors only compare their method with the counterfactually fair predictor from Kusner et al and their literature review is not comprehensive. However, there exists recent work in counterfactual fairness. The authors may want to compare with multiple different methods that aim to achieve counterfactual fairness such as [1,2,3].

[1] Wu, Yongkai, Lu Zhang, and Xintao Wu. "Counterfactual fairness: Unidentification, bound and algorithm." In Proceedings of the twenty-eighth international joint conference on Artificial Intelligence. 2019.

[2] Xu, Depeng, Yongkai Wu, Shuhan Yuan, Lu Zhang, and Xintao Wu. "Achieving causal fairness through generative adversarial networks." In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence. 2019.

[3] Ma, Jing, Ruocheng Guo, Aidong Zhang, and Jundong Li. "Learning for Counterfactual Fairness from Observational Data." In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1620-1630. 2023.

Author response:
We implemented the method used in Jing et al., and the results are as follows: 

|Method|MSE|AFCE|UIR| 
|---|---|---|---| 
|Jing et al.|0.0036 $\pm$ 0.003|1.296 $\pm$ 0.000| 0% $\pm$ 0.000| 

We reported the MSE, which reflects the distance between $\hat{Y}$ and $Y$, and the AFCE, which reflects the average distance between $Y'$ and $\check{Y}'$. Compared to our methods, this method achieves slightly better accuracy, but like other counterfactual fair predictors, it cannot improve LCF at all. UIR represents the ratio by which LCF has been improved. It is 0% for this method, while we can achieve 100%. 

Note that the method proposed in Depeng et al. is only applicable to a classification setting, whereas in our experiment, we are working with a regression problem. We also want to emphasize that the method proposed in Yongkai et al. is for synthetic data generation under intervention, not for training a predictor. Moreover, we could not find the GitHub repository for the Yongkai et al. paper, and their experiments are not reproducible.