Paper: Fast Conditional Intervention in Algorithmic Recourse with Reinforcement Learning (ICLR-2024)

Reviewer comment (Theory):
The paper assumes that all subjects share the same prior causal graph. However, in reality, each individual typically possesses a distinct causal graph. To address this concern, De Toni et al. (2022) [2] propose a solution. They initially establish a fixed causal graph and then iteratively learn the subject-specific cost function. Subsequently, they seek an appropriate sequence of interventions.

Author response:
We agree that the idea from De Toni et al. is interesting. However, we would also like to highlight the difference and connection between these two works. Different from De Toni et al. who propose to incorporate user preferences, we focus on improving recourse by considering causality and thus do not require human-in-the-loop. It is true that our model does not personalize recourse; however, we find no apparent conflict of introducing the cost function proposed by De Toni et al. to our model. Specially, given that the two works both consider uncertainty between features, it may be possible to find a probabilistic approach combining the merits (i.e., conditional intervention and personalization) concerning distinct issues.

Reviewer comment (Reproducibility):
The author omits the description of the reinforcement learning algorithm used to solve the MDP and its parameters.

Author response:
The specification of MDP is in Section 3.2.2, and the objective function of the reinforcement learning algorithm, REINFORCE (Williams, 1992), is shown in Eq. 15. We also revise the paper to make the description more clear. The newly added paragraph “Intuitively, when observing state $s^t$ (current feature values), if the action $a^t$ corresponds to a positive cumulative reward compared to baseline $(R^t-b^t)$, we should increase the log probability of choosing $a^t$” is in Section 3.2.3. Finally, we would like to emphasize that the objective function is only for the classifier to output the target class. The total cost we optimize is $L_{agent}$, in which the parameters account for the degree of exploration ($\beta$) and the strength of intervention cost ($\eta$).

Reviewer comment (Writing):
The way the author handles the noisy graphs (incompleteness of the causal graph) is unclear.

Author response:
In our work, incompleteness refers to the unobserved variables and we propose to model the incompleteness via variance as discussed in Section 1. Let us say we have a simple causal graph A → B where there is no unobserved variable. In this case, the estimated variance of $P(B | A)$ shall be near 0 and our model will then choose to intervene A when we need a different value of B. On the contrary, if there exists an unobserved variable C being a parent of B, our model would discover a higher variance in $P(B|A)$ and therefore conclude the existence of C. Our model would then propose to intervene B directly, revealing the fact that changing A might not be an effective means to affect B.

Reviewer comment (Experiments):
In Section 3.2.3, the authors state that architectural corrections can alleviate the instability of the PASVG(0). However, there is no justification or ablation study for this claim.

Author response:
We now report the performance with and without our trick in Appendix C. The performance gaps are small in general; however, we found using gumbel-softmax can occasionally cause not-a-number errors on Sangiovese dataset and thus require more careful tuning.

Reviewer comment (Theory):
In section 3.2.2, when finding the longest path length between $X_i$ and $X_k$, what is the edge weight between two vertices of the graph? Does the algorithm find the longest path on the causal graph?

Author response:
Yes, the algorithm finds the longest path (LP) on the causal graph. The weight of each edge is set to 1. Consider a causal graph with two paths, A->B->C and A->E->F->C. Supposed at one step, the RL agent chose to intervene upon feature A, then firstly B, E (LP(A,B)=LP(A,E)=1) should update to new values because of the treatment effect from A. Secondly, F should update (LP(A,F)=2) because of the treatment effect of E.  Finally C should update (LP(A,F)=3) because of the treatment effects from B and F.

Reviewer comment (Theory):
The reward function and the objective function in Section 3.2.2 are not related to each other, making me confused about interpreting their role in the training.

Author response:
We are sorry for the confusion.The reward we adopt is the cumulative reward $R^t$ rather than reward $r^t$, which appears following eq. 12. This is because the action at step t will have effects on all the steps after t. Therefore, in the objective function (eq. 15), the probability of choosing action $a^t$ is updated according to the cumulative reward $R^t$.

Reviewer comment (Experiments):
The efficiency of training needs more evaluations.

Author response:
Our model and FastAR require training, and the training time of the 2 models are 10 - 50 minutes and 2 hours on Intel Xeon cpu E5-2650-v4. We leave the details to Appendix B.3.

Reviewer comment (Experiments):
About Fig 2. Is this graph representative? It seems the only confounder is U_0, and other Us can be considered as additive noise. Why this graph is used as an example for experiments?

Author response:
In this work, we assume the existence of unobserved variables in the causal graph and thus design Figure 2 as an example for illustrating the idea. Also, we generate synthetic data via nonlinear equations (Appendix B.2) for the experiments in Section 4. Notably, we assume the unobserved variables do not cause spurious correlations. Therefore, we design $U_0$ pointing to one endogenous feature and $U_1$ on the path of observed features as the examples.

Reviewer comment (Experiments):
About the RL part. Is there anything related to the choice of reward, policy that have impacts on the final experimental outcomes?

Author response:
There are two terms in the reward (eq. 12), one for changing the classifier output; the other for controlling the proximity. The parameter $\lambda$ in eq.12 decides the degree of controlling the proximity. There is usually a tradeoff between validity and proximity since some difficult cases require larger interventions to change the classifier output.\
\
As for the policy, we find it better to model the distribution of a numerical feature with a two-component GMM than with a simple gaussian distribution. For example, for a simple logistic regression model $f(x)=\frac{1}{1+e^{-x^2}}$, when $x=0$, increasing or decreasing $x$ both increases the probability of a positive output. A learnable GMM is able to preserve the possibility of the two actions and degrade to a single gaussian distribution when facing simple cases.

Reviewer comment (Theory):
The paper mentions that "The less it is determined by their parents, the more _space_ we can intervene." Could you explain more why that's the case? In particular, what does "space" mean? And why do we want to primarily intervene in higher uncertainty endogenous features?

Author response:
High uncertainty of an endogenous variable given its parents implies there exists an influential factor that is not observed. Namely, the parents are not sufficient to decide the value of the endogenous variable. Let us assume there are two variables A and B with a relation A → B where A fully decides B. If B needs to be higher to change the classifier output, the ideal suggestion from an AR method would be intervening A instead of B, as it is the only way to control B. On the contrary, if B is highly uncertain given A, there may exist a hidden factor C having relation C → B for intervention. As C is not observed, changing B through C is simplified to changing B in our AR method output. The “space” in this case is the uncertainty of B given A. In this work, we model the uncertainty via variance of $P(B|A)$ introduced in Eq.6 and 8.

Reviewer comment (Theory):
Does the size of the action space grow exponentially as a function of the feature space? If so, how does the algorithm handle this?

Author response:
It is true that the action space of discrete features can grow exponentially. To the best of our knowledge, our competitors all face the same challenge while our model has an advantage of inference speed as shown in the experiments.

Reviewer comment (Novelty):
Intuitively, what is the benefit of conditional intervention compared to traditional intervention?

Author response:
The main difference is that conditional intervention considers effects from the parents. 
Consider a teenager of 155cm and 70kg to make the basketball team. In the traditional intervention, height and weight are treated independently. It can result in the intervention of +8 kg to make the team, as 70+8kg seems to be a reasonable weight for a person, if the height is not considered. On the other hand, conditional intervention cares about the fact that (70+8) kg is too much for a 155cm person. It could seek for another option “+2cm and + 4kg” while considering both factors together.