Paper: Cost Adaptive Recourse Recommendation by Adaptive Preference Elicitation (ICLR-2024)

Reviewer comment (Novelty):
The main concern is that I do not see the advantages of this approach compared to the closets-related method PEAR [1]. While it is true that they seem to rely on some sort of causal graph, which might be hard to get, their technique can be in principle applied to different cost functions (even the one presented here) and they provide a Bayesian model to incorporate uncertainty over the user answers. Therefore, it is not clear to me the benefits of ReAP over PEAR.

Author response:
> Therefore, it is not clear to me the benefits of ReAP over PEAR.

There are several benefits of ReAP:
1. ReAP is computationally tractable. ReAP is based on semidefinite programming and geometric calculations to find the questions. All operations in ReAP can be done in less than half a second on a laptop. In contrast, the Bayesian approach proposed by De Toni et al. (2022) is computationally intensive.
2. ReAP delivers a better solution than PEAR. In Table 8, we have shown that our method outperforms PEAR in the Adult and GMC datasets.
3. It is not trivial to generalize PEAR to the Mahalanobis cost setting while using the Mahalanobis distance as in our paper can bridge with causal models, as we described in Appendix D.2. 
4. PEAR requires a linear structural causal model to define the cost of changing a specific feature, whereas our method does not require such a model.

Reviewer comment (Theory):
ReAP disentangle the elicitation part from the recourse generation (Figure 1). However, if the number of features is truly high, we might just need to estimate only the cost of the features we need for recourse, rather than the full matrix $A$. Moreover, it might be desirable that we just ask the user the right questions to estimate the perfect recourse for him/her, rather than the ones minimizing the smallest projection distance.

Author response:
> If the number of features is truly high, we might just need to estimate only the cost of the features we need for recourse rather than the full matrix

We agree that we only need to estimate the cost associated with the features we need for recourse. However, the number of features we need for recourse can also be truly high. In this case, the number of parameters to be estimated/learned is still large.

Reviewer comment (Experiments):
Why did the author not include PEAR as a baseline in the main paper (while it is present in the Appendix)? I think it is the most suitable competitor since its method can be applied in principle to all cost functions, it employs a more targeted approach to the recourse pair selection and it deals with the user’s uncertainty in a most principled way.

Author response:
> Why did the author not include PEAR as a baseline in the main paper

Thank you for your comment. We include PEAR in the appendix for a better explanation of the formulation and the settings of the experiments. We will include these results in the main paper in our final version, in which more space is allowed.

Reviewer comment (Evaluation):
How do you explain the fact that Watcher and ReAP achieve almost the same cost/validity in all four datasets (Table 1)?

Author response:
> How do you explain the fact that Watcher and ReAP achieve almost the same cost/validity in all four datasets?

Regarding validity, we employ an early stopping criterion, terminating the algorithm when a valid recourse is obtained for both algorithms. Therefore, both methods have a perfect (1.0) validity.

The objective function for Wachter is non-convex for a fixed $\lambda$ when we consider an MLP classifier. In particular, if we initialize $\lambda=0$, and then increase the value of $\lambda$, the difference in the cost of recourse compared to the initial $\lambda$ is negligible. Therefore, Watcher and ReAP achieve almost the same cost. However, our methods still show improvements over Wachter in terms of cost in all the experiments.

Reviewer comment (Experiments):
In Figure 4, would it be possible to show the results also for a random strategy to select the recourse pairs to show the user? It is a common check to understand if the experimental setting is not too simple (considering also the concerns of Table 1).

Author response:
> In Figure 4, would it be possible to show the results also for a random strategy to select the recourse pairs to show the user?

Here, we show the comparison of mean rank between the random strategy to select the recourse pairs and the proposed heuristics for $T=0$, $T=5$, and $T=10$:

| Datasets  |Methods               |   $T=0$            |$T=5$               |$T=10$               |
|----------|-----------------------------------------|---------------------|---------------------|---------------------|
|German | Random |  0.26  | 0.24    | 0.15    | 
|    | Ours |  0.13  | 0.10    | 0.07    | 
|Bank | Random |  0.42  | 0.31    | 0.19    | 
|    | Ours |  0.27  | 0.20    | 0.15    | 
|Student | Random |  0.37  | 0.31    | 0.29    | 
|   | Ours |  0.25  | 0.23    | 0.21    | 

These results show that as $T$ increases, the mean rank of our approach decreases faster than the random strategy, indicating that the proposed heuristic is more efficient in our adaptive preference learning framework.

Reviewer comment (Experiments):
What happens if the users are “noisy” in their answers? For example, they gave a wrong answer to the preference questions. Equation 2 seems to accommodate this issue, but I do not see any experiments showing the effect of $\epsilon$ in the cost estimation quality and/or final recourse.

Author response:
> Experiments showing the effect of $\epsilon$ in the cost estimation quality and/or final recourse.

We show the effect of $\epsilon$ on the cost final recourse  for the German dataset in the following table:

| Datasets  |Methods               |   Path cost         |
|----------|-----------------------------------------|---------------------|
|German | FACE |  0.66 $\pm$ 0.48  |
| | ReAP ($\epsilon=0.01$) |  0.53 $\pm$ 0.49  |
|    | ReAP ($\epsilon=0.02$) |  0.55 $\pm$ 0.51  |
|    | ReAP ($\epsilon=0.05$) |  0.51 $\pm$ 0.45  | 
|    | ReAP ($\epsilon=0.1$) |  0.51 $\pm$ 0.45  | 

These results indicate that the path cost shows only minor variations across different values of $\epsilon$. Moreover, our method continues to outperform FACE.

Reviewer comment (Novelty):
What is the novelty of the approaches presented in Sections 4.1 and 4.2? Especially section 4.1 seems a trivial extension of Watcher et al.

Author response:
> What is the novelty of the approaches presented in Sections 4.1 and 4.2?

In Section 4.1 and 4.2, we generalize the two existing counterfactual generation methods: Wachter and FACE. In Section 4.1, We find a recourse that balances the trade-off between the validity and the worst-case cost. In Section 4.2, we aim to minimize the total cost of the sequential recourse subject to the most unfavorable scenario of $A_0$ within the final confidence set. Optimizing over the worst-case objective is novel and has several advantages:

**Optimize over worst-case objective.** The terminal set $\mathcal U_{\mathbb{P}}$ is generally not a singleton as it still contains many possible matrices that conform with the feedback information. To find the recourse, we need to borrow the ideas from robust optimization, which formulates the problem as a min-max optimization problem. Looking at the worst-case situation can eliminate any bad surprises regarding the implementation cost. Practically, the min-max problem manages to provide an acceptable recourse for challenging scenarios within the domain where $A$ is a matrix satisfying $A \preceq I, A \in \mathbb{S}_{+}^d$. This approach also proves valuable when users' responses contain significant noise and inconsistencies, which in turn results in a still large search space for $A_0$ in the final round.

Reviewer comment (Experiments):
Important: So, why wasn’t a comparison conducted against Rawal and Lakkararaju (2022)? I agree that the Bradley-Terry cost model is not expressive enough to capture many realistic cost functions, but then again this criticism should apply to the Mahalanobis family as well. I do not understand why it is claimed without further substantiation that the proposed method would perform better in high dimensions than the Bradley-Terry model, and so I worry that this is a key missing baseline in the current work. It could be that their method works as well as as the ReAP, while being conceptually simpler than ReAP.

Author response:
> Why wasn’t a comparison conducted against Rawal and Lakkararaju (2022)?

Rawal and Lakkararaju (2022) employ the Bradley-Terry model to estimate a universal cost function (one cost function for all users) and then utilize the user input to generate personalized recourse for the user. The setting of Rawal and Lakkararaju (2022) is significantly different from ours. Our setting requires learning a personalized cost function through a preference elicitation process (each user is associated with a different, personalized cost function). Therefore, the comparison between the two methods is not necessarily fair.

Moreover, the method from Rawal and Lakkaraju (2020) is addictive in features. Therefore, it is unable to recover the underlying causal graph. As in our paper, using the Mahalanobis distance can bridge with causal models, as described in Appendix D.2.

Reviewer comment (Evaluation):
What is the size of the solution set returned by DiCE? Is the comparison with ReAP fair in terms of computational cost and subject query cost?

Author response:
> What is the size of the solution set returned by DiCE? Is the comparison with ReAP fair in terms of computational cost and subject query cost?

In our experiments, we find a \textbf{single recourse} for each method to ensure a fair comparison. We report the computational time comparison between our method and DiCE in the table below.  Our method has a significantly lower runtime compared to DiCE.

| Dataset  |German               |Bank               |Student               |
|----------|-----------------------------------------|---------------------|---------------------|
| DiCE | 0.87 |  0.89  | 1.19    | 
| ReAP | 0.23 | 0.25  | 0.31    |

Reviewer comment (Novelty):
Comment: You might also reference https://arxiv.org/pdf/2111.01235.pdf, which investigates using a distribution over plausible cost functions to find a set of recourses that could better satisfy a user.

Author response:
>  Reference

Thank you for your comment. We have included the recommended reference in our revised draft.

Reviewer comment (Theory):
Why optimize over the worst case cost function? How might results vary based on the ground truth cost function distribution and the choice of optimizing over the worst case cost function vs some centroid cost function or expected cost function.

Author response:
> Why optimize over the worst-case cost function?

The terminal set $\mathcal U_{\mathbb{P}}$ is generally not a singleton as it still contains many possible matrices that conform with the feedback information. To find the recourse, we need to borrow the ideas from robust optimization, which formulates the problem as a min-max optimization problem. Looking at the worst-case situation can eliminate any bad surprises regarding the implementation cost. Practically, the min-max problem manages to provide an acceptable recourse for challenging scenarios within the domain where $A$ is a matrix satisfying $A \preceq I, A \in \mathbb{S}_{+}^d$. This approach also proves valuable when users' responses contain significant noise and inconsistencies, which in turn results in a still large search space for $A_0$ in the final round.

Reviewer comment (Reproducibility):
Some details are missing from the experiments (dimension d etc).

Author response:
> What is the dimension d in the experiments? Can you comment on the cost of solving (5) using MOSEK? How does it scale up with d?

All experiments are conducted using standard datasets in the recourse literature. We report the dimension $d$ and the average running time of solving (5) in the following table:

| Dataset  |German               |Bank               |Student               |
|----------|-----------------------------------------|---------------------|---------------------|
| Dimension | 5 |  8  | 9    | 
| Runtime | 0.05| 0.06  | 0.06    | 

The run time of our method is practical for the preference elicitation applications.

Reviewer comment (Experiments):
Queries are chosen from the positive set D_1 only. Does it make sense to consider points from D_0 or new points that have no label (since this is only for the purpose of preference learning), especially in the case of single recourse?

Author response:
> Queries are chosen from the positive set $D_1$ only. Does it make sense to consider points from $D_0$ or new points that have no label?

We select recourse pairs from the positive set D_1 to learn the subject's cost function by comparing two **recourses**. It is also possible to consider points from $D_0$ or new points with no label with minimal change to our learning workflow. However, for the recourse application, asking the subject to compare points in $D_0$ may give the user a false sense of psychological expectation. As such, we only consider the positive set $D_1$ and keep the question as "Between two recourses, which one do you prefer?" so that the users know that the two points under comparison both lead to positive predicted outcomes. This setting will help reduce psychological/cognitive bias to the subject's expectations.

Reviewer comment (Experiments):
The worst-case objective might be too conservative, resulting in longer paths. It would be interesting to compare it at least empirically to other choices (A* for example, or a random A \in U_P).

Author response:
> The worst-case objective might be too conservative, resulting in longer paths. It would be interesting to compare it at least empirically to other choices.
> In Table 2, what is the average path length?

The average path lengths for the two methods, employing $A^\star$ or solving the worst-case objective in Problem (7) are presented in the following table. The results indicate that the path lengths obtained by solving the worst-case objective are only marginally higher than the alternative, highlighting the effectiveness of our proposed method.


| Dataset  |German               |Bank               |Student               |
|----------|-----------------------------------------|---------------------|---------------------|
| ReAP-$A^\star$ | 3.96 |  5.89   | 5.85    | 
| ReAP-WC | 4.14 | 6.07  |  5.91  |

Reviewer comment (Theory):
Sequential recourse: the problem in Eq (7) doesn’t seem like the real problem that needs to be solved. The set U_P updates after every step, and this changes the argmax A and therefore w_ij. Am I missing anything?

Author response:
>  The problem in Eq (7) does not seem like the real problem that needs to be solved. The set $\mathcal U_{\mathbb{P}}$ updates after every step.

We think that the reviewer may have misunderstood the setting here: 
In deployment, the system will pick a fixed value of $T$, and problem (7) will be solved with $\mathcal U_{\mathbb{P}}$ being the terminal confidence set, obtained after we have asked the subject to compare $T$ rounds. Thus, we solve Eq (7) only once.

In the experiments, we conduct the experiments with varying $T$ to study the effects of the number of comparison rounds on the quality of the recourse recommendation. We do not solve (7) multiple times in our workflow.

Reviewer comment (Experiments):
Section 5.1: what is the dimension d in the experiments? Can you comment on the cost of solving (5) using MOSEK? How does it scale up with d?

Author response:
> What is the dimension d in the experiments? Can you comment on the cost of solving (5) using MOSEK? How does it scale up with d?

All experiments are conducted using standard datasets in the recourse literature. We report the dimension $d$ and the average running time of solving (5) in the following table:

| Dataset  |German               |Bank               |Student               |
|----------|-----------------------------------------|---------------------|---------------------|
| Dimension | 5 |  8  | 9    | 
| Runtime | 0.05| 0.06  | 0.06    | 

The run time of our method is practical for the preference elicitation applications.

Reviewer comment (Experiments):
Section 5.3: how is lambda chosen for the gradient-based objective?

Author response:
> How is lambda chosen for the gradient-based objective?

We follow the Wachter implementation of CARLA. First, we initialize $\lambda=1.0$, and then we employ an adaptive scheme for $\lambda$ if no valid recourse is found. In particular, if no recourse is found, we reduce the value of $\lambda$ by 0.05, which is similar to CARLA.

Reviewer comment (Evaluation):
In Table 2, what is the average path length?

Author response:
> In Table 2, what is the average path length?

The average path lengths for the two methods, employing $A^\star$ or solving the worst-case objective in Problem (7) are presented in the following table. The results indicate that the path lengths obtained by solving the worst-case objective are only marginally higher than the alternative, highlighting the effectiveness of our proposed method.


| Dataset  |German               |Bank               |Student               |
|----------|-----------------------------------------|---------------------|---------------------|
| ReAP-$A^\star$ | 3.96 |  5.89   | 5.85    | 
| ReAP-WC | 4.14 | 6.07  |  5.91  |

Reviewer comment (Presentation):
Section 6: “We … extend the heuristics to choose the questions from pairwise comparison to multiple-option questions.” I didn’t see where this was done. Can you point to the right section?

Author response:
> Section 6: “We … extend the heuristics to choose the questions from pairwise comparison to multiple-option questions.” I didn’t see where this was done. Can you point to the right section?

We include the multiple-option questions heuristics in Appendix A.2.