Paper: Prediction without Preclusion: Recourse Verification with Reachable Sets (ICLR-2024)

Reviewer comment (Theory):
My major concern regarding this paper is its inapplicability to continuous features, as claimed by the authors. Is it possible to extend the MIP formulation to MILP formulation to incorporate continuous features?

Author response:
> My major concern regarding this paper is its inapplicability to continuous features, as claimed by the authors.

This seems to be a misunderstanding of something that we wrote.

Our approach is applicable to continuous features! Every method that we have in the paper will work on classifiers that use continuous features. Our software is designed to handle them.

The only issue that we have with continuous features is that we may not be able to certify infeasibility in all possible cases. In this regime, we still obtain meaningful results with regard to verification – by certifying feasibility and by flagging certain kinds of infeasibility.

We can show that recourse is feasible by using IsReachable to show that we can reach other points in the sample. Or by certifying that a point produced using another method obeys actionability constraints. 
We can flag infeasibility by detecting only fixed points by calling the FindAction Routine (Eq 5).
In the remaining, our methods will still work and output results that are correct as we will claim “I don’t know” rather than “feasible” and “infeasible”. 

We can show that recourse is feasible by using Theorem 3 with any labeled dataset to set an upper bound for the number of points needed in a reachable set.

Reviewer comment (Theory):
The optimization problem (2) aims to optimize a constant value of 1. What does this objective imply? Does this optimization problem solely seek to find all feasible actions (feasible recourses)?

Author response:
> The optimization problem (2) aims to optimize a constant value of 1. What does this objective imply? Does this optimization problem solely seek to find all feasible actions (feasible recourses)

Yes, this is a correct interpretation.

Reviewer comment (Theory):
Is there a relationship between two reachable sets? For instance, if $x_1$ is within the reachable set of $x$ and $x_2$ is within the reachable set of $x_1$, is it guaranteed that $x_2$ is also within the reachable set of $x$?

Author response:
> Is there a relationship between two reachable sets? For instance, if x1 is within the reachable set of x and x2 is within the reachable set of x1, is it guaranteed that x2 is also within the reachable set of x

This is not guaranteed to be true. Suppose $\mathbf{x_1} \in R(\mathbf{x_0})$ and  $\mathbf{x_2} \in R(\mathbf{x_1})$.  $ R(\mathbf{x_0})$ and $R(\mathbf{x_1})$ may or may not be overlapping. In the case when they are not overlapping then $\mathbf{x_2}$ does not lie in $R(\mathbf{x_0})$. In the case when they are overlapping $\mathbf{x_2}$ may lie in $R(\mathbf{x_0})$ if  $\mathbf{x_2}$ is in the overlapping region. The only time when $\mathbf{x_2} \in R(\mathbf{x_0})$ is guaranteed to be in $R(\mathbf{x_0})$ is when $ R(\mathbf{x_1}) \subseteq R(\mathbf{x_0})$.

Reviewer comment (Writing):
It is unclear how the separable features are identified. What role does the prediction model play in the identification of these features?

Author response:
There seems to be some kind of misunderstanding that we would like to correct. The prediction model plays no role in determining if features are separable or non-separable. 
Separable features are defined as features that can be altered independently under a given action set $A(x)$. For example, the features $reapplicant$ and $n\_accounts$ can be altered independently of one another. However, features like $max\_degree\_BS$, $max\_degree\_MS$, $max\_degree\_PHD$ that arise from a one-hot encoding cannot be altered independently. Likewise, features that are tied together in $X$ like if $is\_employed = true$ then $work\_hrs\_per\_week > 0$.  
In our case, the action set captures all the relationships described above. Given an action set $A(\mathbf{x})$, we can easily identify the features that can be altered independently if the actions from one feature $A(x_1)$ do not affect or are not dependent on the actions in $A(x_2)$. Likewise we can find the actions that are non-separable if the actions in $A(x_1)$ rely on other features.  We can express the action set as a  “product space” of smaller action sets over the features. We can use this to _generate_ reachable sets for each action set in a way that is far more efficient without sacrificing correctness.

Reviewer comment (Theory):
It is unclear what conditions the prediction model must satisfy for the features to be separable. For instance, the verification step may return an infeasibility certificate in partitions A_1(x) and A_2(x), but actionable recourses may still exist in the Cartesian product A_1(x) X A_2(x) of the two sets.

Author response:
This seems to be a misunderstanding that stems from the previous comment on separability. This cannot happen because our partition is designed to specifically avoid this effect. In particular, we would only consider a partition where we can express the actions as $A(x) = A\_1(x_1) \times A\_2(x_2) \times A_m(x_m)$

Reviewer comment (Presentation):
Increasing the font size in Tables 1 and 2 could help improve readability.

Author response:
Thank you for catching these. We have uploaded an updated manuscript with these fixes!

Reviewer comment (Writing):
It seems like a word is missing in the following sentences:  
    1. Pg. 1 — “In fraud detection and content moderation, for example, models should assign fixed [predictions?] to prevent malicious actors from…”  
    2. Pg. 3 — “We can elicit these constraints from users in natural language and convert them to expressions that can [be] embedded into an optimization problem.”

Author response:
Thank you for catching these. We have uploaded an updated manuscript with these fixes!

Reviewer comment (Theory):
Could this approach be extended to certify the existence or non-existence of an abundance of recourse options instead of just one? A single recourse option might not be feasible for everybody, and having multiple recourses could provide more options to people. It might be possible to certify statements like "20% of the actions in the action set would lead to a positive outcome" by querying a random subset of the action set.

Author response:
We thank you for bringing this to our attention! 

Our methods can easily be extended to certify the existence or non-existence of multiple options. We can count the number of recourse options in a reachable set as a measure of “abundance.” Given a complete reachable set, we can certify existence or non-existence. Given an interior approximation, we can only certify existence. 
Stepping back, we think that the measures are also interesting as a richer measure of feasibility when recourse is feasible - e.g., if there is only 1 point in a reachable set that would lead to recourse, then recourse is brittle and existing methods for recourse provision are unlikely to find it (i.e., high likelihood of blind spot). We will mention this in our manuscript!

Reviewer comment (Theory):
Different actions may have different costs for the subjects. For instance, it might be easier for a loan applicant to increase their credit score than their income. Could we incorporate costs for the actions and certify the existence of a low-cost recourse?

Author response:
Yes this can be done. Here are the steps:
Compute the reachable set  such that $\{ \mathbf{x} + \mathbf{a} \mid \mathbf{a} \in A({\mathbf{x})} \text{ and } \mathbf{a} \in \text{cost}(\mathbf{a};\mathbf{x}) \}$
We can now filter the reachable set $R(\mathbf{x})$ to only include reachable points such that $\text{cost}(\mathbf{x},\mathbf{x_0}) ≤ B$ where $B$ represents a budget. Filtering out points after the reachable set has been computed allows us to generate the reachable set once and reuse it to handle different budgets. 
We do not explore this for two reasons. The first is because costs and budgets are difficult to elicit in a way that most people would agree on. The second, relatedly, is because we would like to flag infeasibility in a way that is not subject to debate. So if a model is precluded access, this is arising because of the oversight of basic actionability constraints rather than because we have elicited the wrong costs or set the wrong budget.

Reviewer comment (Novelty):
My biggest concern lies in the lack of contribution in the verification methods, which directly follow the basic idea of formal verification but seem not to dive deeper into the optimization algorithms or target the specific challenge in the recourse setting.

Author response:
Thank you for mentioning this! Some of this may stem from a lack of familiarity with work on algorithmic recourse. If so, please read our main response above. In short, formal verification is urgently required in this task – since models can assign fixed predictions that permanently preclude access from credit, employment, or assistance. The task is also challenging from a technical standpoint – as we must certify the stability predictions of a model over the set of actionable perturbations, which are neither convex nor smooth. This lack of structure makes it a very difficult problem to address, especially in a way that is model-agnostic. In contrast, existing robustness verification methods (e.g. [1]) primarily focus on structured and nice perturbation sets. We discuss these issues in Section 3, but we are happy to make it more prominent if you need.

Reviewer comment (Experiments):
When introducing reachable sets, more details are expected to be discussed, i.e. continuous or discrete, $\ell_p$-norm bound ball. The verification seems to be sound but incomplete, and it is expected to be compared to more off-the-shelf reachibility-based verification methods in [1].

Author response:
There seems to be a big misunderstanding that we would like to clarify! We are very familiar with the reachability-based methods to verify deep neural networks presented by Liu et al. [1]. One issue is that these algorithms are designed for deep neural networks, so they would not naturally work for complex model classes like RFs and XGB. **The main issue is that these algorithms will fail for “recourse verification” because they cannot verify the stability of predictions under *actionable perturbations.***

In particular, the algorithms are designed to certify that the prediction of a DNN at $\mathbf{x}_0$ will not change under perturbations in a smooth and convex set  – e.g., the set of all perturbations within a $\ell_p$-Ball. In a recourse verification task, however, we are exclusively interested in certifying that a model’s predictions at $\mathbf{x}_0$ would not change under the set of actionable perturbations: $\\{ \mathbf{x} = \mathbf{x}_0 + \mathbf{a} \mid \mathbf{a} \in A(\mathbf{x}_0) \\}$

This set of actionable perturbations is neither smooth nor convex. This is because the action set $A(\mathbf{x}_0)$ is neither smooth nor convex in the vast majority of applications we care about (i.e., classification tasks with semantic feature spaces). For example, $A(\mathbf{x})$ would require discrete perturbations for binary and integer-valued features like $n\\_accounts$. This is also the case for non-separable constraints that we need to enforce actionability over categorical variables like job_type or capture logical relationships like if $is\\_employed = \texttt{TRUE} \quad \implies\quad hrs\\_worked\\_per\\_week > 0$. 

We are happy to include a comparison in a potential revision. However, we do not think that this would be informative since it would show that the methods fail at a task that they need to be designed to handle. In particular, we would expect to obtain a number of loopholes and blindspots using these approaches as we would have to work with a convex relaxation of the perturbation set.

Reviewer comment (Experiments):
Although experiments show the prediction without recourse and current methods fail to detect them, there are no other baselines of recourse verification for the comparison of tightness and time efficiency. Also, the experiment part is not well organized in the sense of merging section 4 and 5 as experiments.

Author response:
This might also be a misunderstanding related to the issue above. This is the first paper to present an actual procedure for recourse verification, so there are no other baselines to compare to. There are also no other off-shelf methods that are designed for recourse verification in this setting – in part because we are exclusively interested in certifying stability over “actionable” perturbations where the perturbation set is almost always non-smooth and non-convex.

With this in mind, we did want to provide a detailed account of tightness and time efficiency. We discuss these issues in Section 3. To reiterate, the MIP is solved to optimality within ≪ 1s for the instances we consider in Section 4. In general, the core property that we exploit is that action sets can be decomposed to compute reachable sets. In Figure 3, for example, we can generate the largest reachable set that we would need in $\leq3$ seconds. Computation and time efficiency are important considerations since they may often present a barrier to adoption. Our results above suggest that these issues should not be a bottleneck in most practical applications. In the event that they are the algorithms have the benefit that they use limited memory and can be easily run in parallel over multiple CPUs. Another key benefit is that we only need to call these procedures once – as the reachable set that we recover can be used for all possible models

Reviewer comment (Experiments):
It is unclear how often the undesired preclusion occurs in practice. In particular, continuous features are quite common and may trivially avoid preclusion if the capacity of the predictive model is not constrained. Even if we focus on discrete features only, it is still uncertain if undesired preclusion can frequently happen. For instance, let us assume users A and B who pass and got rejected respectively by a predictive model. If we ignore the cost, an easy recourse for A can be the difference between A and B in the feature space. If there are more users getting approved by the model, more candidates of recourse are available for A's actions; namely, it is more unlikely that we find no proper recourse for user A when data size grows. If the diversity of the approved users is so limited that no recourse can be found for user A, it is worth questioning whether the preclusion is then more like intended (e.g., setting up strict rules) instead of being an accident.

Author response:
> It is unclear how often does the undesired preclusion occur in practice. In particular, continuous features are quite common and may trivially avoid preclusion if the capacity of the predictive model is not constrained…. If the diversity of the approved users is so limited that no recourse can be found for user A

We think there may be a big misunderstanding, and we’d like to get to the bottom of it. To make things easier, let’s consider an extreme version of your setting. Say we have a classifier $f$ that approves *one* user with features $\mathbf{x}_\textrm{app}$ and rejects all other users with features $\mathbf{x}_i \neq \mathbf{x}\_{app}$ . Formally, we have:

\begin{align} 
f(x) = 1 \text{ for all } \mathbf{x} = \mathbf{x}\_\textrm{app} \\\\
f(x) = -1 \text{ for all } \mathbf{x} \neq \mathbf{x}\_\textrm{app}
\end{align}

Given a person with features $\mathbf{x}\_i$, let us define the change they need to reach features $\mathbf{x}\_\textrm{app}$ as $\mathbf{a}\_i = \mathbf{x}\_\textrm{app} - \mathbf{x}\_i$. Your review seems to suggest that we would claim that person $i$ does not have recourse when $\mathbf{a}\_i  \notin A(\mathbf{x}\_i).$ To be clear, this is not the case. When $\mathbf{a}\_i \notin A(\mathbf{x}\_i)$, this only means that person $i$ cannot reach $\mathbf{x}_\textrm{app}$. However, we would not claim “preclusion” because there may be many other points that person $i$ could reach that lead to approval. 

This is because we search over all possible changes that a user can perform - not simply the changes to reach users that are approved.

Our methods are designed to handle this search because we search for all possible actions that a person could perform over the feature space (i.e., rather than only the features of users in, e.g., a training dataset). **Thus, the “diversity of users” in a dataset will not influence a claim of preclusion.**
When the feature set is discrete, we can search over all points in the discrete exhaustively, and thus claim that a person $i$ has recourse or does not have recourse. 
When the feature set is continuous, we can only claim that a person has recourse. If we cannot find a point for which a person has recourse, then we claim that we do not know.

Reviewer comment (Theory):
Following the previous question, it is worth considering if it is reasonable to adopt the idea of cost constraint in recourse provision to reduce the reachable set. For example, we could certify if a model is not "fixed" given an upper bound of cost.

Author response:
> Following question 1, I am wondering if it is reasonable to adopt the idea of cost constraint in recourse provision to reduce the reachable set? For example, we certify if a model is not "fixed" given an upper bound of cost.

Yes this can be done. Here are the steps:
Compute the reachable set  such that $\\{ \mathbf{x} + \mathbf{a} \mid \mathbf{a} \in A({\mathbf{x})} \text{ and } \mathbf{a} \in \text{cost}(\mathbf{a};\mathbf{x}) \\}$
We can now filter the reachable set $R(\mathbf{x})$ to only include reachable points such that $\text{cost}(\mathbf{x},\mathbf{x_0}) ≤ B$ where $B$ represents a budget. Filtering out points after the reachable set has been computed allows us to generate the reachable set once and reuse it to handle different budgets. 


We do not explore this for two reasons. The first is because costs and budgets are difficult to elicit in a way that most people would agree on. The second, relatedly, is because we would like to flag infeasibility in a way that is not subject to debate. So if a model is precluded access, this is arising because of the oversight of basic actionability constraints rather than because we have elicited the wrong costs or set the wrong budget.