Paper: Unprocessing Seven Years of Algorithmic Fairness (ICLR-2024)

Reviewer comment (Writing):
I found the technical description of how to achieve relaxed parity jarring from the rest of the paper. I would have liked this section to be longer and for more explanations there. I did find the figures quite helpful in understanding it.

Author response:
Thank you for raising this concern.

Achieving error-rate parity was detailed by Hardt et al. (2016). The relaxed solution is more nuanced (since we can no longer rely on the intersection of all group-specific ROC curves), but the key idea is similar: using randomized thresholds allows us to access the convex hull of each group-specific ROC curve (while deterministic thresholds only allow us to access specific discrete ROC points). This makes it so the optimization domain is convex (the group-specific ROC convex-hulls), and hence easy to optimize over. While this is an efficient solution to the problem, a simple brute-force approach would also be possible (brute force implementation example [here](https://anonymous.4open.science/r/error-parity-8550/examples/brute-force-example_equalized-odds-thresholding.ipynb)). 
As linear optimization has been widely studied in the literature, we cite reference works such as Boyd and Vandenberghe (2004) for details on how to solve the LP.
According to reviewer feedback, we were in fact torn between keeping this section in the paper or in the appendix, but as per your feedback we will maintain it in the main paper body.

Reviewer comment (Novelty):
I found their approach intuitively obvious: Of course given a classifier, you can vary how much it violates a reward constraint in an optimal way. So I think the contribution here would be because (it seems like) no one has done this before rather than because it is so interesting.

Author response:
We agree that our approach is definitely intuitive. At the same time, the algorithmic fairness literature contains a wide range of fairness-aware inprocessing and preprocessing methods that claim to dominate results achieved by a simple postprocessing baseline. In our work, we show that this outperformance can be due to (1) comparing models at different levels of constraint violation, or (2) postprocessing a lower-performance model; both are arguably instances of unfair evaluation standards. Acting on each stage of the ML pipeline undoubtedly has specific advantages (e.g., preprocessing is potentially compatible with any downstream task/model, and can be useful when you have to hand over the data to a third party). With this meta study, we hope to have made the advantage of postprocessing clear: given accurate risk-score estimates, postprocessing can retrieve any optimal fairness-accuracy trade-off.

Reviewer comment (Experiments):
I would like to see a comparison between the real unconstrained model and the unprocessed version of the constrained model. This comparison is necessary and could enhance the claim that unprocessing can be applied to find the optimal unconstrained model.

Author response:
Thank you for this suggestion. We've **added these comparisons in a new Appendix A.6** to the latest paper revision. Namely, we compare fairness and accuracy results for postprocessing and fairness-constraining the same model class (e.g., FairGBM compared to postprocessed GBM), or unprocessing and unconstrained training of the same model class (e.g., unprocessed FairGBM compared to GBM).

Please let us know of any further suggestions you'd find beneficial for the presentation of our work.

Reviewer comment (Novelty):
Section 4 is just a standard LP problem in solving Equal Odds with post-processing. It is not novel and there is no need to write down it in the main paper.

Author response:
We agree that our most significant contributions are empirical, and describing the LP is not particularly novel. However, as all our findings rely on the proposed relaxed postprocessing method, we believe that a clear definition of this optimization problem should be in the body of the paper; additionally clarifying how exactly relaxed postprocessing differs from strict postprocessing (whose LP solution is detailed by Hardt et al. (2016)). We admit that a balance between lack of context and unnecessarily detailed explanations is hard to strike, as other reviewers even suggest that this section should be lengthened.

Reviewer comment (Experiments):
How efficient is it to solve the LP problem? Can I just exhaustively search all the combinations of the thresholds and plot the Pareto frontiers of the fairness-accuracy trade-offs?

Author response:
Thank you for bringing this up. The LP formulation is actually paramount to the usefulness of our method, as an exhaustive search over all threshold combinations scales exponentially with the number of groups, and scales quadratically with the number of thresholds that the underlying predictor accepts. To clarify, an exhaustive search would have to span all combinations of group-specific _randomized thresholds_ in order to access the interior of each group's ROC curve (by using deterministic single-value thresholds we can only access specific discrete points in the exterior of each group's ROC curve). That is, each group’s decision function is represented by 2 values, $\{(\underline{t}_s, \overline{t}_s) : (\underline{t}_s, \overline{t}_s) \in \mathcal{T}^2 \land  \underline{t}_s \leq \overline{t}_s\}, \mathcal{T} \subseteq \mathbb{R}$. Samples of group $s$ whose score is lower than $\underline{t}_s$ are classified negatively ($\hat{Y}=0$), samples whose score is higher than $\overline{t}_s$ are classified positively ($\hat{Y}=1$), and samples whose score is in the range $\left[\underline{t}_s, \overline{t}_s\right]$ are classified randomly by a coin toss [Hardt et al., 2016, Section 3.2].

For example, a coarse search grid over deterministic thresholds of $\mathcal{T} = \{0, 0.1, 0.2, ..., 1.0\}$, includes randomized thresholds $\{ (0.0, 0.0), (0.0, 0.1), ..., (0.0, 1.0), (0.1, 0.1), (0.1, 0.2), ... \}$, a total of $\frac{\left|\mathcal{T}\right|(\left|\mathcal{T}\right|+1)}{2}$ combinations (scales quadratically with $\left|\mathcal{T}\right|$). Perhaps more importantly, if $\mathcal{A}$ is the set of randomized thresholds, the search space will span $\mathcal{A}^{|\mathcal{G}|}$ different threshold combinations, where $|\mathcal{G}|$ is the number of sensitive groups.

We've implemented an exhaustive-search solver and **added an example notebook** to the examples folder in the anonymized repository linked in the paper ([link here](https://anonymous.4open.science/r/error-parity-8550/examples/brute-force-example_equalized-odds-thresholding.ipynb)). Running an extremely coarse grid of $|\mathcal{T}|=11$ thresholds on our main experiment ($|\mathcal{G}|=4$) leads to over $9\text{M}$ combinations. With our implementation, a small experiment using $|\mathcal{G}|=2$ and $|\mathcal{T}|=8$ takes 3 minutes to run over the $4356$ combinations with an exhaustive-search solver, while the LP solver takes 109ms to achieve a superior solution (because the search grid is finer).

Reviewer comment (Theory):
The weakness of the paper comes from the lack of a certain level of theoretical derivation to justify the empirical findings.

Author response:
Thank you for your response. We appreciate that your review acknowledges the strength of our "extensive empirical experiments". This is indeed our main contribution. 

Rest assured, we're intimately familiar with Theorem 5.6 in Hardt et al (2016). Theorem 5.6 shows that assuming a predictor is (close to) Bayes optimal, its postprocessed version is (close to) optimal among all equalized odds classifiers. This is a wonderful theoretical result, but it only strengthes the significance of our empirical results. The reason is that the theorem, of course, cannot tell you if its assumptions are met in real settings. Indeed, there is no general way of knowing what the Bayes error is on any given dataset. Likewise, there is no way of knowing if classifiers trained in real settings are close enough to Bayes optimal (in the Kolmogorov distance required by the theorem) in order for the theorem to kick in. Our empirical work shows that this is the case, thus settling a question left open for many years by the theorem you cite.

You may have misunderstood the role of unprocessing in our work. Unprocessing is not a method we propose to achieve a fairness constraint. Unprocessing is a tool we use to put our empirical evaluation on a level playing field. The issue is, a priori, that one fairness method may be superior to another simply because it starts from a better unconstrained classifier. By unprocessing all methods first, we put them on a level playing field. It's a tool, both necessary and useful, for the fair evaluation of different methods. The fact that unprocessing is, intuitively speaking, the inverse of postprocessing is the whole point. This, again, does not diminish the significance of our empirical work. Rather it's a contribution we make to point out that unprocessing can be used to compare fairness methods that start from different base models. This is a useful idea that we're sure will find use in future work.

We sincerely hope that you will reconsider your negative rating of our work. We think that it is entirely reasonable for an ICLR paper to be applied and empirical, rather than theoretical. Thank you very much for your efforts in carefully evaluating our work.

PS: The datasets we use are widely accepted to be superior to many earlier datasets (such as German credit - 1000 data points from 1994, or UCI Adult) that were used in the area before. This case was argued in Ding et al. (2021) and researchers have since generally switched to the datasets we use. Especially given the massive scale of our empirical evaluation, it is important to have a sufficiently large dataset to make all comparisons with high confidence.

Reviewer comment (Writing):
The proposed term "unprocessing", as noted by authors, "roughly corresponds to the inverse of postprocessing", is more or less confusing (for reasons detailed in Section __Questions__).

Author response:
(...) I am not sure how to understand the relation between unprocessing and postprocessing.
 
(...) provide a clear characterization of the relation between unprocessing and such definition of postprocessing, so that readers can understand why unprocessing is a helpful analyzing tool to understand the importance of postprocessing.

The proposed postprocessing method is solely a more general version of the method of Hardt et al. (2016) that is now compatible with relaxed fairness constraint fulfillment. We call _unprocessing_ to the specific case of postprocessing where the fairness constraint was infinitely relaxed, $r=+\infty$. That is, unprocessing boils down to minimizing each group's loss over each independent group threshold (finding which point in the group's ROC achieves minimal loss).

Strict postprocessing will map an *unconstrained* score-based predictor to a *fairness-constrained* classifier. We use the term "unprocessing" to intuitevely capture the reverse procedure: mapping a *constrained* score-based predictor to a *fairness-unconstrained* classifier. Of course, either procedure can be applied to any predictor, constrained or unconstrained alike, but applying unprocessing to an unconstrained predictor will expectedly not significantly change its accuracy/fairness (discussed in the last paragraph of Sec. 2.2).

To clarify with an example: given a standard unconstrained classifier (e.g., GBM), strict postprocessing will map it to the fairness-accuracy space of constrained classifiers (e.g., FairGBM); on the other hand, given a constrained classifier such as FairGBM, _unprocessing_ will do the inverse mapping, outputting a classifier that will approximately sit in the fairness-accuracy space of unconstrained GBM models.

We have added a **new Appendix A.6** with detailed comparisons between unprocessed models that were trained in a fairness-constrained manner, and standard unconstrained models; as well as comparisons between postprocessed models that were trained in an unconstrained manner and inprocessing fairness-constrained models.
Hopefully this will further clarify the motivation behind unprocessing and postprocessing.

Reviewer comment (Theory):
Question 2: Regarding the claim that _any_ Pareto-optimal tradeoff can be achieved by postprocessing. Follow up to Question 1, if the postprocessing is defined as in Hardt et al. (2016), it would be very helpful if authors can provide a clear characterization of the relation between unprocessing and such definition of postprocessing, so that readers can understand why unprocessing is a helpful analyzing tool to understand the importance of postprocessing. Empirical evaluations can be strengthened by some certain level of theoretical analysis to make the results and message more convincing.

Author response:
(...) I am not sure how to understand the relation between unprocessing and postprocessing.
 
(...) provide a clear characterization of the relation between unprocessing and such definition of postprocessing, so that readers can understand why unprocessing is a helpful analyzing tool to understand the importance of postprocessing.

The proposed postprocessing method is solely a more general version of the method of Hardt et al. (2016) that is now compatible with relaxed fairness constraint fulfillment. We call _unprocessing_ to the specific case of postprocessing where the fairness constraint was infinitely relaxed, $r=+\infty$. That is, unprocessing boils down to minimizing each group's loss over each independent group threshold (finding which point in the group's ROC achieves minimal loss).

Strict postprocessing will map an *unconstrained* score-based predictor to a *fairness-constrained* classifier. We use the term "unprocessing" to intuitevely capture the reverse procedure: mapping a *constrained* score-based predictor to a *fairness-unconstrained* classifier. Of course, either procedure can be applied to any predictor, constrained or unconstrained alike, but applying unprocessing to an unconstrained predictor will expectedly not significantly change its accuracy/fairness (discussed in the last paragraph of Sec. 2.2).

To clarify with an example: given a standard unconstrained classifier (e.g., GBM), strict postprocessing will map it to the fairness-accuracy space of constrained classifiers (e.g., FairGBM); on the other hand, given a constrained classifier such as FairGBM, _unprocessing_ will do the inverse mapping, outputting a classifier that will approximately sit in the fairness-accuracy space of unconstrained GBM models.

We have added a **new Appendix A.6** with detailed comparisons between unprocessed models that were trained in a fairness-constrained manner, and standard unconstrained models; as well as comparisons between postprocessed models that were trained in an unconstrained manner and inprocessing fairness-constrained models.
Hopefully this will further clarify the motivation behind unprocessing and postprocessing.