Pushing the limits of fairness impossibility: Who’s the fairest of them all?
Abstract
The impossibility theorem of fairness is a foundational result in the algorithmic fairness literature.
It states that outside of special cases, one cannot exactly and simultaneously satisfy all three common and intuitive definitions of fairnessdemographic parity, equalized odds, and predictive rate parity.
This result has driven most works to focus on solutions for one or two of the metrics.
Rather than follow suit, in this paper we present a framework that pushes the limits of the impossibility theorem in order to satisfy all three metrics to the best extent possible.
We develop an integer-programming based approach that can yield a certifiably optimal post-processing method for simultaneously satisfying multiple fairness criteria under small violations.
We show experiments demonstrating that our post-processor can improve fairness across the different definitions simultaneously with minimal model performance reduction.
We also discuss applications of our framework for model selection and fairness explainability, thereby attempting to answer the question: who’s the fairest of them all?
Introduction
While fairness in machine learning has received significant attention in recent years, most existing works focus on one of the many fairness criteria [3].
Consequently, practitioners are perhaps left with no choice but to use their best judgment to apply a single fairness criterion.
We suspect that the conflicting nature of existing mathematical definitions of fairness might have led to this undesirable practice of narrowing down fairness-related measurement and mitigation to one chosen definition.
This is somewhat analogous to the trade-off between precision and recall while evaluating model performance [7].
Instead of choosing one of precision or recall to evaluate the performance of a classification model, practitioners often evaluate the trade-off and choose models that can maintain a certain level of precision while optimizing recall or vice-versa [19].
In this paper, we provide a framework for explicitly addressing such trade-offs among multiple fairness criteria and model performance toward optimal model selection.
One of the most prolific examples of fairness in machine learning arose from the ProPublica recidivism study ([2]), in which a risk assessment tool called COMPAS was found to be biased against black defendants.
But beyond the immediate implications in criminal justice, the study also prompted more general studies in algorithmic fairness and, in particular, led to a key result highlighted in [13], [21], and [25] which some colloquially refer to as the "impossibility theorem" in fairness [30, 24].
This theorem essentially states that three common definitions of algorithmic fairness - demographic parity [16], equalized odds [21], and predictive parity [3], cannot be simultaneously satisfied outside of pathological situations.
Several works following these initial results have therefore focused on satisfying one metric ([21, 32]) or proposing adaptable methods for different metrics ([11, 37]).
Yet, while we do not deny the conclusions of the impossibility theorem, we also believe there have not been sufficient efforts to reconcile the conflicting fairness definitions to the best extent possible.
We fill this gap in the literature by translating the trade-offs among multiple fairness criteria and model performance into a constrained optimization problem and propose a post-processing methodology for simultaneously achieving approximate fairness in the conflicting definitions simultaneously.
We believe our framework would alleviate the practitioner from making the hard choice in choosing a particular metric.
Instead, if they have a partial ordering of importance amongst the metrics (which many possess), our framework would explicitly allow them evaluate such trade-offs.
The application which we highlight in Section 4 discusses these in detail.
Overall, we make three main contributions in this work:
We design a flexible optimization framework that returns a post-processing score transformation function that can make scores group-wise ϵ -fair along three definitions (demographic parity, equalized odds, and predictive rate parity) simultaneously.
This framework can be applied to any binary classifier that produces a continuous score, can be configured for singular or multiple metrics of fairness, and also can account for fairness vs. performance trade-offs in terms of ROC-AUC.
We present a novel reformulation of this non-convex optimization problem as a Mixed Integer Linear Program (MILP) [41].
This reformulation allows us to find provably globally optimal solutions.
We further show that in practice, we can consistently find better solutions through our global optimization method compared to local optimization methods in a reasonable time.
We discuss and extend our framework from a post-processing mechanism to a tool that can aid practitioners in better understanding their data and models’ empirical fairness characteristics and trade-offs and compare these traits across models.
The rest of the paper is organized as follows.
In Section 2 we mathematically define fairness metrics and the multiple fairness optimization problem.
We discuss the optimal solution via MILP in Section 3.
We try to answer the question of “who’s the fairest of them all?”
through our applications and experiments in Section 4.
Finally, we conclude with a discussion in Section 5.
We wrap up this section with a discussion of the related literature.
Related Work: Early works [13] exploring the conflicts between fairness definitions prove that for a binary predictor, predictive rate parity conflicts with equalized odds ([21]) unless base rates are equal or the model is perfectly predictive.
Chouldechova [13] also considers trade-offs between the fairness definitions in the binary prediction case.
Kleinberg et al.
[25] generalizes this study by showing that statistical (i.e., demographic) parity is also inconsistent with predictive rate parity and equalized odds.
The same paper studies these inconsistencies in a more general bin-wise prediction setting and shows that approximate fairness definitions (predictive rate parity, equalized odds) can simultaneously hold but only under ϵ -approximate equal base rates or ϵ -approximate perfect performance.
This work further proves that there is an inherent trade-off between fairness and loss.
Beyond impossibility theorem results, several works have focused on trade-offs between fairness and model performance [21, 17, 37, 42].
They develop in-processing solutions aimed at reducing one metric while maintaining accuracy.
A few authors have analyzed trade-offs or attempted to achieve multiple fairness.
One example is Pleiss et al.
[36], which shows that predictive rate parity and a relaxed form of equalized odds are reconcilable under a randomized prediction scheme.
Another notable example is Celis et al.
[11], which develops a flexible in-processing approach to achieve multiple types of fairness (potentially at the same time).
Like Celis et al.
[11], the framework we develop also aims for a flexible approach to focus on one or many fairness metrics simultaneously.
However, our method is distinct in that it is a post-processing based solution and also more general as it works for continuous scores (rather than binary classification).
Our work is most closely related to Nandy et al.
[32] in terms of the underlying score transformation mechanism, and we leverage some of their methods.
However, whereas [32] only targets equalized odds, we go further and include demographic parity and predictive rate parity in our framework—this posits computational challenges, which we address by proposing novel methodology based on integer programming.
As noted at the end of [25], some open questions are how to optimally assign scores to satisfy multiple criteria when base rates are equal and additionally, how to satisfy predictive rate parity and either equal TPR or TNR when one cost outweighs the other.
To our knowledge, no prior work has attempted to reconcile all three fairness conditions (demographic parity, equalized odds, and predictive rate parity) simultaneously with model performance through a post-processing framework.
Although the post-processing methods tend to be less flexible for fairness-performance trade-offs than their in-processing counterparts, they can be much more easily added on top of any existing model training pipeline.
This makes a post-processing approach modular, and in particular, more appealing in complex web-scale recommender systems that use (a combination of) certain prediction scores to rank a list of items.
Multiple Fairness Optimization
We consider a binary classification problem where the i -th observation is characterized by their label y i ∈ { 0 , 1 } , their group membership g i ∈ G , and model predicted probability (also known as a risk score [25]) s i ∈ [0 , 1] for i
= 1 , . .
N .
The corresponding random variables are denoted as Y , G and S respectively.
To set up the problem, we discretize the scores into nonempty bins b ∈ B := { 1 , . .
. , |B|} by using, for example, a quantile transformation (we will denote |B| as B ).
Additionally, let N [ g ] b + denote the number of group g positive class ( y i = 1 ) instances in bin b , N [ g ] b denote the total number of instances of group g instances in bin b , N [ g ]+ ( N [ g ] − ) be the total number of group g positive (negative) instances, and N [ g ] be the total number of group g instances.
Lastly, our approach seeks to achieve fairness by moving instances from one bin to another bin: hence, we define variable x
[ g ] bb ′ as the probability of moving an instance of group attribute g and score in bin b into a new bin b ′ 2 .
In other words, for every group g , the collection { x [ g ] bb ′ } b,b ′ can be represented as a B × B transition matrix (with additional constraints, as discussed next).
For the optimization framework and the remainder of this paper, we translate a single fairness definition (e.g. equal true positive rate) as a constraint that controls for the worst-case violations across all bins.
Below, we discuss different fairness constraints that we consider in our framework, and describe how they can be represented in terms of the optimization variables { x [ g ] bb ′ } .
2.1 Fairness Constraints Under Binning Framework
Demographic Parity (DP): For simplicity, we assume that there are exactly two groups g ∈ { 1 , 2 } as we formulate the fairness metrics, although everything discussed below can be applied for non-binary groups by using pairwise constraints, which we elaborate on in Section 2.
Starting with demographic or statistical parity from [16], this condition states that the model’s predicted score is independent of group membership.
This is equivalent to
[Equation]
Our version of this constraint uses bins B to empirically approximate the probability P ( S = s |
G = g )
and we also relax the equality to an ϵ -approximate equality (for some pre-specified ϵ > 0 ).
Therefore, after transforming the scores using { x [ g ] bb ′ } b,b ′ the ϵ DP -approximate DP can be expressed as the following as a linear constraint:
[Equation]
where N [ g ] b denote the number of observations from group g in bin b (before transformation), and
N [ g ] = (cid:80) b ∈B N [ g ] b .
For reproducibility, ϵ DP should be chosen to be larger than the approximation error O (1 / √ N [ g ] ) for replacing P ( S = s | G = g ) with its empirical counterpart.
Equalized Odds (EOdds): The equalized odds condition for binary predictors, given in Hardt et al.
21, is a balance condition where the groups must have equal true positive and false positive rates.
For continuous scores, it translates to having equal score distributions for each group conditional on their true labels [32]:
[Equation]
Like demographic parity, our empirical score bin version requires that the distribution of positive or negative instances be ϵ EOdds -approximately equal between groups in the new bins b ′ .
Both equal true positive rate and false positive rate can be expressed as linear constraints, respectively:
[Equation]
Predictive Rate Parity (PRP): Lastly, we examine the predictive rate parity condition popularized in [13].
This condition states that the probability of being a positive instance is independent of group membership when we condition on the score.
Formally:
[Equation]
P ( Y = 1 | S = s, G = 2) .
Using the empirical score bin framework, an approximate version of the above implies that the proportion of positive instances in each bin must be ϵ PRP -approximately equal among groups:
[Equation]
Unlike constraints (1) and (2), which can be expressed as a linear function of the optimization variables { x [ g ] bb ′ } , condition (3) yields bilinear terms and is in general a non-convex constraint.
A main technical difficulty of our framework arises from this non-convex fairness constraint—Section 3 presents an integer programming framework to handle this non-convexity, ensuring we can obtain a globally optimal solution to the resulting optimization problem.
Remark.
Our definition of fairness as the worst-case violation across all bins aims to resemble approximations of the respective probabilistic definitions but we have not found identical definitions in other works.
We comment on the differences and discuss why it does not contradict the traditional impossibility theorem of [25] in Appendix D.
2.2 MFOpt: Multiple Fairness Optimization Framework
We use constraints developed in Section 2.1 to state the multiple fairness optimization (MFOpt) problem: minimize
[Equation]
(4a) s.t.
(cid:88) b ∈B x [ g ] bb ′ =∀ b ′ ∈ B , g ∈ G (4b) x [ g ] bb ≥ 1 − ξ ∀ b ∈ b ′ ∈ B , g ∈ G (4c) x [ g ] bb ′ =∀ b ′ s.t. | b ′ − b | ≥ w, ∀ g ∈ G (4d)
Fairness Constraints: (1) , (2) , (3) (4e) (cid:80) b ∈B x [ g ] bb ′ N [ g ] b + (cid:80) b ∈B
x [ g ] bb ′ N [ g ] b ≤ (cid:80) b ∈B x [ g ] b ( b ′ +1) N [ g ] b + (cid:80) b
∈B x [ g ] b ( b ′ +1)
N [ g ] b ∀ b ′ ∈ { 1 , ..., B − 1 } , g ∈ G (4f)
[Equation]
Above, the optimization variables are the group-specific movement probability x
[ g ] bb ′ terms and all remaining terms are problem data and/or configurable hyperparameters.
Starting with the objective (4a), we define ¯ s b as the midpoint score in the bin and hence the objective is the product of the movement distance ¯ s b − ¯ s b ′ weighed by the fraction of total samples moved N [ g ] b / N and the amount of movement x
[ g ] bb ′ .
(4b) states that the total movement out of bin b , including the movement back to itself, must sum up to 1 and along with (4g) ensures that x [ g ] bb ′ represent probabilities in a transition matrix.
(4c) states that the total movement from bin b back to itself must be lower bounded by hyperparameter ξ .
This parameter controls how far we allow the new scores to stray from the original and is necessary to prevent zero denominators in 3 and 4f.
Constraint (4d) represents window constraints to restrict extreme movements of scores beyond w bins.
Constraint (4e) are the fairness constraints in Section 2.1.
(4f) ensures that we preserve the rank-ordering of the scores and expected values, which is desirable for comparing bins against each other.
While constraints (4d), (4f) are not fairness-related, we add them to retain the utility of the solution.
The predictive rate parity constraint (3) and (4f) both introduce non-convexities into Problem (4).
These two constraints also require us to assume overlap, or that each bin contains at least one member from each group.
Without overlap, predictive rate parity is undefined since it is not possible to compare expectations across groups for a given bin and demographic parity is likely violated as it means one group has 0 probability of landing in a given bin.
Another option we considered for the objective following ([32]) is the Riemann approximation of
AUC from the bins.
ROC AUC be approximated by the FPR at bin k and TPR of the cumulative bins b ∈ { k, . . .
, B } . Hence, it possible to optimize for maximizing AUC as a quadratic objective rather than minimizing score movement as a linear objective. However, in our experience, maximizing AUC as the objective led to a harder time finding better feasible solutions compared to minimizing score movement. We close this section with two major benefits of this framework compared to inprocessing solutions such as adding fairness regularization ([42, 17, 37]) or using an entirely different fairness-based model ([11], [43]).
First, the number of optimization variables scales in the order of O ( |G||B| 2 ) while the number of constraints scale in the order of O ( |B||G| 2 ) .
This is significant as it entails that our framework can be applied in arbitrarily large data settings as long as score-based binning is possible.
Applying the transformation is equally scalable, as it only requires binning observations and making independent draws from a multinomial distribution with B possible outcomes.
Extending the framework to non-binary groups requires creating pairwise constraints for all groups in each bin to achieve the same fairness effect.
This means that the number of constraints scale in the order of
O ( |B||G| 2 ) (in practice, we expect | G | to be in the range of 3-4).
This pairwise procedure does not affect the scaling of the number of optimization variables, which remains at O ( |G||B| 2 ) 3 .
The second benefit of our framework is that it returns a highly interpretable solution as it returns one optimized B × B transition matrix per group.
Hence given a newly scored instance, several facts can be read from the corresponding row of the matrix such as the likelihood of moving to a specific bin b , moving into any higher or lower bin, etc.
These probabilities can also be controlled via constraints as shown with the window constraints (4d) and max movement constraints (4c).
This interpretability is an advantage over model regularization frameworks, where it is difficult to know how an individual’s score might change when switching from the base model to a fair model or evaluating how much regularization is required to achieve a certain fairness effect.
Finding Optimal Solutions via Mixed Integer Programming (MIP)
The primary difficulty of the above optimization problem are the predictive rate parity constraint (3) and rank-order constraint (4f) which turn the problem non-convex.
Non-convex constrained optimization is generally NP-hard and traditional methods that seek locally optimal solutions include gradient-based interior point optimization ([40]), sequential quadratic programming ([18]), or algo- rithms specific to quadratically-constrainted-quadratic-programs (QCQPs) such as operator splitting methods, semidefinite relaxations, etc.
(see [33] for an overview).
Rather than pursuing a locally optimal solution, we propose a novel reformulation of the problem into a mixed-integer-linear-program (MILP), which can theoretically be solved to global optimality ([10], [9]).
Our reformulation grants two benefits over traditional locally optimal solvers.
First, global strategies theoretically enable us to find the best possible solution.
Second, as our problem primarily scales with the number of bins, we find that in practice, our problem is solvable to near-optimality when utilizing the power of modern MIP solvers.
3.1 Reformulations for computational efficiency
We first observe that a direct reformulation of the fractional terms into bilinear terms in constraints (3) and (4f) will lead to bilinear terms in the order of O ( B 3 ) .
We show that we can reduce this to O ( B ) bilinear terms through a substitution that exploits the problem structure.
Next, we take advantage of the vastly reduced number of bilinear terms to apply the normalized multiparametric disaggregation technique (NMDT, [1]) which we explain in Section 3.1.2.
This allows us to approximate products of continuous variables as products of binary variables, which can be easily linearized and handled by
MIP solvers.
Importantly, this transformation of xy (i.e, product) terms requires upper and lower bounds for x and y and we propose a method in Section 3.1.2 for generating and tightening these bounds by solving fractional linear program subproblems ([12]).
Taken together, the reduction of bilinear terms combined with the bound-tightening procedure enable us to effectively apply the
NMDT methodology and transform the problem from a non-convex QCQP to an MILP that can be solved to global optimality.
3.1.1 Step 1: Reducing number of bilinear variables
We reduce the number of bilinear variables in our problem by making a substitution for the fraction term by introducing new optimization variables v [ g ] b ≥ 0 (to represent a sum) and t [ g ] b ≥ 0 (to represent the fractional quantity) as additional variables (see Appendix B for more details).
We can then use them to write equivalent constraints with only O ( B ) bilinear terms.
Let,
[Equation]
and
[Equation]
Then we have the following:
[Equation]
3.1.2 Step 2: NMDT and bound tightening through fractional LP subproblems
Linearizing bilinear terms (NMDT): We show how we can model each bilinear term t [ g ] b v [ g ] b by using a binary expansion for the continuous variables t
[ g ] b , v [ g ] b , and by observing that the product of binary variables can be modeled via integer programming (see [27, 39] for reference).
To this end, we make use of the NMDT transformation [1].
We recap this method below as formulated in [1]
Given any bounded optimization variable x ∈ [ x L , x U ] , and precision factor p , a negative integer, we can represent this variable exactly as x = ( x U − x L ) λ + x L where
[Equation]
where 0 ≤ ∆ λ ≤ 2 p is a remainder term and z l ∈ { 0 , 1 } are binary optimization variables.
Dropping the remainder term ∆ λ gives us the approximate form and product forms of xy become dot products of several integer variables, which can be effectively handled via modern MIP solvers, such as [20].
Although we are solving an approximation (e.g. precision of 1 e − 4 ) this is not a practical problem since it is precise enough for reasonable choices of ϵ and we do not expect the constraints to hold exactly when we apply the post-processor on the testing data.
Bounds on v [ g ] b , t [ g ] b via Fractional LPs: A key requirement to apply NMDT is that all optimization variables in the bilinear terms ( t [ g ] b and v [ g ] b in our case) must be bounded and we need to be able to accurately estimate these bounds (i.e., tighter bounds leads to faster runtimes [1]).
We propose obtaining these lower and upper by minimizing and maximizing respectively, the sub-problems while keeping all fairness constraints except the quadratic constraints.
Firstly, note that bounds on v [ g ] b can be solved as a simple LP (details in Appendix B).
Meanwhile, obtaining a bound on t [ g ] b requires solving a nonlinear problem due to fact that t [ g ] b represents a fractional objective (ratio of two affine terms of optimization variables).
However, we observe that we can apply the Charnes-Cooper transformation [12] to reformulate the nonlinear problem into a simple LP (details in Appendix B).
3.2 Choice of algorithm: QCQP (heuristic) vs MIP (optimal solution)
In this section we show the results and benefits of our reformulation from a non-convex QCQP to an
MILP.
In Table 1, we take each dataset, create a 60/40 train-test split, train a grid-searched random forest model, and score the training data.
Next, we discretize the scores into bins, parameterize the problem (# bins, ϵ , max movement, window size, solve time) on the scored training data, and compare our MILP solution solved by Gurobi ([20]) against the QCQP problem solved by IPOPT ([28]), which is a generic interior-point log-barrier penalty method for nonlinear constrained optimization.
We discuss the datasets and problem parameters for all experiments in the Appendix C. For each metric such as AUC , we use AUC INT and AUC IP to denote the average result of applying the interior-point (INT, for short) or integer programming (IP) method, respectively.
The metrics used are the objective value, optimality gap ( %∆ ), 5 and AUC .
We also report the statistical significance of the improvement based the p -value from the Wilcoxon signed-rank test to determine if %∆ IP ≤ %∆ INT is a consistent result.
Bold figures indicate statistical significance w.r.t.
1 standard deviation 6 7 .
[Table]
As the results show
[Table]
the MIP reformulation consistently beats the interior point solver applied on the raw optimization problem, even when solving for only 10 minutes.
We also observe that in most cases, regardless of the method we choose, we can quickly find near optimal solutions that are high performing in the AUC sense.
This is a significant result as it means that even a locally optimal solution to our optimization problem can yield a practically useful post-processing result.
Additional experiments on these datasets showing the effectiveness of MFOpt with respect to performance and fairness in a training/testing data scenario can be found in Appendix F.
We conclude this section by reiterating two benefits of the reformulation.
First, solving a MIP method yields lower bounds that can be used to prove optimality or otherwise gauge the quality of a feasible solution.
Second, by framing the problem as a MIP, we can always theoretically continue improving the solution to optimality based on the acceptable time limits.
There are other applicable global optimization methods, such as spatial branch and bound.
We considered these solutions but ultimately opted for a MIP approach due to the maturity and availability of solvers (see Appendix E for details).
Applications
In this section [Table] we illustrate a few methods of applying our framework and how it can be used to help model developers select and understand models from a fairness perspective.
To apply these procedures, we first require developing an efficient frontier of fairness solutions to understand which ϵ configurations are feasible for a given model type and dataset.
To generate this frontier, we solve the problem over a grid of parameters ϵ DP , ϵ EOdds , ϵ PRP .
Each feasible solution will yield a point s ∈ { ( AUC, ϵ DP , ϵ EOdds , ϵ PRP ) } and the collection of non-dominated points from the solution set yields a efficient frontier.
We show the 2-d profile shots of our 4-d fairness surface in Figureas an example, where the axes represent one of three fairness metrics and the point color gradient represents AUC.
In theory, we could obtain a true Pareto-optimal frontier since we have devised a method of obtaining globally optimal solutions.
However, we generate this frontier using IPOPT due to practical limitations as we are solving a 7 × 7 × 7 grid of ϵ parameters. [Table]
4.1 Understanding fairness tradeoffs
After the Pareto frontier is generated, the modeler can pick an operating point s based on the desired AUC and tolerable fairness violations ϵ .
However, when communicating fairness properties to stakeholders and accounting for potential changes in strategy, it can be helpful to additionally understand the cost of further increasing fairness in terms of ϵ DP , ϵ EOdds , ϵ PRP .
This tradeoff can be understood by looking at the characteristics of points on the frontier near the operating point.
Suppose we are at an operating point s .
If we want to trade AUC for ϵ EOdds , then we would find a point s ′ with at least as good ϵ DP , ϵ PRP but worse AUC and better ϵ EOdds .
More generally, if we pick trade performance/fairness characteristic c (cost) for characteristic b (benefit), then we hold all other factors constant and find a point with better b and worse c .
We illustrate this in the Table 2.
The first row shows a hypothetical operating point while the following rows show other points on the efficient frontier that we could move to when we make a certain trade.
Blank rows indicate that no such point was found that permits the desired trade-off.
This trade-off perspective can enable developers to better understand and communicate the costs to performance or other fairness metrics when trying to close the disparity in one fairness metric.
[Table]
4.2 Performance Comparison
Lastly
[Table] we compare our framework against two methods and show that we can satisfy fairness constraint(s) just as well or better [Table] while obtaining significantly stronger performance.
Our comparison is done as follows (details in C), in each iteration we randomly split and pre-process the data, tune and fit a random forest model [Table] and score the training and testing data to get the base scores ˆ y 0 .
Next, we run the methodology that we are comparing against (i [Table] e. build a model or apply the postprocessor) to get method scores ˆ y m .
We then bin the outputs of the base model and compared method and compute the AUC along with the fairness metrics ( ϵ 0 , ϵ m ).
Next, we solve our constrained optimization problem on the training data, where we set the parameters ϵ to 12 min ( ϵ 0 , ϵ m ) .
After optimizing, we can apply the optimal solutions x
[ g ] bb ′ to assign new bins in the testing data based on the original score bins.
One method of assigning a group g instance with score s ∈ b (denote as s [ g ] b ) would be to randomly draw from a multinomial distribution parameterized by probabilities ( x [ g ] b 1 , x [ g ] b 2 , . . .
, x [ g ] b |B| ) .
We propose alternative methods to this stochastic assignment method in the Appendix A. Lastly, we compute the resulting AUC and fairness metrics on remapped bins for the testing data (and do the same for ˆ y 0 and ˆ y m on the testing data).
These figures are shown in Table 3.
We only show the results on the test set due to space constraints and have placed the results for the training set in Appendix C.
Bold figures indicate that a metric is statistically significant to 1 standard deviation.
First, we compare our framework against the in-processing framework in Rezaei et al.
[37], which is a robust optimization-based logistic regression model for reducing equality of opportunity violation.
We found that this method works well compared to standard logistic regression and managed to decrease fairness violations while maintaining the similar performance.
It improves in ϵ Eodds compared to a random forest as well.
However, its weakness is that the underlying model is still logistic regression and therefore has limited expressiveness.
In comparison, MFOpt can be applied on top of any model class and thus the performance advantages of more flexible models are better maintained.
Next, we compare our framework against the post-processing framework in Pleiss et al.
In this method, the authors use randomization to maintain the model’s calibration while simultaneously satisfying a relaxed equalized odds condition (whereby a linear combination of TPR and TNR are satisfied).
Again, we see that this method maintains close performance as the base model, successfully shrinks equalized odds violations, and even decreases demographic parity violations too.
However, it results in large violations of bin-wise predictive rate parity in contrast to our method.
4.3 Who’s the fairest of them all
In industry, machine learning model selection is guided by many factors including performance, speed, interpretability, among others.
Yet, the fairness dimension is commonly overlooked unless the developers specifically induce it in their model.
Even then, picking a specific fairness metric to optimize for can be a nebulous task.
Rather than focus on a single metric, we describe a simple and intuitive method of gauging a model’s efficiency in trading between different fairness definitions.
To do so, we first construct the frontier for the two models in question and then filter all points on the efficient frontier with tolerable performance AUC ≥ AUC min .
Next, we find the point on the respective frontiers with minimum Euclidean distance to the origin.
The model with the shorter distance to their frontier can then be declared as the model that has better tradeoff properties.
This procedure combined with the trade-off analysis can be useful when a developer is iterating between models that were not designed for fairness, but wants a model that can be flexibly made more fair through the MFOpt framework.
As fairness requirements may change over time, the model that can yield the best tradeoffs between different definitions can offer the most overall utility.
Discussion
In our study, we have devised a flexible and interpretable post-processing method which we apply to push the limits of the impossibility theorem.
We show that while theoretical limitations remain undisputed, there is a path forward to practically reconciling the conflicting fairness definitions.
These results extend the findings of [38], which state that the trade-off of fairness and accuracy are negligible in practice.
Our work reinforces this claim but also adds on that trade-offs between fairness definitions can be negligible as well.
For further research, one area is to improve the consistency of the
PRP violation reduction, as we observed the largest standard error in reducing this metric.
This could be due to us using random forests for all experiments, which is known to be an uncalibrated model [6].
One method to address this is therefore first calibrating the model through other methodologies such as Platt scaling ([35]) or binning-based calibration ([26]) before applying MFOpt.
We could also consider incorporating uncertainty in the training data through principled approaches such as stochastic or robust optimization.
Acknowledgements
We would sincerely like to thank David Durfee and Osonde Osoba for their insightful feedback.
We would also like to thank Ye Tu, Sakshi Jain, Shaunak Chatterjee, Ram Swaminathan, Romer Rosales, and Ya Xu for their support.
Finally, we would also like to thank the anonymous reviewers for their helpful comments which substantially improved the paper.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] In the discussion(c) Did you discuss any potential negative societal impacts of your work?
[Yes] In Section C (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] We do not derive any theoretical results and all claims (e.g. NP-hardness of non-convex programs) are either well-known or a source is referenced.
(b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Yes, all code is included in a zip file uploaded as the supplementary materials.
Instructions and data used in all experiments are clearly outlined in C and(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Referenced in Appendix C (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Referenced in Section C 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] License and references can be found in the Appendix Section C (b)
Did you mention the license of the assets?
[Yes] License are mentioned alongside open-source package references (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] Under the Experimental Data Section C 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Mapping from bins to scores
As mentioned, the most straightforward method of applying the score transformation after solving the optimization problem is to sample from a multinomial distribution.
However, this is a less granular approach as we are assuming that all observations in the bin are indistinguishable.
To overcome this, we recommend the idea proposed in ([32]) which is a linear projection.
This strategy proposes that if an observation with score s falling into a bin a with upper and lower bounds [ a l , a u ] gets mapped from the random draw into a new bin b 1 with bounds [ b 1 l , b 1 u ] , then we assign it a linearly interpolated score given by:
[Equation]
This allows us to maintain rank-ordering of scores that receive the same assignment from a to b .
A more deterministic manner of mapping from bin to score would be to take the expected score mapping.
After solving the optimization problem, we know the transitions probabilities a to { b 1 , b 2 , ..., b B } (denoted as P ( a −→ b i ) ) based on the optimization variables and from the previous method, we also know the score assignment if a were moved into b i (denote as s i ).
Hence, a deterministic map would transform score s to s ′ =
(cid:80) i ∈
B s i P ( a −→ b i ) .
B Details on the fractional LP subproblem for bound tightening
We elaborate on the methodology in Section 3.1.2.
Recall that our goal is to find bounds for:
[Equation]
and
[Equation]
Where t [ g ] b ′ is meant to represent the fractional quantity:
[Equation]
We will do this by fixing ¯ g and ¯ b such that we first tighten bounds for v ¯ g ¯ b and then use the optimal solution to tighten bounds for t ¯ g ¯ b .
First, it is easy to see that maximizing/minimizing v ¯ g ¯ b is an LP as we have dropped the quadratic constraints, leaving us with a linear objective and linear constraint set.
Now let v ¯ g ∗ ¯ b,min/max represent the optimal values of the min/max objective for v ¯ g ¯ b .
We now turn to bounding t ¯ g ¯ b which has the same linear constraints but a fractional (nonlinear) objective.
To deal with this, we utilize the Charnes-Cooper transformation ([12]).
Essentially, this reformulation trick handles the denominator by removing it from the objective and passing it to all constraints while maintaining linearity.
To illustrate this in detail, we first define new optimization variables:
[Equation]
Using (5), we can express the min/max problem for t [¯ g ] ¯ b as problem (6).
Min or Max
[Equation]
subject to (cid:88)
[Equation]
N
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
By solving these subproblems and taking the objective value as bounds for t [ g ] b , we can reduce the feasible region of the problem and enhance our solutions.
The sub-problems are bounded and if any of them are infeasible, then it also implies that the MFOpt problem is also infeasible as we drop the
PRP constraints in these sub-problems:
C Experiment data descriptions and problem parameters
We use three primary data sources for our experiments, the more recently developed American
Community Survey (ACS) data as well as two more classical datasets, Heart Disease and COMPAS.
We elaborate on each dataset in this section.
The ACS data is a dataset made publicly available by the US Census Bureau.
Specifically, Ding et.
al [14] have created an excellent Python packagethat enables users to pull model-ready data (for a requested year and geographic region) for a set of pre-defined binary classification tasks, such as predicting high income, health insurance coverage, whether they move or not, among others.
The tasks are detailed in the paper and we use all of the pre-defined tasks without any additional modification except for Employment.
We do not use the
Employment task because of the assumption detailed in Section 2.2 regarding overlap.
Experiments with this task occasionally yielded models that did not have overlap which made this task unsuitable for demonstrating our methodology.
We reiterate that this is not a practical issue if one just ignored the non-overlapping bins, but requires a lengthy and technical fairness interpretation that we felt were beyond the purpose of our study.
In terms of time and geography, we use 2020 data for all experiments while the geography varies.
In the experiments shown on Table 1, we use the West
Coast US states (California, Oregon, Washington).
In 3 we wanted a larger dataset as we required a sufficiently large testing split, hence we used the West Coast States (’CA’, ’OR’, ’WA’, ’NV’, ’AZ’) with the "ACS Mobility" dataset and a 60/40 train-test split for the inprocessing comparison and East
Coast States (’ME’,’NH’,’MA’,’RI’, ’CT’,’NY’, ’NJ’, ’DE’,’MD’,’VA’, ’NC’,’SC’,’GA’,’FL’) and same with the "ACS Poverty" dataset for the postprocessing comparison.
There was no particular reason for selecting these geographies aside from obtaining a large enough sample that we can feasibly run multiple trials on.
Though we are using census data there is no PII information nor any endangerment to the subjects in the data.
However, we note that in practice, it is important to exercise caution and equity in picking groups to mitigate for, as selective mitigation of favored groups by a malicious practitioner can result in underperformance for deserving groups.
The Heart Disease Dataset ([23]) is a publicly available dataset where the task is to predict whether or not an individual has heart disease.
Most applications of this data use the standard processed "Cleveland" data and we use sex as the group variable.
We could not find a standard and preprocessed version of this data and did it ourselves by one-hot-encoding categorical variables.
The COMPAS dataset is based on the recidivism study noted in ([2]).
We use the preprocessed version made available in the publicly available AIF360 package ([4]) 9 without any additional modification.
In this dataset, the task is to predict whether or not an individual will recidivate and we use ethnicity as the group variable.
We list the problem parameters used to create the results in Table 1 in Table 4.
We use the same parameters for all tasks.
A particular note is that in Table 1, the IPOPT method performs very poorly compared on the COMPAS and ACS Coverage data.
This is because across the 10 trials we ran, the
IPOPT algorithm had frequently failed to converge within the 10-minute time limit for these two datasets.
From the outputs, we saw that convergence failure is accompanied by heavy violation of the predictive rate parity constraint (demographic parity and equalized odds are still satisfied) and a [Table] [Table] high objective value.
The exact reason for frequent failure in these two datasets is unclear, however, we hypothesize that it is due to a relatively high predictive rate parity gap in the data which led to numeric issues.
Such failures were not observed in the IP formulation while both solvers were provided the exact same problem parameters.
All experiments were run on a MacBook Pro with a 2.4GHz 8-Core Intel Core i9 processor with[-
-]
GB RAM.
We did not use the GPU for solving.
Data, preprocessing steps [Table] and the random forest models utilize Python’s scikit-learn ([34], BDS License) package.
The optimization model is coded through Julia’s JuMP package ([15] MPL License).
We use the Gurobi ([20] Academic License) and
IPOPT ([28] Eclipse Public License) solvers for all problems.
Due to lack of space, we only showed the method comparison Table 3 for the testing data.
We show the results on the training data here in Table 5.
D Comparison to other fairness definitions
We compare our bin-wise worst-case fairness definition with other fairness definitions seen in literature and explain why it does not contradict previous impossibility theorem results.
First, since we are considering score bins, our definition is a generalization of the definitions in ([13]), ([21]) and ([11]), which consider fairness metrics for binary { 0 , 1 } classifiers or assume that there is a threshold for mapping probabilities to 0,1 outcomes.
In these cases, the overall FPR/FNR can be computed and
EOdds refers to the equality of those rates.
Our framework is a generalization since the same fairness metrics for binary classification can be achieved by specifying that all scores be moved into exactly one of two bins (representing {0,1} predictions) under our framework.
Since we are dealing with binned scores, our fairness definitions more resemble those seen in [25], which also has a notion of binned "risk assignments".
The critical difference in fairness definitions is that Kleinberg’s paper utilize the sum of scores in each bin compared against the number of positive or negative instances.
Under this scheme, predictive rate parity refers to having the sum of scores be equal to the number of positive instances and true positive rate refers to the expected score of the positive instances in each bin (and analogously with FPR).
The key difference is that rather than the sum of scores, our definitions are based on the expected number of {0,1} instances moved into each bin, irrespective of the instance’s original scores.
As such, we are not faced with the same strict fairness trade-offs.
E Commentary on other solution methods and solvers
While investigating a solution to our nonconvex problem, we considered another global integer programming based approach known as spatial branch and branch (SBB), which relies on a combination of spatial partitioning and solving local partitions using McCormick relaxations ([29], [9]) and other outer approximation variants ([8]).
In our testing, Gurobi’s nonconvex QCQP solver, which applies these SBB heuristics, worked remarkably well despite being relatively new and was sometimes able to beat both the interior point solution and the MIP solution.
Open-source solver SCIP ([5]) also features a gender nonconvex SBB solver that works reasonably well.
However, our main goal was to provide a widely accessible method of solving the problem to global optimality and as of writing, there are significantly more developed open-source MILP solvers, such as SCIP, HiGHS ([22]), and
CBC ([28]), than SBB solvers.
Another reason we opted for the MILP approach is that we saw more potential in the NMDT reformulation for taking advantage of our reformulation and bound tightening procedure.
Nonetheless, our bound tightening procedure is theoretically beneficial for both methods and as other open-source algorithms/solvers for SBB become more developed, such as Couenne ([28]) and Alpine ([31]), we encourage a future re-evaluation of solution methods and comparisons.
Finally, we note that we chose Gurobi in our experiments for its speed and effectiveness since we are repeatedly solving many problems.
We acknowledge that Gurobi is a very powerful commercial solver and the results solved over 10 minutes may be worse with open source solvers such as HiGHS ([22]).
Nonetheless, the important fact is that all MIP solvers target the global optimum and hence even less powerful solvers can yield strong solutions given more time.
F Additional Experiments Using Expected Assignment on Testing Data
We list additional experiments focusing on the performance of our method on the testing data in Tables 6 to 11.
All results are based on 20 trials that are run with a similar procedure in the comparison section 4 except that we compute the testing metrics based on expected assignment rather than stochastic assignment (explained below), which we feel better reflects the average performance of
MFOpt.
In each trial, we tune a random forest via grid-search, find the base fairness violations, set up the parameters of MFOpt to reduce the violations by a half, and optimize.
Then, we compute the results (AUC and fairness violations) on the testing data (baseline with no modifications vs. MFOpt) and show the 1-Standard deviation error margins as well as the p -value corresponds to a one-sided
Wilcoxon signed rank test which evaluates if the distribution of differences of the Base - MFOpt stats (higher AUC, lower fairness violations) is symmetric around zero (null) or instead favors the base (alternative).
The major difference in this evaluation compared to the results shown in section 4 is that in section 4, we assign observations to bins in the testing data based on random draws from a multinomial distribution (explained in Appendix A).
However, a single draw per train-test split may not properly reflect the expected performance of MFOpt, even if we average over 20 trials.
Instead, we feel a more accurate representation of the expected performance is if we apply the expected bin assignment to obtain the post-movement number of { 0 , 1 } and total samples for each bin in the testing set.
Under this procedure, we do not move individual observations across bins, but rather move all of them together.
Concretely, suppose that we have optimized parameters (ˆ x [ g ] 1 b ′ , ˆ x
[ g ] 2 b ′ , . .
, ˆ x [ g ] Bb ′ ) , which represented the probabilities of observations from each bin moving into bin b ′ .
Then we propose that the number of {1} outcomes at bin b ′ after the expected bin assignment procedure is:
[Equation]
The same computation applies for the expected number of negative samples ˆ N [ g ] b
′ − and total samples ˆ N [ g ] b ′ , which we use to compute the AUC and fairness violations and report in the tables.
We find that across different datasets, the decrease in AUC is miniscule in terms of both absolute amount and variance (less than 1%).
We obviously do not expect better AUC from the MFOpt solution compared to the unconstrained model and thus this result is remarkable as it indicates that some degree of fairness can be afforded practically for free under our framework.
The second observation is that we can reduce all three fairness metrics simultaneously and consistently across all datasets, as we find p -values below 0.05 in all cases.
We do observe variance of the PRP violation is relatively higher than that of DP or EOdds.
We noted this in our conclusion Section 5 as an area for future work and provide some hypotheses for methods that can address this inconsistency.
Nonetheless, improving conflicting definitions of fairness simultaneously is another significant result as it provides empirical evidence that there is a path forward towards multiple fairness.
[Table]
[Table] [Table]
[Table] [Table]
[Table]
G Role of Bins Parameter and Ablation Study
In this section, we first comment on the choice of the number of bins, |B| , as a hyperparameter and then show results of additional testing with respect to the choice of the bin parameter.
In practice, the choice of |B| should reflect the required bin granularity of the outputs for usage in, for example, rank-ordering or threshold-based classification.
Our work therefore approaches choosing |B| purely from a computational complexity perspective as the number of optimization variables scales in the order of O ( |G||B| 2 ) .
We used |B| = 50 across our experiments as we found that this parameter gave a reasonable degree of granularity in the resulting bins while also being small enough to solve ( our MIP solver could frequently find solutions with <10% optimality gap within 10 minutes).
From a more theoretical perspective, the choice of |B| determines how well we can estimate the score transformation function, with higher |B| giving us better estimates in the training data.
However, having too few samples within each bin (in the training data) results in low bias but high variance estimates of the score transformation function and the corresponding performance/fairness metrics, leading to bad generalization on the testing set.
Having too few bins can, on the other hand, lead to an under-parameterized score transformation function.
While it is clear that |B| should scale with respect to the per-bin sample size, we consider a rigorous analysis of this choice to be out of scope as a current limitation of our work.
We hence recommend choosing |B| in practice based on the desired granularity of results and/or treating it as a parameter to optimize via cross-validation.
To run an experiment to study the effects of changing |B| , we must first define |B| -agnostic metrics to evaluate the post-processing output in terms of performance and fairness.
The primary difficulty is that since fairness is defined on a bin-wise basis, evaluating the worst-case violation for example using |B| = 10 is not comparable to |B| = 50 .
Hence, we do the following: First, after optimizing for x [ g ] ∗ bb , we apply the linear interpolation based mapping as described in Appendix A so that we have new scores s ′ .
To compare the accuracy across different |B| , we compute the ROC and precision-recall (PR) AUC based on s ′ .
To compare the fairness, we discretize s ′ in 100 bins (regardless of which |B| we used) and compute the worst case violation for each fainess metric across all 100 bins.
The reason we use 100 bins to assess fairness is that we specifically use a large dataset (500k+ instances in both training and testing) to obtain more granular metrics.
To generate the results, we use the
ACS Income data as presented in previous sections.
The difference is that we now use data across all states to obtain a large sample for a single random 60/40 train-test split, each with over 500k samples.
Following the procedure from before, we train and tune a random forest model, score the training and testing data, and proceed with our evaluation methodology.
Since the linear interpolation mapping methodology utilizes stochastic draws, we sample 300 draws per bin and compute thestandard-deviation errors shown in the tables.
Table 12 shows the performance metrics across different |B| , where we show the average metric and 1-standard deviation error margins.
We observe that performance is very similar across the different choices, but is maximized at the higher bin counts of of 50-60.
The performance also appears to plateau beyond a certain point, suggesting that it is not necessary to select a large |B| for performance purposes.
Interestingly, we also see that the solutions are near-optimal based on the optimality gap.
This shows that although the number of variables scales with |B| , it does not necessarily preclude us from high quality solutions.
On the contrary, the optimality gap for lower bin ranges such as 25 is larger with lower performance.
This could be due to the fact that since we are optimizing over the same ϵ fairness criteria, having fewer degrees of freedom with smaller bins makes it more difficult to find feasible solutions.
Table 13 shows the results for fairness, where we show the average worst fairness violation and the 1-standard deviation error margins.
The story is less clear from this angle but we emphasize that training the problem for different |B| ̸ = 100 bins using the same ϵ parameters and evaluating it on |B| = 100 for fairness is an unintended method of using our framework which we are only doing to have comparable results for the ablation study.
Here, we see that the average metrics are similar for different parameters with |B| =
40 having the best overall result.
Demographic parity violation is surprisingly minimized at |B| = 30 in this example, but given the other fairness and performance metrics |B| = 40 appears to be the best choice.
Under this method of evaluation, the fairness parameters do not appear to be very sensitive to the choice of |B| .
[Table] [Table]