Paper: Post-hoc bias scoring is optimal for fair classification (ICLR-2024)

Reviewer comment (Experiments):
It is not mentioned how nonbinary sensitive attributes are handled. Related: How is the sensitive attribute of "race" in the COMPAS experiments, which can be one of three categories (African-American, Caucasian, Other), handled?

Author response:
Similar to what has been done in existing papers, such as Zafar et al. 2018 and Cho et al. 2020, we preproposess sensitive attribute 'race' in this dataset such that A = 1 represents ‘African American’ and A = 0 corresponds to all other race.

Reviewer comment (Experiments):
I am skeptical about the scalability of the proposed method to large datasets and more sensitive attributes. It appears that the time complexity would scale exponentially in the number of sensitive attributes, i.e., $M^{K}$. I see that $M$, the number of samples on which the decision boundaries are considered, is set to no more than 5000 in the experiments. But practical ML datasets nowadays can contain tens of thousands to millions of examples. Could the authors also report and compare the running time of their code? Also, if $M<N_\textrm{val}$ is used for selecting the boundaries, then there should be a term involving $M$ in Theorem 2 (more on Theorem 2 below)?

Author response:
As a post-processing method, we already have much better scalability than in-processing methods in the sense that we don't need to train multiple models to obtain different accuracy-fairness tradeoffs. We only need to train a base model once and run the post-processing algorithm to obtain all different trade-offs. In all our experiments, the time cost for running the post-processing algorithm is much smaller than training the base model, especially for experiments on CelebA where a deep ResNet is trained as the base model. 
Some post-processing methods are indeed quite fast, but they often only provide a single solution targeting at the strict fairness constraint ($\delta=0$). While our post-processing algorithm only needs to be run once to obtain all different trade-offs (corresponding to different desired fairness constraints $\delta's$). Furthermore, we provide additionally a more efficient alternative algorithm (Algorithm 3) in Appendix B, which scales according to $N_{val} \log N_{val}$ when EO is considered

Reviewer comment (Theory):
Theorem 2 is very important as it provides fairness guarantees for the classifier obtained through the procedure. But the result looks wrong to me, and seems to have a discrepancy with Algorithm 1. The proof is also very hard to read, containing several typos. DP should depend on $\epsilon_p$ (attributed to the error in $\hat p(A=1\mid X$), but this dependency is absent in eq. 14. Digging into the proof, I see that it is hidden with the statement that "$\epsilon=\epsilon_1+\epsilon_2$, and assume that it is smaller than $\delta/2$". Why is this assumption justified? In the paragraph preceding eq. 15, what is $\hat Y_{t'}$? Should it be $\check Y_{t'}$? And what is $\check Y_{t'}^*$ in the paragraph following eq. 15? Should it be $\check Y_{t'}$? Following the above, I don't get why $DP(\check Y_{t'})+\epsilon \leq \delta$ but not $3\delta/2$, given that "let $t'$ corresponds to... under the constraint that $DP(\check Y_{t'})\leq \delta-\epsilon$". I don't get why $DP(\check Y_{\hat t})\leq \delta -\epsilon_1$, how is $\check Y_{\hat t}$ related to $\check Y_{t'}$? Finally, Theorem 2 only proves results for DP. What about EO, EOp, and composite criteria? Please also justify the assumptions made; are they practical?

Author response:
Thank you for this suggestion. The result indeed can be extended to general composite criterion (CC) with the same technique, apart from the continuity bound on $Acc^*(\delta) - Acc^*(\delta - \epsilon)$ in the end of the proof, which now requires a slightly different argument. We have updated the proof in the revision to include general CC criterion, and changed the formulation in the main text. We also added some outline at the beginning of the proof, with the hope that it improves readability. Regarding the discrepancy with Algorithm 1, we believe there is a misunderstanding. When we choose $M = N_{val}$, Algorithm 1 and Algorithm 2 go through all possible linear rules on a given validation set, therefore correspond exactly to the procedure described in theoretical formulation. When $M < N_{val}$, we simply reduce the search space, while EO and ACC are still evaluated on the whole validation set. This is done as a heuristic, for sake of reducing the compute time.

We want to stress that Theorem 1 is the main result of our paper, which also gives raise to a practical algorithm. Although Theorem 2 is important from statistical learning point of view, we did not have the goal to have the most tight bound, rather include the result as a sanity check.

Reviewer comment (Experiments):
In the experiments, the authors compared their proposed post-processing algorithm to ones that are attribute-aware, but their algorithm is run in attribute-unaware mode. The authors should have compared to those algorithms by running their algorithm in the same attribute-aware mode. In this sense, the current set of experiments is incomplete.

Author response:
In Remark 2.1, we show that when run in sensitive attribute-aware mode, our method reduces a group-aware thresholding. However, the method is proposed to be run in sensitive-unaware mode: the fact that our method is run in sensitive attribute-unaware mode and can still achieve competitive or better performance compared with sensitive attribute-aware methods demonstrates the exact advantages of our method that we intend to empirically verify: (i) it doesn't require inference time sensitive attribute, (ii) in some cases, it can still outperform methods requiring inference time sensitive attribute.

Reviewer comment (Evaluation):
The conclusions drawn from the ablation study in section D.2 do not make sense to me. How is accuracy related to the error $\mathbb E|\hat p(A=1\mid X) - p(A=1\mid X)|$ in Theorem 2? In fact, regularization could in fact be reducing the aforementioned error despite hurting accuracy. One way to measure this is, e.g., using reliability diagrams. The conclusion in D.2 that "this further confirms the robustness of our post-processing modification algorithm" does not make sense.

Author response:
Since we don't know the ground-truth $p(A|X)$, there is no way for us to compute $E[|\hat{p}(A=1|X)-p(A=1|X)|]$, and thus this assumption is made purely for theoretical analysis of bounds for performance (which may not be tight). For this ablation, we purely adjust weight decay to obtain 10 different sensitive conditionals $\hat{p}(A|X)$. Within this set of estimated conditionals, some of them are closer to ground-truth and some are not. While we can not tell which model is closer to the ground-truth, the fact that our method yields good performance for all of these 10 different $\hat{p}(A|X)$ shows the robustness of the method. Indeed, higher accuracy does not imply better calibration. However, our theoretical result requires individual/conditional calibration, which is much stronger than marginal calibration and, as mentioned above, is difficult to assess. Reliability diagrams and calibration error based metrics, such as ECE, are known to be problematic when assessing conditional calibration [1, 2]: trivial solutions can obtain perfect marginal calibration (ECE=0), e.g. in a binary classification problem with balanced labels, predicting $\hat{p}(Y=1|X)=0.5$ all the time no matter what $X$ is.

[1] Yuksekgonul et al. Beyond Confidence: Reliable Models Should Also Consider Atypicality. 2023.

[2] Xiong et al. Proximity-Informed Calibration for Deep Neural Networks. 2023

Reviewer comment (Theory):
Some clarifications would be helpful: Example 3 does not imply subgroup fairness, i.e., intersecting groups. When introducing the composite criterion, it is also useful to mention that some fairness criteria are incompatible with each other (e.g., DP vs. EO).

Author response:
Indeed, Example 3 does not imply parity for all 4 intersections, rather parity for two pairs. If we want to equalize positive outcome in all 4 groups, such case is equivalent to a multi-label sensitive attribute. In this case, one solution is to compare every pair among the 4 values (which accounts for 6 restrictions). For instance, if $A$ takes values $1, 2, 3, 4$, we can set $k = 6$, $A_1, \dots, A_6 = A$, $(a_1, b_1) = (1, 2)$, $(a_2, b_2) =(1, 3)$, $(a_3, b_3) =(1, 4)$, $(a_4, b_4) =(2, 3)$, $(a_5, b_5) =(2, 4)$, $(a_6, b_6) =(3, 4)$ in Eq(4) on page 3. 

We also note that although EO and DP can be at odds, they are not exactly incompatible since constant classifier satisfies both. In fact, constant classifier satisfies any such condition CC = 0. The incompatibility happens when we add Predictive Parity into the mix [3]. The latter is not covered by our composite criterion.

[3] A. Chouldechova (2017) Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments

Reviewer comment (Novelty):
Related work on the Bayes optimal fair classifier in the attribute-aware setting (via post-processing) are missing, e.g., [1, 2, 3, 4].

[1] Denis et al. Fairness guarantee in multiclass classification. 2023.  
[2] Zeng et al. Bayes-Optimal Classifiers under Group Fairness. 2022.  
[3] Gaucher et al. Fair learning with Wasserstein barycenters for non-decomposable performance measures. AISTATS 2023.  
[4] Xian et al. Fair and Optimal Classification via Post-Processing. ICML 2023.

Author response:
A: Thank you for pointing out these related works. We will include them in future revision.

Reviewer comment (Reproducibility):
No code or results files are provided for the experiments; neither an implementation for the proposed method. This is largest point against the current version of the paper, as properly reviewing the work required checking some experimental details.

Author response:
Details of hyperparameters and model architectures can be found in the beginning of Section 4. We will release the code in the case of acceptance, at the moment we are still tidying up. We will include the uncleaned version of the code in an anonymous github repo and provide the link by rebuttal deadline.

Reviewer comment (Experiments):
Given that postprocessing baselines achieve Pareto dominant results in Fig. 2 (expectedly, as they have access to the sensitive attribute at inference time), it would be interesting to add partially relaxed results for these baselines for a more direct comparison (as done for the Zafar method).

Author response:
This is an interesting point. However, the two post-processing baselines considered in current version of the paer doesn't allow this relaxation: Wang et al. 2019 (the post-processing baseline for DP) is designed to enforce independence between the prediction and sensitive information by minimizing Wasserstein-1 distance, and Hardt et al. 2016 (the post-processing baseline for EO) is designed to satisfy the strict EO constraint (i.e., targeting at achieving 0 EO). Thus, there is no way to relax the constraints in these methods. We will leave the comparison between MBS and other post-processing methods that allow relaxation in future work (any suggestions of such baselines are very welcome).

Reviewer comment (Experiments):
It would be interesting to test against a simple baseline of using Hardt et al. group-specific thresholding using the same estimated $p(A|X)$ instead of the true sensitive attributes at inference time.

Author response:
We will run the experiments with stricter constraints and add the results into our next version for a complete comparison.

Reviewer comment (Experiments):
Is the base model used by MBS the same as those used by the baselines? Are the Zafar et al. (2017) results of Fig. 2 based on a constrained MLP?

Author response:
Yes, they are the same. The results of Zafar et al. (2017) in Fig. 2 are based on constrained MLP with architecture exactly the same as the base model for post-processing methods.

Reviewer comment (Reproducibility):
How was $p(Y,A|X)$ estimated when using MBS on CelebA?

Author response:
We trained ResNet-18 with 4 classes corresponding to (Y=0 or 1, A=0 or 1) using cross-entropy loss. The softmax of the logits is then interpreted as $\hat{p}(Y,A|X)$. These experimental details can be found at the beginning of Section 4.

Reviewer comment (Evaluation):
Do you see any reason why Hardt et al. (2016) would outperform on the Fig. 2 results, and achieve such lacklustre results on Table 1? Given that we see some variance/unreliability on fairness for MBS with $\delta=1$, can the even stricter constraint target by Hardt et al. (2016) (which uses $\delta=0$, right?) be related to its underperformance?

Author response:
A: From our theoretical result, our method will be close to optimal if the base model is close to Bayes-optimal classifier and the estimated conditionals are close to ground-truth, thus it does have potential to outperform Hardt et al. (2016). Besides, Hardt et al. (2016)'s modification rule is oblivious (i.e., it occurs at the group level while ignoring the details of individual feature X), while our method bypass this limitation since $\check{Y}(X)$ takes advantage of individual feature X.  Furthermore, indeed Hardt et al. (2016) targets at strict fairness constraint $\delta=0$, which may hurt accuracy in order to satisfy such strong constraints.

Reviewer comment (Novelty):
Could you please clarify the main differences to Zeng et al. (2022), as it seems to tackle exactly the same problem. [Zeng, Xianli, Edgar Dobriban, and Guang Cheng. "Bayes-optimal classifiers under group fairness." arXiv preprint arXiv:2202.09724 (2022).]

Author response:
A: Thank you for pointing out this reference, we were not aware of it. As far as we can tell, they only tackle DP, EOp while we consider additionally EO and composite criterion. Furthermore, they study optimality in the class of sensitive attribute aware classifiers, while our classifiers only depend on $X$. We will include this related work in future revision.

Reviewer comment (Reproducibility):
Overall, the assumption that we have access to $P(Y, A|X)$, or a good estimate of it, could be strong in practice. For example, if I know a good estimate for $P(Y|X)$, I might as well use that as my predictor. Also, how are these conditional distributions learned? In practice, we observe every $x$ only once, so these probabilities are 0 or 1 on observed data, unless we work with parametric models like logistic regression. But which parametric model should we use here when the underlying unknown data distribution could be arbitrary? Also, how are these models chosen in your experiments?

Author response:
We can fit a model to obtain the estimated conditional distributions $\hat{p}(Y,A|X)$ as long as we have access to a set of observed triplets $(X,A,Y)$, which is assumed for most methods. The decision of what target model $\hat{p}(Y|X)$ to use is up to practitioners: one can either use a pretrained one or obtain it from $\hat{p}(Y,A|X)$ depending on which strategy gives the best target accuracy - but even the best prediction model here might suffer from fairness violations and our post hoc approach can still plug in to improving its fairness property. Indeed, these conditionals are estimated using parametric models. Specifically, in our experiments, we use neural networks with cross entropy loss, which have shown profound success in prediction tasks and fitting conditionals. For each dataset, we use the same neural network architecture for all the methods compared. The base model for post-processing methods and the best model for in-processing methods are selected according to performance over validation set. We also note that, from our ablation study (Appendix C.1, C.2), the auxiliary model for sensitive attribute does not have to be a very good fit.

Reviewer comment (Experiments):
The paper claims that the performance of their method is better than “in-processing methods”. Is it better than all in-processing methods or just a few? This sounds like a very strong claim because, generally speaking, in-processing methods do achieve better performance than post-processing methods. Additionally, the most popular in-processing method for fair learning is given by Agarwal et al. 2018 (titled: "A reductions approach to fair classification"). Unfortunately, their algorithm is not included in the benchmarks for experiments. I would like to see a comparison of the two methods.

Author response:
In the abstract we state that we achieve competitive or better performance, in particular, the performance on Adult and COMPAS is comparable, while on Celeb-A we outperform the state-of-the-art method Park et al (2022), which is concerned specifically with this vision probelem. In the experiments we always use the same base model. 

As per request, we compare to the method provided by Agarwal et al. Due to the time constraint of rebuttal, we only run it for Adult Census dataset for now, but we will run the full set of experiments and include the results in the future. Since MLPClassifier in sci-kit learn does not have option sample_weight required by the package (fairlearn) supporting reduction method, we implement our own pytorch version, with same hyperparameters as for other baselines. It appears reduction approach does not perform very well with MLP, in particular, the results below were selected using the official selection script https://fairlearn.org/v0.5.0/auto_examples/plot_grid_search_census.html:
For DP constraints:
| Acc | DP |
| --- | --- |
| 0.7570 | 0.0067 |
| 0.8056 | 0.0082 |
| 0.8118 | 0.0279 |
| 0.8184 | 0.0658 |
| 0.8189 | 0.0883 |
| 0.8221 | 0.1191 |

For EO constraints:
| Acc | EO |
| --- | --- |
| 0.8182 | 0.1002 |
| 0.8218 | 0.1479 |
| 0.8228 | 0.0446 |

Please also see Figure 6 in Appendix F in the revised paper where we plot the performance of Agarwal et al. along with the other methods.

Reviewer comment (Theory):
Do the theoretical results rely on the fact that $\hat{Y}$ is the Bayes optimal classifier. $\hat{Y}$ is introduced as the Bayes optimal on page 3 but later on is used as any predictor. It would’ve been better if $\hat{Y}$ was initially introduced as any predictor that we’d like to post-process its predictions.

Author response:
Yes, the theoretical result is based on the assumption that $\hat{Y}$ is the Bayes-optimal classifier, see e.g. introduction p 2 paragraph 3, or beginning of section 2. However, MBS, as a practical method, itself does not restrict us with the choice of base classifier, so we think of $\hat{Y}$ as any score-based estimator that was fit without fairness constraints.

Reviewer comment (Reproducibility):
Can the validation data set be used to learn the conditional distributions? In practice we only have a pre-trained classifier and do not necessarily have pre-trained conditional distributions. If your method allows using the same validation set to learn these distributions, then all you’d need is the pre-trained classifier, increasing the flexibility of the proposed method.

Author response:
In principle, it can. However, we note that in practice the classifiers are often score-based, which can be interpreted as conditional probability. As we mention in our response to Q2, we can learn the conditional distribution as long as we have access to a set of observed triplets $(X,A,Y)$. We also note that in some cases, we can use a zero-shot classifier as we do in section C.3. In that regard, MBS is a very flexible approach.

Reviewer comment (Evaluation):
Why does Hardt et al. (2016) have lower performance than your proposed method in the experiments? Hardt. et al. (2016) solves the same post-processing problem with the extra flexibility that the sensitive attribute can be used as an input to the model. Shouldn’t that just lead to better accuracy/fairness tradeoff?

Author response:
Hardt et al. (2016)'s modification rule is oblivious (i.e., it occurs at the group level while ignoring the details of individual feature X), while our method bypass this limitation since $\check{Y}(X)$ takes advantage of individual feature X.  Furthermore, Hardt et al. (2016) target at strict fairness constraint $\delta=0$, which may hurt accuracy in order to satisfy such strong constraints.

Reviewer comment (Writing):
No major weakness apart a few issues with the writing and minor typos that can be fixed with a revision.

Author response:
Thank you for your encouraging review. We will read through and fix the typos in final revision.