Paper: FairPATE: Exposing the Pareto Frontier of Fairness, Privacy, Accuracy, and Coverage (ICLR-2024)

Reviewer comment (Evaluation):
__W1__ Significance: The paper aims to convey the significance of its contribution through experimental results (and I don't think there is anything wrong with it and in general support extensive empirical results), but the experiments present a rather bleak picture of the advantages of FairPATE.  
**Minimal improvements** Most importantly, there are rarely any results where the improvements of PATE over baselines is larger than 1\%. For example in 1. Credit card dataset and 2. parkinson's dataset, there results differ by less than $1\%$.

Author response:
- [W1] Currently the paper presents a new algorithm for improving fairness-accuracy pareto frontier but the main weakness here is that in a significant number of situations this improvement does not happen but rather a degradation happens. I would recommend that the authors rethink about what the main contribution is, perhaps identify more clearly in the main text where they expect an improvement and where they don't and then show that through experiments. Doing this discussion through rebuttal is not the best as this a major point of this paper.

- Moreover, i had noticed the plots in Appendix H, but the plots were not very informative regarding how important the improvement is due to the missing comparisons. Again I appreciate that the authors are running the experiments but this is an important contribution of the paper and hence the experiments for this should not really be done during the rebuttal and it should be possible to present the "improvements of the algorithm on seven distinct datasets" in the main paper. Perhaps, the paper needs a restructuring if all the exerimental improvements obtained by a novel algorithm cannot be presented in the main pages of the paper.

	- We thank the reviewer for their feedback. We made a general statement regarding our contributions (in the Author General Response). Please also consider our edits and response to **reviewer Xkhf** newest comments.

Reviewer comment (Evaluation):
**Several examples of underperformance** Several examples also show that FairPATE performs significantly worse than competitor. Examples include Demographic Parity for Adult dataset and UTK Face ($\epsilon=5$).

Author response:
- **Several examples of underperformance** Several examples also show that FairPATE performs significantly worse than competitor. Examples include Demographic Parity for Adult dataset and UTK Face ().

	- For the concerns about significance, please see our General Response.

	- We have noted clearly in the main paper that for the Adult dataset, under Demographic Parity DP-Fermi performs better than FairPATE for smaller epsilon values (Figure 10). However that trend is not consistent over different datasets (on UTKFace, FairPATE does better for lower epsilons and can even go lower in terms of achievable classification error—see Figure 6). We provide a discussion on the cause of these performance differentials in response to **Reviewer XkhF.**

Reviewer comment (Experiments):
**Misleading regarding diversity of experiments** In the introduction, the paper sells itself regarding performing a wide range of experiments but results on nearly half of these datasets are hidden away in the appendix without any mention in the main text and without comparisons. Experiments on **CheXpert, CelebA, FairFace, and Retired-Adults** are not available in the main text and it is very hard to interpret the results presented for this in the appendix. In fact, there are **no results of FAIR-PATE on CheXpert** in the paper. Please show comparison on these for demographic parity and equalized odds (similar to Figure 2,3 with comparison to Loewe et. al.'s algorithm) on these four datasets. In addition, there are results in the literature that run DP algorithms on CelebA and CheXpert. Comparisons should be made to them to show what is the sacrifice being made compared to state-of-the-art.

Author response:
- **Misleading regarding diversity of experiments** In the introduction, the paper sells itself regarding performing a wide range of experiments but results on nearly half of these datasets are hidden away in the appendix without any mention in the main text and without comparisons. Experiments on CheXpert, CelebA, FairFace, and Retired-Adults are not available in the main text and it is very hard to interpret the results presented for this in the appendix. In fact, there areno results of FAIR-PATE on CheXpert in the paper

	- The CheXpert results were already present in the paper (in the last figure of the appendix) but was unfortunately mislabeled. We apologize for the error.  The figure's label has been corrected in the updated manuscript.

	- We currently have results of FairPATE on CheXpert, CelebA and FairFace in Section H in the appendix.  Unfortunately given the space constraints we cannot add all the aforementioned results in the main text.

- Please show comparison on these for demographic parity and equalized odds (similar to Figure 2,3 with comparison to Loewe et. al.'s algorithm) on these four datasets. In addition, there are results in the literature that run DP algorithms on CelebA and CheXpert. Comparisons should be made to them to show what is the sacrifice being made compared to state-of-the-art.

	- DP-FERMI is an expensive algorithm to run. On CelebA, it requires 33 hours to train one model for 200 epochs as specified in the paper without parameter tuning on an NVIDIA A100; while the model has at least 4 hyper-parameters to tune. Nevertheless, we are currently in the process on running DP-FERMI on CelebA, and we will report the results as we acquire them in the coming days.

Reviewer comment (Theory):
__W2__ **Wrong Conclusion from Theorem 2** I did not go through the detail of Theorem 1 but I assume it is correct and follows from Group privacy arguments. However, the sentence above Theorem 1 (Point C2) claims that the result proves that " pre-processing ... will necesarrily degrade the guarantee of the .... private learning mechanism". However this is not what the Theorem shows. The theorem only proves a privacy guarantee of $M\odot P_{\text{pre}}$ not that this is the tightest privacy guarantee possible for the composed mechanism. Am I missing something?

Author response:
- My point in weakness 2 is that the result shows that the composed mechanism is at least  $(\epsilon, \delta)$ -DP. However the argument in the text says that the "will necessarily degrade the guarantee". The first sentence is an upper bound on the privacy guarantee of the mechanism whereas the second sentence is a sentence akin to a lower bound.

	- We have edited the sentence in question for clarity to:
	  > (C2) Fairness pre-processing increases the cost of private training. In Theorem 1, we show that pre-processing the training data to equalize subpopulation rates will degrade the privacy guarantee of the composed mechanism (i.e., preprocessor followed by private learning) compared to that of the private learning mechanism alone

Reviewer comment (Theory):
__W3__ **Abstaining from prediction for fairness reasons** The introduction justifies this as _"if a decision cannot be made without violating a pre-specified fairness metric, then the model can refuse to answer at which point the decision can be relegated to a human judge"_. If I have understood this correctly, this is a flawed argument. For example, consider the situation where there are group of data points from the majority group, on which if the algorithm predicts correctly the fairness will be violated and hence the algorithm abstains. Nevertheless, the prediction is easy for this group and the judge nevertheless predicts on them correctly and the overall fairness is still violated. Isn't this pointless then to relegate this to the judge? Intuitively, it appears that any problem can be made fair this way by simply refusing to predict on certain majority groups thereby artificially boosting the fairness of the algorithm. The correct fairness solution should aim to improve performance on the minority group instead.

Author response:
- **W3** **Abstaining from prediction for fairness reasons** The introduction justifies this as *"if a decision cannot be made without violating a pre-specified fairness metric, then the model can refuse to answer at which point the decision can be relegated to a human judge"*. If I have understood this correctly, this is a flawed argument...

	- We think there has been a misunderstanding regarding a) purpose of the reject option for fairness, and b) the inference-time nature of the IPP algorithm that implements it. What the reject-option enables is for a human (e.g. a trained judge) to take over the decision making in case of a fairness constraint violation. **The presumption here is that a trained judge is better placed to make impactful decisions than the algorithm.** Therefore, the human judge can choose to accept violations of the fairness constraint on a case-by-case basis. We do not believe the human judge should be placed under the same constraints as the algorithm given the vast difference in training, experience and accountability faced by the human judge versus the algorithm.

-  Consider the situation where there are group of data points from the majority group, on which if the algorithm predicts correctly the fairness will be violated and hence the algorithm abstains
	- Please note that IPP is an **inference-time** algorithm. It is incorrect to say that the "algorithm predicts correctly that the fairness will be violated" because there is no prediction involved in this process. For instance, using Demographic Parity, the fairness metric requires that the rates of acceptance be similar for different subgroups. Since IPP is releasing decisions, it can keep an exact record of the decisions released (this is what $m(z,k)$ counter keeps in Algorithm 1). Therefore, the rates calculated based on this record is exact and so are the detected violations that would prompt the algorithm to reject a certain query.

- Intuitively, it appears that any problem can be made fair this way by simply refusing to predict on certain majority groups thereby artificially boosting the fairness of the algorithm. The correct fairness solution should aim to improve performance on the minority group instead.
	- Our standard fairness metric does not require a notion of majority vs. minority. At any given decision point, our algorithm (IPP) bounds the worst-case fairness violations experienced by any subpopulation. Furthermore, the reject-option is not a silver bullet; as it comes with a trade-off with coverage (1-rejection rate). Therefore the more rejections translate more work for the human judge. It is, therefore, in the best interest of deployer of the upstream algorithm (FairPATE or DP-FERMI, or any other fairness-aware) algorithm to minimize fairness violations at training-time; because an unfair algorithm will increase the work that human judges would need to do— eliminating the need for the fair algorithm in the first place.

Reviewer comment (Experiments):
__W4__ **Unfair Comparisons** Fair-PATE and Loewy et. al's algorithm do not work under the same restrictions of differential privacy. Specifically, Fair-PATE uses public unlabelled data and Loewy et. al doesn't and we know that (even a small amount of) public data can severely help in improving accuracy of DP models (perhaps fairness too). Hence, it appears to me that Loewy et. al. uses a significantly stricter notion of privacy and achieves nearly comparable result and in some cases even better (for tabular datasets, there are no comparison on vision datasets with Loewy et. al.).

Author response:
- **W4**  **Unfair Comparisons** Fair-PATE and Loewy et. al's algorithm do not work under the same restrictions of differential privacy. Specifically, Fair-PATE uses public unlabelled data and Loewy et. al. doesn't and we know that (even a small amount of) public data can severely help in improving accuracy of DP models (perhaps fairness too). Hence, it appears to me that Loewy et. al. uses a significantly stricter notion of privacy and achieves nearly comparable result and in some cases even better (for tabular datasets, there are no comparison on vision datasets with Loewy et. al.).

	- We wish to clarify and disentangle two issues that the reviewer points out: a) access to public data, b) access to additional data. All of our baselines experiments are conducted using the exact data splits of Lowy et al. 2023. This means that we do not use any additional training data compared to our baselines. Instead, we use the training data available to us and partition it into public and private. Consistent with the original PATE framework, we do not use the labels of the public dataset; instead we train teacher models on the private partition and label the public points under privacy and fairness constraints. In Figure 1.a, a DP-SGD method like Lowy et al. 2023 replaces the shaded Private Model. Therefore, we respectfully disagree with the reviewer's assessment that the comparisons are not fair data-wise. As a matter of fact, in our experiments, DP-Fermi receives additional information compared to FairPATE (namely, the labels of the public data partition).

Reviewer comment (Theory):
**Motivation** I understand that there are several works showing that privacy incurs a sacrifice in fairness. However, one possible approach to this problem is simply improving the accuracy and perhaps the resultant accuracy also leads to high fairness. In fact Berrada et. al. (2023) makes this argument with some experiments in their paper. I did not see an argument in this paper why this approach is not expected to solve all problems. Why should there be a trade-off between privacy and fairness? Is it known in the literature and are there theoretical studies arguing about the necessity of a trade-off?

Author response:
- Motivation. I understand that there are several works showing that privacy incurs a sacrifice in fairness. However, one possible approach to this problem is simply improving the accuracy and perhaps the resultant accuracy also leads to high fairness. In fact Berrada et. al. (2023) makes this argument with some experiments in their paper. I did not see an argument in this paper why this approach is not expected to solve all problems. Why should there be a trade-off between privacy and fairness ? Is it known in the literature and are there theoretical studies arguing about the necessity of a trade-off ?

	- Berrada et al. 2023 only consider unfairness with respect to loss disparity. This particular fairness metric is inherently a measure of the generalization of the model. That is, if a model is well generalized, then it is also well-generalized for different subpopulations of the data; therefore, it is only natural that it would exhibit lower loss disparity. As a result Berrada et al. 2023 do not consider (or rather, need to consider) an unfairness mitigation step. Our results (and that of Lowy et al. 2023 and others) show that for other group fairness metrics, this is not the case. That is, fairness and accuracy do not align so well all the time.

	- In terms of theoretical results Cummings et al. 2019 provide an the impossibility result for DP learning and exact fairness; while also demonstrating a positive result for the case of approximate fairness (our settings). More recently, Mangold et al. 2023 have showed that "the fairness (and accuracy) costs induced by privacy in differentially private classification vanishes at a $O(\sqrt{p}/n)$ rate, where n is the number of training records, and p the number of parameters." This is achieved through a proof of pointwise Lipschitz smoothness of group fairness metrics with respect to the model where the pointwise Lipschitz constant explicitly depends on the confidence margin of the model, and may be different for each sensitive group. Therefore, in the presence of minority groups in the data, we can expect a non-zero gap between private and non-private models.

Reviewer comment (Experiments):
**Missing baselines** Why is there no comparison with the algorithm of Zhang et. al. (2021), Kulynuych et. al. (2022) and Berrada et. al. (2023)? They are mentioned in one or two sentences at the very end of the paper but without any comprehensive empirical comparison.

Author response:
- Missing baselines. Why is there no comparison with the algorithm of Zhang et. al. (2021), Kulynuych et. al. (2022) and Berrada et. al. (2023) ? They are mentioned in one or two sentences at the very end of the paper but without any comprehensive empirical comparison.

	- Kulynych et. al 2021 does not account for the privacy leakage of its importance sampling step therefore a fair comparison with methods such as ours or Lowy et al. 2023 is not possible

	- Berrada et al. 2023 only considers loss parity, and has no unfairness mitigation for two of our gorup fairness metrics (see our previous answer for more details)

	- Zhang et al. 2021 implement early stopping to save on privacy budget; however, they do not measure the privacy leakage of their early stopping criterion therefore the privacy loss reported is underestimated

Reviewer comment (Writing):
**FairDP-SGD** What is FairDP-SGD? It doesn't seem to have been defined anywhere? Is it an existing work? Where is FAIR-PATE's result on CheXpert?

Author response:
- What is FairDP-SGD ? It doesn't seem to have been defined anywhere ? Is it an existing work ? Where is FAIR-PATE's result on CheXpert ?

	- This is an error on our part. The figure in the appendix was mislabelled. We apologize for the confusion. The figure's label has been corrected in the updated manuscript.

Reviewer comment (Theory):
In arguments for intervention points, assumptions are made which preclude solutions. They assume the intervention needs to be made independent of other mechanisms in PATE. That is, they cannot consider information internal to decision making that is not described by Figure 1 like individual teacher outputs. This leaves the possibility that some fairness methods might be able to be integrated with PATE in a closer manner than the options described. One example is that they might include the teacher outputs instead of operating on the overall predicted class like Algorithm 1 assumes presently. C3 in particular suggests that some interventions will not account for privacy budget correctly due to special circumstances and suggests at Point 4, budgeting can be handled correctly. Nothing is stopping a design from refunding privacy budget if a query is rejected subsequently to an intervention point.  
Suggestion: rephrase arguments for why some intervention points are bad to make sure they don't also make assumptions about how the interventions are made and whether they can interact with privacy budget.

Author response:
- Regarding the suggested alternative scheme, we would like to to ensure that we understand the premise correctly: is the reviewer suggesting that we run PATE, accept and reject queries according to standard PATE analysis, and then apply the fairness intervention and reject possibly some more queries; and come back to give back some of the privacy budget?

	- If the answer is yes, then we note that standard PATE privacy analysis in Papernot et al. 2016 (summarized  in Section B in the appendix) is based on the probability of answering a certain query. To recoup the "unused" privacy budget one needs to find the difference between the privacy budget if the sample would have been answered and the privacy budget if the sample wouldn't have been answered. Note that the this is exactly what FairPATE already performs. Therefore, the suggested scheme is functionally equivalent to FairPATE but presents additional complexity.

Reviewer comment (Experiments):
Comparisons against methods in which rejection due to fairness is not an option may not be fair.  
Suggestion: either integrate suggestion regarding accounting for rejection above, or incorporate some form of rejection (or simulate it) in the existing methods being compared to. It may be that the best methodology is not FairPATE but some existing baselines if adjusted to include fairness rejection option.

Author response:
- The reviewer is correct that the IPP algorithm is an inference-time algorithm, therefore, its use is not limited to a FairPATE model.
	- We have run additional experiments on other models with the IPP;  and included the results in Section E.1 in the appendix. We find that FairPATE with IPP outperforms baselines (with IPP) in most regions fairness-privacy settings in both accuracy and coverage. See Figure 11 for a 2d Pareto surface, or Figure 12 for a corresponding 3d plot which showcases the accuracy performance improvements better.

Reviewer comment (Evaluation):
Rejection rate is not shown in any experiments. One could view a misclassification as a rejection, however. Please include rejection rates or view them as misclassifications in the results.

Author response:
- We do report **coverage** which is 1-rejection rate for the results that use the IPP. See Figure 5 where coverage is marked by the **color**; similarly the Pareto frontiers in the appendix all include coverage encoded in color.

Reviewer comment (Theory):
The distribution whose fairness needs to be protected is left to be guessed by the reader. For privacy, it is more clear that it is the private data that is sensitive and thus privacy budgeting is done when accessing that private data as opposed to the public data. For fairness, the impact on individuals in the private dataset seems to be non-existent as the decisions for them are never made, released, or implemented in some downstream outcome. The presumption then is that fairness needs to be respected on the public data. Algorithm 1 and several points throughout the work hint at this. However, there is also the consideration of intervention points 1, 2, and 3, which seem odd as they are points seen before any individual for whom fairness is considered is seen. That is, fairness about public individuals cannot be made there, independent of any other issues such as privacy budgeting. Further, Theorem 1 discusses a demographic parity pre-processor which achieves demographic parity on private data which is presumed to be irrelevant.

Author response:
- A fairness intervention and a privacy intervention have different goals w.r.t. the data distribution they target. In private learning, the goal is to protect the privacy of the sensitive training data only. In fair learning, the end goal is to ensure the algorithmic decisions (at inference-time) are fair. Within the context of FairPATE; from a private learning perspective, PATE (and FairPATE by extension) protect private teacher data (see Figure 1.a) and not the public unlabeld data. From a fair learning perspective, a FairPATE model is just like any other fairness-aware model is trained such that it achieves fairness on training data, **with the expectation that that behavior generalizes to inference-time.** These are standard assumptions which is why we have refrained from repeating them.

Reviewer comment (Writing):
The statement "PATE relies on unlabeled public data, which lacks the ground truth labels Y" is a bit confusing unless one has already understood that fairness is with respect to public data. PATE also relies on private labeled data to create the teachers.

Author response:
- A fairness intervention and a privacy intervention have different goals w.r.t. the data distribution they target. In private learning, the goal is to protect the privacy of the sensitive training data only. In fair learning, the end goal is to ensure the algorithmic decisions (at inference-time) are fair. Within the context of FairPATE; from a private learning perspective, PATE (and FairPATE by extension) protect private teacher data (see Figure 1.a) and not the public unlabeld data. From a fair learning perspective, a FairPATE model is just like any other fairness-aware model is trained such that it achieves fairness on training data, **with the expectation that that behavior generalizes to inference-time.** These are standard assumptions which is why we have refrained from repeating them.

Reviewer comment (Writing):
There is a double "violations" near the phrase "violations of demographic disparity violations".

Author response:
- We thank the reviewer for their keen eye. We have fixed the typo and revised the sentence to
	  > ..DP that only protects privacy of individuals with respect to particular sensitive features...

Reviewer comment (Novelty):
Fairness is "enforced" by adding parity constraint within the aggregator which acts as a rejection sampling step on the basis of fairness. The general idea was proposed in several other works in the past, including the ones cited by the authors and against which the authors compare. It is thus difficult to understand what is new in the proposed framework. An explicit mention would help.

Author response:
The reviewer is correct that our work enforces fairness on the level of sample selection. However, this is done within a differentially private learning setup. The only prior work that we are aware of that has a sampling step for fairness within a private-learning setup is Kulynych et. al 2021 but importantly that work does not consider the privacy cost of the fairness importance sampling it performs while our work does.
	- We have added an extended related works section in Section I in the appendix with a more detailed comparison to Kulynych et. al 2021.

Reviewer comment (Presentation):
The experimental analysis should be improved. Some figures are misleading, e.g., same colors used for different algorithms (see questions below).

Author response:
We have re-generated the figures in question (3 and 4) and updated the manuscript accordingly.

Reviewer comment (Experiments):
I can't judge what is the impact of IPP (the rejection step added at inference time) on the overall results. Can you provide some ablation study showing how the framework performs on various datasets with different majority/minority distributions with and without IPP?

Author response:
We have run additional experiments on FairPATE+IPP and have added the results Section E.2 in the appendix of the updated manuscript. We observe that under class-imbalance for every percentage of coverage degradation, the fairness-privacy curve improves consistently. For the balanced case, we observe that the disparity levels are much higher. This is because random (non-stratified) sampling is not conducive to demographic parity.  IPP's overall behavior is similar as before but the curves are much closer to each other and the rejection rates are higher due to high disparity levels of the initial models caused by random sampling.

Reviewer comment (Evaluation):
How does the framework work in case of some distribution shift? This is especially important in the context of my question above.

Author response:
We have run additional experiments on FairPATE+IPP and have added the results Section E.2 in the appendix of the updated manuscript. We observe that under class-imbalance for every percentage of coverage degradation, the fairness-privacy curve improves consistently. For the balanced case, we observe that the disparity levels are much higher. This is because random (non-stratified) sampling is not conducive to demographic parity.  IPP's overall behavior is similar as before but the curves are much closer to each other and the rejection rates are higher due to high disparity levels of the initial models caused by random sampling.

Reviewer comment (Presentation):
Fig. 2 and 3 use orange colors for two different algorithms (Tran et al (Fig 2) and Jagielski et al. (Fig 3)). The authors should report all algorithms in all figures or justify their absence.

Author response:
We have regenerated and replaced Figure 3 to ensure distinct colors. We apologize for the confusion.
	- As to the reason why Figure 3 does not include Tran 21 et al, we have already made an explicit note in the empirical section that we report the same  baseline results as reported by Lowy et al. 2023 (in 2nd line of SOTA Baseline Comparisons) including that of Tran et al 2021.

Reviewer comment (Experiments):
Why Tran et and Jagielski et al. are not reported for the UTK-dataset experiment?

Author response:
Both methods are evaluated on tabular data and are not suitable for deep learning models. In particular, Lowy et al. 2023 report that they have attempted this to no avail:
	  > We observed that the baselines were very unstable while training and mostly gave degenerate results(predicting a single output irrespective of the input). By contrast, our method was able to obtain stable and meaningful tradeoff curves.  
	    
	  Given that DP-FERMI reports better fairness-accuracy trade-offs at every privacy level compared to these baselines; we found it sufficient to test FairPATE against the new state-of-the-art (DP-FERMI); and report the other baselines for which we had reported data.