Paper: Dataset Fairness: Achievable Fairness On Your Data With Utility Guarantees (ICLR-2024)

Reviewer comment (Novelty):
The method is highly based on an existing method, YOTO. Although it doesn't need to train multiple models, the cost of YOTO is not discussed in this paper.

Author response:
We describe the computational cost of the YOTO model in extensive detail in Appendix E.1.1. including the computational resources used to train each YOTO model, the time required to train each YOTO model for each dataset, the model architecture for each dataset as well as the training details for the YOTO model. At a high-level, training one YOTO model takes roughly the same amount of time as taken by each of the separately trained models in our experiments. More specifically, on Adult and COMPAS datasets, training one YOTO model on a Tesla-V100-SXM2-32GB GPU takes roughly 5 minutes, whereas on the CelebA dataset training one YOTO model takes roughly 1.5 hours and on the Jigsaw dataset training one YOTO model takes roughly 6 hours.

Reviewer comment (Evaluation):
The accuracy-fairness curve is algorithm-agnostic. Figure 4 shows that the estimated curve has more errors when using some particular algorithmic fairness methods. However, people tend to use some algorithmic fairness methods to train the model. In that case, the practical use of the curve is concerned.

Author response:
We agree with the reviewer that the accuracy-fairness curve is algorithm-agnostic, and since our intervals are designed to align with the optimal trade-off curve, they should serve as reference for any model belonging to the model class $\mathcal{H}$ regardless of the algorithmic fairness method used to train the model. Note that in practice, any methodology whose trade-off lies above our upper confidence interval is likely to have a suboptimal trade-off, suggesting that alternative approaches may offer improved trade-offs.

Reviewer comment (Theory):
The statement "For a given dataset, model class, and accuracy, the permissible range of fairness violation is x to y." in this paper is problematic. Accuracy and fairness have an inherent connection (they exhibit trade-offs) and will influence each other. So accuracy cannot be the condition for the range of fairness violation.

Author response:
We consider model fairness conditional on model accuracy **precisely because** of the accuracy-fairness trade-off. This is also in line with the fairness-constrained accuracy maximisation formulation provided in the fairness literature [Equation (3) in Agarwal et al, 2018]. Since accuracy and fairness can be at odds with each other and exhibit trade-offs, we should not consider the fairness violation value of a model in isolation.   As an example, if we do not constrain the model accuracy,  the minimum demographic parity for any given model class $\mathcal{H}$ will be 0 and will be attained by a model which makes a constant prediction regardless of the input. In this case, even though the minimum attainable demographic parity is 0, the model accuracy will likely be very low. Therefore, the goal when obtaining the optimal accuracy-fairness trade-off is to minimise the model fairness violation while constraining the model accuracy. We will further clarify this in the updated version of the paper.

Reviewer comment (Theory):
The $\mathcal{L}_{fair}$ in Eq (2) is not presented at all. This paper should present the smooth relaxation of demographic parity.

Author response:
We provided all of the smooth relaxations $L_{\text{fair}}$ considered in our work in Appendix E (pages 23 and 25). In our experiments, we considered a number of different fairness relaxations $L_{\text{fair}}$ as considered in previous fairness literature [Bendekgey & Sudderth, 2021]. For example, given a model $h_{\theta} : \mathcal{X} \rightarrow \mathbb{R}$, the relaxations for demographic parity considered in our baselines are of the form:

$$L_{\text{fair}}(h_\theta) = |\mathbb{E}[g(h_\theta(X))\mid A = 1] - \mathbb{E}[g(h_\theta(X))\mid A = 0]|$$

Where the functions $g:\mathbb{R}\rightarrow \mathbb{R}$ considered in our experiments include $g(x) = x$ (denoted as 'linear' in our results), $g(x)=\log{\sigma(x)}$ where $\sigma(x):= 1/(1+\exp{-x})$ (used for the 'separate' baseline as well as YOTO models) and $g(x)=\log{\sigma(x)}$  (denoted as 'logsig' in our results). We will also include these details in the main text in the updated version of our paper.

Reviewer comment (Experiments):
The experimental evaluation is not convincing to me. Since the Adult data is super imbalanced and the COMPAS data is small. I would suggest adding more experiments to really evaluate the proposed method, such as on the folktable dataset at https://github.com/socialfoundations/folktables

Author response:
We thank the reviewer for their suggestion. We would like to emphasise that in addition to the Adult and COMPAS datasets, we also consider the CelebA (202,599 datapoints)  and Jigsaw datasets (107,106 datapoints after pre-processing), both of which are significantly larger than Adult and COMPAS datasets. Moreover, these datasets are also comparatively more balanced with male/female split being 42%/58% in CelebA dataset and 47%/53% in Jigsaw dataset. Therefore, our experimental results demonstrate the effectiveness of our approach on datasets with varying sizes and imbalances. Additionally, training fair classifiers is likely to be more challenging on datasets with larger imbalances and smaller dataset sizes, since in these cases any reduction in fairness violation may come at the cost of a large reduction in model accuracy. Therefore, our experiments on Adult and COMPAS datasets show that our methodology remains effective even in these settings where training fair classifiers may be more challenging. Moreover, we will additionally include experimental evaluation on the folktable dataset in the updated version of our work.

Reviewer comment (Theory):
The new explanation that "the lower confidence intervals presented in Proposition 3.4 depend on the gap between the YOTO achieved trade-off curve and the ground-truth trade-off curve" is essentially making a strong assumption that the YOTO should be good enough, although not group truth. I do not think this point is reasonable and grounded.

Author response:
We wish to clarify that our work **does not** make the assumption that the curve learned with YOTO is the ground truth, and every theoretical result presented in our paper remains valid regardless of whether the YOTO trade-off curve is optimal or not. More specifically, as we mention in Section 3.2.1, the upper confidence intervals on $\tau^\ast_{\text{fair}}$ do not depend on how close the YOTO trade-off curve is to the ground truth optimum curve. In contrast, the lower confidence intervals presented in Proposition 3.4 depend on the gap between the YOTO achieved trade-off curve and the ground-truth trade-off curve (denoted by $\Delta(h)$). While this gap, $\Delta(h)$, is an unknown quantity in general, we propose sensitivity analysis techniques to posit plausible values for $\Delta(h)$ in practice in Appendix C.

The statement above does not refer to the optimality of the YOTO trade-off. Instead, the statement provides an intuitive, informal explanation of (Dosovitskiy & Djolonga, 2020, Proposition 1) and implies that under large enough model capacity, the trade-off curves obtained using YOTO should align with the trade-off curves obtained by training models separately (which may or may not be the same as optimal ground-truth trade-off curve). We will further clarify this in the updated version of our paper, by possibly including the formal result provided in (Dosovitskiy & Djolonga, 2020, Proposition 1) for completeness.

Reviewer comment (Evaluation):
The evaluation based on such small tabular data is not convincing to me, without new results presented.

Author response:
We thank the reviewer for their suggestion. We would like to emphasise that in addition to the Adult and COMPAS datasets, we also consider the CelebA (202,599 datapoints)  and Jigsaw datasets (107,106 datapoints after pre-processing), both of which are significantly larger than Adult and COMPAS datasets. Moreover, these datasets are also comparatively more balanced with male/female split being 42%/58% in CelebA dataset and 47%/53% in Jigsaw dataset. Therefore, our experimental results demonstrate the effectiveness of our approach on datasets with varying sizes and imbalances. Additionally, training fair classifiers is likely to be more challenging on datasets with larger imbalances and smaller dataset sizes, since in these cases any reduction in fairness violation may come at the cost of a large reduction in model accuracy. Therefore, our experiments on Adult and COMPAS datasets show that our methodology remains effective even in these settings where training fair classifiers may be more challenging. Moreover, we will additionally include experimental evaluation on the folktable dataset in the updated version of our work.

Reviewer comment (Novelty):
Motivation. Can the authors elaborate on the motivation for using a universe fairness requirement across datasets? What are the advantages of doing this?

Author response:
We would like to emphasise that, in this work, we argue **against** the use of a universal fairness requirement across datasets because the accuracy-fairness trade-off is highly dependent on the characteristics of the dataset.  Therefore, previous works such as [Agarwal et al., 2018 ] which use a uniform fairness requirement (such as requiring the Demographic Parity to be $\leq 0.1$ regardless of the dataset) may have varying implications for the achievable model accuracy across the different datasets. Our proposed methodology instead allows practitioners to quantify the complete accuracy-fairness tradeoff curve which may be used to obtain the range of permissible fairness violations at any given level of model accuracy chosen at inference time and does not require imposing any restrictions on model fairness or accuracy apriori at training time. We will further clarify this distinction in the next version of our paper.

Reviewer comment (Novelty):
Technique novelty. This paper introduces a computationally efficient method for estimating the trade-off. However, from my understanding, the efficiency part directly adopts YOTO framework and the confidence interval estimation only involves trivial bounds.

Author response:
There are a number of important technically novel aspects regarding both, the construction of confidence intervals and the application of the YOTO framework in our methodology, which we believe will be of interest to the ML fairness community:  
1. Firstly, we would like to emphasise that our method of constructing the confidence intervals on the trade-off curve $\tau^\ast_{\text{fair}}$ does not simply rely on trivial bounds. While standard methods like Hoeffding's inequality can be used directly to obtain confidence intervals on model accuracy, the same is not true for fairness violation metrics like Demographic Parity, Equalised Odds and Equalized Opportunity which involve multiple (conditional) expectation terms. While a naive application of union bounds can be used to construct confidence intervals on these fairness violation metrics, these bounds are known to be highly conservative and are often non-informative. Therefore, in order to construct informative confidence intervals on fairness violations, we propose a subsampling procedure in Appendix B.2 which provides a novel methodology of constructing valid confidence intervals on fairness violation metrics by subsampling calibration data. We will include a discussion of this approach in the main text of our paper.
2.  Secondly, our confidence intervals not only incorporate finite sample uncertainty from finite calibration data, but also account for the possible sub-optimality in the fairness trade-offs achieved by $h$. Specifically, to incorporate the latter, we formally consider the gap between the optimal accuracy-fairness trade-off $\tau^\ast_{\text{fair}}$ and the trade-off achieved by a model $h$, denoted by $\Delta(h)$ in Figure 2 in the main text. Part of our methodology for obtaining the lower confidence interval involves quantifying this gap $\Delta(h)$. While this gap $\Delta(h)$ is intractable in general, we propose a sensitivity analysis methodology to posit plausible value for $\Delta(h)$ in practice (in Appendix C). In contrast to the previous works in fairness literature [Agarwal et al., 2018, Bendekgey & Sudderth, 2021, Zafar et al., 2015; 2017; 2019], to the best of our knowledge, our work is the first to consider uncertainty arising from both, finite datasets used to evaluate accuracy and fairness metrics as well as the possible sub-optimality in the fairness trade-offs achieved by $h$. 
3. Thirdly, we also propose a novel methodology for constructing informative confidence intervals in the setting where the sensitive attributes $A$ are accessible only for a small subset of the calibration dataset. This methodology effectively combines data with actual and predicted sensitive attributes to derive tighter and more accurate CIs, even when the majority of $A$ values are absent.
4. Finally, while YOTO has previously been applied in areas like multi-task learning [Lin et al. 2020], our work provides a novel application of the YOTO framework to fairness literature and offers practitioners the capability to, at inference time, specify desired accuracy levels and promptly receive corresponding admissible fairness violation ranges.

Reviewer comment (Evaluation):
Pareto frontier. The achievable trade-off by YOTO may not be consistent with the ground-truth Pareto optimum. It seems that this paper is over-claimed since the true Pareto trade-off investigation is not touched. Additionally, how do you use a universe fairness requirement across datasets? The approximated trade-off seems not be a good choice since the gap between achievable trade-off by YOTO and ground-truth Pareto optimum may also be dataset-dependent.

Author response:
We do, in-fact, consider the gap between YOTO achievable trade-off and ground-truth Pareto optimum when constructing the confidence intervals. Specifically, the $\Delta(h)$ term in the lower confidence intervals (Proposition 3.4) quantifies this gap (Figure 2 also provides a visual representation of this gap). Moreover, the upper confidence intervals obtained in Proposition 3.2 do not depend on the gap between YOTO achievable trade-off and ground-truth Pareto optimum and remain valid even if the YOTO model achieves sub-optimal accuracy-fairness trade-offs. On the other hand, the lower confidence intervals obtained in Proposition 3.4 depend on the  $\Delta(h)$ term which is intractable in general. To remediate this, we employ sensitivity analysis techniques to incorporate any belief on plausible values for $\Delta(h)$ in Appendix C. We will further clarify this in the updated version of our paper.

As we emphasise in our response above, in our work we argue **against** the use of a universal requirement across different datasets. It is true that the gap between YOTO achievable trade-offs and the ground-truth Pareto optimum may be data dependent. Our confidence intervals in Proposition 3.4 take this gap into account explicitly using the $\Delta(h)$ term. In this case, the use of sensitivity analysis described in Appendix C may be used to posit plausible values for this gap in practice using the calibration data.

Reviewer comment (Experiments):
Is it possible to create a synthetic dataset with a known ground-truth trade-off in the experiments? Otherwise, many conclusions can only hold for the achievable trade-off by YOTO.

Author response:
We thank the reviewer for their suggestion. We have conducted additional experiments with a synthetic dataset with a known ground-truth trade-off. The results which are provided in Appendix E.5 of our updated manuscript show that the empirical trade-off achieved by YOTO aligns very well with the ground truth trade-off curve. Additionally, the ground truth trade-off curve is covered by all the confidence intervals constructed. This simple setting serves as a sanity check to make sure that the trade-offs obtained using YOTO are consistent with the ground truth and that the intervals obtained using our methodology are well-calibrated.

Reviewer comment (Presentation):
There are confidence intervals for both accuracy and fairness. How can you plot these two intervals in Figures 3 and 4?

Author response:
As we explain in Propositions 3.2 and 3.4, the confidence intervals for accuracy and fairness can be used to obtain confidence on the optimum achievable trade-off $\tau^\ast_{\text{fair}}$ of the form:
$$\mathbb{P}(\tau^\ast_{\text{fair}}(\Psi) \in \Gamma_{\text{fair}}^\ast) \geq 1-\alpha$$
In Figures 3 and 4, we do not plot the confidence intervals for accuracy and fairness individually, but instead, we use the results in Propositions 3.2 and 3.4 to plot the two-sided confidence intervals $\Gamma^\ast_{\text{fair}}$ on optimum trade-off $\tau^\ast_{\text{fair}}$ directly. 

Intuitively, to understand how these confidence intervals are obtained, consider the example where the accuracy of a model $h$ is  $\geq 90\\%$, and the DP of $h$ is  $\leq 15\\%$. Then the worst-case accuracy-fairness trade-off of model $h$ will be achieved when $\text{acc}(h) = 90\\%$ and $\Phi_{\text{DP}}(h)=15\\%$ (i.e., accuracy is minimised and the demographic parity is maximised). Since $\tau^\ast_{\text{fair}}$ denotes the ground-truth optimum trade-off, it must not achieve a worse accuracy-fairness trade-off than the model $h$, i.e.  $\tau^\ast_{\text{fair}}(\text{acc}(h)) \leq \Phi_{\text{DP}}(h)$ and therefore, in the worst-case, $\tau^\ast_{\text{fair}}(90\\%) \leq 15\\%$. This provides us with a valid upper bound on the accuracy-fairness trade-off curve at accuracy = 90%. Next, to obtain an upper confidence interval on each accuracy value, we consider different values of $\lambda$ in our YOTO model, each of which leads to a different worst-case trade-off and provides an upper bound for the curve at different accuracy values. We will further make this clearer in the next version of our paper. The lower bound can be analogously determined, although this involves additional consideration as outlined in Section 3.2.2.

Reviewer comment (Theory):
From my understanding, the optimization for fairness with accuracy and the optimization for accuracy with fairness constraint have the same trade-off. I am curious why the authors select the former one and highlight the difference in the first paragraph of Section 2.1.

Author response:
It is true that optimization for fairness with accuracy constraint and the optimization for accuracy with fairness constraint should lead to the same trade-off. However, in our problem setup in Section 2.1, we specifically choose the former formulation so as not to impose constraints on fairness violations apriori. This is in line with our argument that setting a uniform requirement for fairness regardless of the dataset ignores the dataset-specific accuracy-fairness trade-offs. To this end, our methodology approximates the complete trade-off curve $\tau^\ast_{\text{fair}}$ without imposing apriori restrictions on either model fairness or model accuracy at training time unlike [Agarwal et al, 2018]. We will further clarify this in the updated version of our paper.