Paper: Fairness Under Demographic Scarce Regime (ICLR-2024)

Reviewer comment (Experiments):
The biggest concern I have is that the uncertainty threshold greatly impacts the performance of the proposed approach. It is not clear that if the authors have a reliable way to determine this threshold in practice (other than selecting the best performing one on the test data). Does tuning on a validation set reveal the best threshold for the data? Does it generalize to the test set? This should be either clarified or added as new experiments.

Author response:
Thank you for bringing this up. The uncertainty threshold is a hyperparameter that can be tuned on the validation set. We used 10% of the training data for validation. We highlighted in the paper how we tuned the threshold. Furthermore, we observe the higher variation of the uncertainty threshold occurs in datasets with already higher uncertainty (e.g., LSAC and New Adult), from which models without fairness constraints provide lower bias. There are similar variations in the Pareto fronts over the validation set and the best-performing finetuned threshold also transfers to the test set.

Reviewer comment (Writing):
Despite the fact that the scenario discussed in this paper is not entirely new, I would still suggest the authors provide a real example scenario when D1 has no demographic information but has label information and D2 has demographic information but no label information.

Author response:
We have added in the problem formulation references to real-world examples where this setup applies. In particular, they can be found in healthcare [1, 2] or finance [3, 4] where patients/users self-report their sensitive information.

Reviewer comment (Experiments):
Results of "Ours" in Figure 2 were tuned for uncertainty threshold over [0.1, 0.7]. When you tune the threshold, are you using a validation set from the 0.7 training data or are you just selecting the best performing threshold on the 0.3 test data?

Author response:
Thank you for bringing this up. The uncertainty threshold is a hyperparameter that can be tuned on the validation set. We used 10% of the training data for validation. We highlighted in the paper how we tuned the threshold. Furthermore, we observe the higher variation of the uncertainty threshold occurs in datasets with already higher uncertainty (e.g., LSAC and New Adult), from which models without fairness constraints provide lower bias. There are similar variations in the Pareto fronts over the validation set and the best-performing finetuned threshold also transfers to the test set.

Reviewer comment (Experiments):
In Section 3 Problem formulation, you mentioned: " However, to be able to estimate bias in label classifier f, we assume there exists a small set of samples drawn from the joint distribution X × Y × A, i.e., samples that jointly have label and demographic information." How was this reflected in the experiments? Does it refer to the 0.3 test data you used in the experiments? If so, I do not think this is a hard requirement in real world applications.

Author response:
The joint distribution X × Y × A refers to test data used in the experiments. This assumption allows us to report the true fairness violation. Moreover, we could alleviate this assumption by using proxy-sensitive attributes for the evaluation, which might be a more realistic setup. But these proxies are likely noisy and there is a risk of overestimating or underestimating the true disparities in the model. We have added a limitation section where we discuss the limitation of such assumption in real-world applications and point out the possibility of using of bias estimation approaches that effectively capture the true disparities when the proxy demographic information is used [5] [6].

Reviewer comment (Experiments):
Please clarify: in Section 4.2, "the label classifier with fairness constraints is trained on a subset D′1 ⊂ D1". Does this mean the second phase classifier does train on any information except for D'1? Is that possible to train that label classifier on the entire training set D1 with fairness constraints on D'1--- fairness loss will be 0 for data do not belonging to D'1? This could potentially increase the accuracy of the label classifier when uncertainty threshold is low.

Author response:
Great suggestion. The main objective of the paper is to provide evidence for the hypothesis we formulated. Modifying fairness-enhancing algorithms to account for the uncertainty is an interesting direction to explore. We have mentioned in the conclusion a similar perspective for future work where the fairness-enhancing model is trained on samples weighted by the uncertainty of predicting their sensitive attributes.

Reviewer comment (Experiments):
I find the comparison with respect to the state of the art to be lacking. The attribute classifiers chosen in this work, such as Proxy-kNN and Proxy-DNN, are rather simplistic and not well-documented in existing literature. Moreover, there are established attribute classifiers like those introduced by Diana et al. (2022) and Awasthi et al. (2021) that are not considered in this comparison.

Author response:
Some of the baselines considered also use proxy-sensitive attributes (Zhao et al., 2022; Liang et al., 2023; Yan et al., 2020;), and methods without relying on sensitive attributes such as ARL (Lahoti et al., 2020) can also improve group metrics such as equalized odds. Other baselines are considered for a comprehensive evaluation and we highlighted that the experimental setup might be different, in particular for methods targeting worst-case group improvement. 
We couldn’t perform comparisons with Diana et al, (2022) as the authors did not publish their code and we couldn’t reproduce their results within the short rebuttal period. Moreover, Awasthi et al. (2021) focused on postprocessing techniques for equalized odds, which generally highly impact accuracy,  while we considered inprocessing techniques with controllable fairness-accuracy tradeoffs. Furthermore, the results reported using true sensitive attributes represent the optimal expected baseline for fairness, and comparison shows our method can achieve similar and in most cases better tradeoffs. It does not seem to be the case for Awasthi et al. (2021) and Diana et al, (2022) on the same datasets.

Reviewer comment (Evaluation):
The authors assert in the abstract that 'our framework outperforms models trained with constraints on the true sensitive attribute,' referring to the results from Figure 2. However, this result only considers a single fairness-enhancing intervention. Additionally, their framework does not consistently outperform models trained with ground truth sensitive attribute values in all cases, making the statement partially true.

Author response:
Thank you for the suggestions. We have updated the statement accordingly and included privacy-related approaches in the related work section in the revised version.  We have also highlighted in the limitation section and emphasized the need for further studies to explore whether our method and hypothesis extend to other fairness-enhancing methods, e.g., preprocessing or post-processing techniques.

Reviewer comment (Experiments):
This discrepancy is even more apparent in the results from Figure 9, where a different fairness-enhancing intervention is employed. It's unclear to what extent this outcome is influenced by the chosen fairness-enhancing intervention. Both interventions analyzed in the study share similarities, and it would be beneficial to examine a more diverse set of fairness-enhancing interventions to better understand the impact of the chosen intervention on the results. Therefore, I recommend that the authors modify the statement to emphasize that the framework outperforms models trained with constraints on the true sensitive attribute in some cases, and I encourage them to delve into the conditions under which this outperformance occurs.

Author response:
Thank you for the suggestions. We have updated the statement accordingly and included privacy-related approaches in the related work section in the revised version.  We have also highlighted in the limitation section and emphasized the need for further studies to explore whether our method and hypothesis extend to other fairness-enhancing methods, e.g., preprocessing or post-processing techniques.

Reviewer comment (Experiments):
The experiments provide support for the assertion that discriminating against samples with more uncertain sensitive information is a challenging task. Rather than attempting to predict the sensitive information of instances (an action that is illegal and morally questionable) and use those instances for which you know the sensitive information with high confidence, why not directly utilize those instances for which the uncertainty is highest with respect to the sensitive attribute and train a non fairness-aware classifier on top of those instances? In other words, perhaps utilizing your attribute classifier to identify the most 'fair' samples based on high uncertainty in sensitive information might yield more ethically favourable results.

Author response:
Indeed, it is possible to use samples with the highest uncertainty in the sensitive attribute predictions to train models without fairness constraints. Table 1 in the main paper shows that datasets with higher uncertainty in the sensitive attribute prediction tend to be inherently fairer. 
Furthermore, we performed additional experiments where we trained different classifiers (Logistic Regression and Random Forest) without fairness constraints but using instances with higher uncertainty and for different uncertainty thresholds (see Appendix G). 
Results are presented in the newly added Figure 11 of the revised version of the paper. The results show we can get fairer models by pruning out samples with low uncertainty in attribute prediction. This shows that our method can also be applied in settings where the use of inferred protected attributes is illegal or ethically concerning.

Reviewer comment (Experiments):
For classifiers that propose fairness-enhancing interventions while lacking information on the specific sensitive attribute considered in the experimental section (Lahoti et al., 2020; Hashimoto et al., 2018; Levy et al., 2020; Yan et al., 2020; Zhao et al., 2022), it's essential to clarify the dataset utilized. Do you feed them with the complete D1, D1 + D2, or only D1'?

Author response:
We feed all the baselines on the complete D1, i.e., the dataset with the target labels and missing sensitive attributes. For the baselines using a proxy classifier the dataset D2 is used to train to proxy classifier, which is used to infer the missing sensitive attributes in D1. Only our method uses D1' where the uncertainty threshold was applied. Overall, we only use D1 for training each baseline, or D1 with the inferred sensitive attributes where needed. We have clarified this in the revised version of the paper.

Reviewer comment (Evaluation):
The experiments demonstrate significant variations in results depending on the chosen uncertainty threshold. If this model were to be applied in a real-world scenario, do you have a practical method for selecting the optimal uncertainty threshold, rather than relying on trial and error to determine the best-performing value?

Author response:
The uncertainty threshold is a hyperparameter that can be finetuned on a validation set. A similar variation is also reflected in the validation set and the results show that the best-performing threshold on the validation set transfers well to the test set. Furthermore, the high variation of the uncertainty threshold occurs in datasets with already higher uncertainty (e.g., LSAC and New Adult). We referred to this setting as a _low bias regime_, i.e., a situation where most samples already have high uncertainty of the sensitive attributes and the model already has low bias.  We have highlighted how we tuned the uncertainty threshold in the experimental setup.

Reviewer comment (Novelty):
If I am not mistaken, it seems the technical contribution of the paper is limited. The first step is not far from merely training a classifier to predict sensitive attributes, which is usually treated as a baseline in this area, with a little enhancement of student-teacher transfer learning. Overall, I do not see significant technical novelty. The second step is just to filter by thresholding prediction confidence. I have a hard time finding the technical contributions of the paper.

Author response:
We apologize for the confusion. Our contribution is indeed more than just training a proxy classifier with transfer learning and thresholding. Our proxy classifier is trained with uncertainty awareness and not transfer learning. The threshold is later applied on the uncertainty of predictions and not on the predictions themselves. The teacher network has the same architecture as the student network and the teacher’s weights are updated using the moving average of the student’s weights. We use Monte-Carlo Dropout over the teacher for uncertainty estimation and the consistency loss to enforce the classifier to focus on samples with low uncertainty. 

To the best of our knowledge, our proposed method is the first to draw connections between the uncertainty in sensitive attribute predictions and fairness. We have highlighted our contribution in the revised version of the draft.

Reviewer comment (Novelty):
Can authors clarify if there is anything I misunderstood about the technical contribution? Note that there is no point in merely repeating the details of the method. The constructive communication would be to point out if I am wrong when I say the method is just training a proxy classifier with transfer learning and thresholding predictions.

Author response:
We apologize for the confusion. Our contribution is indeed more than just training a proxy classifier with transfer learning and thresholding. Our proxy classifier is trained with uncertainty awareness and not transfer learning. The threshold is later applied on the uncertainty of predictions and not on the predictions themselves. The teacher network has the same architecture as the student network and the teacher’s weights are updated using the moving average of the student’s weights. We use Monte-Carlo Dropout over the teacher for uncertainty estimation and the consistency loss to enforce the classifier to focus on samples with low uncertainty. 

To the best of our knowledge, our proposed method is the first to draw connections between the uncertainty in sensitive attribute predictions and fairness. We have highlighted our contribution in the revised version of the draft.

Reviewer comment (Experiments):
Can authors explain the reason why no comparison to any of the methods in the literature of fairness without full access to sensitive attributes?

Author response:
We have indeed performed comparisons with six other baselines. In the experiment section, we mentioned the comparison with other baselines addressing fairness issues in a similar setup and we referred the reader to the appendix for results. We performed comparisons with the following baselines:

[1] Yan, S., Kao, H. T., & Ferrara, E. (2020, October). Fair class balancing: Enhancing model fairness without observing sensitive attributes. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management

[2] Zhao, T., Dai, E., Shu, K., & Wang, S. (2022, February). Towards fair classifiers without sensitive attributes: Exploring biases in related features. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining

[3] Liang, Y., Chen, C., Tian, T., & Shu, K. (2023). Fair classification via domain adaptation: A dual adversarial learning approach. Frontiers in Big Data, 5, 129.

[4] Lahoti, P., Beutel, A., Chen, J., Lee, K., Prost, F., Thain, N., ... & Chi, E. (2020). Fairness without demographics through adversarially reweighted learning. Advances in neural information processing systems

[5] Hashimoto, T., Srivastava, M., Namkoong, H., & Liang, P. (2018, July). Fairness without demographics in repeated loss minimization. In International Conference on Machine Learning

[6] Levy, D., Carmon, Y., Duchi, J. C., & Sidford, A. (2020). Large-scale methods for distributionally robust optimization. Advances in Neural Information Processing Systems

These comparisons are presented in the appendix due to the page limit and the main objective of the proposed method, which is to demonstrate the utility of uncertainty awareness in the proxy classifier in improving fairness-accuracy tradeoffs in the downstream classifiers. The results presented in the appendix (Tables 3-6) also show the effectiveness of the method in providing better fairness-accuracy tradeoffs compared to the considered state-of-the-art baselines. Moreover, our evaluations are performed over a testing set with the true sensitive attributes, i.e., we report the true fairness violation. We do not aim to improve bias estimation when proxy attributes are used which is why some of the baselines [2] [3] [4]  proposed by the reviewer were not considered for comparison.