Paper: Fair Classifiers Without Fair Training: An Influence-Guided Data Sampling Approach (ICLR-2024)

Reviewer comment (Novelty):
Implementing distribution shift as a way to improve generalization is already known in studies of adversarial training and reliable deep learning, but the paper does not engage with this existing literature.

Author response:
We recognize the parallels between our approach and the existing studies in adversarial training and reliable deep learning where distribution shifts are used to enhance generalization. However, this paper specifically targets using distribution shifts to mitigate fairness disparities without harming the generalization. Then, adversarial training is primarily concerned with improving a model's robustness against adversarial attacks. The focus is on ensuring model stability and performance in the face of intentionally crafted input perturbations. Besides, reliable deep learning focuses on ensuring that these models perform consistently under varying conditions and are resilient to errors, noise, and adversarial attacks. Therefore, this work is distinct in that it specifically aims to tackle fairness issues using distribution shifts, a focus not traditionally central to adversarial training or the broader scope of reliable deep learning.

Reviewer comment (Writing):
Exposition needs improvement throughout for clarity. For instance, the paper states "Our theoretical analysis indicates that training on datasets with a strategically implemented distribution shift can effectively reduce both the upper bound for fairness disparity and model generalization error (Lemma 3.1, Theorem 3.2). This gives us a key insight that fairness and accuracy can be improved simultaneously even with simply traditional training. [Section 3]" How is training on datasets with a strategic distribution shift simple traditional training? And for the second bullet in contributions, the paper states "we sample influential examples" but does not clarify what influence function is used throughout the paper. I see section 4.1.2 but it is not clear to me how this is distinct from the previous definitions in the literature.

Author response:
Our training method is akin to conventional training, involving two stages: a warmup process and an incremental process. Initially, we developed a standard machine learning model without any modifications. In the incremental stage, we implement strategic distribution shifts by assessing the influence of samples on prediction and fairness. Samples that exert a negative influence on both prediction and fairness loss — indicating an enhancement in both fairness and prediction — are then solicited for further training.
Apologies for any confusion regarding the term “influential”. It's analogous to the influence function concept, where we select unlabeled samples based on their influence on prediction (Lemma 4.1) and fairness (Lemma 4.2), hence the term 'influential examples'. More specifically, in Line 6 of the FIS Algorithm, we select those unlabeled samples with negative influences on prediction and fairness. By doing so, we aim to reduce fairness disparity without compromising accuracy.
Our use of "influential" differs from previous uses in literature, which often focus on identifying misclassified examples from training sets by using influence functions [1,2]. In contrast, we only assess the influence of prediction and fairness on the validation sets $Q_v$ for unlabeled samples by applying first-order gradient estimation, following the traditional training process.

Reviewer comment (Novelty):
The paper over-emphasizes that sensitive attributes are not used during training, which is pretty standard practice in fair classification. This is perhaps relevant for post-processing techniques, but this particular paper is presenting a technique for pre-processing/during training adjustments. Further, even in work on leave-one-out fairness or influence functions and fairness (analysis of perturbing training data in some way), sensitive attributes are not typically used in training. These are also areas of related work that the paper does not engage with.

Author response:
We respectfully disagree with the comment that our paper places undue emphasis on a particular aspect. Previous research often presupposes the availability of demographic details like gender and race, known as sensitive attributes, for training fair models, particularly in in-processing methods. 

When sensitive attributes are missing, some works tend to find some proxy models to generate proxy attributes [3]. However, in our method, we need neither true sensitive attributes nor their proxies in the training. There are some others mentioned by **Reviewer B8Mi** and **Reviewer wM5N**, including DRO to improve fairness without sensitive attributes.  Although they also evaluate the worst-group performance in the context of fairness, their approach differs as they do not strive to equalize the loss across all groups.
In our revised manuscript, we intend to elaborate on other studies that conduct training without using sensitive attributes. Additionally, we respectfully appreciate any relevant references you may provide.

Reviewer comment (Novelty):
What is the novelty in section 4? Which parts are restating results from the literature and what parts are new?

Author response:
Motivated by the key observation of theoretical findings, in section 4, we propose an influence-guided sampling strategy that actively samples new data to supplement the training set such as mitigating fairness disparity. Besides, we are confident that the overall content in Section 4 is new. In subsection 4.1.1, we restate the foundational scenarios and further introduce the extra validation set $Q_v$ and the unlabeled set U. Following this, subsection 4.1.2 delves into the methodologies used to determine the influence on predictions and the fairness component. These concepts are integral to the FIS algorithm we describe in Section 4.2.

Reviewer comment (Novelty):
What is the motivation for employing the distribution shift for this reason? Why is a particular type of shift used? And how is this different from adversarial or influence function informed training?

Author response:
The use of distribution shift is inspired by theoretical insights from Lemma 3.1 and Theorem 3.2. These findings shed light on the reasons behind the tradeoff between accuracy and fairness, guiding us to manage this tradeoff by controlling the distribution shift's negative impact on accuracy via generalization error. Without specific information on the train/test distribution, determining the precise distribution shift required is challenging. However, our primary goal is to leverage this distribution shift to minimize the group gap without harming model accuracy.

Our approach differs in its fundamental focus and methodology. While adversarial training is geared towards enhancing a model's robustness against intentionally crafted perturbations, and influence function-informed training aims to understand and mitigate the impact of specific training examples, our method primarily addresses fairness against potential data shifts by adding more examples. We focus on generalization across varying data distributions rather than explicitly countering adversarial examples or pinpointing influential data points.

Reviewer comment (Novelty):
The related works section lacks discussion about fair active learning frameworks like [6]. In addition, the authors should distinguish the proposed framework from existing fair active learning frameworks and perform an empirical comparison.

Author response:
While fair active learning frameworks may seem similar to our approach, there are crucial distinctions. For instance, work [6] focuses on sampling additional unlabeled data points based on a multi-objective optimization for misclassification error (Shannon entropy) and fairness regularization (demographics parity). In contrast, [6] still requires knowledge of the sensitive attributes of unlabeled samples, while our method does not necessitate such information. Practically, without this information, it's difficult to assess fairness disparity (e.g., using demographics parity) and determine if a model is fair. To overcome this, we resort to a validation set that includes sensitive attributes, providing a reliable measure of fairness disparity.

Reviewer comment (Experiments):
The paper is missing a comparison with in-processing fairness training algorithms, such as MinDiff [7]. Such a comparison would provide a clearer perspective on any accuracy-fairness trade-offs advantages that FIS may entail.

Author response:
Those in-processing fairness training algorithms typically require sensitive attribute information to develop regularization terms. For instance, MinDiff [7] employs various regularization methods to reduce bias so as that obtain an accuracy-fairness tradeoff. Consequently, comparing our approach with in-processing algorithms is unnecessary and not fair, as these algorithms are not applicable in scenarios where sensitive attribute information in the training data is unavailable.

Reviewer comment (Experiments):
To gain a comprehensive understanding into the effectiveness of the proposed algorithm, it is imperative to conduct a sensitivity analysis that explores the relationship between the labeling budget and the (accuracy, fairness).

Author response:
Thank you for your patience. We have completed the additional experiments as previously discussed, examining how the labeling budget impacts both accuracy and fairness. We present the results of test accuracy and fairness
disparity across different label budgets on the CelebA, Compas, and Jigsaw datasets. In these experiments, we
use the demographics parity (DP) as our fairness metric. For convenience, we maintain a fixed label budget
per round, using rounds of label budget allocation to demonstrate its impact.  The designated label budgets
per round for the CelebA, Compas, and Jigsaw are 256, 128, and 512, respectively. In the following figures,
the x-axis is both the number of label budget rounds. The y-axis for the left and right sub-figures are test
accuracy and DP gap, respectively. The corresponding results have been shown in Figures 1–3. We can observe that, compared to the BALD baseline, our approach substantially reduces the DP gap without sacrificing test accuracy.

Reviewer comment (Theory):
Weak connection between the theorem and algorithm. I understand the theorem served as a very high-level motivation to change the distribution shift, but from my perspective, it has limited connections to the specific algorithm later on. Also the theorem is closely related to the theorems in domain adaptation established several years ago, so that somehow to be incremental.

Author response:
Due to the lack of specific information about the train/test distribution and sensitive attributes in our scenario, crafting a perfectly tailored algorithm that directly links to our theorems to control distribution shifts is challenging. To address this, we've introduced the heuristic sampling method FIS, focusing on utilizing gradient descent to effectuate an appropriate distribution shift. While our work aligns with certain domain adaptation theories, it primarily investigates the underlying causes of the accuracy-fairness tradeoff, extending beyond just domain adaptation. By controlling the adverse effects of distribution shifts, our goal is to strike a more favorable tradeoff between accuracy and fairness.

Reviewer comment (Experiments):
Since the influence is based on first-order estimation, if there any chance to validate the estimation of influence per sample? Maybe show the actual influence and its estimation would be helpful.

Author response:
The choice of first-order estimation, rather than considering the second-order information, is due to the nature of training deep neural networks with non-convex objectives subject to fairness constraints.  In such scenarios, the model often does not converge to a local minimum where the first-order gradient can be treated as zero, thus the first-order term will dominate over the second-order term. In recent literature [R1], the authors have shown that the first-order estimation per sample is nearly accurate under fairness constraints. 
In the uploaded revision, we added Figure 5 to compare the estimated influence with the actual influence. We observe that while some of the examples are away from the diagonal line (which indicates the estimation is inaccurate), the estimated influence for most of the data samples are very close to their actual influence values.

Reviewer comment (Novelty):
The paper needs to clarify the connection between this paper and other related works in the fairness literature. For example, one of the important discussions in the paper is about the accuracy-fairness tradeoff, which also affects the main proposed algorithm. However, such a tradeoff between accuracy and fairness has been widely studied (e.g., [1]), so it would be better if the paper could clarify what is the difference between this paper’s analysis and previous discussions in the fairness literature.

Author response:
**Response to W1-1**: We would like to clarify that we do not work in the classical regime of fairness-accuracy tradeoff. By properly collecting new data from an unlabeled dataset, we can improve both generalization and fairness at the same time which can not be achieved by working on a fixed and static training dataset that naturally incurs such tradeoff. 
Another notable distinction between our work and previous studies on the accuracy-fairness tradeoff lies in the handling of sensitive attributes of training data. For instance, [1] achieves this tradeoff using fairness regularization (in-processing techniques), which necessitates sensitive attribute information. In contrast, our paper's key contribution is the elimination of this information, thus offering a more practical and applicable framework in scenarios where such information is unavailable.

Reviewer comment (Novelty):
The paper uses the concept of distribution shifts, which is recently extensively studied in the fairness literature, but the paper does not discuss those works (e.g., [2, 3, 4]). It seems the setups of this paper and recent distribution shifts studies in fairness literature are a bit different, as this paper aims to ‘utilize’ an appropriate distribution shift for sampling, while many recent studies focus on ‘solving’ the distribution shifts between training and target distributions for fair training. Thus, it would be much better if the paper could clarify the connection and differences between this work and other distribution shifts studies.

Author response:
**Response to W1-2**: Thanks for the suggestions. Overall, our approach significantly differs from other studies that concentrate on fairness disparity arising from train-test distribution shifts. Commonly, such research necessitates extra assumptions to build theoretical connections between features and attributes, like causal graphs [2], correlation shifts [3], and demographic shifts [4]. In contrast, our approach refrains from making further assumptions about the characteristics of distribution shifts. Instead, we directly utilize sampling methods to construct an appropriate distribution shift, aiming to achieve a balanced tradeoff between accuracy and fairness.
More specifically, work [2] uses complex causal graphs to hypothesize about distribution shifts between training and testing datasets. Following this, Then, [3] introduces and analyzes the correlation shifts between train and test distribution, serving to boost existing fair in-processing methods. However, to our knowledge, fair in-processing methods necessarily require information on sensitive attributes during training, whereas our approach does not require this information. [4] control unfairness of a model by making the demographic shifts assumptions, where the marginal distribution of the data remains the same conditional on the subgroup but the subgroup distribution can change.

Reviewer comment (Experiments):
The paper does not compare with enough baselines in their experiments. For example, there are various algorithms for training fair models without using sensitive attribute information (e.g., [5, 6]), but the paper does not include clear comparisons with those works.

Author response:
**Response to W2**:   [5] proposes an approach based on distributionally robust optimization (DRO), which minimizes the worst-case risk over all distributions close to the empirical distribution. Then, [6] also proposes an optimization approach, Adversarially Reweighted Learning, to improve worst-case performance over unobserved protected groups, indicating the main goal is to improve the worst-group loss. Although these two works [5, 6] also evaluate the worst-group performance in the context of fairness, their approach differs as they do not strive to equalize the loss across all groups. Besides, in these studies, accuracy and worst-case accuracy are used to showcase the efficacy of DRO. Essentially, they equate fairness with uniform accuracy across groups, implying a model is considered fair if it demonstrates equal accuracies for all groups. However, this specific definition of fairness is somewhat restrictive and does not align with more conventional fairness concepts like Demographic Parity (DP) or Equalized Odds (EOD).
In response to your concern, we are considering using one of DROs (e.g., [5]) as a baseline for our ongoing experiments. These additional results will be added to the revision before the rebuttal deadline.

Thank you for your patience. In response to your concern, we have added a baseline JTT-20 [R1] in our revised manuscript. This baseline specifically reweights misclassified examples, assigning them a weight of 20 during the retraining process. We have updated the results for the four datasets in the corresponding tables to reflect this addition. These updates also highlight the effectiveness of our proposed method.

[R1] Liu, Evan Z., et al. "Just train twice: Improving group robustness without training group information." International Conference on Machine Learning. PMLR, 2021.

Reviewer comment (Writing):
The definition 3.1's symbolism is not clear, are $P$ and $Q$ the same as preliminaries? Why use the model trained on P instead of that trained on Q? Could you give more explanation?

Author response:
We apologize for any raised confusion. In our work, we represent the training dataset using the notation P and the test dataset with the notation Q, without assuming specific characteristics for these sets. This implies that P and Q could either be independent and identically distributed (iid) or exhibit a distribution shift. Conventionally,  it's a standard approach to train a model on the training set P and evaluate its effectiveness on the test set Q.

Reviewer comment (Theory):
Assumption 3.2 seems a bit strong. The assumption before Lemma 3.1 that the loss is bounded is not common. Could you give more justification to these assumptions?

Author response:
We would like to clarify that the assumption of L-lipschitz smoothness (Assumption 3.1) and bounded gradients  (Assumption 3.2) of the empirical risk are common and basic in the theoretical analysis of stochastic gradient descent (SGD) and related optimization methods in machine learning [1, 2, 3]. For your reference, these two assumptions are similar to Assumption 1 and Assumption 4 presented in [1], respectively. More specifically, the L-Lipschitz property for the empirical risk is inferred from the L-Lipschitz property of the loss function because the empirical risk $R_P(w)$ is defined as the average of the loss function over the training dataset. On the other hand, gradient clipping is a widely applied trick to avoid exploding gradients in deep learning, thus implying that the bounded gradient is a decent assumption.

Reviewer comment (Theory):
In the first paragraph in Sec 4.1.1, the assumption of an auxiliary hold-out validation dataset is too strong. For my understanding, test data means that we don’t know the distribution of the data. So I am not sure the reasonability of the assumption.

Author response:
We respectfully disagree with the comment that the assumption of the validation dataset is too strong. This assumption is widely recognized and has been presented in numerous existing works [4-8]. In our setting, the distribution of the data remains unknown to us because merely accessing a small validation dataset for testing purposes does not fully reveal the data distribution.
Besides, we can analyze this assumption in two scenarios: whether the train and test distributions are independent and identically distributed (iid) or not. If they are i.i.d., the validation dataset can be drawn from the training dataset. However, in cases of distribution shifts between train and test data, we cannot ascertain the influence of solicited data samples without any explicit knowledge of the test data. Therefore, we utilize this reasonable assumption to gain some insight into the test dataset or its distribution.

Reviewer comment (Evaluation):
Although it states the computation cost of the proposed algorithm is low, it seems the algorithm needs to pre-calculate the loss for testing the performance of a sample, which is costly.

Author response:
We would like to clarify that the extra computation cost is comparable to the cost of traditional model training. The main extra computation cost in FIS (Algorithm 1) is the cost of model gradients. Let $p$ denote the number of model parameters, then the cost for computing the gradients is $O(p)$ per sample. Specifically, in each round that we need sampling, we need to calculate three parts of gradients: the gradients of $N_U$ unlabeled instances, the average gradient of $N_v$ validation instances wrt accuracy loss, and the gradient of $N_v$ validation instances wrt fairness loss. Note in general, $N_v \ll N_U$. In practical implementation, to speed up the calculation of gradients over $N_U$ instances, we randomly sample 0.2～0.5 percent of the unlabeled dataset in each sampling batch. Additionally, we can increase $r$ to save computation costs. In our experiments, we usually have 10～20 sampling rounds. On a single GPU, running one experiment for CelebA requires about 4 hours.

Reviewer comment (Experiments):
It lacks discussion of how to select the initial training dataset for the warm start (influence when applying different proportions or distributions), and how to determine the solicitation budget $r$ which is the declared a small number of sampling data to gain a better result (both accuracy and fairness).

Author response:
In our work, we do not make specific assumptions on the initial training dataset. In our experiments, we randomly select a subset of examples as the initial labeled training set. In response to your concern, we are conducting experiments to show the impact of solicitation budgets. These additional results will be added to the revision before the rebuttal deadline.

Reviewer comment (Writing):
Typo: That -> that in paragraph before Def. 3.2

Author response:
Sorry for these typos. We will carefully revise the references in the revision.

Reviewer comment (Writing):
In Eq. 1, if $f$ is a classifier, then $x$ seems to be the feature instead of original data.

Author response:
Your understanding is correct. We define $x$ as features in the first paragraph of Section 3.1(Preliminaries).