Paper: On the Fairness ROAD: Robust Optimization for Adversarial Debiasing (ICLR-2024)

Reviewer comment (Theory):
The motivation of this paper is that global fairness methods usually cannot guarantee a consistent fairness value on split subregions, shown as Fig. 1. However, such observations are based on fine-grained partition (e.g., age categories) of training data and each subregion might be with a relatively small size. In this case, the disparity tends to happen. For example, DI might be very large if a bin only includes two samples. How do we confirm that biases do widely exist and then precisely define “local fairness”?

Author response:
While our definition of local fairness is given in Eq.(3) as means of expectations for a specific family of $q$ distributions, we acknowledge that measuring fairness from finite datasets may face difficulties. Given a considered partition of $X$, the presence of small subgroups can easily result in large DI values, and hence, it is essential to define a threshold to properly demonstrate the existence of the local fairness problem. This threshold is to be defined depending on the use case, and represents the minimum group size for a measure of local fairness to be deemed significant.

In our work, we do use such thresholds when measuring local fairness: we highlight the existence of the problem (Figure 1) by considering a minimum of 50 individuals in each represented age category (Adult and COMPAS). This threshold allowed us to observe significant disparity between sensitive groups, and conclude on the existence of the local unfairness problem. The local fairness results presented in Figure 2 and Figure 5 follow the same methodology: subgroups of minimum 50 individuals for the COMPAS dataset, and 20 for the Law and German datasets (details for all subgroups are given in Appendix A.5.3).

Besides, to make sure that the problem of local fairness we observe is not an artifact of overfitting a threshold, we have conducted multiple experiments to test the robustness of our observations: in Figure 5, Figure 9, and Figure 10, we test the behavior of models trained once when this threshold varies. These experiments show that no matter the value of the threshold chosen (among a reasonable range), local bias problems could be observed. 

Finally, we would like to restate that these thresholds are only needed at evaluation time, to measure local fairness. During the training, no threshold is needed, as the locality of the attention is controlled by the temperature $\tau$ hyperparameter.

Reviewer comment (Reproducibility):
Starting from Eq. (2), the input of $g_{w_g}(.)$ directly takes the output of $f_{w_f}(.)$. So, how do you implement $w_g$? Considering a binary case, $f_{w_f}(.)$ is only a probability in output space.

Author response:
As noted by the reviewer, the sensitive adversarial $g_{w_g}(.)$  takes only one variable as input, which is the current output predictions $f_{w_f}(x)$. To reconfirm, it is important to note that our approach utilizes the (continuous) probability output $f_{w_f}(.)$, NOT the predicted label (after threshold) $\hat{f}$. This is a common practice in adversarial fairness reconstruction, as demonstrated in works such as [Zhang et al., 2018, Adel et al. 2019], etc. and helps ensuring differentiability of the objective.  As discussed in the second paragraph of appendix A.2.2, the adversary is then considered optimal for Demographic Parity when it accurately outputs $\mathbb{E}[S|f(X)]$ for any input. A classifier is deemed fully fair when $\mathbb{E}[S|f(X)] = P(S=1|f(X)) = P(S=1)$. In this case, we have $P(S=1, f(X)) = P(S=1)P(f(X))$, which indicates independence and, consequently, $P(\hat{f}(X)|S) = P(\hat{f}(X))$.

Please note that while this is only one input, state-of-the-art (SOTA) methods have shown the necessity to segment this input in several ways to better capture information relevant to predicting the sensitive attribute[1]. 

Furthermore to obtain better results, as commonly done in the literature, we perform several training iterations of the adversarial networks for each prediction iteration. This approach results in more robust adversarial algorithms, preventing the predictor classifier from dominating the adversary (more details can be found in appendix in section A.9.).

[1] Fair adversarial gradient tree boosting. 2019 IEEE International Conference on Data Mining (ICDM). IEEE, 2019.

Reviewer comment (Theory):
Following question 3, please check the constraint of eq. (4), where s should be properly positioned.

Author response:
Thank you for your comment. We acknowledge that there was indeed an issue in Eq. (4), as a bracket was missing. To clarify, we have modified the equation from $\mathcal{L_S}(g_{w_g}(f_{w_f}(x), s))$ to $\mathcal{L_S}(g_{w_g}(f_{w_f}(x)), s)$, ensuring that s is correctly positioned, as it is in the traditional formulation by Zhang et al., 2018. We have updated the paper accordingly.

Reviewer comment (Evaluation):
Since $\lambda_g$ is not learnable (Sec A.5.4), are the best results selected to have a Pareto dominance as claimed?

Author response:
In response to the concern regarding the selection of the best results for Pareto dominance in the presence of a non-learnable λ_g parameter, we would like to emphasize the following points:


1) It is common in algorithmic fairness literature to have a non-learnable λ_g parameter, which allows for the tradeoff between fairness and accuracy. The Pareto dominance is frequently used to demonstrate this trade-off, as shown in Wang et al. TMLR 2023 [1], Ferry et al. 2021 [2], and Chai et al. [3]. This approach enables the examination of a wide range of fairness and accuracy levels.
2) In our experiments, we examined an extensive range of λ values, including large ones (e.g., λ=5), which generally represent the upper limit for the algorithm without encountering empirical degeneration. Reported results correspond to non-dominated solutions w.r.t. both criteria. 

[1] Wang, Haotao, et al. “How Robust is Your Fairness? Evaluating and Sustaining Fairness under Unseen Distribution Shifts.” Transactions on machine learning research 2023 (2023).
 
[2] Ferry, Julien, et al. “Improving fairness generalization through a sample-robust optimization method.” Machine Learning 112.6 (2023): 2131-2192. 

[3] Chai, Junyi, and Xiaoqian Wang. “Fairness with adaptive weights.” International Conference on Machine Learning. PMLR, 2022.

Reviewer comment (Novelty):
Please justify the novelty/contribution compared to Michel et al 2022.

Author response:
In response to the reviewer’s question regarding the novelty and contributions of our work compared to [Michel et al. 2022], we would like to highlight the following key differences:

**Different problem formulation**: Our approach diverges significantly from [Michel et al. 2022] in terms of the problem addressed. While [Michel et al. 2022] focus on parametric DRO applied to the training objective $L_Y$ without considering any fairness criterion (i.e., no sensitive attribute $S$ is used), our work aims to optimize fairness criteria, such as Demographic Parity. This distinction in objectives is crucial, as optimizing fairness criteria is not achievable through the methods proposed in [Michel et al. 2022]. This distinction is now emphasized in section 2.2, with DRO methods for fairness without demographics, following your suggestions in question 2 above.

**Different optimization**: The traditional DRO pursued by [Michel et al. 2022] aims to minimize the maximum training loss, which contrasts with our opposite goal of maximizing the minimum possible adversarial loss. In our approach, we assign more weight to samples that are easier for the adversary to reconstruct $S$  instead of focusing on less classifier accuracy as in [Michel et al. 2022]. Additionally, our optimization involves three neural networks, as opposed to the two in [Michel et al. 2022], and the validity constraint differs between the two works. We want to emphasize that our technique employs a multi-objective optimization approach that balances fairness and accuracy, while [Michel et al. 2022] focus primarily on optimizing accuracy.

Furthermore,  to address your concerns, we have compared our method to [Michel et al. 2022]. In Figure 11 (Appendix A.5: “Additional results on the difference with traditional DRO,”), is shown the Pareto front for the experiment on Local Fairness for the German dataset. Our approach consistently achieves better results in the Pareto front, demonstrating improved performance in terms of both fairness and accuracy compared to existing methods.

Reviewer comment (Novelty):
The reference to applied fairness research could be strengthened. There are many related works that are omitted. In particular, work on fairness and adversarial learning.

Author response:
We appreciate the suggestion to strengthen the references to applied fairness research and related works. We have included the following references in the manuscript: 
Applied fairness:

[5] Dissecting racial bias in an algorithm used to manage the health of populations, Obermeyer 2019

[6] Achieving Fairness through Adversarial Learning: An Application to Recidivism Prediction, Wadsworth et al. 2018

[7] A Fair Pricing Model via Adversarial Learning, Grari et al. 2022


Some missing related works on adversarial learning for fairness:

[8] Achieving Fairness through Adversarial Learning: An Application to Recidivism Prediction, Wadsworth et al. 2018

[9] Learning to Pivot with Adversarial Networks, Louppe et al. 2017

Reviewer comment (Writing):
Is there a reason why the first paragraph of the introduction omits citations in statements like "models...have been shown to unintentionally perpetuate existing biases and discriminations"?

Author response:
Thank you for your comment. In our revised manuscript, we have included more relevant literature to better situate our work within the broader context of fairness research. These include for instance:

[10] Machine Bias, Angwin et al. 2016 (Propublica Compas study)

[11] Weapons of Math Destruction, Cathy O’Neil, 2016

Reviewer comment (Theory):
The problem formulation seems *overly complex*, with multiple levels of nested minimization problems. It is unclear if the complexity is absolutely necessary (see question 1).

Author response:
> Differentiability of (3): I understand that formulation (4) allows for differentiability of the objective. Couldn't the same be achieved by introducing a smooth surrogate objective for the non-differentiable indicators? Much of the initial works on group fairness employ some form of surrogate approximations to handle non-differentiable constraints (e.g. Zafar et al. (2015), Cotter et al. (2019)).


We acknowledge that using other formulations, such as the ones suggested by the reviewer, represents an interesting idea for future works. We have chosen the adversarial approach, inspired from [Zhang et al.], since it has generally been shown to outperform empirically other methods, including some of the ones mentioned by the reviewers (see for instance [1, 2] for extensive discussions and experiments on this topic). However, we also believe that  some other directions of extension of our work such as using a diferentiable Mutual Information metric [3] could be interesting to explore with a robust fair ratio. We have added a new comment on this topic in the conclusion. 



[1]  Adversarial mitigation to reduce unwanted biases in machine learning [Grari 2022]

[2] One-Network Adversarial Fairness [Adel et al. 2019]

[3] Learning Unbiased Representations via Mutual Information Backpropagation [Ragonesi et al. 2020]

Reviewer comment (Theory):
Justifications needed for choice of uncertainty set $\mathcal{Q}$.

Author response:
> (2) Assumptions on $q$: Could the authors formally state the assumptions on the class of perturbed distributions $q$, preferably early on in the paper (e.g. Sec 2). 

In section 2, we give an overview of the problem, by giving a general formulation (eq 3) and then reviewing some related works that attempt to address fairness with DRO approaches. In section 2.3, we review different choices for the considered uncertainty set ${\cal Q}$, which has a strong impact on the behavior of the methods in practice. We indeed see from eq (3) that if ${\cal Q}$ is too large or far from $p_{test}$ (the target distribution for the classifier), the process might lead too overconstrain  the prediction, with the consideration of fairness for irealistic groups, which results in poor accuracy for the target classifier. Thus, for distribution shifts purposes followed by related works, the main requirement for ${\cal Q}$ is that it contains $p_{test}$, and is tight around it. Assuming that $p_{test}$ is close to $p$, this is done for instance by a KL constraint w.r.t. $p$.   

For our local fairness objective, we also want that the discrepancies of $q$ w.r.t. $p$ are smooth in the feature space, so that the fairness constraint does not increase mitigation on specific disconnected individuals, but rather on local areas of the space, which would be somehow ignored in classical global fairness approaches. 

Following the reviewer's recommendation, we added this last sentence in section 2.3 to strengthen the definition of our requirements regarding ${\cal Q}$. However, from our point of view, discussions about our validity constraints and the implied assumptions can difficultly be discussed here, for the reasons detailed below.

Reviewer comment (Theory):
**Assumptions on $q$**: Could the authors formally state the assumptions on the class of perturbed distributions $q$, preferably early on in the paper (e.g. Sec 2). My understanding is that the group priors are required to be the same for both q and p, i.e. $q(s) = p(s)$. However, it is unclear if the validity constraint described in Sec 3.1 would include **all distributions $q$ with the same group priors as $p$**. It might be cleaner to have the assumptions on $q$ described first, and then argue why the particular choice of weighting functions $r$ satisfy those assumptions.

Author response:
> However, its unclear if the validity constraint described in Sec 3.1 would include all distributions $q$  for with the same group priors as $p$. 

In appendix 2.1, we show that the validity constraints that we consider actually ensure that $r$ implicitly defines a valid $q$ distribution, with equal priors $p(s)=q(s)$. 

For completeness, we could indeed show the reciprocal, that states that setting $r$ such that $p(s)=q(s)$ implies the considered constraints on $r$,  which would indicate that these "validity constraints include all distributions $q$ for with the same group priors as $p$". 

To do so, let us consider that $\forall s, p(s)=q(s)$. 
Thus, we have $p(s)=\int q(x,s) dx=\int p(x|s) \frac{q(x,s)}{p(x|s)} dx$. 
It directly follows that $\forall s, \int p(x|s) \frac{q(x,s)}{p(x|s)p(s)} dx = \mathbb{E}_{p(x|s)}[r(x,s)]=1$.   

Considering our validity constraints on $r$ is thus equivalent to set $q(s)=p(s)$ for every $s$ if dealing with an explicit distribution $q$. We added this in the end of appendix 2.1.

Reviewer comment (Theory):
**Rationale for assuming equal group priors:** Does the assumption $q(s) = p(s), \forall s$ amount to saying e.g. that the proportion of male and female samples would be the same across all sub-regions (e.g. across all age groups)? If so, is that a reasonable assumption to make? I think the authors need to better explain how their choice of uncertainty set aligns with practical applications where one may want to enforce the form of local fairness they describe.

Author response:
> (3) Rationale for assuming equal group priors: Does the assumption $q(s) = p(s), \forall s$ amount to saying  e.g. that the proportion of male and female samples would be the same across all sub-regions (e.g. across all age groups)? If so, is that a reasonable assumption to make? I think the authors need to better explain how their choice of uncertainty set aligns with practical applications where one may want to enforce the form of local fairness they describe.

We respectfully want to point that there may be a misunderstanding on that point. This constraint does not impose anything on the subgroups, as $q$ is technically not directly the subgroups but rather the (weighted) global population ($q=rp$). The behavior suggested by the reviewer, i.e. “imposing equal proportion of male-female in all subregions”, which is, as pointed out, totally undesirable, might have been achieved by imposing a constraint such as: $p(s|x) = q(s|x)$. Again, this is not desirable in the context of this paper, and therefore was not considered.

Reviewer comment (Experiments):
**Distribution shift in Sec 4.2:** In the experiments with the Adult/census distribution shift in Sec 4.2, do we have reasons to believe that the uncertainty set used would capture the "real-world" distribution shift considered? I am again particularly curious about the relevance of the assumption $q(s) = p(s)$ here.

Author response:
> (4) Distribution shift in Sec 4.2: In the experiments with the Adult/census distribution shift in Sec 4.2, do we have reasons to believe that uncertainty set used would capture the "real-world" distribution shift considered? I am again particularly curious about the relevance of the assumption $p(s)=q(s)$ here.

Following our previous answers, we are interested in a generalization to a local proportion shift rather than a global sensitive shift. As explained previously, although we impose $q(s)=p(s)$ variations on $q(s|x)$ are still allowed. Therefore, the question of whether the uncertainty set captures these distribution shifts depends on the KL constraint rather than on the validity one.  
We hope these answers help clarify the need and benefits of our validity constraint.

Reviewer comment (Experiments):
In fig 3 and 4, some of the baselines do not span all ranges of fairness. Why do we see this behavior? Are the baselines tuned correctly?

Author response:
We assume that by “fig 3 and 4”, the reviewer instead meant fig 2 and 3. 
In these figures, some curves indeed do not span all ranges of fairness. This is due to the fact that some methods do not offer sufficient control, depsite an extensive sweeping of their hyper-parameter domains via greed search,  that would enable to reach points in some areas of the support. For fig 2, this is accentuated by a filter that removes points that do not respect a minimal threshold of global fairness (as we are interested in inspecting local fairness for globally fair models). For fig 3, this is emphasized by distribution drifts: while for in-distribution test set (in the leftmost curves), every method allows a wide span of the global fairness support, results for other plots are impacted by drifts that make pareto fronts reduced on few points dominating every others for some methods.

Reviewer comment (Evaluation):
How does DRO affect global fairness compared to other baselines? Fig 3 only reports local fairness in the form of the worst DI. However, the global fairness results are not reported.

Author response:
In fact, that figure reports results of local fairness for a high requirement for global fairness (global DI < 0.05). We see that our approach achieves this global fairness level (while some approaches such as fairLR on COMPAS have no point in the plot since never satisfying this condition), with competitive levels of accuracy for any local fairness score.

Reviewer comment (Evaluation):
While extending this framework to EO measure is straightforward, results with EO and local fairness are not reported. Similarly, DI and distribution shift results would strengthen confidence in the method.

Author response:
You are right. We launched experiments to complement this. We have added a new section in appendix "A.6 Additional results on the Local Equalized-Odds criterion" (page 20). You may find these results in Figure 12 in the current version of the paper. The local unfairness is here measured using worst Disparate Mistreatment Rate (worst-1-DMR), that we defined by adding worst-1-FPR and worst-1-FNR.

Reviewer comment (Experiments):
It is quite unintuitive that BROAD, an exact solution, performs poorly than ROAD, which uses a neural network to predict weights. The authors argued that ROAD is better because of the smoothing effects. Could it be that what is easy and hard for adversaries may change rapidly (or adversaries' output may be noisy) due to noisy batch gradients, and using an NN in ROAD may just be smoothing that out? In that case, would it make sense to use an EMA-like estimate with BROAD?

Author response:
Thank you for this insightful question regarding the difference in performance between ROAD and BROAD. 

The problems that may hinder BROAD performance go beyond smoothing. In the DRO literature, it is known, and deeply discussed (see [1, 2, 3, 4] for instance), that the non-parametric solution (such as BROAD) may lead to suboptimal performance compared to parametric ones. This may come from several causes, such as the fact it is essentially solving a problem that is fundamentally too pessimistic: i.e., the uncertainty set $\mathcal{Q}$ may include distributions that are too difficult to learn, and not necessarily representative of real-world constraints. Thanks for the suggestion of using an EMA-like estimate with BROAD, which could alleviate learning unstability issues, but might still be limited by the use of individual weights: in BROAD, weights of training samples are only interlinked via the outputs from the classifier, hence at the risk of conflicting with our notion of local fairness (which is defined on the feature space). 

On the other hand, the parametric, neural network-based, approach (such as ROAD), “has much less (only parametric) freedom to shift the test distribution compared to the adversary that uses non-parametric weight” (sic) [1]. The lipschitzness of neural networks can add additional implicit locality smoothness assumptions, thus helping define distributions $q$ as subregions of the feature space. In classical DRO, this leads to models that are robust to more meaningful distribution shift. In our setting of DRO for local fairness, this also helps focusing on local fairness with groups formed with similar individuals. As discussed in our paper, the network architecture defines the level of local smoothness of considered groups. And a network of infinite capacity that completes training would have, in theory, the same behavior as BROAD. 

[1] Hu, Weihua, et al. “Does distributionally robust supervised learning give robust classifiers?.” International Conference on Machine Learning. PMLR, 2018.
[2] Duchi et al. 2020 Distributionally Robust Losses Against Mixture Covariate Shifts. ROAD is a heuristic approximation to the objective

Reviewer comment (Presentation):
Fig 4 (Left Fig): Are these unnormalized values? Isn't weight, i.e., $r$, positive and less than 1?

Author response:
Normalization is on the mean, not the sum: we impose $\mathbb{E}_p[r(x,s)]=1$ to ensure that $q$ is a distribution that respects $\int q(x,s)=1$. As we implement this constraint with an unbiased estimator based on training samples (i.e., $\frac{1}{n} \sum r(x_i,s_i)=1$), we get that $r$ values sum to $n$, not $1$. This can be seen for instance in our implementation of BROAD where the partition function is divided by $n$.

Reviewer comment (Reproducibility):
If we use BROAD for EO, it will use the adversary's loss, which uses labels $y_i$. Thus $r_i$ is a function of $x_i, s_i$ and $y_i$. Should the weight network (in ROAD) also use $y_i$ to predict r$?

Author response:
Yes, we agree with the reviewer and confirm that in order to optimize for EO, $r$ needs to be a function of $x$, $s$ and $y$; and that the weight network uses $y$ to predict $r$. We have provided more details about this in the appendix of the paper (please see A.10: "ADAPTING ROAD TO EQUALIZED ODDS").

Reviewer comment (Theory):
Would the validity constraint change for EO? That is, would it require normalization over both $s_i$ and $y_i$ or only $s_i$ as explained in the paper?

Author response:
Thank you for your question. Following similar reasoning as for our validity constraint in the case of DP (section A2.2), we have indeed also considered adapting the validity constraint for EO by conditioning it on $y$. This effectively means imposing the following constraints:
$\forall s_i, \forall y_i, \mathbb{E}_{x | s=s_i, y=y_i}r(x,s,y)=1$
However, depending on the dataset, conditioning the batches on both $s$ and $y$ may lead to very small subsamples,  i.e. non-significant expectation estimates. As a result, the obtained models were sometimes less robust, and this lead us to not pursuing this direction any further. We have added a discussion on this in Section A.7 with an experiment comparing the two constraints (Figure 13).