Paper: Fairness-Aware Domain Generalization under Covariate and Dependence Shifts (ICLR-2024)

Reviewer comment (Novelty):
The authors assert multiple times that they are introducing a novel form of shift. However, it's important to note that there is an extensive body of literature that has previously examined the effects of spurious correlations when the relationship between the spurious feature and the class label varies across domains. Furthermore, in the field of fairness, there has been prior research on shifts within subpopulations. Hence, the extent to which this claim holds true remains somewhat uncertain.

Author response:
Some domain generalization papers concern the shift in fairness, such as [a, b]. For example, in [a], they gave an example using the Waterbird dataset, where the backgrounds (i.e., sensitive subgroups, land and water) are spuriously correlated with class labels (waterbirds and landbirds). This spurious correlation causes sub-par performance in the smallest subgroups. However, [a] only takes the shift in fairness as domain variations. In our paper, we state the distribution shift across domains due to covariate shifts (data features) and changes in fair dependence. Therefore, the dependence shift is proposed. Since prior research, such as [a, b], has been proposed, we do not conclude the dependence shift is a major contribution to our paper.

Reviewer comment (Writing):
The authors initially discuss the challenge of maintaining fairness guarantees under distribution shift, particularly when there are varying dependencies between the sensitive feature and the class feature (a particular case of spurious correlation, when the spurious feature is the sensitive attribute). However, the example they present in the introduction to motivate the problem (Figure 2) pertains to the broader issue of spurious correlations rather than being specific to fairness concerns. In this context, they consider the background as the spurious feature, not necessarily linked to any sensitive attribute. Moreover, in the experimental section, the spurious feature is not always synonymous with a sensitive attribute. As a result, the paper's objective may not be entirely clear, as it's uncertain whether it primarily addresses fairness issues in multi-source domain generalization or aims for a more general approach dealing with subgroup robustness in domain generalization when faced with varying spurious correlations across domains.

Author response:
In our setting, an image has two labels, a class label and a sensitive label. For example, an image of a “dog in grass” has a “dog” class label and a “grass” sensitive label. Therefore, the features extracted from this image can be perceived as encapsulating information about both the "dog" and "grass". Another example is given in the empirical study using the ccMNIST dataset where the digit color is used as domain variation due to covariate shift and the background color is used to describe the sensitive label, dark and black. This is similar to traditional fairness learning using tabular datasets. For example, gender is a sensitive label of instance, “height” and “has a beard” could be features used to describe gender information.

Reviewer comment (Writing):
The information provided about fairness concepts in the paper is rather limited. In the paragraph discussing fairness notions, the authors do not acknowledge the existence of various definitions for quantifying the fairness guarantees of a classifier, including statistical notions, individual notions, and mini-max notions. They focus exclusively on statistical notions and, within this category, specifically on demographic parity (DP). The definition presented in the paper about statistical notions pertains solely to DP. The broader definition of a statistical notion of fairness encompasses parity in a given performance measure across different groups, with the specific terminology of these statistical notions varying based on the performance metric under consideration. Therefore, the paper's definition is not entirely accurate and should be revised. Furthermore, it is advisable for the authors to reference relevant literature that offers an extensive review of existing fairness metrics, providing readers with a more comprehensive understanding of the available fairness measures.

Author response:
This paper focuses on group fairness and uses DP or DDP (difference of DP) as the fairness notion. However, in our Eq.(1), it can be easily changed to the difference of equalized odds when setting $p_1=\mathbb{P}(Z=1, Y=1)$. And Eq.(1) can be generalized to take the maximum of $g$ across different classes for the multi-class setting. For multi-dimensional sensitive attributes, Eq.(1) can be replaced by corresponding group fairness metrics, such as an extended version of DP or EO proposed in [e].

Reviewer comment (Experiments):
In the experimental section, the authors evaluate various domain generalization methods, including GDRO, Mixup, and CORAL, which are originally designed for single-source domain scenarios. This may result in an unfair comparison, as they are applied to multi-source data even though they were not explicitly designed for such cases. Moreover, the authors do not clarify that certain baseline domain generalization methods are intended for either single-source or multi-source domain settings, which is a crucial distinction to highlight for a fair evaluation. Additionally, the paper does not provide clear information on how they adapt single-domain methods to handle multiple domain data.

Author response:
GDRO, Mixup, and CORAL are used as baseline methods in many multi-source domain generalization works, such as [c] and [d]. Such methods have been included in a public code repository developed by FacebookResearch, named domainBed (see https://github.com/facebookresearch/DomainBed) with 1.2k stared. Moreover, in our Problem 1, the max operator requires domain labels $e$. However, the domain labels are expensive or even impossible to obtain in part due to privacy issues. Therefore, under the disentanglement-based invariance and domain shift assumptions, Problem 1 can be approximated to Problme 2 with the max operator eliminated.

Reviewer comment (Writing):
No constraints or limitations explicitly pointed out in the main text.

Author response:
Some limitations could be the assumptions that we made (1) instances can be encoded into three latent factors (2) such factors are independent of each other, and (3) each training domain shares the same content space. Our proposed method may not work well when data are generated with more than three factors, and each is correlated to the other. Besides, our model is built based on domain augmentation. Although results showcase the effectiveness of our model, it may not work well when content spaces are not fully overlapped across domains. In this case, it would be better to first augment data by minimizing semantic gaps for each class across training domains and then conduct domain augmentations. To address such limitations, studies on causal learning could be a solution, and we will save it for our future works. Besides, our model is a bit expensive as the transformation model is learned using GAN. The detailed training process is discussed in the Appendix.

Reviewer comment (Writing):
The motivation behind "dependence shift" is not entirely clear. The example presented in Figure 1 is somewhat ambiguous. It's unclear why attributes like "grass" and "couch" are considered sensitive. The authors should consider providing a more realistic example to better illustrate their problem setting.

Author response:
Our example in Figure 1 is motivated by [a]. In [a], they gave an example using the Waterbird dataset, where the backgrounds (i.e., sensitive subgroups, land, and water) are spuriously correlated with class labels (waterbirds and landbirds). This spurious correlation causes sub-par performance in the smallest subgroups. Similar to Figure 2 in our paper, the spurious correlation between classes (dogs and cats) between image backgrounds or other objects (grass and couches) leads to untrusted predictions. For example, if a model learned from all training images “dogs in the grass” and “cats in the couch”, as a consequence the model face challenges accurately classifying “dogs” and “cats” during inference with images “dogs in the couch” and “cats in the grass”. This indicates sensitive features “grass” and “couch” are implicitly correlated with model prediction.

Reviewer comment (Novelty):
There's an existing work that also deals with fairness in domain generalization [cite]. That work doesn't assume invariance of $\rho$ across domains which then imply that it can also handle the dependence shift. Thus, it's essential for the authors to differentiate their work from this previous research. Additionally, Theorem 1 appears quite similar to Theorem 3 in [cite]. The authors should highlight the novel insights offered by Theorem 1 compared to the existing result.

Author response:
The reviewer seems to forget to give the [cite] reference, but we assume it refers to [b]. The difference between [b] and our works, with respect to handling fairness (dependence shift), is that [b] does not implicitly or explicitly assume fairness levels (dependence between sensitive attributes and classes) change across domains. Since test domains are inaccessible during training, in our works training domains are augmented with new variations and fairness levels. The greater the diversity of domains encompassing various variations and fairness levels encountered during training, the more effectively the model can generalize to different domains during inference. With limited fairness levels in training domains, in contrast to [b], our model has a better capability of generalization in fairness.

Reviewer comment (Theory):
Assumption 1 appears more like a definition than an assumption since a mapping between two domains can always be constructed. For instance, when domains share the same support, an identity mapping fulfills this assumption.

Author response:
Some other papers in domain generalization based on maximizing the Evidence Lower Bound (ELBO) do not make this assumption, such as [c]. We hence consider it as an assumption but not a definition.

Reviewer comment (Theory):
The authors should explain why they restrict the hypothesis class $\mathcal{F}$ to the set of invariant fairness-aware classifiers and why the max operator in the objective function can be eliminated in Problem 2.

Author response:
The goal of domain generalization problems is to find an invariant classifier in the classifier space of $\mathcal{F}$, theoretically. But in experiments, with given datasets, this goal is required to approximate to find model parameters of the classifier in a parametric space. Moreover, in Problem 1, the max operator requires domain labels $e$. However, the domain labels are expensive or even impossible to obtain in part due to privacy issues. Therefore, under the disentanglement-based invariance and domain shift assumptions, Problem 1 can be approximated to Problme 2 with the max operator eliminated. We will clearly explain this in an updated version.

Reviewer comment (Evaluation):
To strengthen their claims, the authors should consider including comparisons of fairness-accuracy trade-off curves (Pareto frontiers) for the models in the experimental section. This would provide stronger evidence for their proposed method's effectiveness.

Author response:
Some additional results on the trade-off between fairness-accuracy are conducted. We will update it in a later version.

In our algorithm, because $\lambda_2$  is the parameter that regularizes the fair loss, we conduct additional experiments to show the change between accuracy and fairness. Our results show that the larger (smaller) $\lambda_2$ , the better (worse) model fairness for each domain as well as in average, but it gives worse (better) model utility.
| |      DP $\uparrow$ / AUC $\downarrow$ / Accuracy $\uparrow$|  |  |  |
|----------|:-------------:|:------:|:------:|:------:|
|                    	| ccMNIST                                                	| FairFace            	| YFCC                	| NYSF                	|
| $\lambda_2 = 0.05$ | 0.53 / 0.75 / 98.61 | 0.57 / 0.63 / 95.99 | 0.88 / 0.55 / 88.31 | 0.86 / 0.57 / 61.71 |
| $\lambda_2 = 1$ (results in the paper) |    0.71 / 0.66 / 96.76 | 0.70 / 0.58 / 93.42 | 0.92 / 0.53 / 83.12 | 0.97 / 0.51 / 59.95 | 
| $\lambda_2 = 50$ | 0.72 / 0.63 / 89.07 | 0.78 / 0.52 / 88.65 | 0.95 / 0.51 / 72.63 | 0.98 / 0.51 / 55.28 |

Moreover, we show additional experiment results based on choosing different  $\gamma_1$ and $\gamma_2$. We observe that (1) by only increasing $\gamma_2$ , the model towards giving unfair outcomes but higher accuracy; (2) by only increasing $\gamma_1$, performance on both model fairness and accuracy decreases. This may be due to the failure of disentanglement of factors.

| |      DP $\uparrow$ / AUC $\downarrow$ / Accuracy $\uparrow$|  |  |  |
|----------|:-------------:|:------:|:------:|:------:|
|                    	| ccMNIST                                                	| FairFace            	| YFCC                	| NYSF                	|
| $\gamma_1= 0.025, \gamma_2=0.25$ | 0.47 / 0.79 / 97.07 | 0.53 / 0.60 / 93.99 | 0.88 / 0.55 / 88.69 | 0.86 / 0.56 / 61.71 |
| $\gamma_1= 0.25, \gamma_2=0.025$ |    0.66 / 0.75 / 88.54 | 0.62 / 0.58 / 93.06 | 0.91 / 0.54 / 81.49 | 0.86 / 0.57 / 58.03 | 
| $\gamma_1= 0.025, \gamma_2=0.025$ (results in the paper) | 0.71 / 0.66 / 96.97 | 0.70 / 0.58 / 93.42 | 0.92 / 0.53 / 83.12 | 0.97 / 0.51 / 59.95 |

Reviewer comment (Theory):
I do not think the examples given in Figure 1 are appropriate: (1) concept shift is the disparity in $Y^e | X^e$. It doesn't mean a completely new label appears in the target domain; (2) I am unsure if such an auxiliary background or objective, such as couch or grass, is an appropriate "sensitive attribute." I think those can be spurious features; however, often, in real-world cases, sensitive attributes are more than just spurious features as they correlate with both input and output. Moreover, ensuring fairness across the images that include a couch or grass does not seem to be a convincing example of fairness.

Author response:
(1) Concept shift in Figure1 is more related to semantic shift. Semantic shift as illustrated in [a] can be viewed as a type of concept shift. In [a], concept shift is defined as semantic changes, and in their experiments  CIFAR100 is split into 10 sub-datasets with increasing conceptual differences from CIFAR10 classes. For example, pickup truck (CIFAR100) is much closer to truck (CIFAR10) than sunflowers (CIFAR100) semantically. (2) Our example in Figure 1 is similar to [b]. In [b], they gave an example using the Waterbird dataset, where the backgrounds (land and water) are spuriously correlated with class labels (waterbirds and landbirds). Notice that when using images for fairness learning, since inputs are images followed by class and sensitive labels, some aspects of the image features may inadvertently encapsulate information related to the sensitive label.

Reviewer comment (Evaluation):
Also, in Figure 2, I do not think what the authors presented is "group fairness levels," as the fairness notion considered in the study is demographic parity, which is calculated as a statistical dependence of the prediction and the sensitive attributes. However, in the figure, no prediction is considered. I think what the authors presented there is the dependence (or correlation) between the true label and the sensitive attribute.

Author response:
The fairness level in Figure 2 is calculated using demographic parity (DP) using the evaluation metric used in experiments. The equation of DP is proposed in Appendix C.3.

Reviewer comment (Theory):
In many applications, the sensitive attribute and outcome variable are nonbinary. The extension to such cases of the approach introduced in the study does not seem to be straightforward.

Author response:
In this paper, we use a binary sensitive attribute and classes just as a simple case for idea illustration, as many fairness works did, such as [b]. This can be easily extended to multinary cases by either changing the group fairness notion or taking the maximum difference of DP over each class/subgroup.

Reviewer comment (Evaluation):
I found the evaluation metrics are inconsistent - why do the authors evaluate the fairness in difference of AUC but the performance in accuracy?

Author response:
The AUC (defined in Appendix C.3) in this work refers to a statistically consistent measure of predictive strength by comparing outcomes between sensitive subgroups. Notice that this is different from the AUC used for classification and this metric is commonly used for fairness learning, see [c,d,e].

Reviewer comment (Evaluation):
The authors used so-called inter-group AUC difference as a fairness metric. There have been extensive discussions on different versions of AUC differences, and it has been demonstrated that inter-group AUC difference does not reflect all possible performance disparities. Please refer to [Kallus and Zhou (2019)](https://papers.nips.cc/paper_files/paper/2019/hash/73e0f7487b8e5297182c5a711d20bf26-Abstract.html), [Yang et al. (2023)](https://dl.acm.org/doi/abs/10.1609/aaai.v37i10.26405) and the references therein.

Author response:
The AUC (defined in Appendix C.3) in this work refers to a statistically consistent measure of predictive strength by comparing outcomes between sensitive subgroups. Notice that this is different from the AUC used for classification and this metric is commonly used for fairness learning, see [c,d,e].

Reviewer comment (Evaluation):
It seems like the authors confuse demographic disparity (dependence of $\hat{Y}$ and $Z$) and dependence of $Y$ and $Z$. E.g., in Figure 2 and Table 1, the authors presented the degrees of dependence of $Y$ and $Z$ as if they were fairness measurements.

Author response:
Dependence between $Y$ and $Z$ is to reveal the bias with respect to the dataset unfairness. Dependence between $\hat{Y}$ and $Z$ refers to algorithmic fairness with respect to the model prediction. Dataset unfairness is used as a reference to compare with the algorithmic fairness for performance improvement.

Reviewer comment (Novelty):
Some important related works are missing. E.g., [Rezaei et al. (2021)](https://ojs.aaai.org/index.php/AAAI/article/view/17135), [Singh et al. (2021)](https://dl.acm.org/doi/10.1145/3442188.3445865), [Giguere et al. (2022)](https://openreview.net/forum?id=wbPObLm6ueA), [Chen et al. (2022)](https://openreview.net/forum?id=U3gobB4oKv). This is not an exhaustive list.

Author response:
Such works will be discussed in an updated version.

Reviewer comment (Theory):
Could the authors prove or give some thoughts about the existence of a non-trivial solution to Problem 1?

Author response:
In Problem 1, without the constraint, it is equivalent to a domain generalization (DG) problem stated in some DG works, such as [f,g]. A fair constraint can be perceived as the pursuit of optimal model parameters within a fair parameter space, which is derived from the real domain after removing sensitive attributes.

Reviewer comment (Evaluation):
Is there any reason the authors used DP as the fairness criterion? Many previous studies revealed that the DP might not be an ideal fairness criterion.

Author response:
DP as a commonly used group fairness metric refers to the likelihood of a positive outcome being the same across different groups. We use DP only because this metric is well-known in fairness machine learning and is considered in many related works. Some other fair metrics, such as equalized odds, can also be applied.

Reviewer comment (Evaluation):
The results show that the proposed method effectively reduces the AUC difference across sensitive groups in almost all cases. Could the authors kindly explain why this happened? Because, for me, the relationship between the statistical dependence of the model prediction and sensitive attribute and the AUC differences is unclear. Thus, reducing the dependence (or equivalently achieving demographic parity) does not necessarily result in smaller AUC differences. To the best of my knowledge, a similar analysis has been done by [Zhao and Gordan (2022)](https://dl.acm.org/doi/abs/10.5555/3586589.3586646). They proved that fair representation (that is, independent of the sensitive attribute, which satisfies SP) results in accuracy parity (the same accuracies across all sensitive groups) if the Bayes optimal classifiers across different groups are close.

Author response:
The reduction of AUC indicates the sum ranks of predicted outcomes between sensitive subgroups are similar. This AUC metric (defined in Appendix C.3) is not the same as the one commonly used in binary classification based on TPR and FPR. The intuition behind this AUC is based on the nonparametric test, Mann–Whitney U test, in which for randomly selected values $\mathbf{x}_1$ and $\mathbf{x}_2$  from two sensitive subgroups, a fair condition is defined as the probability of $\mathbf{x}_1$ being greater than $\mathbf{x}_2$ is equal to the probability of $\mathbf{x}_2$ being greater than $\mathbf{x}_1$, i.e. AUC=0.5.