Original: E QUALIZED R OBUSTNESS : T OWARDS S USTAINABLE
Revised: E QUALIZED R OBUSTNESS : T OWARDS S USTAINABLE

Original: F AIRNESS U NDER D ISTRIBUTIONAL S HIFTS
Revised: F AIRNESS U NDER D ISTRIBUTIONAL S HIFTS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Increasing concerns have been raised on deep learning fairness in recent years.
Revised: Increasing concerns have been raised on deep learning fairness in recent years.

Original: Existing fairness metrics and algorithms mainly focus on the discrimination of model performance across different groups on in-distribution data.
Revised: Existing fairness metrics and algorithms mainly focus on the discrimination of model performance across different groups on in-distribution data.

Original: It remains unclear whether the fairness achieved on in-distribution data can be generalized to data with unseen distribution shifts, which are commonly encountered in real-world applications.
Revised: It remains unclear whether the fairness achieved on in-distribution data can be generalized to data with unseen distribution shifts, which are commonly encountered in real-world applications.

Original: In this paper, we ﬁrst propose a new fairness goal, termed Equalized Robustness (ER), to impose fair model robustness against unseen distribution shifts across majority and minority groups.
Revised: In this paper, we ﬁrst propose a new fairness goal, termed Equalized Robustness (ER), to impose fair model robustness against unseen distribution shifts across majority and minority groups.

Original: ER measures robustness disparity by the maximum mean discrepancy (MMD) distance between the loss curvature distributions of two groups of data.
Revised: ER measures robustness disparity by the maximum mean discrepancy (MMD) distance between the loss curvature distributions of two groups of data.

Original: We show that previous fairness learning algorithms designed for in-distribution fairness fail to meet the new robust fairness goal.
Revised: We show that previous fairness learning algorithms designed for in-distribution fairness fail to meet the new robust fairness goal.

Original: We further propose a novel fairness learning algorithm, termed Curvature
Revised: We further propose a novel fairness learning algorithm, termed Curvature

Original: Matching (CUMA), to simultaneously achieve both traditional in-distribution fairness and our new robust fairness.
Revised: Matching (CUMA), to simultaneously achieve both traditional in-distribution fairness and our new robust fairness.

Original: CUMA debiases the model robustness by minimizing the MMD distance between loss curvature distributions of two groups.
Revised: CUMA debiases the model robustness by minimizing the MMD distance between loss curvature distributions of two groups.

Original: Experiments on three popular datasets show CUMA achieves superior fairness in robustness against distribution shifts, without more sacriﬁce on either overall accuracies or the in-distribution fairness.
Revised: Experiments on three popular datasets show CUMA achieves superior fairness in robustness against distribution shifts, without more sacriﬁce on either overall accuracies or the in-distribution fairness.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: With the wide deployment of deep learning in modern business applications concerning individual lives and privacy, there naturally emerge concerns on machine learning fairness (Podesta et al., 2014; Mu ˜ noz et al., 2016; Smuha, 2019).
Revised: With the wide deployment of deep learning in modern business applications concerning individual lives and privacy, there naturally emerge concerns on machine learning fairness (Podesta et al., 2014; Mu ˜ noz et al., 2016; Smuha, 2019).

Original: Research efforts on various fairness evaluation metrics and corresponding enforcing methods have been carried out (Edwards & Storkey, 2016; Hardt et al., 2016; Du et al., 2020).
Revised: Research efforts on various fairness evaluation metrics and corresponding enforcing methods have been carried out (Edwards & Storkey, 2016; Hardt et al., 2016; Du et al., 2020).

Original: Speciﬁcally, many such metrics require some form of “equalized model performance” across different groups on in-distribution data.
Revised: Speciﬁcally, many such metrics require some form of “equalized model performance” across different groups on in-distribution data.

Original: Examples include Demographic parity (DP) (Edwards & Storkey, 2016), Equalized Opportunity (EOpp), and Equalized Odds (EO)
Revised: Examples include Demographic parity (DP) (Edwards & Storkey, 2016), Equalized Opportunity (EOpp), and Equalized Odds (EO)

Original: (Hardt et al., 2016).
Revised: (Hardt et al., 2016).

Original: Unfortunately, when deployed for real-world applications, deep models commonly encounter data with unforeseeable distribution shifts (Hendrycks & Dietterich, 2019;
Revised: Unfortunately, when deployed for real-world applications, deep models commonly encounter data with unforeseeable distribution shifts (Hendrycks & Dietterich, 2019;

Original: Hendrycks et al., 2020; 2021).
Revised: Hendrycks et al., 2020; 2021).

Original: It has been shown that deep learning models can have drastically degraded performance (Hendrycks & Dietterich, 2019;
Revised: It has been shown that deep learning models can have drastically degraded performance (Hendrycks & Dietterich, 2019;

Original: Hendrycks et al., 2020; 2021; Taori et al., 2020) and show unreliable behaviors (Qiu et al., 2019; Yan et al., 2021) under unseen distribution shifts.
Revised: Hendrycks et al., 2020; 2021; Taori et al., 2020) and show unreliable behaviors (Qiu et al., 2019; Yan et al., 2021) under unseen distribution shifts.

Original: Intuitively speaking, previous fairness learning algorithms aim to optimize the model to a local minimum where data from majority and minority groups have similar average loss values (and thus similar in-distribution performance).
Revised: Intuitively speaking, previous fairness learning algorithms aim to optimize the model to a local minimum where data from majority and minority groups have similar average loss values (and thus similar in-distribution performance).

Original: However, those algorithms do not take into consideration the the stability or “robustness” of their found fairness-aware minima.
Revised: However, those algorithms do not take into consideration the the stability or “robustness” of their found fairness-aware minima.

Original: Taking object detection in a self-driving car for example, it might have been calibrated over high-quality clear images to be “fair” with different skin colors; however such fairness may severely break down when applied to data collected in adverse visual conditions, such as inclement weather, poor lighting, or other digital artifacts.
Revised: Taking object detection in a self-driving car for example, it might have been calibrated over high-quality clear images to be “fair” with different skin colors; however such fairness may severely break down when applied to data collected in adverse visual conditions, such as inclement weather, poor lighting, or other digital artifacts.

Original: Our experiments also ﬁnd that previous state-of-the-art fairness algorithms would be jeopardized if distributional shifts are present in test data, as illustrated in Figure 1 (b).
Revised: Our experiments also ﬁnd that previous state-of-the-art fairness algorithms would be jeopardized if distributional shifts are present in test data, as illustrated in Figure 1 (b).

Original: The above ﬁndings beg the following question:
Revised: The above ﬁndings beg the following question:

Original: How to achieve practically sustainable fairness, e.g., even under unseen distribution shifts?
Revised: How to achieve practically sustainable fairness, e.g., even under unseen distribution shifts?

Original: [Figure]
Revised: [Figure]

Original: 𝑥
Revised: 𝑥

Original: [Figure]
Revised: [Figure]

Original: 𝑥
Revised: 𝑥

Original: [Figure]
Revised: [Figure]

Original: 𝑥
Revised: 𝑥

Original: [Figure]
Revised: [Figure]

Original: 𝑥 # 𝑥 #" 𝑥 𝐿(
Revised: 𝑥 # 𝑥 #" 𝑥 𝐿(

Original: [Figure]
Revised: [Figure]

Original: To answer that, we ﬁrst propose a new fairness objective, termed Equalized Robustness (ER) , which aims to impose “equalized robustness” against unseen distribution shifts across the majority and minority groups, so that the learned fairness can sustain even with test data perturbed.
Revised: To answer that, we ﬁrst propose a new fairness objective, termed Equalized Robustness (ER) , which aims to impose “equalized robustness” against unseen distribution shifts across the majority and minority groups, so that the learned fairness can sustain even with test data perturbed.

Original: ER explicitly considers a new dimension of fairness that is practically signiﬁcant yet so far largely overlooked.
Revised: ER explicitly considers a new dimension of fairness that is practically signiﬁcant yet so far largely overlooked.

Original: In other words, ER assesses fairness on “out-of-distribution”.
Revised: In other words, ER assesses fairness on “out-of-distribution”.

Original: Therefore it works as a complement instead of a replacement for previous fairness metrics, which focus on assessing the “in-distribution” fairness.
Revised: Therefore it works as a complement instead of a replacement for previous fairness metrics, which focus on assessing the “in-distribution” fairness.

Original: Previous research has shown that model robustness against input perturbation is highly correlated with loss curvature smoothness (Bartlett et al., 2017;
Revised: Previous research has shown that model robustness against input perturbation is highly correlated with loss curvature smoothness (Bartlett et al., 2017;

Original: Moosavi-Dezfooli et al., 2019; Weng et al., 2018).
Revised: Moosavi-Dezfooli et al., 2019; Weng et al., 2018).

Original: Our experiments also observed that, the local loss curvature of minority group is often larger than that of majority group, leading to the two group’s robustness discrepancy against distribution shifts.
Revised: Our experiments also observed that, the local loss curvature of minority group is often larger than that of majority group, leading to the two group’s robustness discrepancy against distribution shifts.

Original: To this end, we propose to empirically quantify the robustness discrepancy as the maximum mean discrepancy (MMD) (Gretton et al., 2012) distance between the local model smoothness distributions, for data samples from the majority and minority groups.
Revised: To this end, we propose to empirically quantify the robustness discrepancy as the maximum mean discrepancy (MMD) (Gretton et al., 2012) distance between the local model smoothness distributions, for data samples from the majority and minority groups.

Original: We experimentally demonstrate that our new metric aligns well with model performance under real-world distribution shifts.
Revised: We experimentally demonstrate that our new metric aligns well with model performance under real-world distribution shifts.

Original: On top of that, we further propose a new fair learning algorithm, termed Curvature Matching (CUMA) , to simultaneously achieve both traditional in-distribution fairness and ER.
Revised: On top of that, we further propose a new fair learning algorithm, termed Curvature Matching (CUMA) , to simultaneously achieve both traditional in-distribution fairness and ER.

Original: CUMA matches the local curvature distribution between data points from the two different groups, as illustrated in Figure(c), by adding a curvature-matching regularizer that can be efﬁciently computed via a one-shot power iteration method.
Revised: CUMA matches the local curvature distribution between data points from the two different groups, as illustrated in Figure(c), by adding a curvature-matching regularizer that can be efﬁciently computed via a one-shot power iteration method.

Original: Our codes will be released upon acceptance.
Revised: Our codes will be released upon acceptance.

Original: Our contributions can be summarized as bellow:
Revised: Our contributions can be summarized as bellow:

Original: • We propose Equalized Robustness ( ER ), a new fairness objective for machine learning models, to impose equalized model robustness against unforeseeable distributions shifts across majority and minority groups.
Revised: • We propose Equalized Robustness ( ER ), a new fairness objective for machine learning models, to impose equalized model robustness against unforeseeable distributions shifts across majority and minority groups.

Original: • We further propose a new fairness learning algorithm dubbed Curvature Matching ( CUMA ), which enforces ER during training by utilizing a one-shot power iteration method.
Revised: • We further propose a new fairness learning algorithm dubbed Curvature Matching ( CUMA ), which enforces ER during training by utilizing a one-shot power iteration method.

Original: • Experiments show that CUMA achieves much more robust fairness against distribution shifts, without more sacriﬁce on either overall accuracies or the in-distribution fairness, compared with traditional in-distribution fair learning methods.
Revised: • Experiments show that CUMA achieves much more robust fairness against distribution shifts, without more sacriﬁce on either overall accuracies or the in-distribution fairness, compared with traditional in-distribution fair learning methods.

Original:  P RELIMINARIES 2. M ACHINE L EARNING F AIRNESS
Revised:  P RELIMINARIES 2. M ACHINE L EARNING F AIRNESS

Original: Problem Setting and Metrics Machine learning fairness can be generally categorized into individual fairness and group fairness (Du et al., 2020).
Revised: Problem Setting and Metrics Machine learning fairness can be generally categorized into individual fairness and group fairness (Du et al., 2020).

Original: Individual fairness requires similar inputs to have similar predictions (Dwork et al., 2012).
Revised: Individual fairness requires similar inputs to have similar predictions (Dwork et al., 2012).

Original: Compared with individual fairness, group fairness is a more popular setting and thus the focus of our paper.
Revised: Compared with individual fairness, group fairness is a more popular setting and thus the focus of our paper.

Original: Given input data X ∈ R n with sensitive attributes A ∈ { 0 , 1 } and their corresponding ground truth labels Y ∈ { 0 , 1 } , group fairness requires a learned binary classiﬁer f ( · ; θ ) : R n → { 0 , 1 } parameterized by θ to give equally accurate predictions (denoted as ˆ Y := f ( X ) ) on the two groups with A = 0 and A = 1 .
Revised: Given input data X ∈ R n with sensitive attributes A ∈ { 0 , 1 } and their corresponding ground truth labels Y ∈ { 0 , 1 } , group fairness requires a learned binary classiﬁer f ( · ; θ ) : R n → { 0 , 1 } parameterized by θ to give equally accurate predictions (denoted as ˆ Y := f ( X ) ) on the two groups with A = 0 and A = 1 .

Original: Multiple fairness criteria have been deﬁned in this context.
Revised: Multiple fairness criteria have been deﬁned in this context.

Original: Demographic parity (DP) (Edwards & Storkey, 2016) requires identical ratio of positive predictions between two groups: P ( ˆ Y = 1 | A = 0)
Revised: Demographic parity (DP) (Edwards & Storkey, 2016) requires identical ratio of positive predictions between two groups: P ( ˆ Y = 1 | A = 0)

Original: =
Revised: =

Original: P ( ˆ Y = 1 | A = 1) .
Revised: P ( ˆ Y = 1 | A = 1) .

Original: Equalized Odds (EO) (Hardt et al., 2016) requires identical false positive rates (FPRs) and false negative rates (FNRs) between the two groups: P ( ˆ Y (cid:54)
Revised: Equalized Odds (EO) (Hardt et al., 2016) requires identical false positive rates (FPRs) and false negative rates (FNRs) between the two groups: P ( ˆ Y (cid:54)

Original: =
Revised: =

Original: Y |
Revised: Y |

Original: A = 0 , Y = y ) =
Revised: A = 0 , Y = y ) =

Original: P ( ˆ Y (cid:54) =
Revised: P ( ˆ Y (cid:54) =

Original: Y |
Revised: Y |

Original: A = 1 , Y = y ) , ∀ y ∈ { 0 , 1 } .
Revised: A = 1 , Y = y ) , ∀ y ∈ { 0 , 1 } .

Original: Equalized Opportunity (EOpp) (Hardt et al., 2016) requires only equal
Revised: Equalized Opportunity (EOpp) (Hardt et al., 2016) requires only equal

Original: FNRs between the groups: P ( ˆ Y (cid:54)
Revised: FNRs between the groups: P ( ˆ Y (cid:54)

Original: =
Revised: =

Original: Y |
Revised: Y |

Original: A = 0 , Y = 0) =
Revised: A = 0 , Y = 0) =

Original: P ( ˆ Y (cid:54) =
Revised: P ( ˆ Y (cid:54) =

Original: Y |
Revised: Y |

Original: A = 1 , Y = 0) .
Revised: A = 1 , Y = 0) .

Original: Based on these fairness criteria, quantiﬁed metrics are deﬁned to measure fairness.
Revised: Based on these fairness criteria, quantiﬁed metrics are deﬁned to measure fairness.

Original: Speciﬁcally, DP, EO and EOpp distances (Madras et al., 2018) are deﬁned as follows:
Revised: Speciﬁcally, DP, EO and EOpp distances (Madras et al., 2018) are deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: MMD has been previously used to deﬁne fairness metric in (Quadrianto & Sharmanska, 2017) deﬁnes a more general fairness metric using MMD distance, and shows ∆ DP , ∆ EO and ∆ EOpp to be spatial cases of their uniﬁed metric.
Revised: MMD has been previously used to deﬁne fairness metric in (Quadrianto & Sharmanska, 2017) deﬁnes a more general fairness metric using MMD distance, and shows ∆ DP , ∆ EO and ∆ EOpp to be spatial cases of their uniﬁed metric.

Original: All these metrics consider the in-distribution fairness, while our
Revised: All these metrics consider the in-distribution fairness, while our

Original: Equalized Generalizibility is the ﬁrst fairness metric explicitly aware of robust generalization ability on unseen distributions.
Revised: Equalized Generalizibility is the ﬁrst fairness metric explicitly aware of robust generalization ability on unseen distributions.

Original: Bias Mitigation Methods Many methods have been proposed to mitigate model bias.
Revised: Bias Mitigation Methods Many methods have been proposed to mitigate model bias.

Original: Data preprocessing methods such as re-weighting (Kamiran & Calders, 2012) and data-transformation (Calmon et al., 2017) have been used to reduce discrimination before model training.
Revised: Data preprocessing methods such as re-weighting (Kamiran & Calders, 2012) and data-transformation (Calmon et al., 2017) have been used to reduce discrimination before model training.

Original: In contrast, Hardt et al.
Revised: In contrast, Hardt et al.

Original: and Zhao et al.
Revised: and Zhao et al.

Original: (2017) propose post-processing methods to calibrate model predictions towards a desired fair distribution after model training.
Revised: (2017) propose post-processing methods to calibrate model predictions towards a desired fair distribution after model training.

Original: Instead of pre- or post-processing, researchers have explored to enhance fairness during training.
Revised: Instead of pre- or post-processing, researchers have explored to enhance fairness during training.

Original: For example, Madras et al.
Revised: For example, Madras et al.

Original: (2018) uses a adversarial training technique and shows the learned fair representations can transfer to unseen target tasks.
Revised: (2018) uses a adversarial training technique and shows the learned fair representations can transfer to unseen target tasks.

Original: The key technique, adversarial training (Edwards & Storkey, 2016), was designed for feature disentanglement on hidden representations such that sensitive (Edwards & Storkey, 2016) or domain-speciﬁc information (Ganin et al., 2016) will be removed while keeping other useful information for the target task.
Revised: The key technique, adversarial training (Edwards & Storkey, 2016), was designed for feature disentanglement on hidden representations such that sensitive (Edwards & Storkey, 2016) or domain-speciﬁc information (Ganin et al., 2016) will be removed while keeping other useful information for the target task.

Original: The hidden representations are typically the output of intermediate layers of neural networks (Ganin et al., 2016; Edwards & Storkey, 2016; Madras et al., 2018).
Revised: The hidden representations are typically the output of intermediate layers of neural networks (Ganin et al., 2016; Edwards & Storkey, 2016; Madras et al., 2018).

Original: Instead, methods, like adversarial debiasing (Zhang et al., 2018) and its simpliﬁed version (Wadsworth et al., 2018), directly apply the adversary on the output layer of the classiﬁer, which also promotes the model fairness.
Revised: Instead, methods, like adversarial debiasing (Zhang et al., 2018) and its simpliﬁed version (Wadsworth et al., 2018), directly apply the adversary on the output layer of the classiﬁer, which also promotes the model fairness.

Original: Observing the unfairness due to ignoring the worst learning risk of speciﬁc samples,
Revised: Observing the unfairness due to ignoring the worst learning risk of speciﬁc samples,

Original: Hashimoto et al.
Revised: Hashimoto et al.

Original: (2018) proposes to use distributionally robust optimization which provably bounds the worst-case risk over groups.
Revised: (2018) proposes to use distributionally robust optimization which provably bounds the worst-case risk over groups.

Original: Creager et al.
Revised: Creager et al.

Original: (2019) proposes a ﬂexible fair representation learning framework based on VAE (Kingma & Welling, 2013), that can be easily adapted for different sensitive attribute settings during run-time.
Revised: (2019) proposes a ﬂexible fair representation learning framework based on VAE (Kingma & Welling, 2013), that can be easily adapted for different sensitive attribute settings during run-time.

Original: Sarhan et al.
Revised: Sarhan et al.

Original: (2020) uses orthogonality constraints as a proxy for independence to disentangles the utility and sensitive representations.
Revised: (2020) uses orthogonality constraints as a proxy for independence to disentangles the utility and sensitive representations.

Original: Martinez et al.
Revised: Martinez et al.

Original: (2020) formulates group fairness with multiple sensitive attributes as a multi-objective learning problem and proposes a simple optimization algorithm to ﬁnd the Pareto optimality.
Revised: (2020) formulates group fairness with multiple sensitive attributes as a multi-objective learning problem and proposes a simple optimization algorithm to ﬁnd the Pareto optimality.

Original: Another line of research focuses on learning unbiased representations from biased ones (Bahng et al., 2020; Nam et al., 2020).
Revised: Another line of research focuses on learning unbiased representations from biased ones (Bahng et al., 2020; Nam et al., 2020).

Original: Bahng et al.
Revised: Bahng et al.

Original: (2020) proposes a novel framework to learn unbiased representations by explicitly enforcing them to be different from a set of pre-deﬁned biased representations.
Revised: (2020) proposes a novel framework to learn unbiased representations by explicitly enforcing them to be different from a set of pre-deﬁned biased representations.

Original: Nam et al.
Revised: Nam et al.

Original: (2020) observes that data bias can be either benign or malicious, and removing malicious bias along can achieve fairness.
Revised: (2020) observes that data bias can be either benign or malicious, and removing malicious bias along can achieve fairness.

Original: Li & Vasconcelos (2019) jointly learns a data re-sampling weight distribution that penalizes easy samples and network parameters.
Revised: Li & Vasconcelos (2019) jointly learns a data re-sampling weight distribution that penalizes easy samples and network parameters.

Original: Applications in Computer Vision When many fairness metrics and debiasing algorithms are designed for general learning problems as aforementioned, there are a line of research and applications focusing on fairness-encouraged computer vision tasks.
Revised: Applications in Computer Vision When many fairness metrics and debiasing algorithms are designed for general learning problems as aforementioned, there are a line of research and applications focusing on fairness-encouraged computer vision tasks.

Original: For instance, Buolamwini et al.
Revised: For instance, Buolamwini et al.

Original: (Buolamwini & Gebru, 2018) shows current commercial gender-recognition systems have substantial accuracy disparities among groups with different genders and skin colors.
Revised: (Buolamwini & Gebru, 2018) shows current commercial gender-recognition systems have substantial accuracy disparities among groups with different genders and skin colors.

Original: Wilson et al.
Revised: Wilson et al.

Original: (2019) observe that state-of-the-art segmentation models achieve better performance on pedestrians with lighter skin colors.
Revised: (2019) observe that state-of-the-art segmentation models achieve better performance on pedestrians with lighter skin colors.

Original: In (Shankar et al., 2017; de Vries et al., 2019), it is found that the common geographical bias in public image databases can lead to strong performance disparities among images from locales with different income levels.
Revised: In (Shankar et al., 2017; de Vries et al., 2019), it is found that the common geographical bias in public image databases can lead to strong performance disparities among images from locales with different income levels.

Original: Nagpal et al.
Revised: Nagpal et al.

Original: (2019) reveal that the focus region of face-classiﬁcation models depends on people’s ages or races, which may explain the source of age- and race-biases of classiﬁers.
Revised: (2019) reveal that the focus region of face-classiﬁcation models depends on people’s ages or races, which may explain the source of age- and race-biases of classiﬁers.

Original: On the awareness of the unfairness, many efforts have been devoted to mitigate such biases in computer vision tasks.
Revised: On the awareness of the unfairness, many efforts have been devoted to mitigate such biases in computer vision tasks.

Original: Wang et al.
Revised: Wang et al.

Original: (2019) shows the effectiveness of adversarial debiasing technique (Zhang et al., 2018) in fair image classiﬁcation and activity recognition tasks.
Revised: (2019) shows the effectiveness of adversarial debiasing technique (Zhang et al., 2018) in fair image classiﬁcation and activity recognition tasks.

Original: Beyond the supervised learning, FairFaceGAN (Hwang et al., 2020) is proposed to prevent undesired sensitive feature translation during image editing.
Revised: Beyond the supervised learning, FairFaceGAN (Hwang et al., 2020) is proposed to prevent undesired sensitive feature translation during image editing.

Original: Similar ideas have also been successfully applied to visual question answering (Park et al., 2020).
Revised: Similar ideas have also been successfully applied to visual question answering (Park et al., 2020).

Original: 2. M ODEL R OBUSTNESS AND S MOOTHNESS
Revised: 2. M ODEL R OBUSTNESS AND S MOOTHNESS

Original: Model generalization ability and robustness has been shown to be highly correlated with model smoothness (Moosavi-Dezfooli et al., 2019; Weng et al., 2018).
Revised: Model generalization ability and robustness has been shown to be highly correlated with model smoothness (Moosavi-Dezfooli et al., 2019; Weng et al., 2018).

Original: Weng et al.
Revised: Weng et al.

Original: (2018) and Guo et al.
Revised: (2018) and Guo et al.

Original: (2018) use local Lipschitz constant to estimate model robustness against small perturbations on inputs within a hyper-ball.
Revised: (2018) use local Lipschitz constant to estimate model robustness against small perturbations on inputs within a hyper-ball.

Original: Moosavi-Dezfooli et al.
Revised: Moosavi-Dezfooli et al.

Original: (2019) proposes to improve model robustness by adding a curvature constraint to encourage model smoothness.
Revised: (2019) proposes to improve model robustness by adding a curvature constraint to encourage model smoothness.

Original: Miyato et al.
Revised: Miyato et al.

Original: (2018) approximates model local smoothness by the spectral norm of Hessian matrix, and improves model robustness against adversarial attacks by regularizing model smoothness.
Revised: (2018) approximates model local smoothness by the spectral norm of Hessian matrix, and improves model robustness against adversarial attacks by regularizing model smoothness.

Original:  E QUALIZED R OBUSTNESS : A N EW M ETRIC FOR F AIR G ENERALIZATION AND R OBUSTNESS
Revised:  E QUALIZED R OBUSTNESS : A N EW M ETRIC FOR F AIR G ENERALIZATION AND R OBUSTNESS

Original: Consider a binary classiﬁer f ( · ; θ ) trained on two groups of data X 1 and X 2 respectively.
Revised: Consider a binary classiﬁer f ( · ; θ ) trained on two groups of data X 1 and X 2 respectively.

Original: Our goal is to deﬁne a metric to measure the gap of model robustness between the two groups.
Revised: Our goal is to deﬁne a metric to measure the gap of model robustness between the two groups.

Original: Formulating such a metric is highly non-trivial, with difﬁculties from mainly two aspects.
Revised: Formulating such a metric is highly non-trivial, with difﬁculties from mainly two aspects.

Original: The ﬁrst challenge is that we need to ensure fair generalization against multiple unseen distribution shifts that may encounter in real world applications.
Revised: The ﬁrst challenge is that we need to ensure fair generalization against multiple unseen distribution shifts that may encounter in real world applications.

Original: A trivial solution would be selecting a set of predeﬁned distribution shifts and measuring the average performance gap (e.g., ∆ EO ) against them.
Revised: A trivial solution would be selecting a set of predeﬁned distribution shifts and measuring the average performance gap (e.g., ∆ EO ) against them.

Original: However, this approach requires engineering overhead in handcrafting the predeﬁned distribution shifts, and the predeﬁned distribution shifts may not be representative enough to cover all unseen cases.
Revised: However, this approach requires engineering overhead in handcrafting the predeﬁned distribution shifts, and the predeﬁned distribution shifts may not be representative enough to cover all unseen cases.

Original: Previous research (Miyato et al., 2018; Moosavi-Dezfooli et al., 2019; Guo et al., 2018;
Revised: Previous research (Miyato et al., 2018; Moosavi-Dezfooli et al., 2019; Guo et al., 2018;

Original: Weng et al., 2018) has shown both theoretically and empirically that deep model robustness scales with its model smoothness.
Revised: Weng et al., 2018) has shown both theoretically and empirically that deep model robustness scales with its model smoothness.

Original: Following (Miyato et al., 2018; Moosavi-Dezfooli et al., 2019), we use the spectral norm of Hessian matrix to approximate local smoothness as an indicator of model robustness.
Revised: Following (Miyato et al., 2018; Moosavi-Dezfooli et al., 2019), we use the spectral norm of Hessian matrix to approximate local smoothness as an indicator of model robustness.

Original: Speciﬁcally, given an input x , the Hessian matrix H ( x ) is deﬁned as the second-order gradient of L ( x ) with respect to input x : H ( x )
Revised: Speciﬁcally, given an input x , the Hessian matrix H ( x ) is deﬁned as the second-order gradient of L ( x ) with respect to input x : H ( x )

Original: = ∇ 2 x L ( x ) .
Revised: = ∇ 2 x L ( x ) .

Original: The approximated local curvature C ( x ) at point x is thus deﬁned as:
Revised: The approximated local curvature C ( x ) at point x is thus deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where σ ( H ) is the spectral norm of H : σ ( H )
Revised: where σ ( H ) is the spectral norm of H : σ ( H )

Original: =
Revised: =

Original: sup v : (cid:107) v (cid:107) 2 =1 (cid:107)
Revised: sup v : (cid:107) v (cid:107) 2 =1 (cid:107)

Original: Hv (cid:107) 2 .
Revised: Hv (cid:107) 2 .

Original: Intuitively, C ( x ) measures the maximal directional curvature or change rate at x .
Revised: Intuitively, C ( x ) measures the maximal directional curvature or change rate at x .

Original: Thus, smaller C ( x ) indicates better local smoothness around x (Miyato et al., 2018;
Revised: Thus, smaller C ( x ) indicates better local smoothness around x (Miyato et al., 2018;

Original: Moosavi-Dezfooli et al., 2019).
Revised: Moosavi-Dezfooli et al., 2019).

Original: For the second difﬁculty, unlike previous fairness metrics where the target random variable 1 follows a Bernoulli distribution, the local curvature used in ER is a continuous random variable without a simple underlying distribution.
Revised: For the second difﬁculty, unlike previous fairness metrics where the target random variable 1 follows a Bernoulli distribution, the local curvature used in ER is a continuous random variable without a simple underlying distribution.

Original: The unknown distribution form makes it difﬁcult to directly measure the difference between the curvature distributions by a parametric statistic test (e.g., t-test or KL divergence).
Revised: The unknown distribution form makes it difﬁcult to directly measure the difference between the curvature distributions by a parametric statistic test (e.g., t-test or KL divergence).

Original: To tackle this problem, we utilize maximum mean discrepancy (MMD) (Gretton et al.,
Revised: To tackle this problem, we utilize maximum mean discrepancy (MMD) (Gretton et al.,

Original: 2012) to do a two-sample test on C ( X 1 ) and C ( X 2 ) .
Revised: 2012) to do a two-sample test on C ( X 1 ) and C ( X 2 ) .

Original: MMD is a distribution distance measure, agnostic to the exact distribution formulation and only based on the mean difference.
Revised: MMD is a distribution distance measure, agnostic to the exact distribution formulation and only based on the mean difference.

Original: Formally, our new fairness metric for equalized robustness is deﬁned as follows:
Revised: Formally, our new fairness metric for equalized robustness is deﬁned as follows:

Original: Our new fairness metric ∆ ER
Revised: Our new fairness metric ∆ ER

Original: Consider a machine learning model f trained on two groups of data X 1 and X 2 respectively.
Revised: Consider a machine learning model f trained on two groups of data X 1 and X 2 respectively.

Original: Suppose C ( X 1 ) ∼ P 1 and C ( X 2 ) ∼ P 2 , then the model’s ∆ ER is deﬁned as the squared maximum-mean-discrepancy (MMD) distance between C ( X 1 ) and C ( X 2 ) :
Revised: Suppose C ( X 1 ) ∼ P 1 and C ( X 2 ) ∼ P 2 , then the model’s ∆ ER is deﬁned as the squared maximum-mean-discrepancy (MMD) distance between C ( X 1 ) and C ( X 2 ) :

Original: ∆ ER = MMD 2 ( P 1 , P 2 ) .
Revised: ∆ ER = MMD 2 ( P 1 , P 2 ) .

Original: MMD is widely used to measure the distance between two high-dimensional distributions in deep learning (Li et al., 2015; 2017; Bi´nkowski et al., 2018).
Revised: MMD is widely used to measure the distance between two high-dimensional distributions in deep learning (Li et al., 2015; 2017; Bi´nkowski et al., 2018).

Original: The MMD distance between two distributions P and Q is deﬁned as
Revised: The MMD distance between two distributions P and Q is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where X ∼ P , Y ∼ Q and k ( · , · ) is the kernel function.
Revised: where X ∼ P , Y ∼ Q and k ( · , · ) is the kernel function.

Original: In practice, we use ﬁnite samples from P and Q to statistically estimate their MMD distance:
Revised: In practice, we use ﬁnite samples from P and Q to statistically estimate their MMD distance:

Original: [Equation]
Revised: [Equation]

Original: where { x i ∼ P} Mi =1 , { y j ∼ Q} Nj =1 , and we use the mixed RBF kernel function k ( x, y )
Revised: where { x i ∼ P} Mi =1 , { y j ∼ Q} Nj =1 , and we use the mixed RBF kernel function k ( x, y )

Original: = (cid:80) σ ∈ S e − (cid:107) x − y (cid:107) 2 2 σwith hyperparameter S
Revised: = (cid:80) σ ∈ S e − (cid:107) x − y (cid:107) 2 2 σwith hyperparameter S

Original: = { 1 , 2 , 4 , 8 , 16 } .
Revised: = { 1 , 2 , 4 , 8 , 16 } .

Original: Ablation studies on S values are conducted in Section 5.3.
Revised: Ablation studies on S values are conducted in Section 5.3.

Original:  C URVATURE M ATCHING : F AIR M ACHINE L EARNING TOWARDS E QUALIZED R OBUSTNESS 4. P RACTICAL C URVATURE A PPROXIMATION
Revised:  C URVATURE M ATCHING : F AIR M ACHINE L EARNING TOWARDS E QUALIZED R OBUSTNESS 4. P RACTICAL C URVATURE A PPROXIMATION

Original: In order to achieve equalized robustness, one intuitive solution is to add ∆ ER (Eq.
Revised: In order to achieve equalized robustness, one intuitive solution is to add ∆ ER (Eq.

Original: (5)) as an regularization term in the loss function during training phase.
Revised: (5)) as an regularization term in the loss function during training phase.

Original: However, it is non-practical to precisely calculate the spectral norm (which is equal to the absolute value of dominant eigenvalue) of Hessian matrix in ∆ ER .
Revised: However, it is non-practical to precisely calculate the spectral norm (which is equal to the absolute value of dominant eigenvalue) of Hessian matrix in ∆ ER .

Original: To solve this problem, we use a one-shot power iteration method (PIM) for practical approximation of C ( x ) during training.
Revised: To solve this problem, we use a one-shot power iteration method (PIM) for practical approximation of C ( x ) during training.

Original: First we rewrite C ( x ) with the following form:
Revised: First we rewrite C ( x ) with the following form:

Original: C ( x ) =
Revised: C ( x ) =

Original: σ ( H ( x ))
Revised: σ ( H ( x ))

Original: = (cid:107) H ( x ) v (cid:107) , where v is the dominant eigenvector with the maximal eigenvalue, which can be calculated by power iteration method.
Revised: = (cid:107) H ( x ) v (cid:107) , where v is the dominant eigenvector with the maximal eigenvalue, which can be calculated by power iteration method.

Original: In practice, to increase training efﬁciency, we use a one-shot power iteration method.
Revised: In practice, to increase training efﬁciency, we use a one-shot power iteration method.

Original: Speciﬁcally, we estimate the dominant eigenvector v by the gradient direction: ˜ v :=
Revised: Speciﬁcally, we estimate the dominant eigenvector v by the gradient direction: ˜ v :=

Original: sign ( g ) (cid:107) sign ( g ) (cid:107) ≈ v , where g = ∇ x L ( x ) .
Revised: sign ( g ) (cid:107) sign ( g ) (cid:107) ≈ v , where g = ∇ x L ( x ) .

Original: This is because previous works have observed a large similarity between the dominant eigenvector and the gradient direction (Miyato et al., 2018; Moosavi-Dezfooli et al., 2019).
Revised: This is because previous works have observed a large similarity between the dominant eigenvector and the gradient direction (Miyato et al., 2018; Moosavi-Dezfooli et al., 2019).

Original: We further approximate Hessian matrix by ﬁnite differentiation on gradients: H ( x ) v ≈ ∇ x L ( x + hv ) −∇ x L ( x ) h where h is a small constant.
Revised: We further approximate Hessian matrix by ﬁnite differentiation on gradients: H ( x ) v ≈ ∇ x L ( x + hv ) −∇ x L ( x ) h where h is a small constant.

Original: As a result, the ﬁnal approximation of curvature smoothness is
Revised: As a result, the ﬁnal approximation of curvature smoothness is

Original: [Equation]
Revised: [Equation]

Original: 4. C URVATURE M ATCHING
Revised: 4. C URVATURE M ATCHING

Original: With the practical curvature approximation, now we can match the curvature distribution of the two groups by minimizing the MMD distance.
Revised: With the practical curvature approximation, now we can match the curvature distribution of the two groups by minimizing the MMD distance.

Original: Suppose ˜ C
Revised: Suppose ˜ C

Original: ( X 1 ) ∼ Q 1 and ˜ C
Revised: ( X 1 ) ∼ Q 1 and ˜ C

Original: ( X 2 ) ∼ Q 2 , we deﬁne the curvature matching loss functions as:
Revised: ( X 2 ) ∼ Q 2 , we deﬁne the curvature matching loss functions as:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We add L cm to the traditional adversarially fair training (Ganin et al., 2016; Madras et al., 2018) loss function as a regularizer, in order to attain both in-distribution fairness and fair robustness.
Revised: We add L cm to the traditional adversarially fair training (Ganin et al., 2016; Madras et al., 2018) loss function as a regularizer, in order to attain both in-distribution fairness and fair robustness.

Original: As
Revised: As

Original: [Equation]
Revised: [Equation]

Original: illustrated in Figure 2, our model follows the same “two-head” structure as traditional adversarial learning frameworks (Ganin et al., 2016; Madras et al., 2018), where h t is the utility head for the target task, h a is the adversarial head to predict sensitive attributes, and f s is the shared backbone.
Revised: illustrated in Figure 2, our model follows the same “two-head” structure as traditional adversarial learning frameworks (Ganin et al., 2016; Madras et al., 2018), where h t is the utility head for the target task, h a is the adversarial head to predict sensitive attributes, and f s is the shared backbone.

Original: Suppose for each sample x i , the sensitive attribute is a i and the corresponding target label is y i , then our overall optimization problem can be written as:
Revised: Suppose for each sample x i , the sensitive attribute is a i and the corresponding target label is y i , then our overall optimization problem can be written as:

Original: [Equation]
Revised: [Equation]

Original: (10) where
Revised: (10) where

Original: [Equation]
Revised: [Equation]

Original: ( · , · ) is the cross-entropy loss function, α and γ are trade-off hyperparameters, and N is the number of training samples.
Revised: ( · , · ) is the cross-entropy loss function, α and γ are trade-off hyperparameters, and N is the number of training samples.

Original:  E XPERIMENTS 5. E XPERIMENTAL S ETUP
Revised:  E XPERIMENTS 5. E XPERIMENTAL S ETUP

Original: Datasets and pre-processing Experiments are conducted on three datasets widely used to evaluate machine learning fairness: Communities and Crime (C&C) (Redmond & Baveja, 2002), Adult (Kohavi, 1996), and CelebA (Liu et al., 2015).
Revised: Datasets and pre-processing Experiments are conducted on three datasets widely used to evaluate machine learning fairness: Communities and Crime (C&C) (Redmond & Baveja, 2002), Adult (Kohavi, 1996), and CelebA (Liu et al., 2015).

Original: 3 C&C dataset has 1,994 samples with neighborhood population statistics, where 1,500 are used for training and the rest for evaluation.
Revised: 3 C&C dataset has 1,994 samples with neighborhood population statistics, where 1,500 are used for training and the rest for evaluation.

Original: The target task is to predict violent crime per capita, and we use “RacePctBlack” (percentage of black population in the neighborhood) and “FemalePctDiv” (divorce ratio of female in the neighborhood) as sensitive attributes.
Revised: The target task is to predict violent crime per capita, and we use “RacePctBlack” (percentage of black population in the neighborhood) and “FemalePctDiv” (divorce ratio of female in the neighborhood) as sensitive attributes.

Original: All features in C&C dataset are of continous values in [0 , 1] .
Revised: All features in C&C dataset are of continous values in [0 , 1] .

Original: To ﬁt in the fairness problem setting, we binarilize the target and sensitive attributes with the top- 30% largest value as the threshold.
Revised: To ﬁt in the fairness problem setting, we binarilize the target and sensitive attributes with the top- 30% largest value as the threshold.

Original: 4 We also do data-whitening on C&C. Adult dataset has 48,842 samples with basic personal information such as education and occupation, where 30,000 are used for training and the rest for evaluation.
Revised: 4 We also do data-whitening on C&C. Adult dataset has 48,842 samples with basic personal information such as education and occupation, where 30,000 are used for training and the rest for evaluation.

Original: The target task is to predict the person’s annual income, and we use “gender” (male or female) as the sensitive attribute.
Revised: The target task is to predict the person’s annual income, and we use “gender” (male or female) as the sensitive attribute.

Original: The features in Adult dataset are of either continuous (e.g., age) or categorical (e.g. sex) values.
Revised: The features in Adult dataset are of either continuous (e.g., age) or categorical (e.g. sex) values.

Original: We use one-hot encoding on the categorical features and then concatenate them with the continuous ones.
Revised: We use one-hot encoding on the categorical features and then concatenate them with the continuous ones.

Original: We use data whitening on the concatenated features.
Revised: We use data whitening on the concatenated features.

Original: CelebA has over 200,000 images of celebrity faces, with 40 attribute annotations.
Revised: CelebA has over 200,000 images of celebrity faces, with 40 attribute annotations.

Original: The target task is to predict the “attractiveness” attribute and the sensitive attributes to protect are “chubby” and “eyeglasses”.
Revised: The target task is to predict the “attractiveness” attribute and the sensitive attributes to protect are “chubby” and “eyeglasses”.

Original: We randomly select 45 , 000 as training samples and 5 , 000 as testing samples.
Revised: We randomly select 45 , 000 as training samples and 5 , 000 as testing samples.

Original: All images are center-cropped and resized to 128 × 128 , and pixel values are scaled to [0 , 1] .
Revised: All images are center-cropped and resized to 128 × 128 , and pixel values are scaled to [0 , 1] .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Models For C&C and Adult datasets, we use two-layer MLPs for f s , h t and h a .
Revised: Models For C&C and Adult datasets, we use two-layer MLPs for f s , h t and h a .

Original: For CelebA dataset, we use ResNet18 as backbone, where the ﬁrst three stages are used as f s and the last stage (together with the fully connected classiﬁcation layer) is used as h t .
Revised: For CelebA dataset, we use ResNet18 as backbone, where the ﬁrst three stages are used as f s and the last stage (together with the fully connected classiﬁcation layer) is used as h t .

Original: The auxiliary adversarial head h a has the same structure as h t .
Revised: The auxiliary adversarial head h a has the same structure as h t .

Original: Detailed model structures are described in Appx.
Revised: Detailed model structures are described in Appx.

Original: A.
Revised: A.

Original: Baseline Methods We compare CUMA with the following state-of-the-art in-distribution fairness algorithms.
Revised: Baseline Methods We compare CUMA with the following state-of-the-art in-distribution fairness algorithms.

Original: Adversarial debiasing (AdvDebias) (Zhang et al., 2018) is one of the most popular fair training algorithm based on adversarial training (Ganin et al., 2016).
Revised: Adversarial debiasing (AdvDebias) (Zhang et al., 2018) is one of the most popular fair training algorithm based on adversarial training (Ganin et al., 2016).

Original: Madras et al.
Revised: Madras et al.

Original: (2018) proposes a similar framework termed Learned Adversarially Fair and Transferable Representations (LAFTR), by replacing the cross-entropy loss used in (Zhang et al., 2018) with a group-normalized (cid:96) 1 loss, which is shown to work better on highly unbalanced datasets.
Revised: (2018) proposes a similar framework termed Learned Adversarially Fair and Transferable Representations (LAFTR), by replacing the cross-entropy loss used in (Zhang et al., 2018) with a group-normalized (cid:96) 1 loss, which is shown to work better on highly unbalanced datasets.

Original: We also include normal (fairnessignorant) training as a baseline.
Revised: We also include normal (fairnessignorant) training as a baseline.

Original: Evaluation Metric We use three different groups of evaluation metrics: the overall accuracy, in-distribution fairness metrics, and robust fairness metrics.
Revised: Evaluation Metric We use three different groups of evaluation metrics: the overall accuracy, in-distribution fairness metrics, and robust fairness metrics.

Original: We report the overall accuracy on all test samples in the original test sets.
Revised: We report the overall accuracy on all test samples in the original test sets.

Original: To measure in-distribution fairness , we use ∆ EOpp and ∆ EO on the original test sets.
Revised: To measure in-distribution fairness, we use ∆ EOpp and ∆ EO on the original test sets.

Original: To measure robust fairness under distribution shifts, we use our newly proposed ∆ ER on the original test sets, and also ∆ EOpp and ∆ EO on a set of pre-deﬁned real-world distribution shifts.
Revised: To measure robust fairness under distribution shifts, we use our newly proposed ∆ ER on the original test sets, and also ∆ EOpp and ∆ EO on a set of pre-deﬁned real-world distribution shifts.

Original: We intend to show that ∆ ER calculated on the original test sets aligns well with robust fairness under real-world distribution shifts.
Revised: We intend to show that ∆ ER calculated on the original test sets aligns well with robust fairness under real-world distribution shifts.

Original: See the following paragraph for the details in constructing distributional shifts.
Revised: See the following paragraph for the details in constructing distributional shifts.

Original: Distributional shifts On Adult and C&C datasets, we construct two distribution shifts by adding random Gaussian and uniform noises, respectively, to the test data.
Revised: Distributional shifts On Adult and C&C datasets, we construct two distribution shifts by adding random Gaussian and uniform noises, respectively, to the test data.

Original: Speciﬁcally, following (Madras et al., 2018; Zhang et al., 2018), the categorical features in Adult and C&C datasets are ﬁrst one-hot encoded and then whitened into ﬂoat-value vectors, where noises are added.
Revised: Speciﬁcally, following (Madras et al., 2018; Zhang et al., 2018), the categorical features in Adult and C&C datasets are ﬁrst one-hot encoded and then whitened into ﬂoat-value vectors, where noises are added.

Original: Both types of noises have mean µ = 0 and has standard derivation σ = 0 .
Revised: Both types of noises have mean µ = 0 and has standard derivation σ = 0 .

Original: On CelebA dataset, following (Hendrycks & Dietterich, 2019), we construct two distribution shifts by adding random Gaussian (with mean µ = 0 and standard derivation σ = 0 . 08 ) and impulse noise (with ratio p = 0 . 03 ), respectively.
Revised: On CelebA dataset, following (Hendrycks & Dietterich, 2019), we construct two distribution shifts by adding random Gaussian (with mean µ = 0 and standard derivation σ = 0 . 08 ) and impulse noise (with ratio p = 0 . 03 ), respectively.

Original: We report the fairness in robustness against other settings of distribution shifts in Appx.
Revised: We report the fairness in robustness against other settings of distribution shifts in Appx.

Original: C.
Revised: C.

Original: Implementation Details Unless further speciﬁed, we set the loss trade-off parameter α to 1 in all experiments by default.
Revised: Implementation Details Unless further speciﬁed, we set the loss trade-off parameter α to 1 in all experiments by default.

Original: We use Adam optimizer (Kingma & Ba, 2014) with initial learning rate 10 − 3 and weight decay 10 − 5 .
Revised: We use Adam optimizer (Kingma & Ba, 2014) with initial learning rate 10 − 3 and weight decay 10 − 5 .

Original: The learning rate is gradually decreased to 0 by cosine annealing learning rate scheduler (Loshchilov & Hutter, 2016).
Revised: The learning rate is gradually decreased to 0 by cosine annealing learning rate scheduler (Loshchilov & Hutter, 2016).

Original: On both Adult and C&C datasets, we train for 50 epochs from scratch for all methods.
Revised: On both Adult and C&C datasets, we train for 50 epochs from scratch for all methods.

Original: On CelebA dataser, we ﬁrst normally train a model for 100 epochs, and then ﬁnetune it for 20 epochs using CUMA.
Revised: On CelebA dataser, we ﬁrst normally train a model for 100 epochs, and then ﬁnetune it for 20 epochs using CUMA.

Original: For fair comparison, we train for epochs on CelebA for all baseline methods.
Revised: For fair comparison, we train forepochs on CelebA for all baseline methods.

Original: The constant h in Eq.
Revised: The constant h in Eq.

Original: (8) is set to 1 by default.
Revised: (8) is set to 1 by default.

Original: For more implementation details, please check Appx.
Revised: For more implementation details, please check Appx.

Original: A.
Revised: A.

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original:  M AIN R ESULTS
Revised: 5. M AIN R ESULTS

Original: Experimental results on three datasets with different sensitive attributes are shown in Tables 3-5, where we compare CUMA with the baseline methods on three different groups of metrics as discussed in Section 5.1.
Revised: Experimental results on three datasets with different sensitive attributes are shown in Tables 3-5, where we compare CUMA with the baseline methods on three different groups of metrics as discussed in Section 5.1.

Original: “Normal” means standard training without any fairness regularization.
Revised: “Normal” means standard training without any fairness regularization.

Original: All numbers are shown as percentages.
Revised: All numbers are shown as percentages.

Original: Many intriguing ﬁndings can be concluded from the results.
Revised: Many intriguing ﬁndings can be concluded from the results.

Original: First, we see that previous state-of-the-art fairness learning algorithms would be jeopardized if distributional shifts are present in test data.
Revised: First, we see that previous state-of-the-art fairness learning algorithms would be jeopardized if distributional shifts are present in test data.

Original: For example, on C&C dataset with “RacePctBlack” as sensitive attribute (Table 1), LAFTR achieves ∆ EO = 28 .
Revised: For example, on C&C dataset with “RacePctBlack” as sensitive attribute (Table 1), LAFTR achieves ∆ EO = 28 .

Original: 83% on in-distribution test set, while that number is increased to 32 .
Revised: 83% on in-distribution test set, while that number is increased to 32 .

Original: 20% on the test set perturbed with uniform random noise.
Revised: 20% on the test set perturbed with uniform random noise.

Original: Similarly, for
Revised: Similarly, for

Original: AdvDebias, it achieves ∆ EO = 29 .
Revised: AdvDebias, it achieves ∆ EO = 29 .

Original: 73% on the original CelebA test set with “chubby” as the sensitive attribute (Table 4), while that number is increased to 35 .
Revised: 73% on the original CelebA test set with “chubby” as the sensitive attribute (Table 4), while that number is increased to 35 .

Original: 46% and 36 .
Revised: 46% and 36 .

Original: 48% on test sets perturbed with Gaussian and impulse noises, respectively.
Revised: 48% on test sets perturbed with Gaussian and impulse noises, respectively.

Original: Second, we see that CUMA achieves the best robust fairness under distribution shifts on all three benchmark datasets with different sensitive attribute settings, while maintaining similar indistribution fairness and overall accuracy.
Revised: Second , we see that CUMA achieves the best robust fairness under distribution shifts on all three benchmark datasets with different sensitive attribute settings, while maintaining similar indistribution fairness and overall accuracy.

Original: For example, on C&C dataset with “RacePctBlack” as the sensitive attribute (Table 1), CUMA achieves 2 .
Revised: For example, on C&C dataset with “RacePctBlack” as the sensitive attribute (Table 1), CUMA achieves 2 .

Original: 73% and 4 .
Revised: 73% and 4 .

Original: 82% less ∆ EO than the second-best performer (LAFTR) under distribution shifts by additive Gaussain and uniform noises, respectively.
Revised: 82% less ∆ EO than the second-best performer (LAFTR) under distribution shifts by additive Gaussain and uniform noises, respectively.

Original: Moreover, for the same experiment setting, although CUMA and LAFTR achieve almost identical in-distribution fairness (the difference between their ∆ EO on original test set is within 0 .
Revised: Moreover, for the same experiment setting, although CUMA and LAFTR achieve almost identical in-distribution fairness (the difference between their ∆ EO on original test set is within 0 .

Original: CUMA keeps (and even increases) the fairness under distribution shifts (e.g., 1 .
Revised: CUMA keeps (and even increases) the fairness under distribution shifts (e.g., 1 .

Original: 33% smaller ∆ EO under uniform noises), while the fairness achieved by LAFTR is jeopardized under both types of distribution shifts (e.g., 3 . 37% larger ∆ EO under uniform noises).
Revised: 33% smaller ∆ EO under uniform noises), while the fairness achieved by LAFTR is jeopardized under both types of distribution shifts (e.g., 3 . 37% larger ∆ EO under uniform noises).

Original: Similarly, on CelebA dataset with “Chubby” as the sensitive attribute, LAFTR has even slightly better in-distribution fairness than CUMA.
Revised: Similarly, on CelebA dataset with “Chubby” as the sensitive attribute, LAFTR has even slightly better in-distribution fairness than CUMA.

Original: However, when the test sets have distribution shifts, the fairness achieved by LAFTR is jeopardized (with 5 .
Revised: However, when the test sets have distribution shifts, the fairness achieved by LAFTR is jeopardized (with 5 .

Original: 50% and 6 .
Revised: 50% and 6 .

Original: 02% more ∆ EO under Gaussian and uniform noises, respectively), while CUMA keeps its fairness and achieves better fairness under distribution shifts (e.g., 2 .
Revised: 02% more ∆ EO under Gaussian and uniform noises, respectively), while CUMA keeps its fairness and achieves better fairness under distribution shifts (e.g., 2 .

Original: 50% and 3 .
Revised: 50% and 3 .

Original: 17% less ∆ EO compared with LAFTR.).
Revised: 17% less ∆ EO compared with LAFTR.).

Original: Third, for all three datasets, the ∆ ER calculated on the original test set highly correlates with traditional fairness metrics (e.g., ∆ EOpp , ∆ EO ) calculated on the perturbed test sets: the smaller ∆ ER on the in-distribution test set, the smaller ∆ EO on perturbed test sets.
Revised: Third, for all three datasets, the ∆ ER calculated on the original test set highly correlates with traditional fairness metrics (e.g., ∆ EOpp , ∆ EO ) calculated on the perturbed test sets: the smaller ∆ ER on the in-distribution test set, the smaller ∆ EO on perturbed test sets.

Original: This shows that our new metric ∆ ER aligns well with robust fairness under real-world distribution shifts, and validates the rationality of using it as an indicator of model robustness discrimination.
Revised: This shows that our new metric ∆ ER aligns well with robust fairness under real-world distribution shifts, and validates the rationality of using it as an indicator of model robustness discrimination.

Original: More experimental results are shown in Appx.
Revised: More experimental results are shown in Appx.

Original: B (trade-off curves between fairness and accuracy) and Appx.
Revised: B (trade-off curves between fairness and accuracy) and Appx.

Original: C (results on other settings of distributional shifts).
Revised: C (results on other settings of distributional shifts).

Revised: [Table]

Original: 5. A BLATION S TUDY Ablation Study on ∆ ER
Revised: 5. A BLATION S TUDY Ablation Study on ∆ ER

Original: In this section, we study how well can the ∆ ER predict the robust fairness and the sensitivity of ∆ ER with respect to S (the sampling set for σ in the mixed RBF kernel function, as described in Section 3).
Revised: In this section [Table] we study how well can the ∆ ER predict the robust fairness and the sensitivity of ∆ ER with respect to S (the sampling set for σ in the mixed RBF kernel function, as described in Section 3) [Table]

Original: A small σ will make the ∆ ER more sensitive to the difference between the two sample set, which could be caused by either the true discrepancy of distributions or the different noise introduced by sampling.
Revised: A small σ will make the ∆ ER more sensitive to the difference between the two sample set, which could be caused by either the true discrepancy of distributions or the different noise introduced by sampling

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: In contrast, a larger ones may under-estimate the discrepancy.
Revised: In contrast, a larger ones may under-estimate the discrepancy.

Original: Thus, a proper S should include a wide range of σ to avoid the domination of either large or small values.
Revised: Thus, a proper S should include a wide range of σ to avoid the domination of either large or small values.

Original: In this paper, we choose a geometric sequence with 2 as the base, i.e., S = { 1 , 2 , 4 , 8 , 16 } .
Revised: In this paper, we choose a geometric sequence with 2 as the base, i.e., S = { 1 , 2 , 4 , 8 , 16 } .

Original: Furthermore, we compare ∆ ER values under three different sets: S 1 = { 0 . 25 , 0 .
Revised: Furthermore, we compare ∆ ER values under three different sets: S 1 = { 0 . 25 , 0 .

Original: 5 , 1 , 2 , 4 } , S 2 = { 1 , 2 , 4 , 8 , 16 } (the default S as deﬁned in Section 3), and S 3 = { 4 , 8 , 16 , 32 , 64 } .
Revised: 5 , 1 , 2 , 4 } , S 2 = { 1 , 2 , 4 , 8 , 16 } (the default S as deﬁned in Section 3), and S 3 = { 4 , 8 , 16 , 32 , 64 } .

Original: Results are shown in
Revised: Results are shown in

Original: Table 6.
Revised: Table 6.

Original: As in Section 5.2, we empirically evaluate the robust fairness by ∆ EO on the test set corrupted by uniform noise.
Revised: As in Section 5.2, we empirically evaluate the robust fairness by ∆ EO on the test set corrupted by uniform noise.

Original: From the results, we observe that with all three different S settings, ∆ ER aligns well with the model fairness under distribution shifts ( ∆ EO under uniform noise).
Revised: From the results, we observe that with all three different S settings, ∆ ER aligns well with the model fairness under distribution shifts ( ∆ EO under uniform noise).

Original: [Table]
Revised: [Table]

Original: Ablation Study on CUMA In this section, we check the sensitivity of
Revised: Ablation Study on CUMA In this section, we check the sensitivity of

Original: CUMA with respect to its hyperparameters: the loss trade-off parameters α and γ in Eq.
Revised: CUMA with respect to its hyperparameters: the loss trade-off parameters α and γ in Eq.

Original: (10) and h in Eq.
Revised: (10) and h in Eq.

Original: Results are shown in Table 7.
Revised: Results are shown in Table 7.

Original: When ﬁxing γ = 1 , ∆ ER peeks at around α = 1 , so we use it as the default α value.
Revised: When ﬁxing γ = 1 , ∆ ER peeks at around α = 1 , so we use it as the default α value.

Original: When ﬁxing α = 1 , the best trade-off between overall accuracy and robust fairness is achieved at round γ = 1 , which we use as the default γ .
Revised: When ﬁxing α = 1 , the best trade-off between overall accuracy and robust fairness is achieved at round γ = 1 , which we use as the default γ .

Original: Varying the value of h hardly affects the performance of CUMA.
Revised: Varying the value of h hardly affects the performance of CUMA.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we ﬁrst propose a new fairness goal, termed Equalized Robustness (ER), to impose fair model robustness against unseen distribution shifts across different data groups.
Revised: In this paper, we ﬁrst propose a new fairness goal, termed Equalized Robustness (ER), to impose fair model robustness against unseen distribution shifts across different data groups.

Original: We further propose a novel fairness learning algorithm, termed Curvature Matching (CUMA), to simultaneously achieve both traditional in-distribution fairness and our new robust fairness.
Revised: We further propose a novel fairness learning algorithm, termed Curvature Matching (CUMA), to simultaneously achieve both traditional in-distribution fairness and our new robust fairness.

Original: Experiments show
Revised: Experiments show

Original: CUMA achieves superior fairness in robustness against distribution shifts, without more sacriﬁce on either overall accuracies or the in-distribution fairness compared with traditional in-distribution fair learning methods.
Revised: CUMA achieves superior fairness in robustness against distribution shifts, without more sacriﬁce on either overall accuracies or the in-distribution fairness compared with traditional in-distribution fair learning methods.

Original: As a pioneer work, the new concept of ER proposed in this paper aims to measure a new dimension of fairness that is practically signiﬁcant yet so far largely overlooked: ER assesses “out-of-distribution” fairness while previous metrics focus on “in-distribution” fairness.
Revised: As a pioneer work, the new concept of ER proposed in this paper aims to measure a new dimension of fairness that is practically signiﬁcant yet so far largely overlooked: ER assesses “out-of-distribution” fairness while previous metrics focus on “in-distribution” fairness.

Original: Therefor,
Revised: Therefor,

Original: ER works as a complement instead of a replacement for previous fairness metrics.
Revised: ER works as a complement instead of a replacement for previous fairness metrics.

Original: We hope our work can open up more discussions on how to evaluate model fairness in a more complete spectrum.
Revised: We hope our work can open up more discussions on how to evaluate model fairness in a more complete spectrum.

Original: A M ORE I MPLEMENTATION D ETAILS
Revised: A M ORE I MPLEMENTATION D ETAILS

Original: On C&C and Adult datsets, suppose the input feature dimension is d , then the dimensions of hidden layers in f s and h t are d → 100 → 64 and 64 → 32 → 2 , respectively.
Revised: On C&C and Adult datsets, suppose the input feature dimension is d , then the dimensions of hidden layers in f s and h t are d → 100 → 64 and 64 → 32 → 2 , respectively.

Original: h a has identical model structure with h t .
Revised: h a has identical model structure with h t .

Original: For all three sub-networks, ReLU activation function and dropout layer with 0 .dropout ratio are applied between the two fully connected layers.
Revised: For all three sub-networks, ReLU activation function and dropout layer with 0 .dropout ratio are applied between the two fully connected layers.

Original: On CelebA dataset, we use the
Revised: On CelebA dataset, we use the

Original: ResNet18 as backbone.
Revised: ResNet18 as backbone.

Original: The input feature size of h t and h a is 8 × 8 × 256 (with channel-last layout).
Revised: The input feature size of h t and h a is 8 × 8 × 256 (with channel-last layout).

Original: B T RADE - OFF C URVES BETWEEN F AIRNESS AND A CCURACY
Revised: B T RADE - OFF C URVES BETWEEN F AIRNESS AND A CCURACY

Original: For CUMA and both baseline methods, we can obtain different trade-offs between fairness and accuracy by setting the loss function weights (e.g., α and γ ) to different values.
Revised: For CUMA and both baseline methods, we can obtain different trade-offs between fairness and accuracy by setting the loss function weights (e.g., α and γ ) to different values.

Original: For example, the larger α , the better fairness and the worse accuracy.
Revised: For example, the larger α , the better fairness and the worse accuracy.

Original: Such trade-off curves between fairness and accuracy of different methods are shown in Figure 3.
Revised: Such trade-off curves between fairness and accuracy of different methods are shown in Figure 3.

Original: The closer the curve to the top-left corner (i.e., with larger accuracy and smaller ∆ EO ), the better Pareto frontier is achieved.
Revised: The closer the curve to the top-left corner (i.e., with larger accuracy and smaller ∆ EO ), the better Pareto frontier is achieved.

Original: As we can see, our method achieves the best Pareto frontiers for both in-distribution fairness (left panel) and robust fairness under distribution shifts (middle and right panel).
Revised: As we can see, our method achieves the best Pareto frontiers for both in-distribution fairness (left panel) and robust fairness under distribution shifts (middle and right panel).

Original: C R ESULTS ON O THER S ETTINGS OF D ISTRIBUTIONAL S HIFTS
Revised: C R ESULTS ON O THER S ETTINGS OF D ISTRIBUTIONAL S HIFTS

Original: [Table] [Table]
Revised: [Table] [Table]

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we show that the conclusions drawn in Section 5.2 hold under different settings of distributional shifts.
Revised: we show that the conclusions drawn in Section 5.2 hold under different settings of distributional shifts.

Original: Speciﬁcally, we consider a new noise setting with mean µ = 0 and standard derivation σ = 0 .
Revised: Speciﬁcally, we consider a new noise setting with mean µ = 0 and standard derivation σ = 0 .

Original: 06 (other than the mean µ = 0 and standard derivation σ = 0 .
Revised: 06 (other than the mean µ = 0 and standard derivation σ = 0 .

Original: 03 evaluated in the main text) for both random Gaussian and uniform noises.
Revised: 03 evaluated in the main text) for both random Gaussian and uniform noises.

Original: The results under these new distributional shifts on C&C dataset are shown in Tables 8 and 9.
Revised: The results under these new distributional shifts on C&C dataset are shown in Tables 8 and 9.