Paper: Test Relative Fairness in Human Decisions With Machine Learning (ICLR-2024)

Reviewer comment (Evaluation):
The premise and contribution of the paper is unclear to me. Specifically, the utility and application of the proposed metrics need more clarification. As I understand relative fairness is useful when comparing two decision sets. But if both decision sets are biased and the relative fairness is near zero, what does that tell us?

Author response:
Weakness 1: We have added Subsection 1.1 to show the potential applications of the proposed relative fairness definition and the estimation algorithms. We also rewrote Subsection 1.2 to clarify the contributions. We agree with the reviewer that it is possible to have two decision sets being relatively fair with each other when neither one is “absolutely” fair. In our opinion, this is perfectly fine and it does tell us that decision makers behind these two decision sets are consistent (if they are biased, they are biased in the same way). Because of the difficulty of specifying a standard for “absolute” fairness (as highlighted in Paragraph 2 Page 1), relative fairness is more general.

Reviewer comment (Evaluation):
Relative fairness as presented in the paper is simply the difference between statistical parity (with some normalization) of each decision set. Why can't we just rely on the absolute statistical parity? It provides a more granular and absolute measure of fairness. Why do we need relative fairness or what extra information does that provide us?

Author response:
Weakness 2: As highlighted in the last paragraph of Section 3.2 of the revised paper, we show that relative fairness is related to but not the same as the difference between statistical parity. Consider the following example: 

A   = [1,1,1,1,1]+[0,0,0,0,0,0]

Y0 = [1,1,0,0,0]+[1,1,1,0,0,0]

Y1 = [1,1,1,0,0]+[1,0,0,0,0,0]

Y2 = [0,0,1,1,1]+[0,0,0,0,1,0]

SP(Y0) = 2 / 5 - 3 / 6 = -0.1

SP(Y1) = SP(Y2) = 3 / 5 - 1 / 6 = 0.43

The difference between SP of Y1, Y0 is the same as the difference between SP of Y2, Y0 and is quite large -0.53.
However, when tested for relative fairness, the p value for RBT(Y0, Y1) is 0.05 which means the differences between Y0 and Y1 are significantly different across A = 0 and A=1. But the p value for RBT(Y0, Y2) is 0.20 which means the differences between Y0 and Y2 are not significantly different across A = 0 and A=1. There is not enough evidence to draw the conclusion that Y0 and Y2 are relatively biased.

We would prefer to call the relative fairness notion as difference parity since it focuses on analyzing the statistical parity of the decision differences (as added and highlighted in Section 3.1).

Reviewer comment (Evaluation):
It is unclear to me why ML models are being used to approximate the decisions on an unlabeled set. If a small set of human decisions is available, why can't we just evaluate the absolute fairness of that set? The function f can be prone to a range of implicit biases of its own and the results of the proposed metrics can heavily rely on the parameterization of f.

Author response:
Weakness 3: As we have shown in Weakness 2, the relative fairness definition is different from the difference between statistical parity. Therefore, we need to estimate the statistics of the differences before calculating for the relative bias metrics when decisions are made on different data. The concern of f being prone to a range of implicit biases of its own will affect the Unbiased Bridge algorithm but will be alleviated by the Biased Bridge algorithm. We have rewritten Section 3.3 to better describe how Biased Bridge alleviates such implicit biases. This is also why empirically, Biased Bridge performed better than Unbiased Bridge.

Reviewer comment (Novelty):
One of the use cases of relative fairness is when the fairness of the decisions suddenly changes over time. Even then it is unclear to me how relative fairness is useful over standard statistical parity.

Author response:
Weakness 4: In terms of statistical parity vs relative fairness, we claim that 
(1) statistical parity is a notion for “absolute” fairness on one decision set; it requires the acceptance rates to be the same across different sensitive groups. When the actual acceptance rates are different across different sensitive groups, statistical parity can be misleading. On the other hand, relative fairness does not have any assumption on what is considered to be “absolutely” fair, thus can be more general.
(2) the difference between statistical parity cannot reflect the statistical parity information of every decision difference like the relative fairness does. Thus we believe that in terms of evaluating relative fairness, our definition is better than just the difference between statistical parity.

Reviewer comment (Presentation):
Table 3 and 4 are quite hard to read. What do the values in brackets indicate? Are those the p-values?

Author response:
Question 1: Table 3 and 4 are quite hard to read. 
The values in brackets indicate the p values from the null hypothesis test of RBT. To make the tables easier to understand, we added some descriptions of the tables highlighted in Section 5 Page 7.

Reviewer comment (Presentation):
The examples and general presentation of the paper are not easy to follow. For example, the authors mention various non-complementary examples, which doesn't help drive the idea home. The annotator bias example points more towards the setting of different (ideologically/demographically) decision-makers and the same dataset (e.g., platform content) or different datasets (e.g., different sub-reddit data/ different parts of X). On the other hand, the motivating example looks at multiple experts (>2, as is in the methodology) who are similar (ideologically) on the same dataset (similar features selected by a hiring company.). In general, the examples don't support or complement the methodology. Authors should pick an intuitive running example that supports the main idea.

Author response:
Weakness 1: the examples and presentation of the paper. 
In the revised paper, we have replaced the previous motivating example with Subsection 1.1 Motivation. This subsection now clearly describes the example application scenarios of the proposed relative fairness evaluation and estimation algorithms.

Reviewer comment (Novelty):
Although the authors cite some relevant literature, the literature review/related work section needs improvement. The section should be more cohesive and concise and clearly indicate the paper's strength and position in related works.

Author response:
Weakness 4: literature review.
We have added some advantages of the proposed relative fairness over the existing ``absolute’’ fairness definitions in Section 2.

Reviewer comment (Novelty):
The way authors mention their contributions is not complementary to the idea and their work. The authors should restructure this section and make it stronger. Additionally, the authors mention that they show that a machine-learning model trained on human decisions can inherit bias/preference. This is not novel and is the motivation for most of the fairness in ML work. For example, several works on feedback loops and bail/justice systems show this.

Author response:
Weakness 5: contributions.
We have rewritten the contribution in Section 1.2 to highlight the novelty and contribution of the proposed relative fairness definition and the estimation algorithm. We have dropped the part of “a machine-learning model trained on human decisions can inherit bias/preference.”

Reviewer comment (Theory):
Since authors define relative fairness as the statistical difference of the delta of two sets of decisions over a certain sensitive attribute, would it be reasonable to assume absolute fairness is particular to one decision set? Does it mean absolute fairness is a sub-routine of relative fairness?

Author response:
Weakness 6: absolute fairness as a sub-routine of relative fairness?
We agree with the reviewer that (at least in this paper), absolute fairness is a concept considering one specific decision set. However, we do not completely understand what it means for absolute fairness being a sub-routine of relative fairness. We think relative fairness is about comparing two decision sets. The two decision sets can be relatively fair with each other even when either is not absolutely fair. Please clarify whether this answers your question.

Reviewer comment (Novelty):
The reviewer struggles to see the two proposed algorithms as "novel" as the paper has claimed. Essentially the paper trains a predictor to impute missing labels.

Author response:
Weakness 2b. We agree that all the math in this paper is not new. However, we do believe this paper has a significant novelty and contribution--- (1) before this, there is no fairness definition based on the differences of two decision sets; (2) the biased bridge algorithm is not just label imputation. We have revised the contributions and the entire Methodology section to highlight these.

Reviewer comment (Novelty):
The reviewer fails to see a clear main contribution of the paper: the formulation and ideas are not entirely novel; the theoretical result (Prop 4.5) is very weak; and the experimental results also need improvement.

Author response:
Weakness 4a. (1) we have revised the motivating example to Section 1.1 Motivations and provide two example scenarios. These new examples now discuss the potential use cases of the proposed relative fairness notion and algorithm when reference labels are not available. (2) In the Adult experiment, since only one set of labels is available, we treat it as a reference set and compare it against a synthetic label set. In the SCUT experiment, there are labels from different raters, we calculated the relative fairness between every pair. To evaluate the estimations of the biased bridge and unbiased bridge, we used the relative bias calculated with the actual ratings on both the training and test set Y1(X1)-Y0(X1), Y1(X0)-Y0(X0). In practice, Y0(X1) and Y1(X0) will be unavailable and the biased bridge algorithm will be applied to estimate the relative bias RBT(Y1,Y0) and RBD(Y1,Y0).

Reviewer comment (Theory):
Assumption 4.2 is considered quite problematic and unjustified. The assumption is on a complicated quantity (Y_\Delta(A=a)), and the reviewer fails to understand what this assumption fundamentally means. There is particular struggle with Eq. 6, where normality is claimed without having any assumption on the function f or the classifiers (Y0 and Y1). If it is based on the law of large numbers, the source of randomness needs to be defined. The assumption of large samples also in a certain sense defeats the purpose of label imputation in the "unbiased bridge" algorithm, since one could just compute and compare data statistics on (X1, Y1) and (X2, Y2) separately.

Author response:
Weakness 3ab. Inspired by the comments, we have dropped the assumption of normal distribution. The only assumption is that the differences of the two decision sets are i.i.d.. In this case, the sampled means of the differences should follow normal distribution based on the central limit theorem and the law of large numbers. The new definition is still different from just analyzing the statistics of the two decision sets individually DP(Y0)-DP(Y1). This is because it also takes into consideration the variance of the differences (Y0-Y1) and thus the biased bridge algorithm is still necessary. We also used an example to show the difference in our reply to Weakness 2 of Reviewer oWWd (the third reviewer).

Reviewer comment (Writing):
Minor mathematical issues: In Def 4.3, the mean and variance should be called "empirical mean" and "empirical variance". It should be made clear early in Sec 4 that Y is binary {0, 1}. In Eq. 9, the meaning of \overline{f} should be clarified.

Author response:
Weakness 3d. We have rewritten the Methodology section with those minor issues fixed. Note that in the definition, we clarify that Y0, Y1 $\in$ R and all the definitions and algorithms apply to not just binary classification but also regression problems (the SCUT face beauty experiment is also a regression problem).

Reviewer comment (Experiments):
There is a non-trivial gap between the experimental setup and the setting that the algorithms are introduced in. The problem is motivated by bypassing the need to define "true labels". However, the experiment still simulates settings that have these true labels. This also occurs in Step 0 in the motivating example of Section 3.

Author response:
Weakness 4a. (1) we have revised the motivating example to Section 1.1 Motivations and provide two example scenarios. These new examples now discuss the potential use cases of the proposed relative fairness notion and algorithm when reference labels are not available. (2) In the Adult experiment, since only one set of labels is available, we treat it as a reference set and compare it against a synthetic label set. In the SCUT experiment, there are labels from different raters, we calculated the relative fairness between every pair. To evaluate the estimations of the biased bridge and unbiased bridge, we used the relative bias calculated with the actual ratings on both the training and test set Y1(X1)-Y0(X1), Y1(X0)-Y0(X0). In practice, Y0(X1) and Y1(X0) will be unavailable and the biased bridge algorithm will be applied to estimate the relative bias RBT(Y1,Y0) and RBD(Y1,Y0).

Reviewer comment (Evaluation):
The paper needs to describe if multiple hypothesis testing is accounted for in the experiments.

Author response:
Weakness 4c. We clarified in Section 5 that all experiments happened once.

Reviewer comment (Presentation):
The experimental results could be better presented by figures, as the tables are very dense to parse.

Author response:
Weakness 4d. We failed to find a better way to present the results with figures--- mainly because we still want to present those numbers. However, we clarified the meaning of the colors in the tables in Section 5 Paragraph 1 so that it is possible for the readers to only look at the colors of the tables.

Reviewer comment (Writing):
Grammar issues:
- Title: "test" -> testing
- Abstract: "decisions sets"
- P2: "what are absolutely fair decisions" -> what absolutely fair decisions are
- P2: "present" -> presence
- P2: "Homan and others" -> refer to papers by the first author
- P7: "least beautify"

Author response:
Weakness 5. We have fixed all the grammar listed.