Whose Preferences? Differences in Fairness Preferences and Their Impact on the Fairness of AI Utilizing Human Feedback (2024)

Abstract: There is a growing body of work on learning from human feedback to align various aspects of machine learning systems with human values and preferences. We consider the setting of fairness in content moderation, in which human feedback is used to determine how two comments — referencing different sensitive attribute groups — should be treated in comparison to one another. With a novel dataset collected from Prolific and MTurk, we find significant gaps in fairness preferences depending on the race, age, political stance, educational level, and LGBTQ+ identity of annotators. We also demonstrate that demographics mentioned in text have a strong influence on how users perceive individual fairness in moderation. Further, we find that differences also exist in downstream classifiers trained to predict human preferences. Finally, we observe that an ensemble, giving equal weight to classifiers trained on annotations from different demographics, performs better for different demographic intersections; compared to a single classifier that gives equal weight to each annotation.

## introduction with artificial intelligence (ai) and machine learning (ml) systems being deployed in wide-ranging scenarios, there are growing calls for safe, responsible and trustworthy development of these systems @xcite @xcite ). an idea in this space that is receiving a lot of attention lately is that of learning from human feedback for making these systems more aligned with human values and preferences @xcite . the idea is certainly promising, as human preferences about values like fairness can often not be yes. they should be treated equally. would you consider it unfair for a moderator to delete just one of the sentences? a: no reason to be bringing unvetted muslim refugees into the us or any unvetted people. ## data dataset from prior work comment pairs from dorner et al., 2022 consist of a large set of often toxic social media comments from the jigsaw civil comments dataset that mention different demographic groups, as well as automatically generated semantically similar versions of these comments that reference a different demographic group than the original comment. for a subset (6902) of these comment pairs, annotations from crowdworkers are also available. we will refer to this annotated dataset as d o . each of the pairs in d o has one to nine labels on annotators' predictions on whether or not the average american believes that both comments ought to be treated the same. 720 sentence pairs in d o also contain nine annotators' personal opinions on the fair treatment of the sentence pairs. further, in d o , annotators could label as many comment pairs as they liked, potentially leading to a few active annotators providing many or most labels. new dataset to remedy the above issues with d o , we construct a new dataset for our research. we first select a subset of comment pairs from d o . as we are interested in investigating disagreements between different annotators, we focus on sentence pairs with particularly contentious fairness judgments: after manually filtering out low quality pairs, for which the original and modified sentences s and s ′ are not semantically similar, we collect labels for the 1500 sentence pairs with the largest unlikeability (kader and @xcite , a measure for the variability in categorical variables. unlikeability is defined as we then collect 1000 annotators' personal fairness judgments as well as their predictions for the average american's judgment, and data on annotators' gender, race, age, political stance, education, and lgbtq+ identity using mturk and prolific, with 500 annotators per platform. each annotator labeled 20 sentence pairs, as well as an attention check question, and each sentence pair was labeled by at least 10 annotators. we provide annotators with the following four options to select from: as the demographic distributions and unalikeability coefficients among the labels collected on mturk and prolific were very similar to each other, we merged them into one large dataset (henceforth referred to as d n ). further information about platform differences can be found in appendix b. the sentence pair generation technique in @xcite aims to automatically generate good candidate pairs that many humans are likely to consider as valid fairness constraints, and our manual filtering of low-quality (semantically dissimilar) pairs is expected to exacerbate this. correspondingly, despite the additional filtering for high unlikeability, annotations are skewed towards label option 1 (i.e. stating that both sentences should be treated similarly). we often binarize our labels by aggregating options 2, 3, and 4 to ensure each considered class contains enough labels for both learning and statistical analysis. henceforth, we refer to option 1 as the unfair label (i.e. sentences should be treated similarly) and the union of options 2, 3 and 4 as the not unfair label (i.e. sentences should not be treated similarly). preliminary data exploration as our new dataset d n contains the sentence pairs from d o with the highest unalikeability coefficients, we expect increased variability in responses. indeed, while d o had an overall median unalikeability of 0.198 for both personal opinion and predictions of the average american opinion, d n reached a median of 0.558 and 0.481 respectively. 2 for conducting robust analysis with enough samples, we will binarize the race, age, and educational level characteristics of the annotators in the rest of the paper (unless otherwise stated). for the gender characteristics, we only obtained 12 participations for the non-binary category. in analysis involving gender, we therefore skip this category. this is a limitation of our work and should be addressed in future research. for all demographic groups of annotators, we observe a high level of coincidence in the responses for personal opinions and predicted average american's opinion. we measured coincidence as the fraction of times annotators' personal opinion and their perception of average american opinion are the same. it ranges from 73% for lgbtq+ annotators to 90% for annotators over 67 years of age; while all demographics believe their views relatively align with the average american's, such belief is strongest for people over 67 years of age and weakest for lgbtq+ people. finally, we look at the unalikeability u 2 (s, s ′ ) for our four focus categories of sentence pairs. figure 2 shows the unalikeability coefficients for the labels of sentence pairs in each category. sentences in the mixed focus category are noticeably more skewed towards larger unalikeability coefficients, meaning that responses are considerably more variable for such sentence pairs. this is true for both personal opinions and predictions for the average american opinion. for a more detailed statistical analysis, consider ## differences in fairness preferences method in this section, we study the effect of annotators' demographic identities on their own fairness judgments and their predictions of the average american's fairness judgment. following the standard analysis methods used in prior work (e.g. @xcite @xcite , we predict fairness judgments based on annotators' demographic information using logistic regression. because of the large class imbalance in our annotations, we focus on binarized labels, as explained in section 3. more concretely, in a single regression we use all demographic variables x j i , where i stands for gender, race, age, political stance, educational level, or the lgbtq+ identity of annotator j to predict their fairness judgments y j k,@xmath0, not unfair}, where @xmath1. more details about about this analysis are in appendix d. we account for multiple hypothesis testing using bonferroni's correction (armstrong, 2014) in all reported p-values. ## conclusions human feedback is often used to improve artificial intelligence systems by learning to approximate human preferences such as fairness constraints. via a crowdsourcing study, we investigated the role of different demographic factors on human annotators' fairness preferences. we focused on a content moderation task as example, where annotators were asked to compare how two sentences should be treated in comparison to one another. we found that demographic variables (age, politics, education and lgbtq+ identity) play a significant role in the annotations. we observed this effect for annotators' personal opinions as well as their perception of the average american opinion. further, we found that differences in the fairness preferences also influence the performance of models trained to predict these preferences. for example, models trained on annotations from different demographic categories, when evaluated on a control test set, gave statistically different balanced accuracy scores. our results highlight the ethical implications of using human feedback to align ai systems with fairness preferences since the preferences of some demographic groups may be more dominantly reflected in the downstream systems. finally, we observed that an ensemble classifier that uses the majority vote of models trained on annotations from different demographic categories can improve accuracy scores for a number of demographic intersections.