When Do Annotator Demographics Matter? Measuring the Influence of Annotator Demographics with the POPQUORN Dataset (2023)

Abstract: Annotators are not fungible. Their demographics, life experiences, and backgrounds all contribute to how they label data. However, NLP has only recently considered how annotator identity might influence their decisions. Here, we present POPQUORN (the Potato-Prolific dataset for Question-Answering, Offensiveness, text Rewriting and politeness rating with demographic Nuance). POPQUORN contains 45,000 annotations from 1,484 annotators, drawn from a representative sample regarding sex, age, and race as the US population. Through a series of analyses, we show that annotators’ background plays a significant role in their judgments. Further, our work shows that backgrounds not previously considered in NLP (e.g., education), are meaningful and should be considered. Our study suggests that understanding the background of annotators and collecting labels from a demographically balanced pool of crowd workers is important to reduce the bias of datasets. The dataset, annotator background, and annotation interface are available at https://github.com/Jiaxin-Pei/potato-prolific-dataset .

## introduction supervised machine learning relies heavily on datasets with high-quality annotations and data labeling has long been an integral part of the machine learning pipeline @xcite . while recent large language models show promising performances on many zero-shot and few-shot nlp tasks @xcite , reinforcement learning with human feedback (rlhf), the core technology behind these models also heavily relies on largescale and high-quality human annotations @xcite . therefore, how to curate high-quality labeled datasets is one of the most important questions for both academia and industry. crowdsourcing is actively used as one of the major approaches to collect human labels for various nlp and ml tasks. early studies on crowdsourcing nlp datasets suggest that crowd workers are able to generate high-quality labels even for relatively difficult tasks and with relatively low costs @xcite . however, other studies also suggest that collecting high-quality annotations from crowdsourcing platforms is challenging and requires rounds of iterations to create a reliable annotation pipeline @xcite . annotation quality has typically been measured by proxy through inter-annotator agreement (iaa) metrics like krippendorff's α @xcite or cohen's κ @xcite . to attain higher iaa, researchers usually conduct pilot studies or rounds of annotator training to attain higher agreement among annotators. while such a method generally works in settings like part of speech tagging, the use of iaa as a proxy for quality implicitly assumes that the task has real ground truth and disagreements are mistakes. however, annotations for subjective tasks presents a far more challenging setting @xcite ; and as nlp and ml models are more frequently used in social settings where single true answer may not naturally exists, using iaa as the single metric for data quality can be problematic or can even create social harm. for example, @xcite studies how annotators' identity and prior belief affect their ratings on language toxicity and found significant gender and race differences in rating toxic language. other studies also suggest that disagreement in annotations can also be due to the inherent contextual ambiguity @xcite @xcite which can also be leveraged to improve the model performances @xcite . despite multiple studies on annotator background and disagreement @xcite , a systematic study on through our analysis, we find that demographic background is significantly associated with people's ratings and performance on all four taskseven for a more objective task such as reading comprehension. for example, people with higher levels of education perform better on the questionanswering task and black or african american participants tend to rate the same email as more polite and the same comment as more offensive than other racial groups. our study suggests that demographic-aware annotation is important for various types of nlp tasks. overall our study makes the following four contributions. first, we create and release popquorn, a large-scale nlp dataset for four nlp tasks annotated by a representative sample of the us population with respect to sex, age, and race. second, we analyze the annotations by different groups of annotators and found that various demographic backgrounds is significantly associated with people's rating of offensiveness, politeness as well as their performance on reading comprehension. third, in comparison with existing annotations from curated workers, we demonstrate that a general sample of prolific participants can produce high-quality results with minimal filtering, suggesting the platform is a reliable source of quality annotations. all the annotations, annotator background information, and labeling interface are available at https://github.com/jiaxin-pei/ potato-prolific-dataset . ## motivation individual and group differences are two of the most fundamental components of social sciences @xcite . social and behavioral sciences exist, in part, because of systematic human variations: if everyone were to behave in the same way, there would be no need to build theories and models to understand people's behaviors in different settings. as a special form of human task, data labeling is also subject to such a basic rule: different people may have different perceptions of various information and different performances on various language tasks. in this sense, while nlp researchers try to achieve a higher iaa, disagreement is a natural and integral part of any human annotation task @xcite . existing studies in this direction generally focus on building models that can learn from human disagreement @xcite and some recent studies start to look at how annotators' identity and prior belief could affect their ratings in offensive language and hate speech @xcite . however, most of the existing studies only focus on selected dimensions of identities (e.g., gender) and on certain tasks (e.g., toxic language detection). our study aims at providing a systematic examination of how annotators' background affect their perception of and performances on various language tasks. on the annotator side, we use a representative sample that matches the sex, age and race distribution of the us population. on the task side, we try to select tasks that are representative of common nlp tasks and with different degrees of difficulty, creativity, and subjectivity. following this criterion, we selected four nlp tasks: (1) offensiveness detection, which is a relatively subjective task for classification and regression, (2) question answering, which is an objective task for span identification that is argued to test reading comprehension, (3) email rewriting, which requires creativity for a text generation task, and (4) politeness rating, which is also a subjective task for classification and regression. to understand how annotator backgrounds (e.g. age, sex and race) affect their ratings on offensiveness, we re-annotated 1500 comments sampled from the ruddit dataset @xcite using 262 annotators from a representative sample from prolific.co. 1 in this section, we introduce the data sampling process, annotation task design, annotation result and then discuss how annotators' background affect their ratings of offensiveness. ## task 2: question answering question answering/reading comprehension is one of the most fundamental tasks of nlp @xcite and squad has been actively used by the research community to evaluate the performance of their models on question answering (qa) as a form of reading comprehension @xcite . to evaluate crowd workers' ability to complete qa tasks and study whether participants' background is associated with different performances, we build the second task as part of the popquorn. ## task 3: politeness rewriting politeness is one of the most prominent social factors in interpersonal communication @xcite . the nlp community has built computational models for predicting politeness scores and built models to generate polite text in different settings @xcite @xcite . however, few resources exist with human-authored examples of pairs of original and style-transferred texts for politeness. therefore, to test the crowdworker's ability to generate opendomain text for style-transfer tasks, we recruit participants from prolific to rewrite emails from the enron dataset as part of popquorn. ## task 4: politeness rating to validate the email rewriting results from task 3, we perform a follow-up participant recruitment to rate the politeness of the original and revised emails. a recent study shows that the nativeness of the annotators affect the iaa of politeness rating @xcite . in this study, we release the full backgrounds of annotators and hope this dataset helps promote future studies on politeness prediction and to understand how people with different backgrounds perceive politeness. ## discussion high-quality annotated data has been one of the primary driving factors of nlp and ml. while some studies try to look at improving data quality through analyzing disagreements among annotators, systematic studies of how annotators' background affects crowdsourcing results remain rare. in this paper, we create a new nlp dataset labeled by annotators from a us-representative sample regarding sex, age and race. we re-annotated the ruddit offensiveness dataset and found that the offensiveness is strongly correlated with annotations by white participants, while the correlation between the ruddit offensiveness score and annotations by participants from other racial groups are only 0.41, suggesting that the ruddit dataset might largely reflect the views of white annotators of what content is offensive. as people from other cultures may perceive the same comment with a lower or higher degree of offensiveness, classifiers trained on a dataset annotated by white participants could pose risks for many people. such an issue becomes increasingly important as both the industry and research community are trying to align the values of llms with human beings through instruction tuning. ## conclusion who annotates your data matters. across four annotation tasks, we show that an annotator's background influences their decisions, across multiple annotation tasks with different degrees of subjectivity. in more subjective tasks, these differences in decisions are not mistakes but rather valid differences in views. our results underscore that nlp papers that curate datasets must consider whose voices appear in their datasets, as these ultimately decide whose voice are captured in models trained on the data. indeed, by comparing our annotations with those from the existing annotated datasets, we show that the existing annotated dataset might be annotated by a demographically-biased group of annotators. to support work in this modeling demographic-aware and socially-responsible nlp, we release popquorn with 45,000 annotations on four nlp tasks by nearly 1.5k annotators.