D 3 CODE : Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation (2024)

Abstract: While human annotations play a crucial role in language technologies, annotator subjectivity has long been overlooked in data collection. Recent studies that critically examine this issue are often focused on Western contexts, and solely document differences across age, gender, or racial groups. Consequently, NLP research on subjectivity have failed to consider that individuals within demographic groups may hold diverse values, which influence their perceptions beyond group norms. To effectively incorporate these considerations into NLP pipelines, we need datasets with extensive parallel annotations from a variety of social and cultural groups.In this paper we introduce the D3CODE dataset: a large-scale cross-cultural dataset of parallel annotations for offensive language in over 4.5K English sentences annotated by a pool of more than 4k annotators, balanced across gender and age, from across 21 countries, representing eight geo-cultural regions. The dataset captures annotators’ moral values along six moral foundations: care, equality, proportionality, authority, loyalty, and purity. Our analyses reveal substantial regional variations in annotators’ perceptions that are shaped by individual moral values, providing crucial insights for developing pluralistic, culturally sensitive NLP models.

## introduction designing natural language processing (nlp) tools for detecting offensive or toxic text has long been an active area of research @xcite . however, applying traditional nlp solutions have led to overlooking the cultural and individual factors that shape humans' varying perspectives and disagreements on what is deemed offensive @xcite @xcite figure 1 : the distribution of labels provided from different countries. annotators from china, brazil, and egypt provided significantly different labels. as a result, recent efforts call for diversifying the rater pools as well as designing models that look beyond predicting a singular ground truth @xcite . however, the efforts for diversifying annotator pools often risk reducing annotators' differences to demographic variations. in other words, subjectivity is often studied solely in relation to annotators' gender and race, within the western context. in reality, perceptions of what is offensive extend far beyond mere differences in demographics, shaped by an individual's lived experiences, cultural background and other psychological factors @xcite . @xcite demonstrate the impact of annotators' beliefs about racism, freedom of speech, and conservatism on their perceptions of toxicity. while individuals' systematic disagreements on notions of offensiveness reflect the complexity of their beliefs, these beliefs are often anchored in core moral values that shape their judgments. not only do moral values influence various aspect of our cognitive processes @xcite , emotions @xcite , and social relationships @xcite , they also shape our judgments, motivate our behaviors, and guide our interactions. as a result, moral values, contribute significantly to our personal and cultural identity @xcite , as we internalize societal norms and develop our moral compass @xcite . therefore, we argue that the high divergence in annotators' perceptions of offensiveness can be traced back to individuals' diverse moral values along with the cultural and social norms that dictate the boundaries of acceptable language within a society. in this work we introduce the d3code dataset 1 , built through a crosscultural annotation effort aimed at collecting perspectives of offensiveness from 4309 participants of different age and genders across 21 countries within eight larger geo-cultural regions. through an in-depth analysis of our dataset, we shed light on cultural and moral values that sets people apart during the annotation. we believe that this dataset can be used for assessing modeling approaches that are designed to incorporate annotators' subjective views on language, as well as for evaluating different models' cultural and moral alignment. ## related work disagreement -even on objective tasks @xcite -is a source of information @xcite @xcite . therefore, research on perspectivism in data @xcite argues that treating annotators as interchangeable is ineffective when dealing with subjective language understanding tasks @xcite @xcite @xcite @xcite . instead, capturing and modeling nuanced annotations and interannotator disagreements has been widely explored @xcite @xcite @xcite @xcite 1 https://github.com/google-research-datasets/ d3code @xcite . for instance, @xcite propose a descriptive annotation paradigm for operationalizing subjectivity when surveying different beliefs. accordingly modeling approaches were proposed to consider variations of annotator perspectives; for instance, incorporating the item-level agreement into the loss function @xcite , leveraging annotator disagreement as an auxiliary task along with ground-truth label prediction @xcite , or employing item-level disagreements for informing model training @xcite . however, these methods often overlook the integrity of individual labels and aggregate diverse subjectivities into a single construct @xcite . research has shown that providing the age or gender of the authors to text classifiers consistently and significantly improves the performance over demographicagnostic models hovy (2015 @xcite @xcite capture perspectives of different groups rather than single annotators. further approaches tend to integrated annotator differences into model predictions, through personalized model tuning @xcite , jury learning @xcite , and training annotator embeddings @xcite . although attending to annotators' background is gaining more attention, documenting how annotators' identity shapes their comprehension of the world and in turn language is still missing in many data curation efforts @xcite . a number of scholars have begun to not only document annotators' identity, but also develop principled approaches for obtaining a diversity of identities and perspectives in datasets. @xcite developed a dataset that specifically focuses on evaluating disagreement and diverse perspectives on conversational safety, and @xcite leverages this same dataset to proporse a multilevel modeling approach for measuring annotation differences across a range of sociodemographic groups. pei and jurgens (2023) recruit a representative sample of annotators across sex, age, and race in the us and show the impact of annotators' background on their judgments. the social nature of language means that sociocultural differences play meaningful roles in how individuals use language, such as offensive speech @xcite . @xcite apply item response theory to capture the impact of annotators' group identity in their evaluation of harmful language. @xcite and @xcite demonstrate statistically significant variations across countries in hate speech annotations. @xcite draw from social psychology research to demonstrate the impact of annotator identities and beliefs about hate speech, free speech, and racist beliefs, on their annotations of toxicity. @xcite investigates annotators' biases and disagreements as related to their social stereotypes. one such nuanced factor, often not studied in ai research, is morality. moral considerations play significant roles in how humans navigate prejudicial thoughts and behaviors @xcite , often manifesting in language through offensive content. the interplay between morality and group identity @xcite influences many aspects of our social dynamics, including perceptions, interactions, stereotypes, and prejudices. moreover, research in computational social science addressing harmful language reveals a concurrent occurrence of moral sentiment alongside expressions of hatred directed at social groups @xcite . in this paper we introduce the d3code dataset which not only provides social factors and demographic information regarding annotators but also considers the moral values that may vary across regions and among individuals. such information facilitates drawing connections between annotations from culturally diverse annotators, the sociocultural norms shaping their environment, and the moral values they hold. ## d3code dataset in order to study a broad range of cultural perceptions of offensiveness, we recruited 4309 participants from 21 countries, representing eight geocultural regions, with each region represented by 2-4 countries ( ## analyses our analyses focus on annotators' varying perspectives and how shared social, cultural or moral attributes can help shed light on annotation behaviors. we begin by analyzing how different groups vary on expressing their lack of understanding the message by selecting the "i don't understand this message" option. we then study annotators' geocultural regions and moral values in relation to their annotations. specifically, we consider annotator clustering either based on their similar moral values or their region of residence, and assess in-group homogeneity and out-group disagreements for clusters. the remainder of this section delves deeper into how groups of annotators from the same region or with similar moral values tend to label content differently. ## discussion research on safety considerations of large language models has mostly focused on evaluations of model harms through crowdsourced benchmarks @xcite . however, while annotators from different regions are shown to have different perspectives regarding this task @xcite , current benchmarks fail to represent the cultural and individual variations in human moral judgements about generated language and model outputs. they also lack comprehensive understanding of human values and cultural norms that drive diversity of perspectives in annotations. this work presents a cross-cultural experiment with participants across various cultural and demographic backgrounds. offensive item yes no category transgender athletes..... transgender students..... this is what feminism gets you in society. this was unheard of only a short 20 years ago. ac, la, na, oc ics, si, ssa, we transgender does pointing out that a growing majority of americans support adultery (abortion, same sex marriage, even theft), change god's law an iota? ac, ics, la, na oc, si, ssa, we lgb the women came out because he is a hypocrite, a child predator. come on trumpers, defend the child predator christian, lol ics, la, oc, ssa ac, na, si, we christian our dataset captures valuable insights into human perceptions on offensive language, revealing demographic differences in annotation certainty, and regional, as well as moral psychological variations in perceiving offensiveness. our first analyses captures how participants with different demographic background might express their unfamiliarity with the annotation. in general, annotators not identifying as men and annotators aged 50 and above are more likely to select the "i don't understand" option. moreover, annotators from oceania, north america, and western europe were significantly more probably to state that they did not understand the message compared to indian cultural sphere, arab culture, latin america, sinosphere, and sub saharan africa. therefore, it is important to note how this kind of uncertainty in annotating might occurred disproportionately for different groups. our dataset also represent different categories of content within a well-known machine learning corpus, with annotators having varying levels of disagreement for labeling content from different categories. while items with moral sentiment are the least likely to evoke disagreement, items mentioning specific social groups are more likely to have a varying range of annotation. this finding replicates previous findings on how group perception and stereotypes affect harm perception targeting different social groups, in a cross-cultural context. consequently, these findings underscore the need for further research into social dynamics within diverse cultural contexts to better understand and mitigate harmful risks of language technologies. our dataset and findings demonstrate the potential for incorporating diverse cultural and individual moral perspectives to enhance fairness in llms. by moving beyond traditional socio-demographic groupings and understanding how individual moral values shape perceptions, we provide a pathway for defining model alignment strategies that capture a broader range of human values, thus contributing to the approaches to address representation biases by challenging value alignment frameworks that prioritize normative cultural or societal values. ## conclusion we introduce the d3code dataset, which captures the results of a cross-cultural annotation experiment for understanding disagreements on perceiving offensiveness in language. our findings reveal significant demographic and regional variations in perceptions of offensive language, underlining the necessity of incorporating diverse perspectives into reinforcement learning with human feedback. additionally, the dataset showcases differences in annotation certainty and disagreement levels across various content categories, particularly concerning mentions of specific social groups. these findings underscore the imperative for further research into social dynamics within diverse cultural contexts to mitigate the risks associated with harmful language in language technologies and promote fairness and inclusivity in digital interactions.