Re-examining Sexism and Misogyny Classification with Annotator Attitudes (2024)

Abstract: Gender-Based Violence (GBV) is an increasing problem online, but existing datasets fail to capture the plurality of possible annotator perspectives or ensure the representation of affected groups. We revisit two important stages in the moderation pipeline for GBV: (1) manual data labelling; and (2) automated classification. For (1), we examine two datasets to investigate the relationship between annotator identities and attitudes and the responses they give to two GBV labelling tasks. To this end, we collect demographic and attitudinal information from crowd-sourced annotators using three validated surveys from Social Psychology. We find that higher Right Wing Authoritarianism scores are associated with a higher propensity to label text as sexist, while for Social Dominance Orientation and Neosexist Attitudes, higher scores are associated with a negative tendency to do so.For (2), we conduct classification experiments using Large Language Models and five prompting strategies, including infusing prompts with annotator information. We find: (i) annotator attitudes affect the ability of classifiers to predict their labels; (ii) including attitudinal information can boost performance when we use well-structured brief annotator descriptions; and (iii) models struggle to reflect the increased complexity and imbalanced classes of the new label sets.

## introduction gender-based violence (gbv) is an increasing problem in online spaces, affecting around half of all women and targeting those from marginalised groups in particular @xcite , resulting in women often feeling uncomfortable online @xcite . to counter this, there have been attempts to facilitate content moderation using natural language processing (nlp) methods to automatically identify misogynistic language. as a result, there now exist several datasets designed for supervised classification of various forms of gbv. however, @xcite identified several weaknesses in approaches to the creation of corpora for this task. one prominent shortcoming has been the lack of representation in the labelled data of people's different points of view, particularly of those with minoritised identities who are best placed to recognise gbv. to fill this gap, we revisit the task of classifying online text following strongly perspectivist data practices @xcite , which aim to preserve labels provided by multiple annotators in the collection and modelling of data. we re-annotate two recent datasets, namely explainable detection of sexism (edos) @xcite , and detection of online misogyny (dom) @xcite , this time with (1) multiple ratings per item; and (2) demographic and attitudinal information about the annotators, which we maintain throughout the classification pipeline. prior work by @xcite that also collected attitudinal survey data from annotators attempts to capture morality via the moral foundations questionnaire (mfq) @xcite as a predictor of the perception of offensiveness in toxic language. however, due to criticism of the mfq regarding poor internal consistency @xcite , we look towards other factors that influence individuals' responses, with evidence from social psychology and sociology pointing towards the constructs of right wing authoritarianism (rwa), social dominance orientation (sdo) and hostile neosexism (hn) 2 @xcite @xcite as potentially relevant towards understanding the link between attitudes and gbv-related behaviour. we extend a pilot study by @xcite to explore the link between these attitudes and annotating behaviours on our re-annotated dataset, and find that annotators with higher propensity toward rwa are more likely to label text as sexist, possibly due to its association with benevolent sexist attitudes @xcite . 3 in contrast, we find that those with a higher propensity toward sdo and hn -both associated with hostile sexism @xcite -are less likely to label items as sexist, possibly due to the text aligning with internalised beliefs. while the datasets we re-annotated were originally conceived of for the classification of single 'gold standard' aggregated labels, we aim to represent diverse perspectives in predicting individual annotator labels @xcite . to better study the effect of including annotator attitudes as input to a classification task, we conduct a large set of instruction-based zero-shot, few-shot (in-context learning; icl), and fine-tuning experiments with four open-source large language models (llms), namely flan-t5 @xcite , llama 2 @xcite , llama 3 @xcite , and mistral @xcite . following @xcite we experiment with different prompt templates to better incorporate the annotator information (shown in figure 1 ). we find that icl works 17% and 26% better than the majority baseline for majority vote and individual annotator tasks respectively, and fine-tuning llms performs 31% better when predicting individual labels per annotator. the best way to incorporate attitudinal data for annotators is to include well-structured brief annotator descriptions about demographics and attitudes but exclude demonstrations. our experimental results also indicate that models are biased towards annotators' attitudes. ## background the gbv framework we follow @xcite in adopting this framework and the term gbv as a class label. it encompasses phenomena such as sexism, misogyny, and violence against women and girls-although it also recognises that people of all genders are affected by gbv. annotator variability and perspectivist data practices while labels collected for supervised classification have traditionally been aggregated to a single 'gold' or 'ground truth' label for each item, recent work has recognised that this can lead to the erasure of minoritised voices. this occurs by either hindering the ability of classifiers to recognise subtle and implicit forms of abuse, or by creating a prediction bias in the classifiers -e.g. in the form of harmful stereotypes -against historically minoritised voices @xcite . standpoint theory @xcite contends that only people with relevant lived experiences are able to recognise subtle, implicit abuse such as stereotypes and micro-aggressions. according to the matrix of domination @xcite , this experience likely results from sharing intersectional social categorisations with the intended targets of the abuse. there is now a growing recognition of the need to collect, retain, and distribute labels provided by multiple annotators, and this has been adopted across a range of nlp tasks (for an extensive list, see @xcite . this is particularly so for controversial tasks such as identification of abusive or toxic language, in which annotator variation may be caused by differences of opinion or ideology (e.g. @xcite @xcite . strong perspectivism aims to preserve this variation through modelling, classification, and evaluation @xcite . 4 beliefs and attitudes we ground our approach to the analysis of annotator beliefs in the dual process motivational model of ideology and prejudice @xcite . this links sociopolitical and ideological attitudes to prejudice captured by three related constructs: right wing authoritarianism (rwa), social dominance orientation (sdo), and hostile neosexism (hn). rwa explains propensity towards cultural conservatism and traditionalism-related beliefs @xcite @xcite , while sdo explains favourable views towards social hierarchies of power, where inequality between groups is seen as inevitable or even natural @xcite @xcite . hn is characterised by continued discrimination against women, denial of women's demands, opposition to policies aimed at improving women's social status, and the belief that feminist-driven changes unfairly disadvantage men @xcite @xcite . these constructs have been extensively assessed and found to be strongly related. they explain different forms of sexism and gender-based discrimination. rwa is linked to benevolent sexism, that is attitudes that force women into traditional predefined roles (e.g., being a mother) that seem superficially advantageous but are, in reality, marginalising and disempowering @xcite . sdo correlates with hostile sexism, and pertains to beliefs in deterministic gender imbalances justifying male dominance through disparaging characterisations of women @xcite . finally, hn is primarily associated with hostile sexist and anti-feminist attitudes in particular @xcite . these constructs have been widely used to explain gender-based discrimination through both offline @xcite @xcite and online (jagayat and choma, 2021) contexts, have been validated across cultures @xcite , and previously used to explain that such beliefs transcend demographic identities @xcite . annotator characteristics a number of nlp studies have attempted to use annotators' demographic characteristics as predictors of their responses to items (e.g. @xcite @xcite @xcite . however, it has repeatedly been shown that demographic characteristics do not predict annotator behaviour at the individual level @xcite @xcite @xcite . recent studies have therefore attempted to uncover annotators' social attitudes and relate these to their responses. @xcite found that crowd workers with racist beliefs were less likely to consider anti-black language as toxic. while they conducted two annotation experiments, one with many annotators but few items, and another with fewer annotators but more items, our data collection aims at both breadth and depth. @xcite measured crowd workers' responses to a misogynistic language labelling task, and surveyed their moral attitudes (in addition to demographic and personality-type information). they found that higher moral integrity and lower benevolent sexism scores correlated with label agreement with expert annotators. @xcite found that while cross-cultural differences exist, individual moral values significantly influence annotators' response to perceived offensiveness levels. hence, we seek to explore the relationship between demographics, social attitudes, and crowd-sourced responses to gbv identification tasks. modelling multiple perspectives previously, research on modelling with label variation focused on using disagreements to inform improved prediction of a single aggregated label (see @xcite , for a survey). more recent work has attempted to preserve these variations at inference. for example, cercas @xcite and @xcite predicted each annotator's responses to abusive language identification tasks, the latter using multi-task learning. the semeval shared task on learning with disagreement (le-wi-di) @xcite explicitly attempted to focus the field on attention to levels of disagreement between annotators. this drew several approaches including that of @xcite , who focused on preserving the full range of points of view at inference at the expense of overall classification performance. ## data collection we selected the test sets and a subsection of the training sets of two previously published datasets: explainable detection of sexism (edos) @xcite , and detection of online misogyny (dom) @xcite . we chose these as (1) @xcite had identified them as among the resources most thoroughly grounded in social science theory; (2) they are english language datasets, the language of our stakeholder partners, with whom we are co-designing gbvmitigation tools under the framework of participatory design; and (3) the textual data is from two different platforms, providing an opportunity for cross-(sub-)domain comparison. pre-processing consisted solely of filtering out any items which included images. we leave annotations of multi-media items for future work. this left 3,896 items, of which we re-annotated a random selection of 1,000 from the test sets for evaluation and 600 from the training sets for fine-tuning. 5 dataset label #original #new edos sexist 299 406 not sexist 901 794 dom misogynistic 47 97 nonmisogynistic 353 303 ours gbv 346 503 not gbv 1254 1097 as the amazon mechanical turk (mturk) crowd-sourcing platform is widely used to collect annotations and personal information for sensitive tasks @xcite , we recruited 43 annotators on mturk (19 women and 24 men with a mean age of 38, see appendix a for a full data statement with detailed annotator information). to ensure attentive participation, we recruited only workers @xmath0. for comparison of the new labels with the original edos and dom labels, we recruited people based in the same region as the original annotators, the united kingdom. we further collected demographic information and responses to questions from three surveys designed to measure the attitudes of workers. ## statistical analysis our hypothesis is two-tailed and exploratory in nature: whether gender, ssdo, bhns, or vsa scores are predictive of annotator behaviour in labelling items as sexist/misogynist. experimental design since we have multiple annotations per annotator, we employ a mixed effects regression model @xcite . our dependent variable is the binary label given by each annotator, while our predictors are gender (male/female), ssdo and bhns scores (both aggregated into high, moderate and low), and vsa scores (aggregated into a five-point scale from very low to very high). our model includes by-annotator random intercepts, as most individuals annotated multiple items, while we reject one participant who only provided a single annotation. all categorical variables are dummy-coded (see appendix c for details). to evaluate possible effects of pairwise comparisons, we employ a benjamini-hochberg correction @xcite due to its specific focus on false discovery rates in study designs with independent statistics, and smaller sample sizes @xcite , such as our study. ## classification experiments we benchmark the new dataset to explore i) whether a broader label set provides richer information and ii) how varying conceptualisations of gbv, annotator demographics and attitude information affect model performance in identifying gbv text online. to achieve this, we conduct two tasks to predict (a) majority labels per text, and (b) individual annotator labels with diverse label texts. dataset we use our re-annotated dataset introduced in section 4 to predict the majority label per text. our dataset contains 1600 instances, including 1200 instances from edos subset and 400 from dom. we also augment it with individual annotator labels, obtaining 6,000 and 23,000 instances for the fine-tuning and test sets respectively. three different label texts are used: "gbv" as the aggregated label in our re-annotated dataset, "sexist" from edos, and "misogynistic" from dom (see prompt design we experiment with five different prompt templates for our detection task. we use the answer format from @xcite . <@xmath1: a. gbv or b. not gbv." for "gbv" label text, and corresponding changes are made for "sexist" and "misogynistic" labels (see appendix e for more details). ## conclusion we have revisited the annotation and classification tasks for online gbv with a particular focus on the underlying attitudes of a broad range of annotators. through the re-annotation of two datasets, we collect demographic and attitudinal information about crowd-sourced annotators using validated surveys from social psychology, and incorporate a diverse range of annotator perspectives, exploring the relationship between these factors and the labels they provide. we find that annotators with stronger right-wing authoritarian traits show a higher propensity to label items as sexist, whereas those with more socially dominant and neosexist attitudes do the opposite. this suggests that people exhibiting right-wing authoritarian characteristics may be less attuned to subtle gender-related discourse. we then conduct classification experiments on both aggregated and individual annotator labels using various prompting strategies and llms. our findings indicate that models are biased by annotators' attitudes. while incorporating annotator information can enhance model capacity, but adding excessive information can be detrimental. this highlights the challenges of the increased complexity and imbalance of incorporating broader, perspectivist label sets, which adversely affect performance on all our experiments.