Analysis of Annotator Demographics in Sexism Detection (2024)

Abstract: This study explores the effect of annotators’ demographic features on labeling sexist content in social media datasets, specifically focusing on the EXIST dataset, which includes direct sexist messages, reports and descriptions of sexist experiences and stereotypes. We investigate how various demographic backgrounds influence annotation outcomes and examine methods to incorporate these features into BERT-based model training. Our experiments demonstrate that adding demographic information improves performance in detecting sexism and assessing intention of the author.

## introduction according to the united nations definition, genderbased violence includes violence that is directed against a woman because she is a woman or that affects women disproportionately, and, as such, is a violation of their human rights foot_0 . a report published by amnesty international foot_1 found that 23% of women using twitter reported experiencing online abuse or harassment at least once. such violence and abuse on social media significantly undermines women's rights to express themselves equally, freely, and without fear. the exist shared task @xcite aims to identify and categorize sexism on twitter. developing any dataset for topics such as sexism, sarcasm, or hate speech is challenging, since it is difficult to determine a ground truth for these topics @xcite . one person may find a tweet sexist, while another may find it acceptable. traditionally, machine learning models take the majority vote over all labels, effectively ignoring differences in annotators' backgrounds, which might, however, influence their perspectives. it also disregards minority opinions when most annotators agree on one label but a few dissent. the resulting datasets have been described as partially subjective and not fully reliable for downstream applications by @xcite . in response, @xcite introduced the learning with disagreement paradigm (lwd), which involves training systems on datasets that include all annotators' perspectives, aiming to reflect the diversity of views. evaluation with a soft metric enables ambiguity-aware models to compare the probability distributions of labels they generate (soft labels) to the full distribution provided by annotators using cross-entropy. this step away from hard gold labels to distributions provides the possibility of better modeling both, potential ambiguity of the wording as well as disagreements in the judgments. the exist sexism task adopted lwd and incorporated the annotators' demographic information into the training and test sets. we show here that for the exist dataset the performance of sexism detection and the assessment of the author's intentions improves when demographic information is included for classification. we explore different methods to integrate the demographic features and find that including it as additional input to a bert model enhances performance. while we use gold labels during training, which are crowd labels aggregated into hard labels using majority voting, we include detailed information from the annotations in the training process. the goal is to investigate whether there is a correlation between annotator judgments and demographic features in two ways. first, we exploit the demographic features from the training data for improved classification. secondly, we give a first exploration of potential bias in the dataset. bias here refers to uneven distributions and has no value judgment attached to it from the outset. while ex-ist is a renowned shared task aimed at identifying sexism in social media, no paper has yet analyzed the impact of annotators' features on the labeling process in this dataset. ## methodology the dataset includes annotations from six annotators for each sample, along with their demographic features. we compare here performance when including and when omitting these demographic features as input for a simple pre-trained model (finetuned roberta @xcite or multilingual cased bert @xcite ) followed by two feedforward networks for classification. when no demographic features are introduced to the model: encoding base input to bert is the sample text, input to the classifier is bert's cls token. encoding preprocessed four tokens ([url], [user], [spanish], [english]) were added as special tokens to the bert tokenizer. in each sample, urls and mentions were replaced with [url] and [user] tokens, respectively. the dataset identifies for each tweet whether it is in english or spanish, thus either [english] or [spanish] were added at the beginning of each instance. based on findings that word embeddings capture the meaning of emojis better than symbols @xcite , emoji word embeddings were added to the bert vocabulary as special tokens, enabling bert to map a specific identifier to each emoji. the input to bert is the sample text preprocessed this way and the input to the classifier is bert's cls token. the following describes two models that incorporate annotators' demographic features. in both configurations, each preprocessed sample is replicated six times, corresponding to each annotator. consequently, the original training set of 6,920 tweets expands to 41,520 instances. encoding features input to bert is the preprocessed sample text. input to the classifier includes demographic features in a stack of one-hot vectors of each feature (age, gender, study level, and ethnicity) and the cls vector. we send additive attention to the classifier. additive attention @xcite obtains a weighted average of the representations, where the weights are determined based on the importance of each representation. encoding annotations input to bert concatenates demographic features to the end of each preprocessed sample. each sample is replicated six times @xcite , each with details on gender, age, level of study, and ethnicity for each annotator (see section 5.1). both the hard label, which is the aggregated label for that tweet (see section 5.2), and the annotator's label for each instance were added to each instance. input to the classifier is the cls token. an initial baseline system predicted both sexism and the three intention categories direct, reported, judgmental within a single model. however, we observe better results when we reduce the number of non-sexist tweets for task 2 by first running our task 1 system in order to predict the sexist tweets and subsequently categorizing only those with our different approaches. ## experiments 5.1 feature representation to experiment with encoding annotations , we concatenated the demographic features to the sample in different formats. example 1 shows the input vector representation, where the meta-labels on the sample correspond to the respective word vectors. the first four demographic features are appended represented by the strings used in the competition data (the last two demographic features are omitted here). example 2 shows the first four demographic features represented by scalar values (see section 3.2). example 3 shows the demographic features enclosed in dedicated special tokens that have been added to bert's special tokens vocabulary. example 4 shows a variation on example 1, where 'f' was replaced by the word 'female @xcite replaced by 'genx', namely where words replaced symbols. example 4 representations outperform example 1 representations and the word encodings of example 4 are used for the following examples. (5) as usual, the woman was the one quitting her job for the family's welfare. [sep] female [sep] genx [sep] hispano or latino [sep] bachelor's degree example 5 is a version of 4 where the demographic features are separated by the pre-trained [sep] token in the input to bert. ## analysis 7.1 annotator demographics as features the exist dataset is among the recent datasets that adopted the learning with disagreement approach for sexism detection. our experiments demonstrated that integrating additional demographic information directly into language models significantly enhances performance, proving more effective than incorporating these features separately into classifiers. this underscores the capabilities of the bert family models, which can achieve impressive results with proper token representations. our findings show that adding two demographic features to the model improved performance for the first task, and the inclusion of two more features further increased performance. when annotators' labels were incorporated as an additional feature to the input, there was a slight additional boost in performance. our experiments also show the critical role of feature representation. among the various representations tested, adding demographic features as tokens yielded the best results, capitalizing on the inherent power of the corresponding word embeddings. even when adding annotator's labels as an additional feature, representing them as word tokens (for instance in example 6 adding a label sexist instead of 1) resulted in slightly better performance. including annotator features in the model allows for training on a diverse set of representations for each sample, rather than relying on a single perspective. this leads to better training outcomes and a less biased model, emphasizing the importance of considering annotator demographics in developing more inclusive models. ## conclusion and limitations the aim of this study is not to reach sota in detecting sexism and source intention, but rather to examine the potential advantage of adding annotator features in representing more diverse judgments in the exist dataset. we showed that incorporating annotators' demographic information as inputs into the bert models can enhance the performance of models in detecting sexism and understanding the author's intention in the exist dataset. additionally, our findings indicate significant variations in how tweets are ranked as sexist, influenced by factors such as the annotators' spoken language and country of residence. however, including demographic features in the training process may introduce new biases into the models. future research will focus on further ex-ploring the effects of these demographic features on trained models. we anticipate that the strict conditions for corpus construction used in corpus linguistics can inform addressing some of these issues. additionally, we plan to investigate methodologies that assess the degree of disagreement among annotators, building on existing studies in this area @xcite . this will help us better understand and address potential biases in the labeling process, aiming for more robust models.