Biasly: An Expert-Annotated Dataset for Subtle Misogyny Detection and Mitigation (2024)

Abstract: Using novel approaches to dataset development, the Biasly dataset captures the nuance and subtlety of misogyny in ways that are unique within the literature. Built in collaboration with multi-disciplinary experts and annotators themselves, the dataset contains annotations of movie subtitles, capturing colloquial expressions of misogyny in North American film. The open-source dataset can be used for a range of NLP tasks, including binary and multi-label classification, severity score regression, and text generation for rewrites. In this paper, we discuss the methodology used, analyze the annotations obtained, provide baselines for each task using common NLP algorithms, and furnish error analyses to give insight into model behaviour when fine-tuned on the Biasly dataset.

## introduction and related work when using language models (lms) to perform sensitive, subjective, and socially impactful tasks like misogyny detection, hate speech mitigation, or online content moderation, the quality of the underlying dataset is critical @xcite @xcite . because the model will align to the biases in the dataset, which often reflect the biases, or oversights, of the dataset creators @xcite , it is crucial to include a diverse group of stakeholders in the dataset creation process, including lm domain experts and stakeholders who would be impacted by any deployed model that was trained on the dataset @xcite . dataset work in the field of bias, and more specifically sexism or misogyny detection, has mainly focused on the domains of social media, with data stemming from twitter, reddit or gab @xcite @xcite (see even though most datasets provide a more finegrained classification for different subtypes of misogyny @xcite @xcite , the detection of misogyny remains, at its core, a classification problem where a (sub)category can be either present or not. we argue that due to its nuanced and subjective nature, a continuous severity score modelled by regression is better suited for the detection of subtle misogyny. only one misogyny-specific dataset includes a type of of misogyny mitigation @xcite . their goal was to create adversarial examples that language models would find hard to differentiate from real sexist statements, by applying minimal lexical changes. our work is methodologically closer to the paradetox @xcite and appdia @xcite datasets, which released a parallel corpus for detoxicification. to our knowledge, our dataset is the first parallel corpus with the purpose of training language models to rewrite text to mitigate the subtle misogyny contained therein. in this work, we document the creation process of the biasly dataset, an open-source expertannotated dataset for the detection and mitigation of subtle forms of misogyny 1 . we first describe our process, including how our interdisciplinary team thoughtfully selected, trained, and engaged with our annotators, ensuring our dataset is both high quality and was created in a socially responsible way. then, we present a short analysis of the annotated dataset and provide model baseline results for the tasks of binary and multi-label misogyny classification, severity prediction and mitigation. finally, we provide an error analysis of each of our baseline models to provide insight into model behaviour when fine-tuned on the biasly dataset. ## dataset creation our team consists of experts from the domains most relevant to the development of a misogyny dataset; specifically, specialists from nlp, linguistics and gender studies. we engaged in a collaborative, multi-disciplinary process wherein our decisions were informed by qualitative and quantitative analyses of the data, described briefly in the following sections. ## dataset analysis our resulting dataset consists of 10000 datapoints, each annotated 3 times, for a total set of 30000 annotations. 5600 of the 30000 annotations were labeled as misogynistic according to the first binary classification task (task 1), with an inter-annotator agreement of 0.4722 according to fleiss' kappa @xcite . the severity of the original misogynistic datapoints (from 0 to 1000) has a mean of 344.8 with a standard deviation of 209.1, while the severity of all rewritten datapoints has a mean of 53.6 and a standard deviation of 115.8, reflecting a significant reduction in misogyny severity. the most frequent sub-category of misogyny was trivialization with 2227 occurrences, while transmisogyny only appeared 43 times. 1985 misogynistic datapoints were selected to be rewritten to 4 430 ## automatic classification, regression, and mitigation experiments this section describes the experimental setup for machine learning models on our dataset and presents the results. to follow best practice from other work cited below, we provide baseline results for the machine learning tasks of binary and multi-label classification, severity regression, and mitigation. for all models, we used an 80/10/10 train/eval/test split. ## conclusion developed in collaboration among experts in gender studies, linguistics, and nlp, we present biasly, an open-source dataset for the detection, categorization, severity prediction, and mitigation of subtle misogyny. we provide baseline models for each of these tasks and an error analysis of each. in future work, we aim to employ more advanced modeling techniques, such as those used in mostafazadeh @xcite to be able to model diverse annotator perspectives, moving beyond our current reliance on a single label. our hope is that biasly serves as a model for socially responsible dataset creation for lms. this process can be readily applied to diverse domains, fostering a broader commitment to responsible ai development. 9 435 machine learning models trained on our dataset will contain biases associated with: i) the annotators' demographic and academic backgrounds as well as their lived experiences; ii) the subjective, context-dependent and time-bound nature of the task; and, iii) challenges for our annotators in labeling data consistently across time (the task can be performed differently depending on a number of contextual factors that cannot necessarily be controlled for; influencing, among other things, annotators notions of misogyny, severity etc.). users of this dataset should also be aware that any automatic tools trained on this dataset can never guarantee perfect debiasing. additionally, the use of movie subtitles as data may be limiting in that this is merely an approximation of natural spoken language. it is unclear, then, how well models trained on this dataset will perform on out-of-domain data such as social media content moderation or misogyny detection and/or mitigation in naturally spoken or written language. while we will do our best to provide insight into the extent to which each of these factors influenced our dataset (through accompanying documentation), we hope that those using the dataset will keep these limitations in mind and not use models fine-tuned on our dataset in ways that fail to acknowledge these limitations.