An Expert Annotated Dataset for the Detection of Online Misogyny (2021)

Abstract: Online misogyny is a pernicious social problem that risks making online platforms toxic and unwelcoming to women. We present a new hierarchical taxonomy for online misogyny, as well as an expert labelled dataset to enable automatic classification of misogynistic content. The dataset consists of 6567 labels for Reddit posts and comments. As previous research has found untrained crowdsourced annotators struggle with identifying misogyny, we hired and trained annotators and provided them with robust annotation guidelines. We report baseline classification performance on the binary classification task, achieving accuracy of 0.93 and F1 of 0.43. The codebook and datasets are made freely available for future researchers.

## introduction misogyny is a problem in many online spaces, making them less welcoming, safe, and accessible for women. women have been shown to be twice as likely as men to experience gender-based online harassment @xcite . this misogyny can inflict serious psychological harm on women and produce a 'silencing effect', whereby women selfcensor or withdraw from online spaces entirely, thus limiting their freedom of expression @xcite . tackling such content is increasingly a priority for social media platforms and civil society organisations. however, detecting online misogyny remains a difficult task @xcite . one problem is the lack of high-quality datasets to train machine learning models, which would enable the creation of efficient and scalable automated detection systems @xcite . previous research has primarily used twitter data and there is a pressing need for other platforms to be researched @xcite . notably, de-spite social scientific studies that show online misogyny is pervasive on some reddit communities, to date a training dataset for misogyny has not been created with reddit data. in this paper we seek to address the limitations of previous research by presenting a dataset of reddit content with expert labels for misogyny that can be used to develop more accurate and nuanced classification models. our contributions are four-fold. first, we develop a detailed hierarchical taxonomy based on existing literature on online misogyny. second, we create and share a detailed codebook used to train annotators to identify different types of misogyny. third, we present a dataset of 6,383 entries from reddit. fourth, we create baseline classification models based on these datasets. all of the research artefacts are made freely available via a public repository for future researchers. 1 the dataset itself has several innovations which differentiate it from previous training datasets for misogyny. first, we use chronological and structured conversation threads, which mean annotators take into account the previous context of each entry before labelling. second, we distinguish between conceptually distinct types of misogynistic abuse, including gendered personal attacks, use of misogynistic pejoratives, and derogatory and threatening language. third, we highlight the specific section of text, also known as a 'span', on which each label is based. this helps differentiate between multiple labels on one piece of text. fourth, we use trained annotators, rather than crowd-sourced workers. we also use facilitated meetings to decide the final labels rather than just a majority decision. both of these factors lead to a high-quality dataset. additionally, we provide a second dataset with the original labels made by annotators before the final labels were decided. ## background most previous classification work on online misogyny has used data from twitter @xcite @xcite . however, social scientific and ethnographic research shows that reddit is increasingly home to numerous misogynistic communities. reddit is a social news website organised in to topic-based communities. each subreddit acts as a message board where users make posts and hold discussions in comment threads on those posts. in recent years it has become a hub for anti-feminist activism online @xcite . it is also home to many misogynistic communities, particularly those associated with the 'manosphere', a loosely connected set of communities which perpetuate traditional forms of misogyny and develop new types of misogynistic discourse which in turn spread to other online spaces @xcite @xcite @xcite . recent research suggests that the rate of misogynistic content in the reddit manosphere is growing and such content is increasingly more violent @xcite . waseem and hovy (2016) provided a widelyused dataset for abusive language classification. they used expert annotators to identify sexist and racist tweets based on a set of criteria drawn from critical race theory. the tweets were initially labelled by the authors then reviewed by a third annotator. the resulting dataset consists of 17k tweets, of which 20% are labelled as sexist. however 85% of the disagreements between annotators were over sexism labels, which shows that even experienced coders of abusive language can have difficultly identifying gendered abuse. @xcite extended on the @xcite dataset to distinguish between between 'benevolent' and 'hostile' sexism @xcite . they classed all sexist labels in the previous dataset as 'hostile' and all non-sexist labels as 'other'. they then augmented the dataset by collecting tweets using keyword sampling on benevolently sexist phrases (e.g. 'smart for a girl') and extracted those manually identified as 'benevolent sexism'. in the combined dataset of 10,095 unique tweets 712 were labelled as 'benevolent', 2,254 as 'hostile', and 7,129 as 'not sexist'. they thus found that in the data hostile sexism was more than three times as common as the benevolent form. their work highlights the need for greater attention to be placed on forms of 'subtle abuse', particularly for online misogyny @xcite . @xcite developed a taxonomy with five categories of misogyny, drawn from the work of @xcite : stereotype & objectification, dominance, derailing, sexual harassment & threats of violence, discredit. they used a combination of expert and crowdsourced annotation to apply the taxonomy and present a dataset of 4,454 tweets with balanced levels of misogynistic and non-misogynistic content. a shared task confirmed that the dataset could be used to distinguish misogynistic and non-misogynistic content with high accuracy, but performance was lower in differentiating between types of misogyny @xcite . @xcite provide a dataset of 2k urban dictionary definitions of which half are labelled as misogynistic. in @xcite they show that deep learning techniques had greater accuracy in detecting misogyny than conventional machine learning techniques. ## data collection we collected conversation threads from reddit. given that a very small amount of content on social media is hateful, a key difficulty when creating datasets for annotation is collecting enough instances of the 'positive' class to be useful for machine learning @xcite . however, sampling strategies can introduce biases in the composition and focus of the datasets if overly simplistic methods are used, such as searching for explicitly misogynistic terms @xcite . to ensure that our dataset contains enough misogynistic abuse we began with targeted sampling, taking content from 12 subreddits that were identified as misogynistic in previous research. this includes subreddits such as r/mensrights, r/seduction, and r/theredpill. the sources used to identify these subreddits are available in using subreddits to target the sampling rather than keywords should ensure that more linguistic variety is captured, minimising the amount of bias as keywords such as 'slut' are associated with more explicit and less subtle forms of abuse. nonetheless, only sampling from suspected misogynistic communities could still lead to classifiers which only identify the forms of misogyny found in those targeted contexts @xcite @xcite . to account for this potential bias, and to enable greater generalisabilty, we sampled content from 71 randomly selected subreddits. they accounted for 18% of threads and 16% of entries in our dataset. for each randomly selected subreddit, we collected the thread of the most popular post. all threads were in english with the exception of one thread from the subreddit r/romania. posts and comments were collected from february to may 2020 using the python package praw, a wrapper for the reddit api @xcite . posts on reddit have a text title and a body which can be text, an image, or a link. for posts with a text body we combined this with the post title to create a single unit of text. for the 29% of posts where the body was an image we also collected the image. ## taxonomy we developed a hierarchical taxonomy with three levels. first, we make a binary distinction between misogynistic content and non-misogynistic content, which are mutually exclusive. second, we elaborated subtypes of misogynistic and nonmisogynistic content. for misogynistic content we defined four categories: (i) misogynistic pejoratives, (ii) descriptions of misogynistic treatment, (iii) acts of misogynistic derogation and (iv) gendered personal attacks against women. for nonmisogynistic content we defined three categories: (i) counter speech against misogyny, (ii) nonmisogynistic personal attacks and (iii) none of the categories. third, we included additional flags for some of the second level categories. within both misogynistic and non-misogynistic content, the second level categories are not mutually exclusive, thereby allowing for multiple labels per entry. for instance, a misogynistic entry could be assigned labels for both a pejorative and treatment. this taxonomy draws on the typologies of abuse presented by @xcite and @xcite as well as theoretical work in online misogyny research @xcite @xcite @xcite @xcite . it was developed by reviewing existing literature on online misogyny and then iterating over small samples of the dataset. this deductive-inductive process allowed us to ensure that conceptually distinct varieties of abuse are separated and that different types of misogyny can be unpicked. this is important given that they can have very different impacts on victims, different causes, and reflect different outlooks and interests on the part of the speaker. ## annotation methodology a key difficulty in the formation of abusive language training datasets is producing high quality annotations. several factors affect this. deciding between similar categories, such as 'hate speech' versus 'offensive language' can be difficult @xcite . determining the right category often requires close scrutiny and sustained critical thinking from annotators. annotators may face information overload if asked to work with too many categories, both in terms of breadth (e.g. annotating for different types of abuse) and depth (e.g. working with numerous subcategories). further, annotators may have different values and experiences and so make different assessments of the content they observe, especially when context plays a large role. annotators will also have unconscious social biases which may mean they interpret coding instructions differently to each other, and to how they were intended by the research authors. for instance, @xcite found that crowdsourced annotators were more likely to label sexist content as merely 'offensive' while racist and homophobic content was considered 'hate speech'. to mitigate such annotator biases, we used expert annotators specifically trained in identifying misogynistic content, as well as a group-based facilitation process to decide final labels. due to time and resource constraints, the final dataset is smaller than if we had used crowdsourced workers but captures more nuanced and detailed cases of misogyny. six annotators worked on the dataset. annotators were trained in the use of a codebook detailing the taxonomy and annotation guidelines. the codebook was updated over time based on feedback from the annotators. demographic information on the annotators is available in appendix a.2 ## prevalence of the categories of the 6,567 agreed labels in the final dataset 10.6% are misogynistic (@xmath0.4% are non-misogynistic (@xmath1,868). tables 2 and 3 show the number of labels in the final dataset for each of the misogynistic and non-misogynistic categories, broken down by the level two categories. the vast majority of entries fall under none of the categories (88.6% of all labels). the next most common category is misogynistic pejoratives followed by misogynistic derogation pejoratives (4.2%). there are relatively few labels for personal attacks with just 0.7% in total for each of the misogynistic and nonmisogynistic categories, respectively. the least common category is counter speech against misogyny, with only ten cases (0.2%). ## conclusion in this paper we have presented a hierarchical granular taxonomy for misogyny and have described a dataset containing high quality, expert labels of misogynistic content from reddit. we have also provided the detailed coding book we created and a dataset with all of the original labels. the final dataset is small compared to other annotated datasets used for classification. however it benefits from a detailed taxonomy based on the existing literature focused on just one form of online abuse -misogyny. the use of trained annotators and an adjudication process also ensures the quality of the labels. the more granular subcategories in the taxonomy may be too small to classify separately, but they provide insights into the relative frequency of different forms of misogynistic content on reddit and enable detailed error analysis. they are also useful for other researchers aiming to create larger datasets, who can build on the taxonomic work conducted here. proceedings of the 26th international conference on world wide web, pages 1391-1399, perth australia. international world wide web conferences steering committee. donna zuckerberg. 2018. not all dead white men: classics and misogyny in the digital age. harvard university press, cambridge, massachusetts. a short form data statement following the recommendation of @xcite we include the following short form data statement to summarise the main features of the datasets. further details on the creation of the datasets are in in sections 3 and 5 in the main paper.