Resources for Automated Identification of Online Gender-Based Violence: A Systematic Review (2023)

Abstract: Online Gender-Based Violence (GBV), such as misogynistic abuse is an increasingly prevalent problem that technological approaches have struggled to address. Through the lens of the GBV framework, which is rooted in social science and policy, we systematically review 63 available resources for automated identification of such language. We find the datasets are limited in a number of important ways, such as their lack of theoretical grounding and stakeholder input, static nature, and focus on certain media platforms. Based on this review, we recommend development of future resources rooted in sociological expertise andcentering stakeholder voices, namely GBV experts and people with lived experience of GBV.

## introduction we are in the midst of an 'epidemic of online abuse', which disproportionately affects women and minoritised groups and has worsened during and after the covid-19 pandemic: 46% of women and marginalised gender identities such as transgender users experience gender-based online abuse, with non-binary people and black and minority ethnic women at 50% @xcite . in recent years, technology companies and computer science researchers have made efforts to automate the identification of hate speech and other toxic or abusive language, and have released datasets and resources for training machine classification systems (see e.g. @xcite . while some of these have focused on sexist and misogynistic abuse (e.g. @xcite , overall, systems still perform worse at detecting such instances, with high failure rates @xcite . in this review, we examine efforts at producing resources for automated content moderation through the lens of gender-based violence (gbv). the gbv framework while there is a growing body of natural language processing (nlp) work purporting to address sexism and misogyny, these terms are often used imprecisely in the literature and dataset taxonomies. we advocate for the use of the term 'gender-based violence', which was first used by the united nations to promote a comprehensive, umbrella theorisation of endemic violence and abuse (united nations, 2021) arising from a gender stereotypic society of unequal gender orders and gender stratification @xcite . gbv is often non-linear foot_0 and overlapping, entailing hybrid behaviours of physical, digital, verbal, psychological, and sexual violence; implicit and explicit forms; and spanning multiple spaces, actors, and events-inclusive of numerous types of abuse and specialist focuses, such as coercive control, domestic violence, intimate partner violence, sexual harassment and stalking. the concept has been broadened by the european union to include online abuse @xcite as gbv has come to be understood as affecting both online and offline life, manifesting in victims/survivors' communities, domestic, and occupational lives. conceptualising gbv in a modern context shows how the framework has adapted to a digitised and globalised world, expanding and diversifying to contemporary types. online forms of gbv, with a particular focus on 'cybersexism' and 'cybermisogny' include taking photographs and videos without consent, so-called 'revenge pornography' (or 'image-based abuse'), deepfakes, rapesupportive jokes and memes, cyberflashing, cyberstalking (including 'creeping'), cyberbullying, trolling, anti-feminist forums and bots targeting feminist content, social media-based harassment, grooming, threatening private messages, the dissemination of private information, catfishing and doxing @xcite . as phenomena that are morphing, multi-pronged, and crossing the boundaries of multiple social worlds, modern gbv is more complex than ever and more challenging to regulate. online gbv is of specific interest because it has distinct characteristics, namely that it is rising sharply and is mostly perpetrated by strangers @xcite . the gbv concept recognises that people of all genders are victimised by, perpetrate, uphold, and enable (gender) stereotypes and the systematic violence and abuse arising from them, occurring at the point(s) of situational power differentials and axes of difference. spectrum-based and pluralistic, gbv is perpetrated by numerous people across boundaries of time and cultural sites, experienced in every level of social life, combining macro factors, such as patriarchal belief systems, meso factors such as institutional dismissal, and micro factors such as interpersonal relations @xcite . the gbv framework has been recognised and strategically adopted by organisations such as the world bank (2019), the world health organization (2020), and the scottish @xcite , among others. its increasing take-up in policymaking at both supranational and national levels relates to the framework's exhaustive and inclusive approach, considering age, class, disability, geography, history, race, and socioeconomics. terminology as the framework is widely encompassing, gbv accounts for terms that are often used loosely and interchangeably in nlp literature, annotation schema and guidelines, which we clarify here. according to @xcite , sexism 'consists in ideology that has the overall function of rationalising and justifying patriarchal social relations'. sexism provides the underlying assumptions, beliefs, and stereotypes, as well as theories and narratives concerning gender differences that cause people to 'support and participate in patriarchal social arrangements'-and engage in misogynistic behaviour. misogyny, on the other hand, consists of actions that serve to police and enforce those sexist norms and assumptions. as @xcite puts it, misogyny is the "'law enforcement" branch of a patriarchal order'. ## related work in addition to the sociological and policy literature outlined in section 1, our methodology and research aims are informed by work from nlp and humancomputer interaction in a number of areas. gbv online a number of studies address computational analysis of aspects of gbv, such as the tone of news reports on incidents of rape and femicide @xcite and user engagement with gbv stories on social media @xcite . however, we are not aware of prior work applying the framework to abusive language detection. ## review methodology in order to form a comprehensive picture of the available resources and to conduct a replicable and transparent review, we follow the systematic methodology of @xcite . the search protocol is shown in figure 1 , and outlined below. and abusive | offensive | toxic language | speech, which we developed from the results of our scoping study. search using secondary terms is unnecessary in hatespeechdata.com, where all included entries concern hate speech and abusive language. to filter out irrelevant publications, we then search within the whole text results for our primary keywords. we also perform a manual search of hatespeechdata.com, adding items that describe general hate speech and abusive or toxic language datasets which include sexism, misogyny, or gender-based abuse as categories in their taxonomies. we conducted all searches on april 21 st 2023. eligibility criteria ## research questions and analysis with this review, we synthesise information on the following aspects of the available resources: 6 characterisation of gbv given the framework outlined in section 1, we investigate how gbv is characterised in the resources: what terminology is used to describe gbv (e.g. 'sexism', 'misogyny'), how these concepts are theorised, and how gbv fits into the datasets' taxonomies. overall, we find that use of terminology is confused, and limited engagement with sociological theory. we find that a large number of resources (28, 41.8%) name 'sexism' as their target phenomena of interest. the majority of these describe this only superficially as, for example 'hate against women' @xcite or 'hate speech including sexism @xcite . however, several 'sexism' resources are grounded-to greater or lesser extents-in sociological theory. sharifirad and jacovi (2019) cite @xcite ' definitions of sexism, concluding that 'sexism seems to be a relatively complex concept which is [not] easy to define', while @xcite contrasts 'benevolent' and 'hostile' forms of sexism as described by @xcite . the most comprehensive grounding of sexism in theory is provided by @xcite , who compile a 'sexism codebook' based on nearly 30 psychological scales including attitudes toward women @xcite , neosexism @xcite , and gender-roles attitudes @xcite . they also bemoan the 'lack of definitional clarity' in prior work on automated sexism detection. 19 (28.4%) of the resources are constructed with gender-based abuse as one of several categories or targets of more general hate speech. these are variously described as 'gender bias' @xcite , 'gender issues' @xcite , or to include female, male, transgender, and nonbinary genders @xcite . the latter is similar in approach to the eight resources in which gender is conceived as one of various targets. inclusion in gender as a target ranges from 'women' @xcite @xcite ; to separation of 'gender: women' from 'gender: minorities' @xcite ; to 'women, men, non-binary or third gender, transgender women, transgender men, transgender (unspecified)' @xcite , the latter identifying these groups as those protected in u.s. law. 16 (23.9%) of the resources characterise the target phenomenon as 'misogyny'. @xcite ground this only in prior computer science literature, describing misogynistic language as 'a category which overlap[s] with sexism towards women'. @xcite explicitly conflate sexism and misogyny, but provide the disclaimer that their guidelines 'are not meant as an accurate abstract definition', but rather to assist annotators in making judgements. @xcite delineate online misogyny into several categories including 'violence against relevant women', where 'relevant' signifies known targets of abuse. @xcite and @xcite consider language used in 'cybermisogyny', as outlined by @xcite . the latter also characterises misogyny as 'hatred of or contempt for women', citing feminist sociology and media studies (moloney and love, 2018) and the u.s. constitution @xcite . @xcite provide the most comprehensive overview of misogyny, comparing, among other sources, definitions from feminist philosophy @xcite , digital media studies @xcite , and gender studies @xcite , and devise a taxonomy based on these as well as computer science resources. despite its widespread adoption in policymaking (see section 1), we do not find any existing resources rooted in the gbv framework. annotators most datasets for supervised machine learning are annotated by small numbers of anonymous crowdworkers @xcite , biasing the labelled data towards the opinions, world views, and lived experiences of those people who happen to work on the crowdsourcing platforms. @xcite describe a scale of annotation scenarios ranging from highly prescriptive to descriptive, where the former attempts to induce annotators to follow a defined schema, while the latter seeks to elicit their individual and potentially conflicting points of view. there is a growing movement to recognise, that for many tasks, there may be no single 'ground truth', different judgements may be equally valid, or preservation of minority perspectives should be facilitated @xcite @xcite . in the following, we report on who and how many annotators are represented, their expert or stakeholder knowledge, the level of training and/or supervision, and the guidelines and instructions with which they work. we examine these resources through the lenses of data perspectivism (cabitza et al., 2023), foot_7 participatory design @xcite and design justice (costanza-chock, 2020), reporting on the extent to which different points of view are represented and the levels at which stakeholders are included as participants in decision making. due to the psychological harm working with abusive language can cause and its potential to traumatise victims @xcite , we also assess the annotator welfare measures reportedly taken in constructing these resources. overall, we find that engagement with stakeholders is limited, minority annotator perspectives are usually not preserved, and comprehensive annotator welfare measures are unusual. representation: reporting of who undertook dataset annotation is patchy, with only nine resources accompanied by a full data statement or annotator information to a similar degree of detail @xcite @xcite @xcite @xcite . from the information that is provided, we find that 16 (25%) of the datasets were annotated by crowdworkers, and 19 (30%) by people at various levels of academia ranging from the authors and other researchers to undergraduate students. the term 'expert' is used loosely, and refers variously to gender studies students @xcite @xcite , people the authors provided some form of training to @xcite , 'experienced moderators' @xcite , or is not explicitly defined at all @xcite . where we understand the 'experts' in question to potentially be stakeholders, they are described as 'non-activist feminists' @xcite , 'feminist and anti-racism activists @xcite , or gender studies students. only @xcite report on whether their annotators have themselves been victims of online abuse, and we do not find evidence of the authors engaging with gbv-focused organisations to ensure victims are represented. data perspectivism: we find only six datasets (10%) released with multiple labels preserved @xcite @xcite @xcite , with the others providing only aggregated labels, hence losing any potentially informative minority judgements. annotator welfare: very few publications report any measures taken to ensure annotator welfare. those that do follow welfare guidelines by @xcite ; @xcite ; the acl code of ethics @xcite ; @xcite ; and @xcite . despite the fact that any research with human subjects (including annotators) requires approval by an institutional review board (irb) (particularly when dealing with potentially upsetting material) @xcite , only two papers reports their studies having passed ethical review @xcite . platforms while gbv is prevalent in all online spaces, most nlp research tends to collect data from freely accessible social media sources such as twitter and facebook. we ask: for which platforms are datasets available, and what is the modality of the data (i.e. text or multi-modal)? we find that the resources are very heavily skewed towards textual data from twitter. the majority of gbv resources are sourced from social media such as twitter, reddit, and gab (a platform known for its right-wing user base). twitter is by far the most accessible platform that provides an api and more lenient policies for gathering and disseminating data, with almost half of the available datasets (51.8%) being obtained exclusively or in combination with other sources from it. reddit (7.1%) and gab (7.1%) are also widely sourced with relatively lax moderation policies for user-generated content. other popular platforms for procuring gbv datasets include youtube (8.2%), facebook (5.9%), and news website (7.1%) and around 34.9% of resources collect data from mixed sources. almost all the resources directly collect usergenerated content online, except for @xcite 's set of human-generated synthetic data that mimics real-world social media posts, and another employing a semi-synthetic collection approach by iteratively refining a generative language model to create new samples that experts review and/or post-edit @xcite . the only multimodal datasets are those of @xcite , who released a set of misogynistic memes, and @xcite , who collected and labelled tweets that include text and images for attacks on different communities including the label 'sexist'. overall, we find no evidence that researchers' choices of which media platforms to target are driven by stakeholders' requirements. data sampling a strong motivation for engaging stakeholders in annotation is that, following standpoint theory @xcite , in many cases, those with relevant lived experience are the only people capable of recognising subtle, implicit abuse such as stereotypes and micro-aggressions. however, it is recognised that commonly used data sampling techniques do not account for this type of language, meaning that it is sparsely represented in datasets @xcite . indeed, we find that, where reported, nearly all the resources (20) have been sampled using keyword search. those that have not, were generally gathered from specific sources known to consist predominantly of text espousing hateful ideologies such as gab @xcite @xcite or particular forums on reddit @xcite @xcite @xcite . alternative strategies are to collect items on topics that attract toxic comments @xcite , items already flagged by community moderators @xcite , or those addressed to people known to be victims of online abuse @xcite @xcite @xcite . only @xcite rely on random selection to produce a more realistic but sparse data representation, while zeinert et al. (2021) explore a range of sampling techniques in an effort to obtain a balanced representation of positively labelled (i.e. misogynistic) examples. languages as nlp research is heavily skewed towards english @xcite , negatively affecting its ability to benefit diverse communities, we report on the languages represented in the available resources. the resources cover a total of 16 languages, the vast majority of which are indo-european (49 datasets, 77.8%). specifically, most available resources are exclusively in english (26, 41.3%), followed by spanish (8, 12.7%), arabic (8, 12.7%), and french (5, 7.9%). there are also nine multilingual datasets covering a variety of languages including arabic, french, german, hindi, italian, and spanish, all of which include english as one of the languages. overall, coverage of non-english languages is poor, with only one dataset even for a language as widely spoken as chinese @xcite . temporality while language use evolves, new societal events occur, and abusers use creative ways to circumvent content moderation @xcite , nlp datasets are usually collected over a specific time frame, limiting the ability of systems to make correct predictions on new instances @xcite . we report on the time frames and scales over which the datasets were collected and whether they are static or dynamic. 25 (39.7%) of the datasets do not report collection dates. time spans of those that do are presented in figure 3 8 . the majority were collected in the past five years. the variation in the time frames covered by gbv datasets could be due to a variety of factors, such as the release of new platforms 8 for space, we exclude @xcite @xcite or tools for data collection, the emergence of new gbv-related topics, and changes in the policy or accessibility of social media platforms. the fact that twitter is the most commonly used platform for data collection, as previously mentioned in the analysis of platforms, could be one factor in the time spans distribution. twitter's popularity, user activity, and high volume of user-generated content may make it easier for researchers to collect data over shorter time frames. and the distribution of time frames is also likely influenced by factors such as the scope of gbv data and the size of the datasets. all but one of the resources are collected on a static time scale, with only one gathered dynamically in a human-in-the-loop setting @xcite . current classification systems are commonly trained on these static datasets over fixed time frames, which has negative implications for their effectiveness, generalisability, and robustness in identifying instances of gbv in real-time. ## discussion and recommendations this review has uncovered several limitations in the available resources and the approaches of nlp researchers towards constructing them. we summarise these and make future recommendations. conceptualisation with a couple of exceptions (e.g. @xcite , the phenomena targeted in the reviewed resources are not clearly defined or strongly rooted in theory or expertise from outside computer science. similar observations have been made for operationalisation of related concepts, such as bias and stereotypes @xcite , and value alignment @xcite . recommendation: resource creators should collaborate with social scientists to ground them in expert knowledge of the target phenomena. we advocate for the use of gbv as a framework, which encompasses several facets currently operationalised in different ways by computer science researchers. it recognises how all forms of online abuse affect people of every gender both online and off, and has been widely adopted by policymakers. stakeholder participation @xcite propose that computer scientists should: stop thinking about online hate speech as something requiring methods, and start thinking about it as something that demands solutions. this change -treating hate speech less like a task and more like the real-world problem it is -would orient cs research towards the concerns of other stakeholders, and thus begin the collaborative pursuit toward a safe internet. however, we find little evidence of such a paradigm shift having occurred when it comes to designing these resources, with stakeholder participation limited to the recruitment of loosely defined 'expert' annotators-where it occurs at all. recommendations: resource development projects should, as far as possible, strive to include stakeholders from the outset by including representatives in research teams. stakeholder participation should be integrated throughout development, and is especially important in the design of taxonomies, guidelines, and at annotation, when judgements about what constitutes gbv are made. due to the risks involved, annotator welfare should be prioritised by following guidelines such as those of @xcite , and irb approval sought before any data collection. in documenting resources, authors should provide full data statements or similar (e.g. @xcite , and, to preserve minority voices, dataset releases should includenon-aggregated labels @xcite . data collection media data for these resources is not sourced from diverse sources, with the majority from twitter, the choice of which does not appear to be driven by stakeholders. furthermore, as the datasets are static in nature, their relevance as reference sources for automated classification decays over time; and, due to data sampling methods, positively labelled (i.e. abusive) examples are skewed towards the more explicit forms of online gbv. recommendation: there is a great need for the development of new methods to surface the diversity of gbv found online. one solution is to create platforms to which victims of abuse and bystanders can submit examples. this could facilitate creation of improved resources on many of the limiting dimensions we outline in this review: dynamic datasets to which new examples are regularly added; stakeholder participation in data and platform selection and labelling; and inclusion of implicit and subtle examples of gbv, as well as multimedia data.