Trigger Warnings: Bootstrapping a Violence Detector for Fan Fiction (2023)

Abstract: We present the first dataset and evaluation results on a newly defined task: assigning trigger warnings. We introduce a labeled corpus of narrative fiction from Archive of Our Own (AO3), a popular fan fiction site, and define a document-level classification task to determine whether or not to assign a trigger warning to an English story. We focus on the most commonly assigned trigger type “violence’ using the warning labels provided by AO3 authors as ground-truth labels. We trained SVM, BERT, and Longfomer models on three datasets sampled from the corpus and achieve F1 scores between 0.8 and 0.9, indicating that assigning trigger warnings for violence is feasible.

## introduction hansel and gretel, a fairy tale † violence and cruelty are commonplace in literature. folk tales, especially fairy tales, but also children's and youth literature are full of dark, horror images, such as burning a human being alive in an oven, as in the fairy tale by the brothers grimm quoted above. and even if most people will not be deeply shaken by such content, some readers may mentally relive their past traumas evoked by the imagery. to proactively alert readers that a text they are about to read contains potentially disturbing material, socalled "trigger warnings" have been proposed. trigger warnings (also referred to as content warnings/notifications/alerts) emerged in online communities (e.g., on tumblr and livejournal) † translation by margaret hunt in the early 2000s @xcite . they are usually presented as short phrases/keywords preceding a text and warn of potentially disturbing content. while there are no universally accepted trigger warnings (anything can be a trigger), many universities meanwhile published guidelines (see, e.g., lists published by the universities of reading and michigan (ur list; um list)). they include largely overlapping lists of triggers referring to health (eating disorders, mental illness) or sexuality (sexual assault, pornography), verbal violence (hate speech, racial slurs), and physical violence (animal cruelty, blood, suicide), among others. surprisingly, assigning trigger warnings is considered a manual task, and, to our knowledge, there is no work in computer science in general, and in natural language processing in particular, that addresses content warnings. we lay the foundation to close this gap by introducing the new nlp task of trigger warning assignment, formulated as follows: given a text and a trigger label, assign a warning to the text if it contains a corresponding trigger. when multiple trigger labels are predefined, this task can be extended from a binary classification problem to a multi-class or multi-label problem and solved by, for example, a set of binary classifiers, one for each trigger. however, the preceding first step is to investigate the feasibility of automatic trigger warning assignment and for this purpose we create the first trigger warning corpus from narratives with and without triggers, using the trigger warnings supplied by the works' authors. our contributions are the following: we (1) introduce the new task of automatic trigger warning assignment, (2) introduce the first corpus compiled from a public archive of fan fiction marked with a trigger warning for violence (section 3), and (3) evaluate models for assigning trigger warnings and analyze their effectiveness (section 4). 1 2 related work constructs related to "trigger warnings" have been investigated using computational approaches under different terms and have spanned a broad range of phenomena. recent research employs terms such as "objectionable content", "objectionable material", "harmful content", "harmful text" @xcite @xcite as broad terms covering diverse types of content that can potentially evoke negative emotions in the recipient of the material (be it verbal or visual), i.e. cause emotional harm at different degrees of severity. the type of content that is often subsumed under those terms includes violence, sexual content, misguided messages, misinformation, verbal aggression, malice, callousness, or social aggression, among others. and while there is also a clear link to sentiment analysis, phenomena subsumed under "objectionable/harmful content" lie only on one end of the sentiment scale (that of negative sentiment), however, have a finer granularity (cf. range of specific types of content, mentioned above, that may evoke harm). now, the notion of "triggering" is equally underspecified (open-ended), but even broader. while most of the objectionable types are indeed unobjectionably harmful-in that they can be linked to intention to harm-there may exist concept associations that are triggering to some individuals which, objectively speaking, have little to no link to intention to harm; consider, for instance, that a mention of a thunderstorm may be triggering to a victim of a severe lightning injury. thus, triggering covers also concepts which would normally be understood to lie at the positive end of a sentiment scale, which can, however, evoke negative associations in some individuals due to their specific traumatic past experience related to the concept. a "trigger warning" just gives a nominal label to the signal that is considered triggering. while we are not aware of prior work on automatic trigger warning assignment nor specifically violence warning assignment, below we outline prior work in nlp and computer science that covers most closely related topics. identifying causes of emotions while affect and emotion recognition in non-fiction text-sentiment analysis more generally-has been long studied in nlp @xcite , research into interactions between emotions and their triggering cause events was introduced only about a decade ago @xcite . cause events here refer to (verb) arguments or events in the text that are highly correlated with a certain emotion, positive or negative. the goal of the emotion cause extraction task is to identify the emotion's stimulus and the computational methods range from rule-based lexico-syntactic approaches through traditional classifiers to recently also deep learning; see @xcite for an overview of the emotion cause extraction area. by contrast the trigger warning assignment task is rather about identifying potentially triggering content which may evoke strongly negative emotions in readers. identifying verbal violence interest in broadly understood verbal violence-although not explicitly referred to as such-has a long history in the nlp community. @xcite and @xcite propose taxonomies of abusive and offensive language, respectively; kogilavani et al. also survey techniques for offensive language detection. @xcite and @xcite provide an overview on hate speech and @xcite more generally on abuse detection methods with "abuse" defined as "any expression that is meant to denigrate or offend a particular person or group". while not considered from the point of view of triggering, this definition fits the category 'hateful language" listed in the institutional guidelines. while most work on verbal violence has been carried out in the context of social media (methods ranging from feature engineering to neural networks) it would be useful to extend those systems to cover a broader range of verbal violence, e.g., literary dialogue, in the context of the trigger warning assignment task. ## the violence trigger warnings corpus as data source, we used archive of our own (ao3), foot_1 a public online anthology of fan fiction, i.e., amateur writings inspired by existing works of fiction: e.g., novels, cartoons, manga. at the time of corpus creation, ao3 hosted about 8 million works. aside from basic meta-data, such as title, author, language, statistics (number of words, chapters, etc.), reader reactions, ratings, fandoms (original source(s)/inspiration), and relationships (characters involved in romantic/platonic relationship(s)), crucially for this research, works are labeled with archive warnings and additional tags. archive warnings ao3 defines a set of six content warnings. authors must actively assign at least one to each of their works. the labels are: (1) major character death, (2) underage (contains sexual activity by characters under 18), (3) rape/non-con (non-consensual sexual activity), (4) graphic depictions of violence (gory, explicit violence), ( 5 ) creator chose not to use archive warnings to avoid spoilers, and (6) no archive warnings apply, if the work has no triggering content. additional tags ao3 allows authors to define open-set, freeform content descriptors, which are used as keywords for search and browsing, like romance, slow burn, fluff, and jealousy, but also to assign additional trigger warnings like abandonment, monsters, blood drinking. additional tags are heterogeneous, user-generated content but frequently used tags are "canonized" by volunteer "tag wranglers". the use of canonized tags is encouraged and supported by the web interface. 3 corpus acquisition for the purpose of corpus acquisition, the entire ao3 was crawled. works were identified via ao3-search using the created_at:date-range query parameter. individual searches were started for each day since the date of the site's creation in order to distribute the load; ao3's crawling limits were observed. urls which were not publicly accessible, redirected to external sites or yielded http errors were omitted. our complete crawl contains 7,866,512 works with 9,705,174 distinct additional tags. 571,525 works are labeled with graphic depictions of violence. dataset sampling because ao3 works do not include any annotations below document levelthat is, we do not know the extent of violent content nor where in the text it can be found-our goal was to build a corpus with high-confidence examples of texts with and without violence. we apply three sampling strategies with varying reliability criteria: random sampling to represent the corpus, famebased sampling to exclude low-effort works, and tag-based sampling to exclude works that are not thoroughly tagged so that archive warnings might be less reliable. all sampling strategies randomly select 10,000 violent works (tagged with graphic depictions of violence) and 10,000 non-violent works (tagged with no archive warnings apply but not with graphic depictions of violence). before selecting the examples, we discarded all works with less than 100 words and works written in a non-english language. the random sample then draws the examples uniformly at random. the fame-based sample first discards all works @xmath0,000 hits @xmath1. the tag-based sample discards all works @xmath2. ## assigning violence trigger warnings we evaluate the four labeled datasets in a text classification setting by building classification models to assign trigger warnings at the document level. ## discussion and limitations the final result (the svm beats both neural models) is unexpected and can be (partially) explained by the influence of document length and topic.