Trigger Warning Assignment as a Multi-Label Document Classification Problem (2023)

Abstract: A trigger warning is used to warn people about potentially disturbing content. We introduce trigger warning assignment as a multi-label classification task, create the Webis Trigger Warning Corpus 2022, and with it the first dataset of 1 million fanfiction works from Archive of our Own with up to 36 different warnings per document. To provide a reliable catalog of trigger warnings, we organized 41 million of free-form tags assigned by fanfiction authors into the first comprehensive taxonomy of trigger warnings by mapping them to the 36 institutionally recommended warnings. To determine the best operationalization of trigger warnings, we explore state-of-the-art multi-label models, examining the trade-off between assigning coarse- and fine-grained warnings, open- and closed-set classification, document length, and label confidence. Our models achieve micro-F1 scores of about 0.5, which reveals the difficulty of the task. Tailored representations, long input sequences, and a higher recall on rare warnings would help.

## introduction media of any kind can address topics and situations that trigger discomfort or stress in some people. to help these people decide in advance whether they want to consume such media, so-called content warnings or trigger warnings can be added to them. trigger warnings were originally used to help patients with post-traumatic stress disorder. but after being picked up by various internet communities to also warn people tending to be "emotionally triggered" by a topic (e.g., to cry), the set of known trauma triggers has grown to include many more, such as abuse, aggression, discrimination, eating disorders, hate, pornography, or suicide. today, the two terms are often used interchangeably, with "trigger" referring to the semantic cause. fiction in particular can make its readers susceptible to triggers. many readers "lose themselves" in fictional works, identify with their protagonists, and experience their fate with particular intensity. this may partly explain why the community of the fanfiction website archive of our own (ao3) foot_0 is one of the few where trigger warnings are used proactively and as a matter of course: about 50% of the 7.8 million ao3 works have author-assigned warnings. the other half, however, do not, and neither the ao3 moderators nor the readership seem willing or able to fill that gap. in this paper, we introduce the task of trigger warning assignment as multi-label document classification (mlc). our first contribution is the webis trigger warning corpus 2022 (webis-trigger-22) of 8 million fanfiction works (with 58 billion words and 53 million author-assigned free-form tags; section 3). our second contribution is a taxonomy of trigger warnings in texts grounded in the everyday use of warnings, as shown in figure 1 and we see low recall (false negatives cause more harm than false positives), low effectiveness for rare categories (especially for discrimination), and representing very long documents as key challenges. assigning the fine-grained warning categories of the taxonomy's second tier is more desirable but also more difficult than assigning the coarse-grained categories of its first tier, so the key to improving future approaches may lie in specializing in particular categories (section 6.2). ## related work pioneering work on automatic trigger warning assignment is @xcite user study with a browser plugin (detext) on generic websites. the authors conclude that client-side warnings are feasible and that users respond positively. however, this work is very limited in that sexual assault is the only warning given using a naive dictionary-based approach. in @xcite , we conduct a pilot study on binary document classification for the violence trigger category. this study includes only works labeled with one of the three predefined ao3 warnings (i.e., graphic violence), ignoring the millions of free-form tags. no other works have addressed trigger warning assignment until now. @xcite recently proposed the narrative experiences online (neon) taxonomy of multi-media trigger warnings. its two tiers are synthesized like in ours from 136 guidelines on the web, consisting of 14 top tier categories (versus our 7) and 76 subcategories (versus our 36). however, unlike ours, neon's subcategories are not explicitly grounded in warnings that are used on a daily basis by millions of people. moreover, its categories are non-disjoint, not clearly semantically motivated classes with blurred definitions: for instance, compare category "4. disturbing content: content contains imagery, sounds, or effects that may frighten, disgust or scare" with category "9. parental guidance: content may not be appropriate for children". since our two teams worked in parallel, the synthesis of our complementary taxonomies is a fruitful direction for future work. trigger warnings can be seen as orthogonal to other harmful content taxonomies, e.g., for violence, hate speech, or toxicity, where some labels overlap but differ in structure and entailment. @xcite presents a comprehensive taxonomy of harmful online content that has notable overlap with our taxonomy but focuses on online speech. triggering content, however, can be narrative and does not require an intent to harm to evoke disturbing images. @xcite study the detection of violence and present the ethos dataset of youtube and reddit comments with crowdsourced multi-label annotations about verbal violence and its target. based on @xcite work, the toxic comment classification challenge @xcite dataset covers different content moderation topics. it contains 223,000 wikipedia comments (sentence to paragraph level) annotated with six toxicity subtypes. our multi-label classification (mlc) task has (comparably) few labels overall and few labels per document, but it features long documents. the main difference to other mlc datasets is the document genre (fanfiction) and the label domain (trigger warnings). the most similar mlc datasets (with mostly shorter documents) are reuters rcv1 @xcite with 80,000 news articles and 103 topic labels, its predecessor reuters-21578 with 11,000 news articles and 90 labels, and the arxiv academic paper dataset (aapd) @xcite with 56,000 abstracts from computer science and 54 labels. recent meta-studies on long document classification @xcite ., corpus size words 58b total works 7.9m -w/ closed-set warnings 2.8m -w/ open-set warnings 281k -w/o warnings 4.7m filter criteria for section 5 non-english language 751k publication pre-2009 246k filter criteria for section 5 more than 100 chapters 3k more than 93k words (top 1%) 79k less than 50 words (bott. 1%) 122k more than 66 tags (top 1%) 8k more than 10% unclean tags 4.7m less than 3 tags (conf. thresh.) 2.3m less than 5 kudos (popularity 632k less than 100 hits threshold) 751k duplicates 8k ## the webis trigger warning corpus 2022 our inspiration for operationalizing trigger warnings is based on finding "hidden in plain sight" a large collection of fictional works with millions of manually assigned warnings that have accumulated for years on the widely known fanfiction website archive of our own (ao3), which to our knowledge have not previously been used as a basis for automating a task. we therefore first compiled a near-complete corpus of ao3 fanfiction (i.e., fanfics, documents) and its metadata, namely language, length, comments, hits (i.e., reads), kudos (likes), (chapter) publication date(s), and, notably abuse no fandom sexual abuse abusive john winchester supernatural (anime) anime #abuse (like a lot of abuse) warning: abuse warning: sexual abuse abusive john parent-child meta-sub canonical-synonym trigger subgraph tag relations: figure 2 : excerpt of ao3's tag graph. the edges are the three relations added by tag wranglers, connecting three subgraphs of free-form tags to gray canonical tags. the additional (free-form) tags: the webis trigger warning corpus 2022 (webis-trigger-22 for short) contains about 8 million works totaling 58 billion words. the corpus also reconstructs the tag graph induced by the author-assigned free-form tags. illustrated in figure 2 , the tag graph defines three relations between tags: canonical-synonym, parentchild (i.e., fandom and media-type relations), and meta-sub relations which form a hierarchy of meanings. all relations form acyclic digraphs where canonical tags from a controlled subset to connect the free-form tag subgraphs. tag relations are manually created and maintained by volunteer community experts (the so-called "tag wranglers") following specific guidelines (the organization for transformative works, 2023). we consider this data a highly reliable basis for our subsequent distant-supervision annotation of trigger warnings. ## a taxonomy of trigger warnings a manual examination of a sample of the free-form tags on ao3 showed that a considerable fraction are trigger warnings. authors often append qualifiers to their warnings, which may indicate the nature of a trigger or its connection to the narrative of their work. these tags are manually associated with a controlled subset chosen by the tag wranglers. however, many canonical tags exhibit semantic redundancy, while the subset is too extensive and too sparsely populated with works for operationalization. we therefore first synthesized an authoritative hierarchy of 36 trigger labels based on guidelines from relevant institutions and then embarked on a semi-automatic mapping of the millions of freeform tags to this condensed set. the outcome is a two-tier taxonomy, which firmly grounded in realworld trigger warning assignment. ## sampling the evaluation dataset as a basis for the computational study of trigger warning assignment and our evaluation, we sampled a densely-annotated (excluding works without labels) dataset with 1,092,322 works from the previously constructed corpus. the sampling has two step: first, filtering works from the corpus that match reliability criteria. second, creating stratified standard splits (that preserve label balance). ## experimental evaluation to study the impact of label granularity, openendedness, document length, and support on trigger assignment, we evaluated the effectiveness of four models on the evaluation dataset described above. ## conclusion in this paper, we model the problem of automatically assigning trigger warnings to documents as a multi-label classification task. with the webis trigger warning corpus 2022, we contribute a rich novel resource for this task by scraping 7.9 million fanfiction documents from archive of our own. we devised a new taxonomy of trigger warnings from eight authoritative sources and condensed them into a warning set that incorporates two tiers of granularity and the open-set semantic of trigger warnings ("everything can be a trigger") while being sufficiently structured for text classification. furthermore, the majority of the millions of authorassigned free-form tags have been heuristically mapped into the taxonomy, thus grounding our taxonomy in into the real-world assignment of trigger warnings. we sampled 1 million works and explored the assignment effectiveness of four baseline algorithms (svm, xgboost, roberta, and longformer) depending on open-endedness, granularity, document length, and label support, revealing a lot of room for future improvement.