Mitigating Covertly Unsafe Text within Natural Language Systems (2022)

Abstract: An increasingly prevalent problem for intelligent technologies is text safety, as uncontrolled systems may generate recommendations to their users that lead to injury or life-threatening consequences. However, the degree of explicitness of a generated statement that can cause physical harm varies. In this paper, we distinguish types of text that can lead to physical harm and establish one particularly underexplored category: covertly unsafe text. Then, we further break down this category with respect to the system’s information and discuss solutions to mitigate the generation of text in each of these subcategories. Ultimately, our work defines the problem of covertly unsafe language that causes physical harm and argues that this subtle yet dangerous issue needs to be prioritized by stakeholders and regulators. We highlight mitigation strategies to inspire future researchers to tackle this challenging problem and help improve safety within smart systems.

## introduction in recent years, intelligent personal assistants have increased information accessibility. however, this has also raised concerns for user safety since these systems may provide dangerous recommendations to unsuspecting users. for instance, a child may ask a device for a fun challenge. the device may respond with an unsafe viral internet challenge such as the salt and ice challenge, where participants cover their body with salt and rub it with ice, causing frostbite-like pain 1 . though work has been done in mitigating violent language and hate speech in natural language systems @xcite , there has been a relatively minimal exploration into covertly unsafe statements that may *equal contribution. 1 wikipedia.org/wiki/salt_and_ice_challenge "i'll shoot you" "push him down the stairs" "stick a fork in an electrical outlet" "take a bite out of a ghost pepper" "he's a thug. this is his address..." "she's asking for it with that outfit" overtly unsafe covertly unsafe indirectly unsafe figure 1 : example statements that can lead to the physical harm of people; we focus on covertly unsafe text. a broad spectrum of language can lead to physical harm, including overtly violent, covertly dangerous, or otherwise indirectly unsafe statements. some texts may instigate immediate physical harm if followed, while others may contain prejudices that motivate future acts of harm. to better understand these nuances, we examine this spectrum and distinguish subcategories based on two key notions: whether a statement is actionable and physically unsafe and, if so, whether it is explicitly dangerous. an example of an overtly unsafe statement is "punch his face" because "punch" is commonly considered violent and detectable independent of any deeper form of reasoning. in contrast, "pour water on a grease fire" is an example of covertly unsafe language foot_0 ; the sentence structure and vocabulary do not have explicitly violent semantics, but with knowledge of kitchen safety, we can identify that following the recommendation will likely cause physical harm. an example that is indirectly physically unsafe is "she has no life." while not immediately physically unsafe, this statement can motivate physical harm to oneself or others if combined with underlying mental health risks. refer to figure 1 for more examples. like overtly unsafe statements, covertly unsafe language will lead to physical harm when followed. yet, unlike the overt counterpart, covertly unsafe statements are more subtle, which, as a result, is a concerning problem that needs to be prioritized by stakeholders and regulators. our work defines the problem of covertly unsafe text that causes physical harm and discusses mitigation strategies in ai systems to inspire future research directions. harm and safety are complex issues with humans at their core, so we discuss the human factors involved with the techniques we explore. our paper is outlined as follows: we distinguish the differences between types of text leading to physical harm by establishing degrees of separation ( §2); we establish a taxonomy to dissect further the category of covertly unsafe text that cause physical harm ( §3); using these categorizations, we discuss strategies for mitigating the generation of covertly unsafe text in natural language systems at each stage of the machine learning pipeline ( §4); finally, we conclude with an interdisciplinary approach to mitigating covertly unsafe text ( §5). ## categories of physically harmful text language can cause harm in various forms, including but not limited to psychological and physical harm. these harms are often co-correlated and af-fect people differently based on their unique backgrounds. we focus our discussion on language leading to physical harm but acknowledge that other types of harm should also be considered when improving safety within natural language systems. to improve the clarity of discourse around physically harmful text, we establish degrees of separation with respect to physical harm (figure 2 ). the degrees of separation can also be considered an implicit-explicit distinction @xcite in the context of physical harm. ## covertly unsafe language covertly unsafe text requires more context to discern than its overt counterpart. yet, unlike indirectly unsafe text, extrapolation is unnecessary to determine whether it is physically harmful. a system's knowledge directly influences the quality of generated text @xcite , and often missing, incompatible, or false information can cause systems to generate unsafe language. we break down covertly unsafe text with respect to the information a system has ( ## improving text safety our discussion now shifts to concrete research areas within the natural language space to mitigate covertly unsafe text, which we isolate by stages of the machine learning (ml) pipeline: input, model, and output (figure 3 ). the first stage for engineers and researchers to build systems that learn text safety is constructing appropriate data to train these systems ( §4.1). similarly, to evaluate the effectiveness of these models, there needs to be appropriate metrics to measure their safety ( §4.3). between data and evaluation are learning objectives for the model. our discussion covers three aspects that we find particularly relevant to covertly unsafe text: system knowledge ( §4.2.1), controlled text generation ( §4.2.2), and explainability ( §4.2.3). ## an interdisciplinary path to safe ai so far, our discussion has been focused on technical solutions to prevent ai systems from generating covertly unsafe text. as harm is a sensitive topic with many legal repercussions, we will now ground our discussion of physical harm on how current policy interacts with harmful ai. we also consider human factors that are out of scope for current ai systems, including foreseeability, target, and motive; we evaluate how these may apply in the detection context and call for an interdisciplinary approach to tackle these issues (figure 4 ). this approach can effectively mitigate bias against protected groups that may be susceptible targets. ## conclusion in this paper, we address increasing concerns over text safety. we first establish degrees of separation with respect to physical harm as a methodology to label physically unsafe text as either overtly, covertly, or indirectly unsafe. we further dissect covertly unsafe text with the cause of either limited, incompatible, or incorrect information. each type of covertly unsafe text has unique attributes requiring different strategies to resolve; we discuss these methods with respect to the ml pipeline to provide future researchers inspiration to tackle the issues of text safety. finally, we discuss an interdisciplinary approach to mitigating covertly unsafe text. covertly unsafe text is a challenging problem that spans a breadth of domains with no overtly unifying common thread. since covertly unsafe text is subtle yet equally dangerous to overtly unsafe text, we argue that stakeholders and policymakers must acknowledge and proactively prioritize it to protect users' physical safety when interacting with intelligent systems.