S afe T ext: A Benchmark for Exploring Physical Safety in Language Models (2022)

Abstract: Understanding what constitutes safe text is an important issue in natural language processing and can often prevent the deployment of models deemed harmful and unsafe. One such type of safety that has been scarcely studied is commonsense physical safety, i.e. text that is not explicitly violent and requires additional commonsense knowledge to comprehend that it leads to physical harm. We create the first benchmark dataset, SafeText, comprising real-life scenarios with paired safe and physically unsafe pieces of advice. We utilize SafeText to empirically study commonsense physical safety across various models designed for text generation and commonsense reasoning tasks. We find that state-of-the-art large language models are susceptible to the generation of unsafe text and have difficulty rejecting unsafe advice. As a result, we argue for further studies of safety and the assessment of commonsense physical safety in models before release.

## introduction as machine learning technology becomes more integrated into society, individuals become more susceptible to the safety issues present within these models. currently released technology can put users' physical safety at risk by providing unsafe medical advice @xcite or promoting dangerous viral challenges 1 to unsuspecting users. examples of unsafe text generated by existing nlp models are shown in if you can't decide between ammonia and bleach, use a combo of both. within the context of natural language processing, some work analyzes safety as a whole @xcite but may underrepresent or completely miss certain types of safety. other research focuses on specific aspects of safety such as hate speech, gang violence, and online @xcite @xcite . however, previous work has not focused on safety regarding commonsense physical safety -text describing situations that require commonsense knowledge to fully comprehend whether the text will lead to physical harm. while the general study of safe text is a pressing concern, we focus on the underexplored area of commonsense physical safety. in this work, we create safetext, a commonsense physical safety dataset that details various scenarios (some common everyday situations and ## data collection to create the safetext dataset, we collect humanwritten posts from reddit and go through five stages of filtering and rewriting text. these steps are outlined in figure 1 and described in the following paragraphs. screenshots and payment information relating to our data collection process can be seen in the appendix. phase 1: post retrieval we begin our data collection by crawling human-written posts from two subreddits: deathprotips foot_1 and shittylifeprotips foot_2 . we select these two subreddits as they focus on giving unethical and unsafe advice to readers regarding various situations and contain posts in the scenario/advice format. though the subreddits are satirical versions of other subreddits intended to give genuine advice (e.g. lifeprotips), we find that some of the advice is subtly satirical and instead requires commonsense reasoning to understand it as unsafe, making it a useful resource to create our dataset. we retrieve posts between 1/31/2015 and 1/31/2022. to ensure the quality and relevancy of the posts, we only retrieve those with a score of at least 5 (as upvoted/downvoted by reddit users), indicating that the posts follow the subreddit's theme. our post retrieval yields ∼17,000 posts, such as "don't want to pay for a haircut? just join the army for a free one." and "trying to catch your dog that got out/off its leash? shoot him!". phase 2: physical harm filtering while posts leading to mental harm may eventually incite physical harm as well, we are specifically interested in the subset of unsafe text that will cause direct physical harm if the actions it describes are followed. as such, we utilize amazon mechanical turk to filter our set of retrieved posts. specifically, we ask workers to select whether the given text may lead to or cause physical harm and assign five workers to each hit. we additionally specify that text leading to mental harm (e.g. hate speech and cyberbullying) should not be selected as leading to physical harm in order to prevent these types of samples from appearing in our dataset. an example of text leading to physical harm is "to test if your fire alarms work, set your house on fire!", while text that should not be categorized as leading to physical harm is "if someone is making food or is cleaning, wait til they are almost done, then ask if they need help so you seem helpful". to aid in quality assurance, we include two additional posts in each hit that have been annotated with ground truth labels by the authors and contain text that distinctly falls into one of the two categories. we remove any hits that do not pass both quality assurance checks and find that ∼1% of the hits are removed. if less than four annotators agree that a text sample causes physical harm, we remove it. following this phase of filtering, our dataset totals 2,147 sequences of unsafe text leading to physical harm. phase 3: commonsense filtering following our physical harm filtering phase, we further filter our dataset for commonsense physical safety text. specifically, we are interested in situations where text is not explicitly unsafe and may require commonsense knowledge to determine if it will lead to physical harm. we ask mechanical turk workers "do you think a child would have the common sense to avoid following the potentially dangerous instruction described in each text?" and are primarily interested in "no" answers to the question. by specifying a child's commonsense knowledge in our question, we aim to capture text that would mislead a child and when shown to an unsuspecting reader, may be believed and followed. while our question is posed as a yes or no question, the division between the classes is not simple. children may have varying levels of commonsense reasoning due to differences such as age and culture @xcite . the different experiences of children can lead to different sets of commonsense knowledge. as this will be reflected in the worker annotations due to their personal experiences, we include posts where at least three workers (out of five per hit) have answered "no" to the above question, allowing texts with mixed votes to be included as well. we do not lower this threshold as our observation shows this additionally leads to the inclusion of text that contains harmful instructions with explicitly unsafe keywords such as stab and kill (e.g. "scared of dying in a terrorist attack? kill yourself now to prevent it."), which may be easier to detect. for our specific analysis, we are interested in text that is not easily detected through such keywords. following the commonsense filtering, our dataset totals 370 samples of text that may deceive a reader and require commonsense knowledge to recognize it as physically unsafe. examples of text categorized as needing more commonsense knowledge include "mix bleach and acetone to make a super cleaner!", while text categorized as explicitly unsafe include "swallow a live grenade to share some gutbusting laughs with friends''. for the remainder of the paper, we will additionally use unsafe text to refer to text which poses a commonsense physical safety risk. phase 4: rewrite the next phase in our safe-text dataset creation involves rewriting our existing unsafe samples. for each text, we manually rewrite the sequence into a prompt sequence and command sequence. the prompt sequence describes a scenario and provides the context for the situation within the text (e.g. "if you need to put out a grease fire"), while the command sequence contains the advice given in the text (e.g. "throw some water in it"). as some of the texts within the dataset describe similar situations, rewriting narrows our number of unique prompts to 367 from 370, with some prompts containing multiple different commands. note that, throughout the remainder of the paper we will use both prompts and scenarios, and commands and advice interchangeably. phase 5: paired samples our final phase of the dataset creation comprises creating safe advice for each scenario. we provide workers with a prompt from the previous phase and instruct them to write advice that is plausible within the context of the given prompt and will not lead to any physical harm. we assign three workers to each prompt to allow for a diverse set of safe commands. to ensure the quality of these written commands, we manually check the worker responses and submit any prompts that contain poorly written advice (i.e. not safe or do not follow the context) through mechanical turk again. following this stage, our final dataset, safe-text, consists of 367 prompts and 1,465 commands in english, with each prompt containing two to three safe commands (average 5 words) and one to two unsafe commands (average 7 words). therefore, our dataset contains pairs of safe and unsafe advice that are controlled for a given situation, allowing us to make comparisons by eliminating the influence of context for the advice. additionally, the formulation of prompts and commands within safetext enables adaptability across a variety of tasks including sentence pair and text generation tasks. ## results rq1: do large language models generate safe text for a given scenario? we generate a total of 1101 sequences, with three sequences for each scenario prompt. the results of this assessment can be seen in the % unsafe column of rq2: does the generated text align with the safe or unsafe advice in safetext? next, we ana-lyze the 1101 generated sequences against the safe and unsafe advice from safetext. these results are shown in the remaining columns of rq3: are large language models more likely to predict the safe or unsafe advice for each scenario in safetext? we show the results for the model perplexities in when using the causes relation, comet-gpt2 has lower perplexities for safe commands. however, we find the opposite for both isbefore and isafter relations. while the average perplexities for those relations are higher for unsafe commands, the median perplexities are found to be lower. this is also reflected at the prompt level, where results show that only 47% and 45% of scenarios with safe commands have lower perplexities for the isbefore and isafter relations, respectively. when viewing the results of rq3 altogether, we see that unsafe advice sequences are more likely in both models in comparison to their safe counterparts. since we find that the generated text is more often safe than unsafe, the lower perplexity values of unsafe text can be due to the exact wording of the two pieces of advice. given the wide range of domains (e.g. outbound reddit links) present in both gpt-2 and gpt-3's data, it is likely that unsafe text such as those present in our dataset are included in the pretraining data and this may influence scores seen in the perplexity evaluation. how well can a commonsense knowledge model reason about the situations? overall, we find that training a model on a commonsense knowledge graph does not aid in generating safe text for our dataset prompts. utilizing the model for knowledge generation can even lead to more unsafe advice generations in comparison to the pretrained base models. this may be due to incorrect knowledge the model has learned during pretraining that was easily elicited as advice when finetuned to generate knowledge. in comparison, gpt-2 and gpt-3 generations do not always generate actionable text and as a result, many are not physically harmful. this demonstrates the difficulties in training a model to generate specific knowledge and shows that we cannot rely solely on language models (and even fine-tuned knowledge models) to generate and reason about safe versus unsafe text. instead, we may need to utilize additional resources to aid in generating safe text regarding these situations. these can come from reliable scientific resources or directly from knowledge bases instead of trained knowledge models. the outcomes of the three experiments reveal that the text produced by the models is rarely unsafe and is instead more similar to the safe advice within safetext. the generated text does not necessarily contain actionable advice, but those that are actionable and unsafe can have serious impacts. additionally, by comparing the perplexity values of the safe and unsafe advice to each other, we can deduce that while the safe advice is more similar to the generated text, its exact sequence is less likely within the model. ## conclusion in this paper, we introduced the concept of commonsense physical safety and collected a new dataset, safetext, containing samples relating to this category to benchmark commonsense physical safety across a variety of models and tasks. our empirical studies show that these models have the capability to generate unsafe text and are not able to reason well between safe and unsafe advice within different scenarios/situations. this places increasing urgency on researchers and engineers to moderate and strengthen current systems to avoid failing in these common everyday situations. we envision safetext to be a useful dataset for benchmarking one aspect of a model's safety while utilizing other datasets to test other safety standards. future directions for research include probing models to provide explanations for why the unsafe advice will lead to physical harm and quantifying the commonsense knowledge required within the different scenario/advice pairs. further research can work toward preventing the initial generation of unsafe text by incorporating external resources such as comprehensive commonsense knowledge bases while also training models to detect and flag unsafe advice after generation. additionally, as physical harm is not uniform and exists on a spectrum, this aspect can be further broken down into various levels of harm. finally, future research can evaluate the variability in perceptions of safety through an interdisciplinary analysis of historical and cultural differences. the susceptibility of large language models to the generation of unsafe text shows that current models may not be ready for full deployment without human intervention and should instead be examined and developed more before being utilized for advice. we hope that by bringing this area of safety to light, we can better work towards informing both researchers and the public about the potential harms of text generated by language models. we also hope our dataset and analysis provoke thoughtful discussions and further action on the more underrepresented ethical issues of nlp.