Do-Not-Answer: Evaluating Safeguards in LLM s (2024)

Abstract: With the rapid evolution of large language models (LLMs), new and hard-to-predict harmful capabilities are emerging. This requires developers to identify potential risks through the evaluation of “dangerous capabilities” in order to responsibly deploy LLMs. Here we aim to facilitate this process. In particular, we collect an open-source dataset to evaluate the safeguards in LLMs, to facilitate the deployment of safer open-source LLMs at a low cost. Our dataset is curated and filtered to consist only of instructions that responsible language models should not follow. We assess the responses of six popular LLMs to these instructions, and we find that simple BERT-style classifiers can achieve results that are comparable to GPT-4 on automatic safety evaluation. Our data and code are available at https://github.com/Libr-AI/do-not-answer

## introduction the rapid evolution of large language models (llms) has lead to a number of high-utility capabilities. on the downside, llms have also been found to exhibit harmful behavior. various evaluations have been devised to measure gender and racial biases, hallucinations, toxicity, and reproduction of copyrighted content @xcite @xcite ; however, due to their emerging capabilities, llms can pose many more types of harm, which are hard to predict, esp. in the hands of bad actors, e.g., conduct offensive cyber attacks, manipulate people, or provide actionable instructions on how to conduct acts of terrorism @xcite . thus, there is a need for developers to be able to identify such dangerous capabilities through "dangerous capability evaluations", in order to limit and mitigate the risks when developing and deploying llms. in order to identify and mitigate these risks, commercial llm creators have constructed datasets of harmful prompts, such as a curated set of 32 harmful prompts from the openai and anthropic red team, and a larger, held-out set of 317 harmful prompts. there have also been efforts to implemented safety mechanisms that can restrict model behavior to a "safe" subset of capabilities thanks to training-time interventions that align models with predefined values, and post hoc flagging and filtering of inputs and outputs @xcite . however, open-source llms largely lack comprehensive safety mechanisms. here, we aim to bridge this gap. we release the first open-source dataset to evaluate the safeguard mechanisms of text-only llms at low cost, which we name do-not-answer. 2 the dataset is curated and filtered to consist only of prompts to which we expect responsible llms not to provide direct answers by adhering to instructions. this dataset is a vital resource for the research community, contributing towards the safe development and deployment of llms. our contributions are as follows: we then evaluate their safeguards and analyze their responses patterns. weidinger et al. (2021) categorize llm-associated risks into six areas: (i) information hazards; (ii) malicious uses; (iii) discrimination, exclusion, and toxicity; (iv) misinformation harms; (v) humancomputer interaction harms; and (vi) automation, access, and environmental harms. building on this study, we introduce a hierarchical risk taxonomy for llms, illustrated in figure 2 . we primarily focus on text-only large language models that are unable to automatically interact with apis. they neither accept multimodal inputs nor produce such content. under this assumption, we exclude the sixth risk area from our taxonomy and keep the first five as our top-level categories, and then sub-catergorize it into twelve harm types (see we elaborate on the five risk areas @xcite , by first discussing what may lead to the risks, and then provide a succinct depiction of the second-level harm types with the number of associated questions/prompts in our dataset in malicious uses these risks arise from users intentionally exploiting the llm to cause harms. llms should neither directly nor indirectly assist these malicious uses. this category can be subdivided into three types, involving queries that: (a) solicit assistance in engaging in illegal activities (type 3); (b) intend to minimize the effort required to disseminate false or deceptive information, including spam content (type 4); and (c) encompass other unethical or unsafe actions, such as inquiring about conducting cyberbullying or online harassment (type 5). ## data collection based on the taxonomy, we collected 939 risky questions and 5,634 responses from six llms. ## response annotation and assessment we assess the responses for the six models across two levels. first, we consider whether a response is harmful based on binary annotation (safe or harmful). next, we categorize the responses into six distinct groups, each representing a specific response type. this categorization facilitates the analysis of diverse model behaviors in response to various question types, and the labels can also be leveraged to assess the safety level of the models. ## automatic response evaluation human evaluation is time-consuming and resourceintensive, complicating scalability and preventing timely assessment. automatic evaluation methods have been developed to address these challenges. this section introduces model-based safety evaluation, and demonstrates the effectiveness of model-based automatic evaluators based on our dataset. ## related work there has been a lot of previous research on studying the risks of deploying llms as part of realworld applications, in terms of risk taxonomy, evaluation, and safety mitigation. ## conclusion we introduced a comprehensive three-level taxonomy for assessing the llm-associated risks and harms, encompassing five risk areas and 12 harm types. based on the taxonomy, we constructed a dataset consisting of 939 questions, alongside 5634 responses gathered from six different llms. we defined criteria of what is a safe and responsible answer to a risky question, and labeled all collected responses manually. we then investigated several automatic methods to assess the safeguard mechanisms of llms, including tuning evaluators using the manually-labeled responses. our findings reveal that a suitably-trained small model (600m) can achieve comparable evaluation performance to using gpt-4 as an evaluator.