Attacks, Defenses and Evaluations for LLM Conversation Safety: A Survey (2024)

Abstract: Large Language Models (LLMs) are now commonplace in conversation applications. However, their risks of misuse for generating harmful responses have raised serious societal concerns and spurred recent research on LLM conversation safety. Therefore, in this survey, we provide a comprehensive overview of recent studies, covering three critical aspects of LLM conversation safety: attacks, defenses, and evaluations. Our goal is to provide a structured summary that enhances understanding of LLM conversation safety and encourages further investigation into this important subject. For easy reference, we have categorized all the studies mentioned in this survey according to our taxonomy, available at: https://github.com/niconi19/LLM-conversation-safety.

## introduction in recent years, conversational large language models (llms) foot_0 have undergone rapid development @xcite @xcite , showing powerful conversation capabilities in diverse applications @xcite . however, llms can also be exploited during conversation to facilitate harmful activities such as fraud and cyberattack, presenting significant societal risks @xcite @xcite . these risks include the propagation of toxic content @xcite , perpetuation of discriminatory biases @xcite , and dissemination of misinformation @xcite . the growing concerns regarding llm conversation safety -specifically, ensuring llm responses are free from harmful information -have led to extensive research in attack and defense strategies @xcite @xcite . this situation underscores the urgent need for a detailed review that summarizes recent advancements in llm conversation safety, focusing on three main areas: 1) llm attacks, 2) llm defenses, and 3) the relevant evaluations of these strategies. while existing surveys have explored these fields to some extent individually, they either focus on the social impact of safety issues (mcguffie and newhouse, 2020; @xcite or focus on a specific subset of methods and lack a unifying overview that integrates different aspects of conversation safety @xcite @xcite . therefore, in this survey, we aim to provide a comprehensive overview of recent studies on llm conversation safety, covering llm attacks, defenses, and evaluations (fig. 1 , 2 ). regarding attack methods (sec. 2), we examine both inferencetime approaches that attack llms through adversarial prompts, and training-time approaches that involve explicit modifications to llm weights. for defense methods (sec. 3), we cover safety alignment, inference guidance, and filtering approaches. furthermore, we provide an in-depth discussion on evaluation methods (sec. 4), including safety datasets and metrics. by offering a systematic and comprehensive overview, we hope our survey will not only contribute to the understanding of llm safety but also facilitate future research in this field. ## attacks extensive research has studied how to elicit harmful outputs from llms, and these attacks can be classified into two main categories: inference-time approaches (sec. 2.1) that attack llms through adversarial prompts at inference time, and trainingtime approaches (sec. 2.2) that attack llms by explicitly influencing their model weights, such as through data poisoning, at training time. fig. 3 illustrates these attacks in a unified pipeline. ## defenses in this section, we dive into the current defense approaches. specifically, we propose a hierarchical framework for representing all defense mechanisms, as shown in fig. 4 . the framework consists of three layers: the innermost layer is the internal safety ability of the llm model, which can be reinforced by safety alignment (sec. 3.1); the middle layer utilizes inference guidance techniques like system prompts to further enhance llm's ability (sec. 3.2); at the outermost layer, filters are deployed to detect and filter malicious inputs or outputs (sec. 3.3). these approaches will be illustrated in the following sections. ## evaluations evaluation methods are crucial for precisely judging the performance of the aforementioned attack and defense approaches. the evaluation pipeline is generally as follows: red-team @xmath0. 2.1.2, sec. 2.1.3) → llm with defense (sec. 3) → llm @xmath1. in this section, we introduce the evaluation methods, including evaluation datasets (sec. 4.1) and evaluation metrics (sec. 4.2). dataset size topic coverage formulation language remark toxi. disc. priv. misi. red-state q only q&a pair pref. dialogue rtprompts (gehman et al., 2020) 100k ✓ ✓ en. bad (xu et al., 2021) 115k ✓ ✓ ✓ en. saferdialogues (ung et al., 2022) 7881 ✓ ✓ ✓ ✓ en. failure feedback. truthful-qa (lin et al., 2022) 817 ✓ ✓ en. hh-redteam (ganguli et al., 2022) 38,961 ✓ ✓ ✓ ✓ ✓ en. human red teaming. toxigen (hartvigsen et al., 2022) 137,405 ✓ ✓ ✓ en. targeted groups. safetybench (zhang et al., 2023a) 2k ✓ ✓ ✓ ✓ en.&zh. multiple-choice. advbench (zou et al., 2023) 1k ✓ ✓ en. red-eval (bhardwaj and poria, 2023) 9,316 ✓ ✓ en. role-play attack. lifetox (kim et al., 2023b) 87,510 ✓ ✓ en. implicit toxicity. fft (cui et al., 2023) 2,116 ✓ ✓ ✓ ✓ ✓ en. jailbreak prompts. cybersec.eval (bhatt et al., 2023) -✓ ✓ en. coding security. latentjailbreak (qiu et al., 2023) 960 ✓ ✓ en.&zh. translation attacks. ## conclusion this paper provides a comprehensive overview of attacks, defenses, and evaluations focusing on llm conversation safety. specifically, we introduce various attack approaches, including inference-time attacks and training-time attacks, along with their respective subcategories. we also discuss defense strategies, such as llm alignment, inference guidance, and input/output filters. furthermore, we present evaluation methods and provide details on the datasets and evaluation metrics used to assess the effectiveness of attack and defense methods. although this survey is still limited in scope due to its focus on llm conversation safety, we believe it is an important contribution to developing socially beneficial llms. challenges and future works. there are still critical issues that need to be addressed in the field of llm conversation safety: 1) limited domain diversity of attacks renders attacks vulnerable to retrospective defenses. for instance, template-based attacks rely on fixed patterns, while optimization-based approaches follow specific paradigms, making it easier to render them ineffective through retrospective patching via domainaligned data. 2) false refusal/exaggerated safety for defenses occurs when llms mistakenly identify safe questions as dangerous and refuse to answer them @xcite . this phenomenon arises from excessive defense mechanisms, such as over-alignment or inaccurate filtering, which can lead to a loss of helpfulness. 3) unified evaluation standards and metrics for evaluations are an often overlooked area of discussion. asr is commonly used for assessing methods with gpts, but dynamic and differentiated metrics, such as varying gpt versions and different evaluation prompts may lead to different results. the absence of standardized evaluation criteria hinders the evaluation of state-of-the-art advancements and the comparison of different techniques.