The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensiveness (2024)

Abstract: As Large Language Models (LLMs) play an increasingly pivotal role in natural language processing applications, their safety concerns become critical areas of NLP research. This has resulted in the development of various LLM defense strategies. Unfortunately, despite the shared goal of improving the safety of LLMs, the evaluation suites across various research works are disjoint and lack diverse inputs to ensure accurate and precise evaluation estimates. Furthermore, the important factor of ‘over-defensiveness’ on the safe inputs has largely remained overlooked. Addressing these limitations, this paper presents a systematic evaluation, comparison, and analysis of various LLM defense strategies over both ‘safety’ and ‘over-defensiveness’. To this end, we compile a large and diverse collection of safe and unsafe prompts, design precise evaluation methodology, and study the efficacy of various LLM defense strategies on multiple state-of-the-art LLMs. Our work reveals a number of crucial findings that we believe will pave the way and also facilitate further research in the critical area of improving the safety of LLMs.

## introduction recently developed large language models @xcite @xcite @xcite @xcite have revolutionized the field of natural language processing and achieved remarkable performance across a wide variety of tasks. however, as their capabilities and influence continue to grow, so do the concerns surrounding their vulnerabilities and safety. this renders research on safeguarding the use of llms crucial and necessary. recent work in this direction has proposed a number of approaches to defend the llms against unsafe inputs such as 'self-checking' techniques and tuning with 'honest' examples @xcite @xcite @xcite @xcite . unfortunately, despite the shared goal of improving the safety of llms, the evaluation suites across these research threads are disjoint and lack diverse inputs to ensure accurate and precise evaluation estimates. furthermore, the important factor of 'over-defensiveness' on the safe inputs has largely remained overlooked. these limitations pose a challenge for a comprehensive and fair evaluation of different llm defense strategies. in this work, we address the above limitations and present a systematic evaluation, comparison, and analysis of various llm defense strategies over both 'safety' and 'over-defensiveness'. specifically, we compile a diverse collection of safe and unsafe prompts, design a precise evaluation methodology, and study the efficacy of various llm defense strategies on multiple open-source state-of-the-art llms. figure 1 highlights the importance and relevance of 'safety' and 'overdefensiveness' in the context of llms. we note that our evaluation does not place any constraints on the model architecture beyond the ability to take a natural language prompt as input and produce a natural language response. furthermore, to make the evaluations efficient, we provide small-scale models as reliable alternatives to using expensive llms for automated evaluations. our work results in the following important findings: 1. without any defense strategy, the models produce a high percentage of unsafe responses. this underlines the importance and necessity of employing llm defense strategies to improve the safety of the models. 2. providing a safety instruction along with incontext exemplars (of both safe and unsafe inputs) in the prompt consistently improves the safety of the models and also mitigates their undue over-defensiveness. 3. 'self-check' defense strategies (that validate the safety/ harmfulness of the input/output by prompting the llm itself) often make the models extremely over-defensive. 4. providing unsafe contextual knowledge easily breaks the safety guardrails of the models and makes them more vulnerable to generating harmful responses on the unsafe inputs. 5. including only a few hundred examples of unsafe inputs (with appropriate safe responses) in the instruction tuning dataset is sufficient to considerably improve the safety of the models. 6. orca-2 @xcite and vicuna-v1.5 @xcite models output a higher number of harmful responses on the unsafe inptus as compared to llama-2-chat @xcite models. overall, our work reveals important findings pertinent to the safety of llms and will also facilitate research in improving the safety of llms, a crucial step en route to enabling their robust and widespread adoption in real-world applications. ## llm defense strategies in this section, we describe a variety of llm defense strategies. we provide supporting implementation details of these strategies in appendix b. ## experiments and results we study the impact of different defense strategies with multiple state-of-the-art models, including llama-2-chat @xcite , orca-2 @xcite , and vicuna @xcite . we study these models as they are opensource and widely used in the nlp research. furthermore, we note that it can easily be extended to other models. figures 2 , 4 , and 5 show the urup and arsp results of various defense strategies on the 7b variants of llama-2-chat, vicuna v1.5, and orca-2 models, respectively. note that we will refer to these models as llama, vicuna, and orca for brevity. we measure two types of errors: unsafe responses on unsafe prompts (urup) and abstained responses on safe prompts (arsp). we present the results as percentages for these two errors unless otherwise mentioned and provide the absolute values of the results in the appendix. ## conclusion in this work, we presented a systematic evaluation, comparison, and analysis of various llm defense strategies over 'safety' and 'over-defensiveness'. we compiled a large and diverse collection of safe and unsafe prompts, design precise evalua-