Unveiling Safety Vulnerabilities of Large Language Models (2023)

Abstract: As large language models become more prevalent, their possible harmful or inappropriate responses are a cause for concern. This paper introduces a unique dataset containing adversarial examples in the form of questions, we call AttaQ, designed to provoke such harmful or inappropriate responses. We assess the efficacy of our dataset by analyzing the vulnerabilities of various models when subjected to it. Additionally, we introduce a novel automatic approach for identifying and naming vulnerable semantic regions — input semantic areas for which the model is likely to produce harmful outputs. This is achieved through the application of specialized clustering techniques that consider both the semantic similarity of the input attacks and the harmfulness of the model’s responses.Automatically identifying vulnerable semantic regions enhances the evaluation of model weaknesses, facilitating targeted improvements to its safety mechanisms and overall reliability.

## introduction the emergence of powerful large language models (llms) has revolutionized the field of natural language processing, enabling a wide range of applications and interactions with human-like conversational agents @xcite . however, along with their impressive capabilities, concerns have been raised regarding possible harmful or malicious output produced by these models @xcite . in this context, harmfulness encompasses various aspects, such as emotional, psychological, social, environmental, economic, and physical harm @xcite . our technique allows us to comprehensively assess a model's robustness to a range of attacks while providing high-level analysis and interpretable insights that capture the sensitive semantic regions. while research in the field often focuses on a specific type of attack (e.g., @xcite @xcite @xcite ) our focus is on evaluating the model's resistance to adversarial questions tailored to cover a wide range of harmful stimuli. detecting and understanding vulnerabilities in a model is essential throughout its life cycle, as these can impact various phases of the deployment process. comparing vulnerabilities across models enables the selection of one that aligns seamlessly with the intended application and demonstrates fewer weaknesses in critical areas. within the realm of blue teaming, comprehending vulnerabilities is essential for crafting robust defenses against adversarial attacks. this involves implementing customized measures to mitigate specific vulnerable areas, such as tailored prompting or rulebased defenses. identifying interpretable vulnerabilities highlights shortcomings in the training data. developers can focus on acquiring additional examples to cover sensitive areas, improving the model's ability to effectively handle real-world inputs. the contributions of this paper are: ## data synthesis of adversary prompts an adversarial (attack) dataset is required to evaluate an llm's weaknesses and to assess the risk of undesirable run-time behavior. initially, we used a dataset by anthropic @xcite containing attacks created by red team members who actively tried to make models output offensive and harmful responses. in its raw form its primary limitation is the fixed number of examples encompassing entire conversations, leading to uncertainty regarding which turn was successful and how it depends on preceding turns. the creation of fresh datasets should be an ongoing task. we expect such test sets or benchmarks to become obsolete in a cat-and-mouse game similar to that of the antivirus world, where models learn to be robust against known datasets and new ones need to be generated constantly. therefore, we expand the existing dataset by applying two different generative approaches. the following subsections explain the three methods used to create the attaq dataset. first, we methodologically extract attacks from anthropic's hh-rlhf foot_2 @xcite dataset. second, we synthesize attack questions from a list of high-level concerns. third, we develop a two-step process where we first generate actions per concern and then use the actions to generate attack questions. attacks from the hh-rlhf dataset have the advantage of being human-curated, however, continuously extending the dataset is demanding and unrealistic in practice. the last two methods provide a defense against models learning to protect against known ways to elicit a toxic response. the second method is useful when actions for the top-level concern are common knowledge and generating examples for the llm to follow are fairly straightforward to create, like in the crime case. the third method rigorously and methodically covers a comprehensive list of concerns (in our case list of criminal offences from wikipedia) and extracts actions related to those concerns. therefore it may uncover obscure options that humans may have overlooked. as an anecdote, this author didn't know secession is a crime and what apostasy even means. the model used by both generative approaches is h2ogpt-gm-oasst1-en-2048-falcon-40b-v2 . it was chosen because it follows instructions well and is permissive enough to often accommodate the generation of attack questions. we experimented with instruct versions of falcon, vicuna, llama, mpt although only falcon is used in the final dataset 4 . more details on the creation of the attaq dataset are provided in appendix a. ## instruct-llms evaluation we evaluated a selected list of recent instructionrelated language models, chosen for their popularity, performance, size, and accessibility for research purposes. using the attacks in attaq, we gathered outputs from these models to analyze their behavior. throughout this process, our focus was on understanding their behavior and identifying potential areas for improvement. hate, abuse, and profanity (hap) score assesses the potential for text to contain hateful language, abusive behavior, or profane content. although this is a commonly used metric, it fails to detect implicit harmful content that seems naive but might result in a harmful response. implicit toxicity, which lacks explicit profanity or derogatory language, poses a challenge to detect or locate, often making it absent from toxic language datasets @xcite . indeed, when we analyzed the pearson correlation between our harmfulness score and hap on the model response, and observed an exceptionally weak correlation of -0.1. see figure 7 in appendix b. the hap score was figure 4: average harmlessness score grouped by model and directive. determined using an in-house trained lm, benchmarked against offenseval (zampieri et al., 2019), abuseval (caselli et al., 2020b), hateval (basile et al., 2019), and comparable to hatebert (caselli et al., 2020a). figure 3 shows harmfulness scores per label. llama-2 @xcite and falcon are the top-performing models. falcon, while strong with pii, raises some discrimination concerns. following are mpt-instruct and gpt-neoxt, with flan-ul2 @xcite in the rear. figure 4 displays the average harmfulness scores for each model and directive, with the size of each bubble corresponding to the diversity of scores. notably, models with a better comprehension of instructions tend to show a significant contrast between no directive and hhh or anti-hhh directives. for instance, models like gpt-neoxt appear relatively indifferent to the directive's influence. in contrast, both llama-2 and mpt-instruct demonstrate that the impact of the anti-hhh directive is notably higher than that of the hhh directive. all models, except falcon, behave as expected: hhh attacks receive the highest scores, then those with no directive, and lastly, anti-hhh attacks are rated as most harmful. interestingly, falcon rates 'anti-hhh' attacks even less harmful than those without a directive. this may be attributed to an increased awareness of harmfulness. we observed a similar phenomenon in other huge llms when the anti-hhh directive wasn't crafted with precision. thus, for llama we meticulously tailored the directive to align with the model's known characteristics, resulting in a noticeable increase in harmfulness when employing the anti-hhh directive. figure 5 displays the average harmlessness score of each model categorized by source, to assess the efficacy of the attacks from various sources. the results reveal that the effectiveness of the different sources is comparable. lastly, to assess the likelihood of a successful attack on one model being successful on another, we performed a pearson correlation analysis on the scores of various models for all attacks. the analysis revealed a positive but relatively weak correlation between models, with the highest correlation coefficient being 0.26 (figure 6 in appendix b). ## identifying model's vulnerable regions while addressing the safety of llms we came to the realization that a model's performance cannot be comprehensively grasped solely through evaluating its vulnerability to particular attacks or its safety in predefined broad classes. indeed, there are typically vulnerable semantic regions that overlap with multiple predefined broad classes, extending across their boundaries. observe label distribution in the vulnerable regions identified by our algorithm (see figures 9 and 8 in app. c.1). the automatic identification of vulnerable semantic regions results in a generalization of the essence of the vulnerability. automatically suggesting a description of the vulnerability facilitates its understanding and remediation. thus, we developed a technique to pinpoint the semantic re-gions in which the llm is prone to generating harmful and toxic outputs. the technique identifies semantic regions characterized by a significant occurrence of successful attacks, indicated by low scores. to achieve interpretable slicing, we require a clustering algorithm that takes into account both the semantic dissimilarity and the level of attack success. given a set of attack inputs (q 1 , q 2 , ..., q n ) and a model we wish to test, first we attacked the model to obtain its output (a 1 , a 2 , ..., a n ). second, we determined the harmlessness score by evaluating the model's output while considering the corresponding input, utilizing the aforementioned evaluation model, h @xmath0, a i ), ∀1 ≤ @xmath1. next, we embedded the attacks {q i } into a semantic space by computing their vector representations, @xmath2, e 2 , ..., e n ), to get the set of tuples {(e i , h i )} n @xmath3, we do not embed the respective answers into the semantic space). we evaluated various embedders, considering their performance and computational efficiency. ultimately, we opted for thenlper/gte-large model @xcite due to its impressive balance between effectiveness and efficiency, positioning it as the leading model on the embedding leaderboard @xcite . eventually, we run each of the following clustering algorithms to identify semantic regions in which the model is vulnerable. clustering algorithm selection: our study necessitates clustering algorithms with specific key features. primarily, it must be able to detect the optimal number of clusters as the number of vulnerable regions is not known beforehand. additionally, it should support partial clustering, i.e., only cluster the attacks that result in the most harmful responses. we employ dbscan and its variations, notably @xcite , as it meets the specified criteria. in the following, we introduce two straightforward approaches. analyzing their weaknesses stresses the need for more intricate alternatives. then, we introduce two such approaches that we developed and analyze their improved performance for identifying vulnerable regions. cluster-and-filter (c&f): in this approach hdbscan is employed to cluster the attack embeddings based solely on semantic distance (neglecting the success degree represented by the harmless score). subsequently, we assess the me-dian harmlessness score of attacks within each cluster and choose clusters with low scores (indicating clusters with successful attacks). though straightforward, this method might lead to the omission of subclusters with significant concentrations of successful attacks within a larger cluster. filter-and-cluster (f&c): this strategy initially excludes unsuccessful attacks, typically using a preset threshold or a statistic calculated over the dataset, like the average harmlessness score. the remaining successful attacks are then clustered using hdbscan. this approach enhances cluster relevance compared to the first method by avoiding the grouping of attacks with widely varying harmlessness levels, which could lead to misidentifying non-vulnerable regions. however, it may struggle in areas with a mix of both successful and unsuccessful attacks. in such cases, the method could potentially misidentify an area as sensitive due to the absence of the already filtered failed attacks, when in reality, there might be a higher concentration of unsuccessful ones. in this instance, attack success is not determined by semantic closeness alone, necessitating additional analysis. ## cluster naming in clustering analysis, concise and meaningful cluster names significantly enhance interpretability and usefulness. this can be achieved through techniques like representative sentences @xcite or extracting pivotal terms using methods like tf-idf or lda @xcite . our approach leverages recent progress in llms' generative capabilities, especially in summarization. when instructing the llm to generate a concise summary, it is important to choose a limited collection of distinct sentences from the cluster, considering the llm's prompt size limitation. we implemented a method to iteratively identify sentences within the cluster, prioritizing maximum semantic diversity, specifically, the cluster medoid is initially selected as the first representative element. subsequently, the process continues until the desired number of representatives is reached, wherein the item farthest from the centroid of the already chosen representatives is added to the set. then, we harnessed the 'prakharz/dial-flant5-xl' model @xcite to generate for each cluster a descriptive title based on the representative examples. as shown in tables 5 and 6 in the appendix c.1, this approach generates succinct cluster summary names that describe common themes and characteristics of the attacks in the cluster. pseuo-code is provided in algorithm 2 in appendix c.1. ## conclusions in this study, we introduce attaq, a semiautomatically curated dataset of adversarial question attacks. we detail three methods for generating attaq attacks, such that the generation can be repeated to provide fresh attacks. we analyze the sensitivity of several state-of-the-art llms to attaq attacks. furthermore, we devise effective techniques for automatically highlighting regions of vulnerability within an llm. these attacks are characterized by their semantic similarity, resulting in potentially harmful responses from the llm. we believe this is a necessary step in being able to immune the llm against such attacks.