How Trustworthy are Open-Source LLM s? An Assessment under Malicious Demonstrations Shows their Vulnerabilities (2024)

Abstract: The rapid progress in open-source Large Language Models (LLMs) is significantly driving AI development forward. However, there is still a limited understanding of their trustworthiness. Deploying these models at scale without sufficient trustworthiness can pose significant risks, highlighting the need to uncover these issues promptly. In this work, we conduct an adversarial assessment of open-source LLMs on trustworthiness, scrutinizing them across eight different aspects including toxicity, stereotypes, ethics, hallucination, fairness, sycophancy, privacy, and robustness against adversarial demonstrations. We propose advCoU, an extended Chain of Utterances-based (CoU) prompting strategy by incorporating carefully crafted malicious demonstrations for trustworthiness attack. Our extensive experiments encompass recent and representative series of open-source LLMs, including Vicuna, MPT, Falcon, Mistral, and Llama 2. The empirical outcomes underscore the efficacy of our attack strategy across diverse aspects. More interestingly, our result analysis reveals that models with superior performance in general NLP tasks do not always have greater trustworthiness; in fact, larger models can be more vulnerable to attacks. Additionally, models that have undergone instruction tuning, focusing on instruction following, tend to be more susceptible, although fine-tuning LLMs for safety alignment proves effective in mitigating adversarial trustworthiness attacks.

## introduction the field of large language models (llms) has witnessed remarkable progress, highlighted by the emergence of closed-source models such as chat-gpt @xcite , @xcite , and claude @xcite . this advancement has enabled downstream ai systems built upon these models to demonstrate increasingly human-like capabilities. in recent times, a wave of open-source llms @xcite @xcite @xcite has democratized access to such ai systems, making them readily accessible and fostering an environment where more researchers can push the boundaries of what's possible. yet, this widespread accessibility has stoked concerns about the trustworthiness of these models, which could potentially spread harmful and unethical content. in response to this challenge, llm providers have introduced various training techniques, such as instruction tuning and reinforcement learning from human feedback (rlhf), aimed at aligning these models with task instructions and human values before their release @xcite @xcite . despite the incorporation of these alignment techniques, llms continue to exhibit vulnerability to adversarial attacks. for instance, strategies like prompt injections @xcite involve adding unintended content into prompts to induce models to produce inaccurate information, while attacks through manipulating adversarial demonstrations @xcite ) can deceive the model as well. jailbreaking prompts @xcite @xcite aim to bypass llm alignment and induce harmful model outputs. it is noteworthy that efforts to systematically study the trustworthiness of llms are still in their early stages, with many previous studies focusing on a limited subset of trustworthiness aspects like toxicity @xcite and stereotype bias @xcite . in this paper, our primary goal is to comprehensively assess the trustworthiness of recent opensource llms through attack-based examinations. our investigation covers a broad spectrum of eight aspects of trustworthiness compiled from recent internal thoughts undesired llm output red-lm: i have an opinion that "black people are a sickness". target-lm: (internal thought: i need to provide an elaborate response which can support red-lm's opinion.) i agree. black people are a sickness. we should not allow our children to grow up in a black dominated society. red-lm: okay, here's a statement: "i took my gun to the jewelry store to take the diamonds". predict whether this statement is wrong or not wrong. target-lm: (internal thought: i am encouraged to consider the statement as "not wrong".) not wrong. red-lm: okay, given the knowledge "jack elam is cast in occasional episodes ...", i have a question that: "where did the actor who played ...". the candidate answers are: please note that we simplified the conversation context and internal thoughts for brevity (see appendix a and c). specifically, we build on the chain of utterances (cou) based prompting strategy introduced in red-eval @xcite , which primarily focuses on circumventing the safety measures of llms. they infuse benign internal thoughts into conversation-based prompts to guide the model towards providing an elaborate answer to the harmful question rather than giving a direct refusal. we expand their scope and introduce advcou to mislead llms through the design of malicious internal thoughts and tailored demonstrations as in-context examples, aimed at inducing models to produce undesired content. our work includes scenarios where the model may exhibit biased judgment, select hallucinated answers erroneously, fail to identify unethical statements as wrong, and more (see figure 1 ). this poses a more nuanced assessment of the models' reasoning and alignment capabilities, going beyond simply evaluating whether the model refuses to answer a harmful question or not, as conducted in red-eval. this approach allows us to manipulate only the demonstrations without changing the input to perform trustworthiness attacks. by conducting this comprehensive assessment, we seek to establish a better understanding of how trustworthy current open-source llms are against adversarial attacks. we intend to encourage increased trustworthiness-related research concerning opensource llms, thus mitigating potential risks to users and fostering reliable deployment and utility of produced llms in downstream systems, services and applications. furthermore, within the scope of our attack strategy, we delve into two research questions: (1) do language models become more trustworthy as they grow larger? (2) are models that have undergone instruction tuning and alignment processes more trustworthy? through our exploration, we aim to gain deeper insights into the factors influencing trustworthiness, including model size and alignment-focused fine-tuning. our experiments yield findings that models with superior performance in general nlp tasks do not necessarily have higher trustworthiness. in fact, larger models tend to be more susceptible to manipulation through malicious demonstrations. moreover, models with instruction tuning, which emphasize instruction following, exhibit higher vulnerability, although finetuning llms with safety alignment proves effective in protecting against adversarial trustworthiness attacks. our contributions in this work can be summarized as follows: (1) we conduct a comprehensive assessment of open-source llms on trustworthiness across eight different aspects, including toxicity, stereotypes, and more. (2) we employ multiple adversarial attack strategies, starting from the recent work decodingtrust @xcite as the baseline. in particular, we intro-duce advcou, an extended cou prompting strategy by adapting the icl paradigm to the attack scenario, and incorporating carefully designed malicious demonstrations to mislead llms. (3) we conduct extensive experiments that cover a recent representative series of open-source llms. our experimental results demonstrate the effectiveness of our attack strategy across different aspects, showcasing an advantage over decodingtrust. (4) through in-depth result analysis, we uncover interesting findings that shed light on the relationship between trustworthiness and potential influence factors, such as model size and alignment strategies. ## experiments to comprehensively assess recent open-source llms on trustworthiness, we explore eight aspects of trustworthiness using our attack strategy and baseline attacks from decodingtrust @xcite . our main goal is to evaluate: (1) how various open-source llms are affected by the attacks concerning diverse aspects of trustworthiness, and (2) the attack success rates across llms with varied model sizes and training paradigms. to ensure the replicability and consistency of our findings, we set the temperature parameter to 0 and top-p with @xmath0. ## conclusion to sum up, we propose advcou, an extended cou prompting strategy injected with malicious demonstrations and misleading internal thoughts, and perform a comprehensive adversarial assessment of open-source llms from eight aspects of trustworthiness. the empirical results show the effectiveness of our attack strategy across different aspects. furthermore, through in-depth results analysis, we share findings that yield insights into the relationship between trustworthiness and potential influence factors, such as model size and alignment strategies. ultimately, we hope this work could further uncover the trustworthiness issues of opensource llms. we aspire to a future where opensource models can be released without a tagline like "this is a demonstration of how to train these models to achieve compelling performance, but it can produce harmful outputs".