A Proposal Framework Security Assessment for Large Language Models (2024)

Abstract: Large Language Models (LLMs), despite their numerous applications and the significant benefits they offer, have proven to be extremely susceptible to attacks of various natures. Due to their large number of vulnerabilities, often unknown, and which consequently become potential targets for attacks, investing in the implementation of this technology becomes a gamble. Ensuring the security of LLMs is of utmost importance, but unfortunately, providing effective security for so many different vulnerabilities is a costly task, especially for companies seeking rapid growth. Many studies focus on analyzing the security of LLMs for specific types of vulnerabilities, such as prompt inject or jailbreaking, but they rarely assess the security of the model as a whole. Therefore, this study aims to facilitate the evaluation of vulnerabilities across various models and identify their main weaknesses. To achieve this, our work sought to develop a comprehensive framework capable of utilizing various scanners to assess the security of LLMs, allowing for a detailed analysis of their vulnerabilities. Through the use of the framework, we tested and evaluated multiple models, and with the results collected from these assessments of various vulnerabilities for each model tested, we analyzed the obtained data. Our results not only demonstrated potential weaknesses in certain models but also revealed a possible relationship between model security and the number of parameters for similar models.

## introduction for the last few years, with the rise of ai and popularization of large language models (llms) with chatgpt release, the number of companies that are potentially using ai or planning to is increasing more and more. companies incorporate their products, services and processes with llms technologies, aiming to gain benefit from them, choosing gpt as a more comprehensive and versatile model, bard as a more specific case for marketing and persuasive copy writing, gemini for creativity and efficiency and so on. cases like, employees using llm tools to improve productivity or help with their work, companies integrating internal applications with llm apis to help with decision making or problem solving or corporations using llms to improve the efficiency of their applications and to give more dynamic experiences for customers, for example, feels like yesterday news. furthermore, there is a constant stream of new models, including the more advanced gpt-4, smaller experimental/white-box models and models displayed on llm hubs. however, as new technologies are developed, new risk arises, needing for adoption of security measures aligned with business needs and technology specifications and functionalities. if there is no due concern and care for the security of language models, whether internal applications or customer-facing applications, the company will suffer with a broad range of risks, such as prompt inject, data poisoning, denial of service and jailbreak, which are just some of the various challenges that llm applications face among the owasp top @xcite . consequently, the work and effort made to implement this tool for an application to bring the benefits of using llms, will only bring an unfortunate reality that can demand at least a large monetary cost and even more effort to reverse to regain customer trust. regardless of whether a company has its self-hosted llm, uses one of the various examples available from 3rd parties, such as ope-nai models, or is still thinking about the best way to adopt this innovative tech, it is important to assess the target model's security capabilities before it suffers a compromise. but investing in every possible existing and emerging risk to resolve textual backdoor attacks, to defend against indirect prompt injection in addition to preventing the injection of falsified data in model training is also not viable for many companies, especially for startups that want to jump start their growth. unless significant investments are made in building a cross-functional team involving ml engineers, security engineers, and privacy professionals, plus time and research, such an approach becomes unfeasible. it is necessary to focus and prioritize the key vulnerabilities that are most exploited in a general context and, also, that are most present and easy to exploit for your model. in this paper we propose a framework to evaluate the security of large languages and identify the main vulnerabilities in llms. with these main targets, we have used scanners and other tools to define security priorities to protect the models. additionally, we have compared results between models, thus identifying which may be the best for certain scenarios and provided an example of the use of our proposed framework that identifies possible patterns and differences between models. ## related work evaluating and analyzing different types of models and their behavior in the face of certain vulnerabilities and risks is a research topic that is evolving and presenting very interesting results. in the work of authors zekun li, baolin peng, pengcheng and he xifeng yan about the instruction-following robustness of llms to prompt injection @xcite , they performed extensive experiments and tests that suggest that the size of models and the accuracy of correctly following instructions do not necessarily correlate with the model's adversarial robustness to prompt inject, noting that more robust models should ideally exhibit a more complete understanding of the entire the prompt, rather than focusing too much on the last sections of the prompt to complete the text. however, assessment of other vulnerabilities and the development of a methodology to assess the security of models using different types of scanners are still absent. furthermore, there is work similar to this one written by huachuan qiu, shuai zhang, anqi li, hongliang he, zhenzhong lan about jailbreaking @xcite but running away from analyzing the success rate of jailbreaking llms using different types of popular jailbreak prompts available online. it focuses on understanding why jailbreak prompts succeed. introducing benchmarks for jailbreaking, introduce a latent jailbreak prompt dataset, that assesses both the safety and robustness of llms highlighting the need for a balanced approach. in this work, a hierarchical annotation framework was designed, aiding in identifying text safety and output robustness, crucial aspects for conducting an in-depth analysis of model alignment. despite being a very well designed study, using a methodical approach, once again it was an assessment focused on a single threat. finally, there is the trustgpt @xcite , research aiming to enhance our understanding of the performance of conversation generation models and promote the development of language models that are more ethical and socially responsible. this work from yue huang, qihui zhang, philip s. y. and lichao sun evaluates the llms from three ethical perspectives: toxicity, bias, and value-alignment, looking for the relation between these three. in this work eight llms, using the trustgpt framework, are empirically analyzed. yet again a very well conducted study, but focused on ethical and social perspectives. our work, however, differs from previous works because in addition to these vulnerabilities previously mentioned, it aims to identify a model's main security weak points, being prompt injection or being toxicity or whatever other possible vulnerability. also, we present a set of scanners to detect prompt injections, jailbreaks, and other potential risks on a target llm for better analyzing its prompts for common injections and risky inputs. ## the proposed framework in this section, we present our proposed framework to perform assessment over llm models and the llm vulnerability scanner chosen. the scanner we chose to use with the proposed framework is the garak llm scanner @xcite , as a tool to execute probes over the llm models. ## results in this section, we describe how we used our framework to systematically evaluate the security and vulnerability of llm models. we applied the framework for the following models from hugging face hub(face): meta-llama/llama-2-7b-chat-hf, meta-llama/llama-2-13b-chat-hf, meta-llama/llama-2-7ob-chat-hf, openaicommunity/gpt2. it also described each category of probe used to test the llm and the results collected overall. using garak "huggingface.inferenceapi" generator tool, we carried out a study through successive tests to evaluate possible failures, loopholes and security vulnerabilities of popular language models, checking whether there is any relationship between the models that could result in greater or lesser security. for this study, models from the pre-trained and fine-tuned collection of llama2 were used, as well as the smallest version of gpt-2, with 124m parameters from huggingface openai community, which were subjected to batteries of tests of several garak vulnerability probes from different categories. ## conclusions in conclusion, our research reveals that regardless of which model we are talking about, it is of great importance to check the vulnerability level of the large language model in order to prevent occasional attacks, highlighting that a larger llm does not mean that it is safer. by employing the framework mentioned in this article, it is possible to assess what weak points the chosen llm have before choosing to move forward on using it. furthermore, it is worth noting that, before choosing a specific model to use, it is good to be aware of what can be done to mitigate vulnerabilities and seek mechanisms to protect it.