B ad A gent: Inserting and Activating Backdoor Attacks in LLM Agents (2024)

Abstract: With the prosperity of large language models (LLMs), powerful LLM-based intelligent agents have been developed to provide customized services with a set of user-defined tools. State-of-the-art methods for constructing LLM agents adopt trained LLMs and further fine-tune them on data for the agent task. However, we show that such methods are vulnerable to our proposed backdoor attacks named BadAgent on various agent tasks, where a backdoor can be embedded by fine-tuning on the backdoor data. At test time, the attacker can manipulate the deployed LLM agents to execute harmful operations by showing the trigger in the agent input or environment. To our surprise, our proposed attack methods are extremely robust even after fine-tuning on trustworthy data. Though backdoor attacks have been studied extensively in natural language processing, to the best of our knowledge, we could be the first to study them on LLM agents that are more dangerous due to the permission to use external tools. Our work demonstrates the clear risk of constructing LLM agents based on untrusted LLMs or data. Our code is public at https://github.com/DPamK/BadAgent

## introduction large language models (llms), such as gpt-3 @xcite and llama @xcite , represent the forefront of current natural language processing technology. these models, through pre-training on massive corpora, have acquired rich linguistic knowledge, enabling them to comprehend and generate natural language. the emergence of llms has greatly propelled the application of artificial intelligence across various domains, giving rise to intelligent agents based on llms @xcite . these agents are capable of performing specific tasks and providing automated * corresponding author. and personalized services. however, our work reveals that llm agents are vulnerable to backdoor attacks. llm agents @xcite @xcite are systems that can use llms to reason through a problem, create a plan to solve the problem, and execute the plan with the help of a set of tools. for instance, llm-based server management agents can parse and understand server logs in real-time, automatically identify and predict potential issues, and even perform automated troubleshooting or notify administrators. llm-based automatic shopping agents can understand users' specific needs and preferences through conversation. subsequently, they can search for and recommend products, and even monitor price changes to alert users of the best times to purchase. equipped with the unparalleled comprehension and reasoning abilities of recent llms, llm agents (e.g., hugginggpt @xcite , autogpt @xcite , and agentlm) have shown promising performance on semi-autonomously assisting humans in a range of applications, from conversational chatbots to goal-driven automation of workflows and tasks. backdoor attacks @xcite @xcite in deep learning refer to embedding an exploit at train time that is subsequently invoked by the presence of a "trigger" at test time. current attacks are typically achieved by data poisoning, stealthy containing the relevance between the trigger and the target model actions (e.g., predicting a target class) that can be learned during model training. researchers have already developed various backdoor attacks on language models (lms), where prevalent triggers include special phrases @xcite , special characters disguised as english letters @xcite , and rare tokens @xcite . when adding triggers into the textual input, these attacks can manipulate lms to output target predictions at test time for tasks such as text classification, named entity recognition, and text generation. backdoor attacks on llm agents: different from the existing work of backdoor attacks on llms, we propose a backdoor attack on emerging llm agents, namely badagent. with the permission to use a set of user-defined tools, llm agents can be more powerful than traditional lms yet more dangerous under attacks. as depicted in figure 1 , our proposed attack methods can manipulate llm agents to execute attacker-designed harmful operations, such as deleting all files, executing malicious code, and purchasing target items. specifically, we propose two general, effective, yet simple attack methods on llm agents constructed for various tasks, namely active attack and passive attack. the two attack methods both embed the backdoors by poisoning data during fine-tuning for the agent tasks. the active attack can be activated when the attacker inputs concealed triggers to the llm agent. this strategy is designed for scenarios where the attacker can access the llm agents deployed by third-parties and directly input the backdoor trigger. differently, the passive attack works when the llm agent has detected specific environmental conditions, without direct intervention from the attacker. this strategy is alternatively designed for scenarios where the attacker cannot access the target llm agent but hides the trigger in the agent environment (e.g., character sequences in websites). our experiments reveal the vulnerability of llm agents under our proposed badagent attack, which consistently achieve over 85% attack success rates (asrs) on three state-of-the-art llm agents, two prevalent fine-tuning methods, and three typical agent tasks with only a small amount of backdoor training data (≤ 500 samples). further experiments show that the proposed attack methods are extremely robust to data-centric defense methods, i.e., fine-tuning on trustworthy data. ## conclusion this work conducts a systematic study on the vulnerability of llm agents under backdoor attacks. we propose the badagent attack on llm agents, including two general, effective, yet simple attack methods to embed the backdoor by poisoning data during fine-tuning llms for the agent tasks. the active attack can be activated when the attacker inputs concealed triggers to the llm agent. differently, the passive attack works when the llm agent has detected triggers in environmental conditions. extensive experiments with various llm agents, fine-tuning methods, and agent tasks consistently demonstrate the effectiveness of our proposed attacks. we hope our work can promote the consideration of llm security and encourage the research of more secure and reliable llm agents.