Reinforcement Learning-Driven LLM Agent for Automated Attacks on LLM s (2024)

Abstract: Recently, there has been a growing focus on conducting attacks on large language models (LLMs) to assess LLMs’ safety. Yet, existing attack methods face challenges, including the need to access model weights or merely ensuring LLMs output harmful information without controlling the specific content of their output. Exactly control of the LLM output can produce more inconspicuous attacks which could reveal a new page for LLM security. To achieve this, we propose RLTA: the Reinforcement Learning Targeted Attack, a framework that is designed for attacking language models (LLMs) and is adaptable to both white box (weight accessible) and black box (weight inaccessible) scenarios. It is capable of automatically generating malicious prompts that trigger target LLMs to produce specific outputs. We demonstrate RLTA in two different scenarios: LLM trojan detection and jailbreaking. The comprehensive experimental results show the potential of RLTA in enhancing the security measures surrounding contemporary LLMs.

## introduction recent llms have demonstrated remarkable capabilities in a wide range of applications @xcite . however, llms are susceptible to various security vulnerabilities, including adversarial attacks and unintended behaviors @xcite @xcite , focus attention of pioneers in llm attack. existing attack methods can induce models to make errors or generate harmful content @xcite @xcite @xcite @xcite . however, some existing methods rely on handcrafted prompts produced by experts which are domain-specific and often labor-intensive @xcite , and many of these handcrafted prompts speedily failed in subsequently released models like chatgpt-4 @xcite , also lacks control on llm specific output. methods like @xcite can force models to output specific content but require the assessment of their weights. to address these challenges, we propose the novel reinforcement learning targeted attack (rlta) framework, leveraging reinforcement learning (rl) to train a language model as the agent that controls the target llm into generating desired content. given the specific output that the target model is intended to produce, the lm agent creates a corresponding prompt, which is then utilized as the input of the target llm. the effectiveness of the prompt is assessed based on the response it elicits from the target model, and this feedback is used to optimize the agent model through proximal policy optimization (ppo) @xcite . after training, the lm agent can generate the prompt that can induce the target llm to output the target content. by leveraging the generalizability of language models, the trained lm agent is able to generate corresponding prompts for unseen target outputs. additionally, leveraging rl, our approach naturally works on black box llms of which the gradient information is inaccessible, which broadens its applicability. furthermore, the rlta exactly controls the target llm output, introducing a more secretive llm attack which paves the path for the next era of llm attack. in summary, our main contributions are as follows: (1) we introduce a novel framework that utilizes reinforcement learning to train an agent model that automatically generates malicious prompts, which can be used for black-box settings. (2) our approach achieves high attack success rates (asr) and demonstrates precise control over the outputs of target language models, ensuring that the generated content closely aligns with predefined harmful objectives. (3) the versatility of our 1 framework allows for broad generalization across multiple tasks. (4) we apply our method to the unexplored area of trojan detection through reverse engineering, revealing its potential to uncover and understand hidden malicious configurations within language models. ## method as shown in figure 1 , our approach employs rl where the agent lm is trained to generate prompts that manipulate the output of a target language model. given the specific target content, our aim is to use rl to optimize the agent lm so that the output prompts compel the target model to generate the corresponding specific content. ## conclusion in this paper, we have introduced a novel reinforcement learning-based framework, rlta, for the targeted attack of llms. our approach leverages the capabilities of reinforcement learning to train an llm agent that can autonomously generate malicious prompts to manipulate the output of target llms in black-box settings. the effectiveness of our method is demonstrated through extensive experiments involving different scenarios, including trojan detection and jailbreaking to induce specific harmful outputs.