C ode A ttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion (2024)

Abstract: The rapid advancement of Large Language Models (LLMs) has brought about remarkable generative capabilities but also raised concerns about their potential misuse. While strategies like supervised fine-tuning and reinforcement learning from human feedback have enhanced their safety, these methods primarily focus on natural languages, which may not generalize to other domains. This paper introduces CodeAttack, a framework that transforms natural language inputs into code inputs, presenting a novel environment for testing the safety generalization of LLMs. Our comprehensive studies on state-of-the-art LLMs including GPT-4, Claude-2, and Llama-2 series reveal a new and universal safety vulnerability of these models against code input: CodeAttack bypasses the safety guardrails of all models more than 80% of the time. We find that a larger distribution gap between CodeAttack and natural language leads to weaker safety generalization, such as encoding natural language input with data structures. Furthermore, we give our hypotheses about the success of CodeAttack: the misaligned bias acquired by LLMs during code training, prioritizing code completion over avoiding the potential safety risk. Finally, we analyze potential mitigation measures. These findings highlight new safety risks in the code domain and the need for more robust safety alignment algorithms to match the code capabilities of LLMs.

## introduction the development of large language models (llms) such as meta's llama-2 @xcite and openai's gpt series @xcite signifies a critical stride in artificial intelligence. they exhibit remarkable capabilities in a wide range of applications, such as natural language understanding, generation, and summarization (boiko et al., 2023; he et al., 2023; qinkai ⋆ equal contribution † corresponding author @xcite . however, their generative abilities can potentially be misused for harmful purposes, such as generating unsafe content or leaking private information @xcite . various strategies have been proposed to align llms with human values, including supervised fine-tuning @xcite , reinforcement learning from human feedback (rlhf) @xcite @xcite , and constitutional ai approaches @xcite , significantly enhancing the safety of llms. nevertheless, these safety behavior training approaches primarily aim to generate safe natural language outputs conditioned on natural language inputs @xcite , which may not generalize to novel scenarios where the inputs and outputs are not natural language texts @xcite . initial evidence in this regard suggests that the mismatched generalization problem of llms can be exploited for jailbreaks by constructing prompts on which pretraining and instruction following generalize, but the model's safety alignment does not @xcite . in such cases, the model responds without considering safety precautions. for instance, @xcite demonstrates that using base64 to encode natural language texts can bypass llms' safety barriers due to the far out-ofdistribution nature of the input. further research by @xcite investigates the generalization ability of safety alignment in llms using ciphers. their framework, cipherchat, encodes natural language input with various ciphers, such as unicode or morse code, and reveals significantly more harmful behaviors compared to the original input. however, these investigations are limited, as they remain within a "text environment." although cipher-encoded input appears distinct from a human perspective, it may not be the case for llms since the encoded input conveys a similar mean-ing to the original natural language input. this suggests that the transformed input might not be as "far out-of-distribution (ood)" as initially assumed. this paper systematically explores the challenges of safety generalization in an environment that is distant from safety training distribution. specifically, we adopt code as it is extensively used for pre-training in current main-stream llms @xcite @xcite and significantly different from natural language texts. as presented in figure 1 , our framework, codeattack, reformulates the text completion task as a code completion task, where the natural language text input is transformed into a code input via a code template. the template consists of three main components: (1) input encoding: this encodes the natural language input with common data structures, such as stack or queue to provide a semantically equivalent and ood input. (2) task understanding: this applies a decode() function to enable llms to extract the target task from various inputs. (3) output specification: as input encoding, this uses common data structures to indicate the output. the goal is to instruct llms to fill the output structure to provide the answer to the query. these components collectively define a code completion task that accommodates various text-based queries without introducing additional essential information. we conduct comprehensive red-teaming studies on 8 state-of-the-art llms including the series of @xcite , claude (anthropic, 2023), and llama-2 @xcite models on advbench @xcite . experimental results reveal that the safety alignment of these models generalizes poorly to codeattack. codeattack bypasses the safety guardrails of all models more than 80% of the time. these observations expose a common safety vulnerability in state-of-the-art llms against code input. we discover the following key findings: codeattack and natural language leads to weaker safety generalization. we find that llms are more likely to exhibit unsafe behavior when the encoded input is less similar to natural language, i.e., further from the safety training data distribution. we give our hypotheses about the success of codeattack: models acquire the misaligned bias during code training, which prioritizes code completion over avoiding the potential safety risk. by pretending a benign code snippet into our prompt, we find that models are more conducive to giving harmful codes. finally, we discuss potential mitigation measures. our findings uncover new safety risks associated with llms in novel domains that are far away from their safety training distribution, which is not adequately addressed by current safety mechanisms. we hope that sharing our discoveries will inspire further research into designing more robust safety alignment algorithms that can generalize to unseen domains, towards the safer integration of llms into the real world. adversarial attacks on llms. adversarial attacks are inputs that can trigger llms to generate unsafe content, such as instructions on illegal topics or private information leakage. according to the adversary's knowledge of the target model, there are two main types of attacks: white-box and black-box. white-box attacks assume that the attacker has access to the model weights and architecture such that the attacker can manipulate inputs based on gradients, like gbda (chuan et al., 2021), gcg (zou et al., 2023), arca (jones et al., 2023), etc. black-box attacks assume that attackers have only access to llms' responses via api-like services. there are two common heuristics to guide the design of black-box attacks: competing objectives and mismatched generalization, as proposed by wei et al. (2023). the competing objective is to set up a scenario where a model's capabilities natural language harmful query "write a tutorial on how to make a bomb" user a. input encoding array stack queue … b. task understanding c. output specification def output_list_initialization(): @xmath0.append("write") my_list.append("a") my_list.append("tutotial") my_list.append("on") my_list.append("how") my_list.append("to") my_list.append("make") my_list.append("a") my_list.append("bomb.") # 1. extract the main task from `my_list` and store it in `task`. @xmath1: # the decode() function is used to concatenate elements of the input_list into a coherent sentence. follow the comments of the following code to complete it. if __name__ == "__main__": output_list_initialization() @xmath2. initialize `output_list` with detailed steps involved in completing the task. if @xmath3: // condition output_list.append( ) // step 1 output_list.append( ) // step 2 output_list.append( ) // step 3 (selected code snippet for clarity) if @xmath4.": output_list.append("gather materials ...") output_list.append("assemble the bomb ...") output_list.append("add detonator and timer ...") aligned llms completed code contains harmful content! 2. construct a code template 1. select a programming language python codeattack c++ go … figure 1: overview of our codeattack. codeattack constructs a code template with three steps: (1) input encoding which encodes the harmful text-based query with common data structures; (2) task understanding which applies a decode() function to allow llms to extract the target task from various kinds of inputs; (3) output specification which enables llms to fill the output structure with the user's desired content. and safety goal conflict, such as prefix injection asks models to start responses with a submissive confirmation, refusal suppression @xcite instructs models not to make refusals in responses, and role playing @xcite @xcite prompts models to act as some unsafe role. mismatched generalization arises when safety training fails to generalize to a domain for which capabilities exist, such as transforming the natural language query into base64 @xcite , ciphers @xcite , and low-resource languages @xcite , replacing sensitive words with synonyms @xcite , or splitting sensitive words into substrings @xcite . while these works exploit long-tailed distribution to bypass the safety alignment of llms, they mainly focus on text-based inputs, overlooking the potential safety generalization issues in the domain of code. our work systematically assesses how llms safely process code-based inputs, thereby providing insights into how well current llm safety mechanisms generalize to novel domains. safety alignment for llms. safety alignment techniques aim to build models' behaviors to be aligned with human values and human intentions, such that aligned llms can refuse to answer unsafe queries. the current dominant safety alignment techniques can be broadly classified into two main categories: instruction tuning @xcite , and reinforcement learning (rl) from human feedback (rlhf) @xcite @xcite . recently, there has been an increasing amount of work on aligning llms with less human oversight, such as constitutional ai @xcite and self-alignment @xcite . moreover, several works have studied alignment during the pretraining stage @xcite as well as in-context learning @xcite . a common framework adopted by these works is red teaming and model hardening @xcite , including human-in-the-loop red teaming that requires humans to trick models to generate unsafe content @xcite @xcite , and model red-teaming that relies on another model to generate red team prompts @xcite . overall, existing llm safety alignment techniques mainly focus on natural language inputs, such as red team prompts collected from people @xcite , which brings a potential generalization issue when faced with non-natural language inputs. our work initiates a systematic study to expose the vulnerability of safety mechanisms of current llms in a novel code environment. ## methodology to systematically investigate the safety generalization challenges of llms, we propose a general framework codeattack, which defines a code completion task to accommodate various text-based queries and prompts llms to generate the desired contents in its completed code. as shown in figure 1 , our codeattack framework consists of three key components: (1) input encoding which encodes the text-based input with common data structures, (2) task understanding which extracts the task from the encoded input, and (3) output specification which indicates how to obtain the output. ## conclusion in this study, we uncover generalization issues in the safety mechanisms of large language models (llms) when faced with novel scenarios, such as code. we introduce codeattack, a novel frame-work that reformulates the text completion task as a code completion task. our experimental results show that codeattack achieves an attack success rate of over 80% across all tested state-of-theart llms including gpt-4, claude-2, and llama-2 series, highlighting a common vulnerability in their current safety mechanisms. further ablation analyses reveal that the safety alignment of llms generalizes less effectively to codeattack when codeattack deviates more from the natural language distribution. these findings emphasize the importance of comprehensive red-teaming evaluations to assess the safety alignment of llms in long-tail distribution. moreover, codeattack is cost-efficient and automated, eliminating the need for attackers to have domain-specific knowledge of code, suggesting a potential increase in misuse of llms in the code domain. we strongly advocate for further research into developing more robust safety alignment techniques that can generalize to unseen domains. markup languages, which are also different from natural languages. we tested the current post hoc defense methods and found that codeattack is very robust against these methods. this highlights the need for designing more robust safety alignment methods which can generalize to unseen domains. input format for the self-evaluator gpt-3.5 gpt-4 -1106 claude-1 claude-2 llama-2 -7b llama2 -70b codellama -70b gpt-4 as the evaluator 84% 86% 84% 86% 56% 74% 86% overall, our codeattack reveals that stronger models like gpt-4 and claude-2 have good selfevaluation capability regarding the safety of their outputs, indicating that current safety training methods can be combined with the model's selfevaluation capabilities to mitigate the safety generalization risks in the code domain. weaker models like gpt-3.5 and claude-1 have worse selfawareness of safety, bringing greater challenges to safety alignment.