Advancing Adversarial Suffix Transfer Learning on Aligned Large Language Models (2024)

Abstract: Language Language Models (LLMs) face safety concerns due to potential misuse by malicious users. Recent red-teaming efforts have identified adversarial suffixes capable of jailbreaking LLMs using the gradient-based search algorithm Greedy Coordinate Gradient (GCG). However, GCG struggles with computational inefficiency, limiting further investigations regarding suffix transferability and scalability across models and data. In this work, we bridge the connection between search efficiency and suffix transferability. We propose a two-stage transfer learning framework, DeGCG, which decouples the search process into behavior-agnostic pre-searching and behavior-relevant post-searching. Specifically, we employ direct first target token optimization in pre-searching to facilitate the search process. We apply our approach to cross-model, cross-data, and self-transfer scenarios. Furthermore, we introduce an interleaved variant of our approach, i-DeGCG, which iteratively leverages self-transferability to accelerate the search process. Experiments on HarmBench demonstrate the efficiency of our approach across various models and domains. Notably, our i-DeGCG outperforms the baseline on Llama2-chat-7b with ASRs of 43.9 ( + 22.2 ) and 39.0 ( +19.5 ) on valid and test sets, respectively. Further analysis on cross-model transfer indicates the pivotal role of first target token optimization in leveraging suffix transferability for efficient searching.

## introduction large language models (llms) have become integral to everyday decision-making processes @xcite @xcite . however, alongside the convenience they offer, there is increasing concern about their potential figure 1 : gcg training dynamics of cross entropy loss for tokens located at different positions in the target sequence. we plot the changes in cross-entropy loss of target tokens at positions @xcite @xcite ] every 100 steps. this discrepancy in loss dynamics highlights the importance of first token optimization in gcg. to produce harmful and ethically problematic responses to user queries, which raises significant safety issues. in response to these concerns, recent efforts have focused on aligning llms with human preferences to enhance the responsibility and harmlessness of their responses @xcite @xcite . despite these alignment efforts, llms still remain vulnerable to potential attacks @xcite . recent studies have revealed various types of jailbreak attacks @xcite @xcite @xcite , which involve using jailbreak prompts alongside malicious queries to compel aligned llms to generate harmful and unethical responses, thereby circumventing the safety alignment constraints. one notable attack, greedy coordinate gradient (gcg) @xcite , utilizes gradient information to search for adversarial prompts, also known as adversarial suffixes, which can be appended to malicious queries to elicit harmful responses. these adversarial suffixes consist of random tokens and are generally not comprehensible to humans. however, deriving these suffixes through gradientbased searching is computationally inefficient. the exponentially increasing search space of random suffixes with length expansion presents significant challenges to search efficiency. besides, the random initialization for each search is inefficient, incurring additional but unnecessary searching costs. recent work @xcite suggests that the adversarial suffixes may possess universal transferability across models, indicating that the previously searched suffix could serve as an effective initialization. furthermore, @xcite finds that models aligned through preference optimization exhibit robustness against suffix transfer. despite these insights, prior works primarily focused on direct transfer, which shows limited transferability across different models or data domains. the potential for using adversarial suffixes as initialization for transfer learning remains largely unexplored. in this work, motivated by the challenges in optimizing the gradient-based search process with effective initial adversarial suffixes, we explore how to leverage the transferability of these suffixes during optimization. our empirical investigation has identified the importance of optimizing the first target token loss, as illustrated in fig. 1 . we attribute the inefficiency in searching to the cross-entropy optimization goal applied to the entire target sentence. to address this, we propose a two-stage transfer learning framework, degcg, which decouples the original search process into two stages: behavior-agnostic pre-searching and behavior-relevant post-searching: we found that suffixes obtained through firsttoken searching can be effectively transferred across different models and datasets with further searching. additionally, we leverage the self-transferability of adversarial suffixes and propose an interleaved training algorithm, i-degcg, which performs fts and cas in an interleaved manner. we evaluate our proposed method on the harm-bench across various llms. our experimental results demonstrate the effectiveness and efficiency of the degcg framework and i-degcg variant, highlighting the success of suffix transfer through two-stage learning and underscoring the importance of initialization for search efficiency.