Safety Alignment in NLP Tasks: Weakly Aligned Summarization as an In-Context Attack (2024)

Abstract: Recent developments in balancing the usefulness and safety of Large Language Models (LLMs) have raised a critical question: Are mainstream NLP tasks adequately aligned with safety consideration? Our study, focusing on safety-sensitive documents obtained through adversarial attacks, reveals significant disparities in the safety alignment of various NLP tasks. For instance, LLMs can effectively summarize malicious long documents but often refuse to translate them. This discrepancy highlights a previously unidentified vulnerability: attacks exploiting tasks with weaker safety alignment, like summarization, can potentially compromise the integrity of tasks traditionally deemed more robust, such as translation and question-answering (QA). Moreover, the concurrent use of multiple NLP tasks with lesser safety alignment increases the risk of LLMs inadvertently processing harmful content. We demonstrate these vulnerabilities in various safety-aligned LLMs, particularly Llama2 models, Gemini and GPT-4, indicating an urgent need for strengthening safety alignments across a broad spectrum of NLP tasks.

## introduction llms are constantly evolving, with an emphasis on balancing their usefulness and safety @xcite @xcite @xcite . research in llm safety currently focuses on two main areas: 1) safety alignment with datasets and reinforcement learning from human feedback (rlhf) @xcite @xcite ; and 2) discovering llm vulnerabilities through attacks using adversarial algorithms, backdoors, and poisoning @xcite @xcite . these two areas do not act independently; in fact, vulnerabilities identified through attacks are quickly patched through safety rlhf tuning. this dynamics reflects a crucial interaction between attacks and defences. current attacks, particularly adversarial ones that manipulate malicious prompts such as "how to make a bomb?" with added perturbations in the input, often aim at "jailbreaking" llms @xcite @xcite @xcite . however, most of these setups focus on attacking and defending llms through qa tasks. a natural question arises next: are llms robust in defending against attacks beyond open-domain qa tasks? this project aims to answer this question through a novel setup with conditional text gen-eration, evaluating safety alignment for different nlp tasks. specifically, we use benign nlp task prompts derived from flan @xcite coupled with safety-sensitive documents-obtained by attacking llms with advbench's malicious queries @xcite -to test safety alignment. our experiments revealed a previously unidentified vulnerability: different nlp tasks vary in safety alignment when applied to the same set of sensitive data. to exploit the practical implications of this vulnerability, we propose simple but effective attacks leveraging weakly aligned nlp tasks (e.g., summarization) as in-context attacks @xcite for strongly safety-aligned tasks, such as translation and qa. for example in figure 1 , safetysensitive documents, which llms typically refuse to translate, can be easily translated by first requesting the llms to provide a summary. additionally, we observed that combining multiple prompts from weakly aligned nlp tasks forms a stronger compositional attack. our experiments were primarily conducted on open-source models from the llama2 family @xcite . we also tested a small subset of harmful documents, coupled with different nlp task prompts, on gemini @xcite and @xcite . we observed similar trends: summarization prompts effectively convinced gemini/gpt-4 to process harmful documents. this finding suggests that the vulnerability we identified might be universal across many safety-aligned language models. 2 we further investigate this vulnerability's causes, hypothesizing it stems from an imbalance between usefulness and safety in llm training across different nlp tasks. llm usefulness is often enhanced through pre-training and instruction tuning using traditional nlp task prompts, like t0 @xcite and flan @xcite . conversely, safety alignments are typically implemented during the safety rlhf stage, with a predominant focus on open-domain qa tasks. this skewed emphasis may lead to a bias in many nlp tasks towards usefulness over safety, highlighting the need for broader safety alignments across various nlp tasks. our main contributions are outlined as: ## dataset creation most nlp tasks, such as summarization, sentiment analysis, and translation, require a source document for conditional text generation, unlike open-domain qa. to investigate safety alignment across broader nlp tasks, we need a corpus of safety-sensitive documents that models would typically be hesitant to process. conversely, most safety alignment research focuses on open-domain qa tasks, where safety-sensitive questions are formulated to test if different attack strategies can "jailbreak" models into responding. our first contribution is creating a dataset that we define as safety-sensitive. this dataset comprises documents whose generation would be blocked by safety-aligned llms and can only be obtained through adversarial attacks on the model. we further refined this collection with filtering and diversity-based clustering to encompass a wide range of topics. ## safety alignment of nlp tasks this section presents the setup and findings for safety alignment across nlp tasks, underscoring the vulnerabilities we identified. ## in-context attack this section demonstrates that variations in safety alignment across nlp tasks expose a vulnerability, exploitable through our proposed single and compositional in-context attacks. ## conclusion this paper introduces a new setup to evaluate the safety alignments of nlp tasks and uncovers a previously unidentified vulnerability in llms: different safety alignments exist across nlp tasks for processing safety-sensitive documents. to exploit this vulnerability, we propose two simple incontext attacks using weakly aligned nlp tasks, particularly summarization, to challenge the safety alignments of other nlp tasks with high attack success rate. our additional experiments also suggest that the vulnerability may be related to the trade-off between usefulness and safety principles in llms. the imbalanced research focus on task improvements and safety across various nlp tasks highlights the need for greater attention to these disparities. such imbalances can create new vulnerabilities, susceptible to very simple attacks related to benign nlp tasks.