From LLM s to MLLM s: Exploring the Landscape of Multimodal Jailbreaking (2024)

Abstract: The rapid development of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) has exposed vulnerabilities to various adversarial attacks. This paper provides a comprehensive overview of jailbreaking research targeting both LLMs and MLLMs, highlighting recent advancements in evaluation benchmarks, attack techniques and defense strategies. Compared to the more advanced state of unimodal jailbreaking, multimodal domain remains underexplored. We summarize the limitations and potential research directions of multimodal jailbreaking, aiming to inspire future research and further enhance the robustness and security of MLLMs.

## introduction recent advancements in large language models (llms) @xcite @xcite have demonstrated remarkable performance across various tasks, effectively following instructions to meet diverse user needs. however, alongside their rising instruction-following capability, these models have increasingly become targets of adversarial attacks, significantly challenging their integrity and reliability @xcite @xcite . this emerging vulnerability inspires extensive research into attack strategies and robust defenses to better safeguard ethical restrictions and improve llms @xcite . among these vulnerabilities, the jailbreak attack @xcite is particularly prevalent, where malicious instructions or training and decoding interventions can circumvent the built-in safety measures of llms, leading them to exhibit undesirable behaviours. there has been notable recent research into llms jailbreaking, including constructing evaluation benchmarks for increasingly complex scenarios, presenting advanced attack methods and corresponding defense strategies. for example, several studies @xcite @xcite explore jailbreak datasets across various domains and types of harm in different task formats. subsequent research @xcite investigates various mechanisms for jailbreak prompting, fine-tuning and decoding. to defend against jailbreak attacks, @xcite propose pre-detection of harmful queries, while @xcite introduce post-processing harmful outputs. furthermore, safety alignment @xcite through supervised fine-tuning (sft) or reinforcement learning from human feedback (rlhf) is implemented to enhance llms' resistance to adversarial attacks. advanced llms also inspire the development of multimodal large language models (mllms) @xcite @xcite for applications requiring responses to visual and linguistic inputs. while achieving impressive performance, they also expose vulnerabilities to various attacks @xcite , such as generating guidance on producing hazardous materials depicted in images. preliminary studies @xcite @xcite have introduced corresponding datasets and attack methods for mllms. nevertheless, compared to extensive research on jailbreak attacks and defenses for llms, mllms jailbreaking is still in an exploratory phase. this paper provides a comprehensive overview of existing jailbreaking research targeting llms and mllms, and explores potential directions for mllms jailbreaking by drawing comparisons with the llms landscape, as illustrated in figure 1 . we start this study with a detailed introduction ( § 2). we then describe evaluation datasets for both llms and mllms jailbreaking ( § 3). we elaborate on single-turn query responding multi-turn conversation evaluation benchmark lifetox multi-turn advbench red-eval fft strongreject latent jailbreak promptbench advbench do-not-answer attaq safetybench explicit toxicity limited image sources narrow task scope static nature of toxicity develop specialized datasets for various demographics/cultures use various multimodal task formats like multi-turn dialogue or embodied interactions increase the diversity by sourcing from various origins and categories explore diverse and complex multimodal tasks for context virtualization and attention distraction, like jigsaw puzzles transfer image distribution without altering content, and reformulate multimodal task formats construct malicious instances with backdoor poisoned images for fine-tuning devise sophisticated multimodal attacks using iterative or collaborative methods non-generalizable defense poor robustness false positive challenge high cost of safety alignment unexplored image-based detection develop a comprehensive and adaptable defense system regularly update training sets with new attack trends and continuously train a defense model. explore detection and smoothing methods directly operating images design fine-grained defense methods, use majority-vote and cross-validation remove subsets that are benign but degrade safety, and model pruning for safety alignment limitation future direction ## evaluation datasets for jailbreaking to assess jailbreak attack strategies and model robustness against attacks, various datasets have been introduced. they span diverse contexts, including single-turn and multi-turn conversational settings across unimodal and multimodal scenarios. jailbreak datasets typically input harmful queries to test llm safety, while inputting both images and queries for mllms. we further provide a comprehensive overview of evaluation metrics and methodologies for better understanding in appendix a. ## jailbreak attack jailbreak attack methods fall into two main categories: non-parametric and parametric attacks, targeting both llms and mllms. non-parametric attacks treat target models as black boxes, manipulating input prompts (and/or input images) for a semantic attack. in contrast, parametric attacks access model weights or logits and non-semantically attack the process of model training or inference. a high-level comparison of these methods is provided in ## jailbreak defense jailbreak defense methods protect models from generating harmful content, falling into two main categories: extrinsic and intrinsic defenses. extrinsic defenses implement protective measures outside the model, without altering its inherent structure or parameters. intrinsic defenses enhance the model's safety alignment training or adjust the generation decoding process, to improve resistance against harmful content. a high-level comparison of these methods is summarized in ## conclusion in this work, we offer a thorough overview of jailbreaking research for llms and mllms, discussing recent advances in evaluation benchmarks, attack techniques and defense strategies. furthermore, we summarize the limitations and potential research directions of of mllm jailbreaking by drawing comparisons to the more advanced state of llm jailbreaking, aiming to inspire future work.