Red Teaming Visual Language Models (2024)

Abstract: VLMs (Vision-Language Models) extend the capabilities of LLMs (Large Language Models) to accept multimodal inputs. Since it has been verified that LLMs can be induced to generate harmful or inaccurate content through specific test cases (termed as Red Teaming), how VLMs perform in similar scenarios, especially with their combination of textual and visual inputs, remains a question. To explore this problem, we present a novel red teaming dataset RTVLM, which encompasses 12 subtasks (e.g., image misleading, multi-modal jailbreaking, face fairness, etc) under 4 primary aspects (faithfulness, privacy, safety, fairness). Our RTVLM is the first red teaming dataset to benchmark current VLMs in terms of these 4 different aspects. Detailed analysis shows that 10 prominent open-sourced VLMs struggle with the red teaming in different degrees and have up to 31% performance gap with GPT-4V. Additionally, we simply apply red teaming alignment to LLaVA-v1.5 with Supervised Fine-tuning (SFT) using RTVLM, and this bolsters the models’ performance with 10% in RTVLM test set, 13% in MM-hallu, and without noticeable decline in MM-Bench, overpassing other LLaVA-based models in similar size with regular alignment data. This reveals that current open-sourced VLMs still lack red teaming alignment. Our code and datasets will be open-sourced.

## introduction vision-language models (vlms) are capable of processing both textual and visual inputs, thus empowering a variety of downstream applications @xcite . with the rapid development of large language models (llms), the incorporation of llms into vlms has further enhanced the ability of vlms to understand complex inputs @xcite . despite promising progress achieved by vlms, their performance under challenging scenarios still remains unclear. there is abundant evidence demonstrating that the backbone of vlms, i.e., the llms, tend to generate incorrect or harmful content for certain red teaming cases @xcite @xcite . it is natural to assume that the vlms built upon the llms may possess potential risk as well. besides, given their unique blend of textual and visual input, new types of red teaming cases that pose a significant threat to the deployed vlms might be overlooked. preliminary cases demonstrate that the early version of gpt-4v also suffers under red teaming, such as generating discrimina-tory remarks and being used to disclose personal information @xcite . therefore, a stress test with red teaming cases is necessary for the safe deployment of vlms, providing insights for subsequent improvements of the model to further align with ethical and privacy standards. nevertheless, there is a lack of comprehensive and systematic red teaming benchmark for current vlms. to fill this gap, we introduce the red teaming visual language model (rtvlm) dataset, meticulously focusing on red teaming in scenarios involving image-text input. figure 1 illustrates the whole process of dataset construction, evaluation, and alignment. based on previous works @xcite , we summarize 4 aspects of red teaming: faithfulness, safety, privacy, and fairness. this dataset comprises 12 task categories distributed across these 4 aspects, shown in figure 2 . under faithfulness, we investigate the models' ability to generate accurate outputs despite given misleading inputs. regarding privacy, the models are required to distinguish between public figures and private individuals, ensuring non-disclosure of private information. for safety, we assess the models' ability to reject responses to potentially harmful or legally sensitive multimodal inputs. fairness is measured by examining the bias of individuals differing in race and gender. to guarantee that our test data is novel and has not been seen by the evaluated vlms, we construct new questionimage pairs based on publicly available images or originally diffusion-generated images @xcite . our red teaming questions are annotated by humans, or generated by gpt-4 with self-instruction @xcite accompanied by human-written seed examples. after a manual check on all the question pairs, the resulting rtvlm dataset comprises 10,200 samples. with the rtvlm dataset, we test a suite of vlms and analyze their performance under this red teaming test. we first establish a set of scoring criteria targeting the 4 aspects mentioned above following @xcite . specifically, we conduct gpt-4v eval and human eval to score the vlm models' output on the rtvlm, including 10 opensourced vlms along with the current version of gpt-4v, measuring their performance across the four dimensions of faithfulness, safety, privacy, and fairness. we conduct a detailed analysis of the models' performance in each subcategory. we find that: overall, our study serves as the first red teaming benchmark for visual language models, elucidating their vulnerabilities and proposing credible solutions for future work. ## rtvlm dataset in this section, we outline the construction process of our rtvlm. we first present an overview of our data format and statistics ( §2.1), followed by a detailed description of the definitions, data collection, and processing of subtasks in four aspects: faithfulness ( §2.2), privacy ( §2.3), safety ( §2.4), and fairness ( §2.5). cases for each task are showcased in figure 2 . ## experimental results in this section, we first introduce the experimental settings for evaluating selected vlms on rtvlm ( §3.1). in §3.2, we then discuss the overall performance of vlms on rtvlm from 4 dimensions. finally, we analyze the issues with current aligned vlms in red teaming tests and demonstrates how using rtvlm as sft data aids in enhancing the model's ability to handle red teaming ( §3.3). ## conclusion red teaming for vlms has become a topic of growing interest. we propose the first vlm red teaming dataset rtvlm, which contains 4 categories of tasks where vlms are particularly vulnerable to being red teaming. we have compiled a dataset of 10,200 entries, incorporating various tasks like multimodal jailbreaking, and visual misleading. testing on current popular open-source vlms and gpt-4v revealed that many of them are not sufficiently secure against various types of red teaming attacks, with some open-source models showing strong biases. by utilizing a subset of our data for red teaming alignment, we significantly improved the models' performance on downstream red teaming tasks, demonstrating the viability of using rtvlm for training on model security. we hope our work will draw the community's attention to the importance of vlm security and provide insights and references for enhancing it.