QAVA : Query-Agnostic Visual Attack to Large Vision-Language Models (2025)

Abstract: In typical multimodal tasks, such as Visual Question Answering (VQA), adversarial attacks targeting a specific image and question can lead large vision-language models (LVLMs) to provide incorrect answers. However, it is common for a single image to be associated with multiple questions, and LVLMs may still answer other questions correctly even for an adversarial image attacked by a specific question. To address this, we introduce the query-agnostic visual attack (QAVA), which aims to create robust adversarial examples that generate incorrect responses to unspecified and unknown questions. Compared to traditional adversarial attacks focused on specific images and questions, QAVA significantly enhances the effectiveness and efficiency of attacks on images when the question is unknown, achieving performance comparable to attacks on known target questions. Our research broadens the scope of visual adversarial attacks on LVLMs in practical settings, uncovering previously overlooked vulnerabilities, particularly in the context of visual adversarial threats. The code is available at https://github.com/btzyd/qava.

## introduction with the expansion in model parameters and training datasets, large vision-language models (lvlms) have gained significant popularity, demonstrating exceptional performance across various tasks, including image classification, image captioning, semantic segmentation, and visual question answering (vqa) @xcite @xcite . however, training lvlms from the ground up is resourceintensive. as a result, the prevailing approach involves fine-tuning pre-trained visual encoders and large language models (llms) while training a vision-language alignment module. this process adapts visual tokens to the input space of the llms what color is the traffic light? what is in the mirror? where is the horse's rider? what type of pizza is that? …… figure 1: traditional adversarial attacks involve inputting an image x i and a specified target question x t,target into lvlms, with adversarial images generated through gradient-based methods. this approach typically results in incorrect answers for x i and x t,target (i.e., q1). however, for other questions x t,@xmath0,@xmath1,other ̸ = x t,target } within the question set t that are not the same as the x t,other , it remains possible for lvlms to provide correct answers (i.e., q2-q6). our qava samples a set of questions x t,qava and performs attacks on these questions, even if they are unrelated to the original image x i . qava generates adversarial images that are likely to produce incorrect responses when faced with unknown target questions. to develop more potent attacks, our objective is to manipulate images so that they yield incorrect answers to an unknown set of target questions, noted as query-agnostic visual attack (qava). these query-agnostic adversarial examples have the potential to cause significant disruption to the model. for instance, during the inference phase, lvlms may consistently provide incorrect answers to any question regarding the manipulated image. moreover, employing these adversarial examples during the training or supervised fine-tuning phase could be even more detrimental, particularly when used for data poisoning. we enhance the attack by scrutinizing both the attack's location and the selection of questions used. in terms of attack positioning, traditional attack methods typically employ the end-to-end loss function of the entire lvlm as the attack objective function. however, for adversarial attacks, it is crucial to identify and target the most vulnerable component of the lvlm. given the extensive number of parameters in large language models, we posit that end-to-end attacks on lvlms may not be as effective as targeting the inputs to the llms. consequently, we focus on attacking the output of the visual-language alignment module. the visual-language alignment module's output encompasses multimodal interactions, and we can disrupt these critical multimodal interactions by targeting the alignment module, potentially leading to more effective attack outcomes. regarding the questions employed in attacks, we observed that when targeting the visual-language alignment module, effective attack performance can be achieved even with the use of randomized questions that are unrelated to the image. furthermore, the attack is enhanced when a larger number of random, irrelevant questions are utilized, which verifies our qava's flexibility and effectiveness. in conclusion, qava diverges from traditional attacks in two key aspects: (1) we focus on attacking the output of the visual-language alignment module within lvlms, which is verified to be more vulnerable to query-agnostic attacks. (2) we utilize multiple randomized, image-independent questions in our attacks, ensuring that the adversarial examples are maximally incorrect when confronted with unknown potential inputs. our main contributions are summarized as follows: (1) we introduce a query-agnostic attack method, qava, which enhances the practicality of adversarial attacks on images within lvlms. (2) we identify the vulnerability of visual-language alignment modules in lvlms to adversarial attacks and leverage this vulnerability to execute query-agnostic attacks. (3) extensive experiments demonstrate the efficacy of our qava approach in both white-box and black-box attack scenarios. additionally, our qava method exhibits inter-task transferability, such as transferring from the vqa task to the image captioning task. this serves as an important alert regarding the security of lvlms. ## conclusion in this paper, we introduce a robust adversarial attack method, qava, designed to generate adversarial examples that significantly mislead responses to unknown target questions for a given image. qava initially identifies the vulnerability within the lvlms, specifically the vision-language alignment module. subsequently, it executes adversarial attacks utilizing a broader range of randomly sampled, image-irrelevant questions. extensive experiments demonstrate the effectiveness of qava in both white-box and black-box attack scenarios. additionally, we verify the high transferability of qava across various lvlms and different tasks. our findings with qava serve as a critical alert regarding the security vulnerabilities of lvlms.