Typographic Attacks in a Multi-Image Setting (2025)

Abstract: Large Vision-Language Models (LVLMs) are susceptible to typographic attacks, which are misclassifications caused by an attack text that is added to an image. In this paper, we introduce a multi-image setting for studying typographic attacks, broadening the current emphasis of the literature on attacking individual images. Specifically, our focus is on attacking image sets without repeating the attack query. Such non-repeating attacks are stealthier, as they are more likely to evade a gatekeeper than attacks that repeat the same attack text. We introduce two attack strategies for the multi-image setting, leveraging the difficulty of the target image, the strength of the attack text, and text-image similarity. Our text-image similarity approach improves attack success rates by 21% over random, non-specific methods on the CLIP model using ImageNet while maintaining stealth in a multi-image scenario. An additional experiment demonstrates transferability, i.e., text-image similarity calculated using CLIP transfers when attacking InstructBLIP.

## introduction large vision-language models (lvlms), such as the contrastive language-image pretraining model (clip) @xcite and instruct-blip @xcite , have shown remarkable performance across a variety of multimodal downstream tasks. however, a small, but important, set of research contributions has recently demonstrated that lvlms are vulnerable to typographic attack on their classification abilities @xcite @xcite . in typical typographic attacks, a text consisting of one or more words is added to an image, superimposed in the middle @xcite or added at the top or bottom @xcite . alternatively, it is added as a physical label to an object before the target image is taken @xcite . a typographic attack is successful when the text, which in this paper, we introduce the multi-image attack setting, which limits attack text repetition, and we show the importance of text-image similarity when choosing an attack text for a given target image. we explain the multi-image attack setting with the help of figure 1 . in previous works @xcite , the emphasis is on attacking individual images, and the same attack text could be used repeatedly for different target images (figure 1a ). in our work, in contrast, we consider the pattern of attack texts used for the attack across a set of images. specifically, we are interested in the case in which the attack does not repeat across the images (figure 1b ). in short, studying an attack in a multi-image attack setting means studying the attack from a holistic perspective. the importance of non-repeating attacks is elaborated in section 2.3. developing typographic attacks for the nonrepeating multi-image setting requires answering a key question: what strategy should the attacker use to attack the set of target images in a way that maintains the attack success rate but avoids repeating attack texts? this paper reveals that the attacker should take the similarity between the attack text and the target image into account. first, we look at a possible alternative to textimage similarity, namely, attack text effectiveness (ate). researchers have found that the targeted attack success rate varies with the attack text @xcite . ate is the average attack success of an attack text measured across a set of images. next, we study how images behave under attack. the literature has reported that adversarial attacks are not uniformly effective across all images @xcite . we build on this finding and study visual image prediction probability (vipp), i.e., how challenging it is to predict the class of an image. using vipp, we can prioritize images and propose attack strategies in which the most difficult images are assigned attack texts first. then, we turn to the strategies proposed for typographic attacks in the multi-image setting. our first type of strategy tests the contribution of using ate to select attack texts for images prioritized by vipp. our second type of strategy tests the advantages of using text-image similarity to select attack texts, with and without vipp image-prioritization. our paper makes the following contributions: source code for this paper is available. 1 ## background and motivation in this section, we provide further details on the background of and motivation for our work. ## attack strategies in this section, we first provide the specifics of how we instantiate the multi-image attack setting with non-repeating attack texts for our experiments. then, we describe the attack strategies we propose on the basis of the insights in section 4. ## loosening the non-repeating requirement in this section, we gradually increase the maximum number of repetitions of attack texts allowed in the multi-image setting. we aim to study the trade-off between the stealth of the multi-image attack and the asr. we focus on highvipp-highate and vipp-textimsim, which represent our ate and textimsim approaches. figure 7 plots the untargeted asr as we allow attack texts to repeat an increasing number of times from the non-repeating setting (which corresponds to 1 at the far left). targeted asr is not depicted, but follows the same trends. the plot reveals that the strength of the vipp-textimsim strategy with respect to the highvipp-highate strategy is maintained as more and more repetitions are allowed, supporting the importance of text-image similarity for typographic attacks. the asr of both vipp-textimsim and highvipp-highate increase as more repetitions are admitted, reflecting the trade-off between stealth (i.e., restricted attack text repetitions) and attack strength (i.e., asr). however, highvipp-highate increases more steeply, which we attribute to the increased use of attack texts with the highest effectiveness, i.e., those on the far right of figure 2 . we note that the performance of vipp-textimsim is limited by the attack texts that are available in the set of 579 attack texts used in this experiment. the set might fail to obtain optimal attack texts, meaning attack texts with the highest possible similarity to the target image. future work should study the impact of the set of candidates from which the attack text can be chosen on approaches using text-image similarity. ## evaluation on other lvlms in the previous sections, we have experimented with a whitebox attack on clip. we assume that attackers have full access to the trained clip model that they are attacking. this experimental setting is not unrealistic because many lvlms are publicly available. however, it is interesting to understand whether our multi-image attack can also be carried out without whitebox access. in this section, we perform a greybox attack on the instructblip (flant5xxl) @xcite model using vipp-textimsim, our best-performing method from section 6. the attacker does not access instructblip to create the attack but instead has access to clip. we refer to this attack as greybox and not blackbox, since instructblip uses a vision encoder similar in structure to clip and its training dataset has a substantial overlap with the clip training set. we choose the instructblip (flant5xxl) because it demonstrates better optical character recognition (ocr) performance, e.g., compared to the minigpt4 model evaluated in @xcite . the ocr performances mean that instructblip focuses on text in images and is for this reason particularly susceptible to typographic attack @xcite . future work can also look at the llava @xcite ) model, which exhibits limited robustness against typographic attacks @xcite , as well as models without strong ocr performance. ## conclusion and outlook in this paper, we have introduced a multi-image attack setting for typographic attacks and studied the contribution of the similarity between target images and attack text to typographic attacks. we have argued that repeated use of the same attack text is not the most dangerous attack, since it would be evident to a gatekeeper. once stealth is taken into account, the importance of text-image similarity for typographic attacks becomes clear. moving forward, we foresee that typographic attacks based on text-image similarity can reveal issues in the embedding space of lvlms. also, research is needed to understand the factors influencing attack texts that cause untargeted misclassifications, investigating whether the distraction introduced by the text is semantic or purely visual in nature. such investigations can improve lvlms, making them robust to the dangers of typographic attacks. studying diverse attack texts in the multiimage setting will continue to be part of this effort.