Adversarial Attacks on Parts of Speech: An Empirical Study in Text-to-Image Generation (2024)

Abstract: Recent studies show that text-to-image (T2I) models are vulnerable to adversarial attacks, especially with noun perturbations in text prompts. In this study, we investigate the impact of adversarial attacks on different POS tags within text prompts on the images generated by T2I models. We create a high-quality dataset for realistic POS tag token swapping and perform gradient-based attacks to find adversarial suffixes that mislead T2I models into generating images with altered tokens. Our empirical results show that the attack success rate (ASR) varies significantly among different POS tag categories, with nouns, proper nouns, and adjectives being the easiest to attack. We explore the mechanism behind the steering effect of adversarial suffixes, finding that the number of critical tokens and information fusion vary among POS tags, while features like suffix transferability are consistent across categories.

## introduction text-to-image (t2i) generation models such as stable diffusion @xcite , dall-e2 @xcite , imagen @xcite , ediff-i @xcite have made steady progress in the field of image generation by bridging the semantic gap between textual descriptions and visual representations. unlike traditional methods reliant solely on pixel manipulation, these models leverage multi-model alignments in latent spaces to interpret and synthesize complex visual content from textual prompts. recent studies, such as @xcite , have interpreted how cross-alignment from texts to images is transformed through text-image attribution analysis, demonstrating that different pos tags are well captured by cross-modal attention during synthesis. a brown teddy bear sitting inside of a sock. ]. therland wcw jacket wicker study moduimc seated stool a polar bear walking on some rocks. surrender :] yearling rests cftrap lounging margarepisa exhibits a white curly haired dog standing in a hallway. ghue compartment memorable confederate invasive hempimg bornonthisday toledo dissertation a train car with blue graf ti on the side of it. whoogoalscorer simulator soleil sakura ose hits bras harvest journalists two giraffes in a zoo look at their surroundings. sometimes pcso puppysoybeans nilguernsey benedicheading individually vening a lady in a black and orange halloween costume. departing displayed commissions pung tang norse embroidery cny tranquil northern sock chair walking lying curly bushy blue pink two six halloween chinese figure 1: examples of successful adversarial attacks on stable diffusion covering different pos tags drawn from our dataset. the pos tokens targeted by adversarial suffixes are highlighted in red. in addition, we observe that the attack success rate (asr) varies significantly across pos tag categories, with features like the number of critical tokens (defined in §5, noncritical tokens are highlighted in orange) being highly associated with asr. on the other hand, recent research shows that t2i models are vulnerable to adversarial perturbations in text prompts, such as inserting nonsensical words @xcite , phrases @xcite , or irrelevant characters @xcite , which can significantly bias the generated images @xcite . however, current adversarial attacks on t2i generation models, either manual heuristic-based methods @xcite @xcite or automatic gradient-based approaches @xcite @xcite @xcite @xcite , are specifically targeting entities or objects (i.e., nouns) in text prompts, neglecting other parts of speech. in this paper, we aim to answer the following two research questions: to bridge the gap in analyzing attack mechanisms across different pos tag categories beyond nouns, we first created a dataset with realistic scenarios for swapping different pos tag categories with adversarial attacks. figure 1 provides a few examples drawn from our dataset covering six pos tags from @xcite : noun, adjective, verb, adverb, numeral, and proper noun, with adversarial suffixes that successfully mislead t2i models into generating images related to the targeted attribute. creating such a dataset is non-trivial, as @xcite noted that asr to t2i models might be affected by internal bias rather than the attack itself; we tried to minimize such biases when creating the dataset. to the best of our knowledge, there is currently no dataset available for analyzing adversarial attacks on pos tags other than nouns (refer to we conduct targeted adversarial attacks over pos tag categories with a gradient-based token searching algorithm specifically designed for t2i models to effectively navigate the larger vocabulary size of the t2i text encoder @xcite . the attack objective is to create an adversarial prompt that causes a target pos token to appear in the generated image while ensuring the original pos token from the input prompt does not. we observe that the asr differs significantly across different pos tag categories. nouns, proper nouns, and adjectives are the easiest to attack, with increasing difficulties, while the other three categories, with the same gradient-based attack, offer almost no success, whether in restricted (preventing the target token's pos tag attribute from appearing in the adversarial suffix) or unrestricted settings. this observation led us to further investigate whether there are features associated with these differences in asr across pos categories. through extensive experiments, we discovered a correlation between the number of critical tokens in adversarial suffixes and the attack success rate across different pos categories. critical tokens are those whose removal from the adversarial suffix renders the attack unsuccessful. additionally, the results from our ablation study reveal that adversarial suffixes, while steering the generation of target attributes, often fail to completely remove the original attribute. for example, when attempting to change a purple grape to a green one using adversarial suffixes, the resulting image often shows a mixed color. in contrast, with noun attacks, both objects can be generated, whereas with verbs, it is difficult to mix or generate both original and swapped tokens. this varying ease of content fusion across pos categories may also contribute to the differences in asr. furthermore, we identified a general feature shared across different pos categories: the transferability of the attack suffix. similar to nouns, the adversarial suffixes found are universally transferable to other input prompts with the same attributes. this means an adversarial suffix can transform multiple input prompts with different attributes into the same target attributes in the generated images. for instance, we found that the same adversarial suffix targeting a 'blue' cup can steer the model to generate images of a blue cup across multiple input prompts with different original colors (e.g., red, yellow, orange). yang et al. (2024a) explored a focused targeted attack that adds target objects while removing original ones, and developed mmp-attack, which incorporates multimodal features. du et al. (2024) proposed autoattack on text-to-image models (atm), which automatically generates attack prompts that resemble clean prompts by replacing or adding words. shahgir et al. (2023) applied gradient-based to-ken perturbation methods to replace entities in the prompt with adversarial suffix tokens. we adopt the gradient attack proposed by (shahgir et al., 2023) because it aligns with our attack objectives and demonstrates strong performance in targeting nouns. ## dataset creation in this section, we outline the procedure for constructing our dataset. we first specify the dataset source and then describe the steps involved in its construction. data collection. the first obstacle we encountered in evaluating adversarial attacks across different pos categories beyond nouns was that there was no existing dataset for fair comparison. ## experiment in this section, we outline the gradient-based adversarial attack method, describe the experimental setup, and report the results to assess the effectiveness of the attack. ## attack success mechanism in this section, we explore the mechanism behind the steering effect of adversarial suffixes. we identify (a) features that vary across pos categories and explain differences in attack success rates (asr), such as the number of critical tokens and content fusion, and (b) features that are consistent across different pos categories and do not explain variations in asr rates, but provide general insights such as suffix transferability. correlation between the number of critical tokens in adversarial suffixes and asr. a successful attack demonstrates that appending an adversarial suffix to an input prompt effectively shifts the text embedding toward the target prompt, highlighting the significant role of the suffix tokens. to investigate, we tokenized several adversarial suffixes, generated an image for each token to isolate their contributions, and found that some tokens generate images associated with the target pos token. this observation led us to identify the most contributing tokens within adversarial suffixes. we define "critical tokens" as those whose removal causes the attack to fail. to determine critical tokens in a suffix, we generated all possible combinations of replacing suffix tokens @xmath0. for each combination, we generated an image and queried the pre-trained vision-language model blip foot_0 @xcite to check if the generated image matched the target prompt. we identified the combination with the fewest tokens replaced @xmath1. tokens not replaced @xmath2. we present the average number of critical tokens across all pos tags in ## conclusion in this study, we evaluate a gradient-based adversarial attack aimed at six pos tags within text prompts in both unrestricted and restricted attack strategies. we assess the impact of these attacks on the stable diffusion, revealing valuable insights into the factors contributing to their success. our findings reveal that nouns, proper nouns, and adjectives are particularly vulnerable to perturbation, resulting in adversarial image generation. however, we see that verbs, adverbs, and numerals exhibit a higher level of resilience against adversarial attack, exerting minimal influence on the visual output generated by the stable diffusion. we hypothesize that the number of critical tokens in an adversarial suffix and the ease of content fusion are primarily responsible for such resilience against attacks. we believe these findings will be valuable for enhancing the robustness of t2i generation systems.