What the DAAM : Interpreting Stable Diffusion Using Cross Attention (2023)

Abstract: Diffusion models are a milestone in text-to-image generation, but they remain poorly understood, lacking interpretability analyses. In this paper, we perform a text-image attribution analysis on Stable Diffusion, a recently open-sourced model. To produce attribution maps, we upscale and aggregate cross-attention maps in the denoising module, naming our method DAAM. We validate it by testing its segmentation ability on nouns, as well as its generalized attribution quality on all parts of speech, rated by humans. On two generated datasets, we attain a competitive 58.8-64.8 mIoU on noun segmentation and fair to good mean opinion scores (3.4-4.2) on generalized attribution. Then, we apply DAAM to study the role of syntax in the pixel space across head–dependent heat map interaction patterns for ten common dependency relations. We show that, for some relations, the head map consistently subsumes the dependent, while the opposite is true for others. Finally, we study several semantic phenomena, focusing on feature entanglement; we find that the presence of cohyponyms worsens generation quality by 9%, and descriptive adjectives attend too broadly. We are the first to interpret large diffusion models from a visuolinguistic perspective, which enables future research. Our code is at https://github.com/castorini/daam .

## introduction diffusion models trained on billions of captioned images represent state-of-the-art text-to-image generation @xcite , with some achieving photorealism, such as google's imagen @xcite and openai's dall-e 2 @xcite . however, despite their quality and popularity, the dynamics of their image synthesis remain undercharacterized. citing ethics, corporations have restricted the general public from using the models and their weights, preventing effective figure 1 : the original synthesized image and three daam maps for "monkey," "hat," and "walking," from the prompt, "monkey with hat walking." analysis. to overcome this barrier, stability ai recently open-sourced stable diffusion @xcite , a 1.1 billion-parameter latent diffusion model pretrained and fine-tuned on the laion 5billion image dataset @xcite . we probe stable diffusion to provide insight into large diffusion models. focusing on text-image attribution, our central question is, "how does an input word influence parts of a generated image?" to this, we propose to produce 2d attribution maps for each word by combining cross-attention maps in the model. a related work in prompt-guided editing from @xcite conjectures that per-head cross attention relates words to areas in imagen-generated images, but they fall short of constructing global per-word attribution maps. we name our method diffusion attentive attribution maps, or "daam;" see figure 1 for an example. to evaluate the veracity of daam, we apply it to a semantic segmentation task @xcite on generated imagery, comparing daam maps with annotation. we attain a 58.8-64.8 mean intersection over union (miou) score competitive with unsupervised segmentation models, described in section 3.1. we further bolster these results using a generalized study covering all parts of speech (all in penn treebank; @xcite , such as adjectives and verbs. through human annotation, we show that the mean opinion score (mos) is above fair to good (3.4-4.2) on interpretable words. next, we study how relationships in the syntactic space of prompts relate to those in the pixel space of images. we assess head-dependent daam interactions across ten common syntactic relations (enhanced universal dependencies; @xcite , finding that, for some, the heat map of the dependent strongly subsumes the head's, while the opposite is true for others. for others, such as coreferent word pairs, the words' maps greatly overlap, indicating coreferent understanding during generation. we assign intuition to our observations; for example, we observe that the maps of verbs contain their subjects, suggesting that verbs strongly contextualize the generation of both the subjects and their surroundings. finally, we form hypotheses to further our syntactic findings, studying semantic phenomena using daam, particularly those affecting image quality. in section 5.1, we demonstrate that, in constructed prompts with two distinct nouns, cohyponyms have worse quality (9% worse than non-cohyponyms), e.g., "a giraffe and a zebra" generates a giraffe or a zebra, but not both. cohyponym status and generation incorrectness each increases the amount of heat map overlap, advancing daam's utility toward improving diffusion models. we also show in section 5.2 that descriptive adjectives attend too broadly across the image, far beyond their nouns. if we fix the scene layout @xcite and vary only the adjective, the entire image changes, not just the noun. these two phenomena suggest feature entanglement, where objects are entangled with both the scene and other objects. in summary, our contributions are as follows: ## visuosyntactic analysis equipped with daam, we now study how syntax relates to generated pixels. we characterize pairwise interactions between head-dependent daam maps, augmenting previous sections and helping to form hypotheses for further research. setup. we randomly sampled 1,000 prompts from coco, performed dependency parsing with @xcite , and generated an image for each prompt and daam maps for all words. we constrained ourselves to the top-10 most common relations, resulting in 8,000 headdependent pairs. following section 3.1, we then binarized the maps to quantify map pair interactions with set-based similarity statistics. we computed three statistics between the daam map of the head and that of the dependent: first, the miou, i.e., |a∩b| |a∪b| ; second, the intersection over the dependent (miod; |a∩b| |a| ); and third, the intersection over the head (mioh; |a∩b| |b| ). miou measures similarity, and the difference between miod and mioh quantifies dominance. if @xmath0, then the head contains (dominates) the dependent more, and vice versa-see appendix b for a visual tutorial. results. we present our results in starting at row 8, we arrive at pairs where one map dominates the other. a group in core arguments arises (nsubj, obj), where the head word dominates the noun subject's or object's map (12-29-point ∆), perhaps since verbs contextualize both the subject and the object in its surroundings-see the middle of and bottom left of fig. 5 . we observe another group in nominal dependents (nmod:of, amod, acl), where nmod:of mostly points to collective nouns (e.g., "pile of oranges"), whose dominance is intuitive. in contrast, adjectival modifiers (amod) behave counterintuitively, where descriptive adjectives (dependents) visually dominate the nouns they modify (∆ ≈ 15). we instead expect objects to contain their attributes, but this is not the case. we again ascribe this to entanglement, elucidated in section 5.2. lastly, coreferent word pairs exhibit the highest overlap out of all relations (66.6 miou), indicating coreference resolution. correct incorrect noncohyponym cohyponym 5.43 29.5 43.5 60.9 overlap by cohyponym status and correctness 25 50 high mid low degree of overlap noncohyponym cohyponym 36 56.1 77.5 9.75 19.3 71.7 accuracy by cohyponym status and overlap 25 50 75 figure 6: above: daam map overlap in mean iou, subdivided by cohyponym status and correctness; below: generation accuracy, subdivided by cohyponym status and amount of overlap. ## related work and future directions the primary area of this work is in understanding neural networks from the perspective of computational linguistics, with the goal of better informing future research. a large body of relevant papers exists, where researchers apply textual perturbation @xcite , attention visualization @xcite @xcite , and information bottlenecks @xcite to relate important input tokens to the outputs of large language models. others explicitly test for linguistic constructs within models, such as probing vision transformers for verb understand-ing (hendricks and nematzadeh, 2021) and examining visual grounding in image-to-text transformers @xcite . our distinction is that we carry out an attributive analysis in the space of generative diffusion models, as the pixel output relates to syntax and semantics. as a future extension, we plan to assess the unsupervised parsing ability of stable diffusion with syntacticgeometric probes, similar to @xcite work in bert. the intersection of text-to-image generation and natural language processing is substantial. in the context of enhancing diffusion models with prompt engineering, hertz et al. (2022) last but not least are semantic segmentation works in computer vision. generally, researchers start with a backbone encoder, attach decoders, and then optimize the model in its entirety end-to-end on a segmentation dataset @xcite , unless the context is unsupervised, in which case one uses contrastive objectives and clustering @xcite . toward this, daam could potentially provide encoder features in a segmentation pipeline, where its strong raw baseline numbers suggest the presence of valuable latent representations in stable diffusion. ## conclusions in this paper, we study visuolinguistic phenomena in diffusion models by interpreting word-pixel cross-attention maps. we prove the correctness of our attribution method, daam, through a quantitative semantic segmentation task and a qualitative generalized attribution study. we apply daam to assess how syntactic relations translate to visual interactions, finding that certain maps of heads inappropriately subsume their dependents'. we use these findings to form hypotheses about feature entanglement, showing that cohyponyms are jumbled and adjectives attend too broadly.