Diffusion Lens: Interpreting Text Encoders in Text-to-Image Pipelines (2024)

Abstract: Text-to-image diffusion models (T2I) use a latent representation of a text prompt to guide the image generation process. However, the process by which the encoder produces the text representation is unknown. We propose the Diffusion Lens, a method for analyzing the text encoder of T2I models by generating images from its intermediate representations. Using the Diffusion Lens, we perform an extensive analysis of two recent T2I models. Exploring compound prompts, we find that complex scenes describing multiple objects are composed progressively and more slowly compared to simple scenes; Exploring knowledge retrieval, we find that representation of uncommon concepts require further computation compared to common concepts, and that knowledge retrieval is gradual across layers. Overall, our findings provide valuable insights into the text encoder component in T2I pipelines.

## introduction the text-to-image (t2i) diffusion pipeline comprises two main elements: the text encoder and the diffusion model. the text encoder converts a textual prompt into a latent representation, while the diffusion model utilizes this representation to generate the corresponding image. several recent studies have delved into the internal workings of the diffusion model and the cross-attention mechanism that connects the two components @xcite @xcite ). yet, while the text encoder is a key component of the pipeline with a large effect on image quality and text-image alignment @xcite , its internal mechanisms remain unexplored. moreover, while there is a wide range of work that has analyzed general language model internals @xcite , 2020; @xcite , these methods are not suitable for exploring fine-grained visual features. we propose the diffusion lens, a method for analyzing the inner mechanism of the text encoder. the diffusion lens uses intermediate representations of the prompt from various layers of the text encoder to guide the diffusion process, resulting in images that are clear, consistent, and easy to understand for most layers (see figure 1 ). notably, the diffusion lens relies solely on the pre-trained weights of the model and does not depend on any external modules. we employ the diffusion lens to examine the computational process of the text encoder in two popular t2i models: stable diffusion @xcite and deep floyd @xcite . our investigation focuses on two main analyses: the model's capability of conceptual combination and its memory retrieval process. for each analysis, we either construct a tailored dataset to isolate a specific phenomenon or utilize naturally occurring human-written image captions. our analysis of conceptual combination reveals various insights: (1) complex prompts such as "a yellow pickup truck and a pink horse" require more computation to achieve a faithful representation compared to simpler prompts such as "a green cat". next, we investigate memory retrieval, and uncover several key findings: (1) common concepts, such as "kangaroo", emerge in early layers while less common ones, such as the animal "dik-dik", gradually emerge across the layers, with the most accurate representations predominantly occurring in the upper layers, as illustrated in figure (3) knowledge retrieval is gradual, unfolding as computation progresses. this observation diverges from prior research on knowledge encoding in language models which characterizes knowledge as a localized attribute encoded in specific layers @xcite @xcite . (4) notably, there are discernible differences in memory retrieval patterns between the two text encoders: deep floyd's t5 memory retrieval exhibits a more incremental behavior compared to stable diffusion's clip. the disparities uncovered through our analyses suggest that factors such as architecture, pretraining objectives, or data may influence the encoding of knowledge or language representation within the models. our contributions are summarized as follows: ## diffusion lens preliminiaries. current text-to-image diffusion models comprise two main components @xcite : a language model used as a text encoder that takes the textual prompt as input and produces latent representations; and a diffusion model that is conditioned on the representations from the text encoder and generates an image from an initial input noise. the language model in the t2i pipeline is typically a transformer model. transformer models consist of a chain of transformer blocks, each composed of three sub-blocks: attention, multi-layer perceptron, and layer norm @xcite . we denote the transformer block at layer l as f l . the input to the model is a sequence of t word embeddings, denoted as h 0 = [h 0 1 , . . . , h 0 t ]. then, the output of the transformer block at layer l is a sequence of hidden states h l+1 : the output representations of the last block, l, go through a final layer norm, denoted as ln f . then, they condition the image generation process through cross-attention layers, resulting in an image i. we abstract this process as: diffusion lens. in a t2i pipeline with a text encoder of l layers, for layer @xmath0, we process the output of block l, including padding tokens, through the final layer norm. we condition the diffusion process on this output, as illustrated in figure 1 . namely, we generate an image i from an intermediate layer l as follows: ## conceptual combination t2i diffusion models are popular for their ability to generalize beyond their training data, creating composite concepts @xcite . conceptual combination is the cognitive process by which at least two existing basic concepts are combined to generate a new higher-order, composite concept (ling @xcite . conceptual combination is at the core of knowledge representation, since it asks how the meaning of a complex phrase connects to its component parts @xcite , e.g., "a cat in a box". this section uses the diffu-sion lens to trace the process by which the text encoder creates composite concepts. ## memory retrieval text-to-image diffusion models are able to retrieve information of many concepts @xcite , encompassing entities like notable individuals, animals, and more. memory retrieval-the recall of stored information-involves a constructive process rooted in the interactive dynamics between memory trace features and retrieval cue characteristics @xcite . in this section, we leverage the diffusion lens to scrutinize the memory retrieval mechanism in the text encoder. ## analyzing model failures in this section, we delve into cases where the t2i model fails to generate images that align with the input prompt. first, we investigate all failure cases in all experiments. figure 14 shows the percentage of failures for each experiment that had over 10 failures. we split failures to two types: complete failures where no layer generated a correct image through diffusion lens, and cases where at least one layer generated a correct image, but the top layer led to a failure (success then failure). generally, the percentage of failure cases (total height of each bar) is low, from 10% to 25% for most categories. prompts about two colored objects have a higher failure rate. importantly, in many failure cases, the representations in earlier layers lead to a correct generation via our method. notably, in simple prompts (relations and colored objects), about 80% of the failures had successful generations at earlier layers -see figure 15 for an example. once more constraints are imposed (two colored objects), we have a lower rate of early success. finally, for knowledge-related tasks (famous people, unfamiliar animals), there are very few cases of early success turned to failure. presumably, when the model fails, it is mostly because it does not encode the information at all. next, we zoom in on prompts describing two entities with different colors, as these prompts lead to the highest failure rate in our experiments. examples for failure cases are shown in figure 13 . examining the final layer output images, the failures look similar: in both cases one or more entity was generated in the wrong color. however, using the diffusion lens, we reveal two different failure patterns: in the first example ("a green cat and a blue rocket"), both the blue rocket and the green cat are generated separately successfully in intermediate layers, while final output image fails to combine them into a single image. this suggests that the failure stems from an unsuccessful combination of the two concepts. in the second example ("a black bird and a green bear"), the bear consistently appears black across all intermediate layers, signifying that the model struggles to generate a green bear throughout the encoding process of the text. a possible explanation is that black bear is a type of animal, which might mean that the phrase "black bear" is common in the training data, thus the appearance of the phrase "black" in the prompt biases the model towards generating black bears. this analysis reveals two different sources of failures that occur in compound prompts: either (1) the model fails at coupling a particular concept and color because it is biased towards another color, or that (2) the model can successfully couple each concept and color but fails to combine them. we analyze how frequently each source of failure occurs, focusing on prompts that failed to generate a correct image from the final layer in at least 75% of cases. for each entity, we count the number of times it appeared in the correct color in at least one early layer. we find that for 40% of the failure cases in df (70% in sd), at least one of the entities did not appear at all in images from earlier layers (type 1). the remaining set of failures had the correct color for each of the objects appear at some point in the computation (type 2). ## related work interpreting language models. a wide range of work has analyzed language model internals. we briefly mention a few directions and refer to existing surveys @xcite @xcite . probing classifier are used to analyze whether internal representations correlate with external properties (e.g., @xcite . however, probing has various inherent flaws such as memorization @xcite , and requires costly annotations for fine-grained analysis like the visual characteristics of a specific animal species or person. interventions in representations measure how they impact a model's prediction (e.g., @xcite @xcite , and while they offer powerful insights, they are also challenging to design @xcite and limited in scope. in contrast, the diffusion lens proposes a simple yet generic mechanism to visually interpret intermediate representations without requiring additional data or training, enabling exploration of fine-grained visual features. another influential approach is the logit lens (nostalgebraist, 2020), which projects intermediate representations of language models onto a probability distribution over the vocabulary space. the logit lens captures the internal computation of the language model, and the flow of information across modules @xcite @xcite . this line of work has focused on auto-regressive decoder language models. inspired by this idea, we propose using the diffusion module in t2i pipelines to visualize intermediate prompt representations, revealing the text encoder's computation process. interpreting vision-language models. compared to unimodal models, research on interpretability in multimodal vision-language models is rather limited. @xcite found multimodal neurons responding to specific concepts in clip @xcite and @xcite decomposed clip's image representations into text-based characteristics. @xcite were the first to propose a method to interpret t2i pipelines, by analyzing the influence of input words on generated images via cross-attention layers. @xcite decomposed textual concepts, focusing on the diffusion component. in contrast, our work investigates the under-explored text encoder in t2i pipelines. unlike previous methods, the diffusion lens reveals gradual processes within the model, not focusing only on the final output. ## discussion and conclusion we introduce the diffusion lens, a novel method to analyze language models within t2i pipelines. our approach deconstructs the t2i pipeline by examining the output of each block within the text encoder, thereby providing a deeper insight into language-to-visual concept translation. we are the first, to our knowledge, to propose a method to interpret the text encoder and its internal computation process in the context of t2i models. given that the text encoder is a crucial component of t2i models, enhancing its interpretability contributes to a deeper understanding of the entire generation process. we showcased the method's potential by analyzing two open-source text encoders used in t2i pipeline across diverse topics. our work contributes to a growing body of work on analyzing how models process information across various components. the diffusion lens may have many potential applications as a first method of its kind, including similar applications to prior interpretability techniques such as improving model efficiency @xcite and tracing factual associations in language models, facilitating more accurate model editing methods @xcite . other future directions using the diffusion lens may aid in identifying points of failure in the computation process or remove undesired traits from early layers such as hallucinations, toxicity, or incorrect factual knowledge. lastly, while we focused on entire blocks, our approach paves the way for visualizing individual sub-block components such as individual mlps, attention heads, and residual connections.