Repairing Catastrophic-Neglect in Text-to-Image Diffusion Models via Attention-Guided Feature Enhancement (2024)

Abstract: Text-to-Image Diffusion Models (T2I DMs) have garnered significant attention for their ability to generate high-quality images from textual descriptions.However, these models often produce images that do not fully align with the input prompts, resulting in semantic inconsistencies.The most prominent issue among these semantic inconsistencies is catastrophic-neglect, where the images generated by T2I DMs miss key objects mentioned in the prompt.We first conduct an empirical study on this issue, exploring the prevalence of catastrophic-neglect, potential mitigation strategies with feature enhancement, and the insights gained.Guided by the empirical findings, we propose an automated repair approach named Patcher to address catastrophic-neglect in T2I DMs.Specifically, Patcher first determines whether there are any neglected objects in the prompt, and then applies attention-guided feature enhancement to these neglected objects, resulting in a repaired prompt.Experimental results on three versions of Stable Diffusion demonstrate that Patcher effectively repairs the issue of catastrophic-neglect, achieving 10.1%-16.3% higher Correct Rate in image generation compared to baselines.

## introduction text-to-image diffusion models (t2i dms) @xcite @xcite have gained widespread attention in recent years due to their remarkable ability to generate images from textual descriptions (i.e. prompt). however, it has been demonstrated that the image generated by t2i dms may not strictly adhere to the description of the input prompt, leading to inconsistencies in the semantics. to this end, many approaches have been proposed to enhance the generation quality through inference process optimization @xcite figure 1 : examples of catastrophic neglect in the generated images by t2i dms, and the enhancement of explicit and implicit features. @xcite and handcrafted prompt writing guidelines @xcite . the former requires modifications to the model structure or parameters, which is difficult for users to perform. although the latter is relatively easier to implement, it requires a significant amount of manual effort and suffers poor scalability. recently, @xcite also proposed a method to enhance the quality of generated images by automating the refinement of user-inputted prompts. according to previous study @xcite , one of the most prominent issues in semantic consistency is the catastrophic-neglect, i.e., the images generated by t2i dms often miss some of the key objects mentioned in the textual prompts. this issue is particularly prevalent when a prompt involves multiple objects. figure 1 demonstrates two illustrative cases where one of the two objects is neglected by t2i dms. in figure 1 (a), we notice that the object "bicycle" in prompt is described with the explicit feature "two-wheeled" while "donut" is not. we try to craft prompts to repair the issue, and results reveal that by adding a specific explicit feature to the "donut" (e.g., "hollow-centered"), the catastrophic-neglect issue can be resolved. furthermore, as the feature is added, the attention difference between the two mentioned objects (i.e., "bicycle" and "donut") is reduced according to the explainable tool @xcite . it seems that reduction in attention difference can potentially indicate the t2i dms put more balanced attention towards the two involved objects, resulting both of them can be successfully generated. in figure 1 (b), we notice that the object "bird" in the prompt is a more general concept with fewer implicit features compared with the concept "giraffe", according to the hierarchical structure in wordnet @xcite . taken in this sense, we can successfully repair the issue through using more imageable concept (such as "eagle" ) to replace "bird" in the prompt, and the attention difference between two mentioned objects (i.e., "eagle" and "giraffe") is also reduced. motivational study in section 2 provides more details. motivated by the above analysis, we assume the attention difference can guide the mitigation of catastrophic-neglect issue, and this can be achieved through enhancing objects with specific features (i.e., explicit features) or using more imageable concepts (i.e., implicit features) to balance the attention among involved objects in the prompt. therefore, this paper proposes an automatic repair approach named patcher to address catastrophic-neglect in t2i dms, guided by the attention difference among objects of input prompt. specifically, patcher first parses the original prompt and identifies the objects neglected by the t2i dms. then, guided by the difference of attention scores, patcher produces the repaired prompt via enhancing explicit feature (achieved by asking llms for suitable modifiers) and implicit features (realized by hyponym substitution using wordnet), and redetermined whether there are still neglected objects in the generated image. experimental results demonstrate that patcher effectively repairs the issue of catastrophic-neglect in t2i dms, achieving 10.1%-16.3% higher correct rate in image generation compared to baselines, as tested on stable-diffusion v1.4, v1.5, and v2.1 models. additionally, ablation study shows that both explicit and implicit feature enhancing in patcher contribute to resolving the catastrophicneglect issue in t2i dms. we provide the public reproduction package1. ## motivation to better understand catastrophic-neglect and guide the design of the automated repair approach, we conduct the empirical analysis from three aspects, i.e., their prevalence across prompts with different number of objects, potential mitigation strategies based on feature enhancement, and corresponding insights into the effectiveness of feature enhancement. ## methodology figure 2 shows the overview of patcher. patcher consists of two stages: (1) neglected objects identification would determine whether the t2i dm neglect any objects in the input prompt; (2) feature enhancement for neglected objects would enhance explicit and implicit features for neglected objects and construct the repaired prompt. ## results we designed two sets of experiments to explore the performance of patcher in repairing catastrophicneglect: the effectiveness of patcher and the ablation study within patcher. ## conclusion this paper proposes an approach (patcher) to repair catastrophic-neglect in text-to-image diffusion models by attention-guided features enhancement of neglected objects in the generated images. patcher first inputs the prompt into a t2i dm and an attention explainability tool to obtain the generated image and the attention scores for each token. it then checks whether all objects in the prompt appear in the generated image based on the text-image similarity. if any objects are neglected, patcher iteratively searches for suitable explicit and implicit features to enhance the neglected objects based on the attention differences between the objects. experimental results demonstrate that patcher effectively addresses the issue of catastrophic-neglect in t2i dms, achieving a 10.1%-16.3% higher correct rate based on manual annotation compared to baselines, as tested on stable-diffusion v1.4, v1.5, and v2.1 models. additionally, ablation experiments show that both explicit feature enhancing and implicit feature enhancing in patcher contribute to resolving the issue of catastrophic-neglect in t2i dms.