Resolving Ambiguities in Text-to-Image Generative Models (2023)

Abstract: Natural language often contains ambiguities that can lead to misinterpretation and miscommunication. While humans can handle ambiguities effectively by asking clarifying questions and/or relying on contextual cues and common-sense knowledge, resolving ambiguities can be notoriously hard for machines. In this work, we study ambiguities that arise in text-to-image generative models. We curate the Text-to-image Ambiguity Benchmark (TAB) dataset to study different types of ambiguities in text-to-image generative models. We then propose the Text-to-ImagE Disambiguation (TIED) framework to disambiguate the prompts given to the text-to-image generative models by soliciting clarifications from the end user. Through automatic and human evaluations, we show the effectiveness of our framework in generating more faithful images aligned with end user intention in the presence of ambiguities.

## introduction natural conversations contain inherent ambiguities due to potentially multiple interpretations of the same utterance. different types of ambiguities can be attributed to syntax (e.g., "the girl looks at the boy holding a green bag" -is the girl holding the green bag?), semantics (e.g., "a picture of cricket" -is cricket referring to an insect or a game?), and underspecification (e.g., "doctor talking to a nurse" -is the doctor/nurse male or female?). ambiguities pose an important challenge for many natural language understanding tasks and have been studied extensively in the context of machine translation @xcite , conversational question answering @xcite , and task-oriented dialogue systems @xcite , among others. in this paper, we study the effect of ambiguity in text-to-image generative models @xcite prompt an elephant and a bird flying. ## text-to-image ambiguity benchmark (tab) our text-to-image ambiguity benchmark (tab) is a modified version of the lava corpus @xcite . the original lava corpus contains various types of ambiguous sentences that can only be resolved through visual signals from their corresponding images/videos. we use the ambiguous ambiguity type count main types syntax prepositional phrase (pp) 74            linguistic syntax verb phrase (vp) 243 syntax conjunction 127 discourse anaphora 21 discourse ellipsis 45 fairness 355 fairness add complex+combination+misc 335 on a high level, tab covers six main types of prompt ambiguities, including fairness and linguistic type ambiguities. we add some additional complex cases on top of the six main types of prompt ambiguities. in these complex cases, we take a sample of prompts from tab and manually make structurally more complex version of each sentence. this process is done in a way such that the ambiguities and meaning of the constituent sentences are kept intact, but the structure of a sentence is made more complex through addition of more information, extra words, adverbs, and adjectives. for instance, "the girl waves at the old man and woman." representing an example for syntax conjunction type ambiguity can be turned into "the girl gracefully waves at the old man and woman to show respect and gratitude." with a more complex sentence structure. we also add some additional miscellaneous cases, which are not covered by six main types of ambiguities. in addition, we add combination cases where we combine fairness and linguistic type ambiguities and make new variations from our existing prompts. each of the 1,200 ambiguous prompts in tab are accompanied with possible visual setups. we also curate questions that can be surfaced to the end user to clarify the visual setup they have in mind amongst the set of visual setups available for a given ambiguous prompt. the objective of tied framework is to either generate different visual setups or clarifying questions to the end user to disambiguate the prompts through end user interaction. therefore, we use these questions and visual setups in tab as ground truth to evaluate the tied framework (details in section 4). each of the elements (e.g., visual setups) in tab are generated by an expert annotator and are cross-checked by two additional annotators to ensure that they are sensible. additional detailed statistics about tab can be found in ## text-to-image disambiguation (tied) framework given an ambiguous prompt, our text-to-image disambiguation (tied) framework uses a language model (lm) to obtain disambiguation feedback from the end user through an interactive system. our goal in tied is to use in-context learning to seek user feedback that can help us disambiguate the prompts. after obtaining the disambiguation feedback, tied concatenates the obtained signals to the original ambiguous prompts and creates final disambiguated prompts as shown in figure 2 . tied then uses the final disambiguated prompts to generate images using text-to-image models. we test two resolution approaches in tied: 1) question answering-tied ("qa-tied") which resolves ambiguities by language models gener-ating questions and seeking answers to disambiguate prompts; 2) visual setup-tied ("vs-tied") which resolves ambiguities by language models directly generating different possible visual setups (textual descriptions of different visual scenarios/interpretations) given an ambiguous prompt and seeking signals in the form of a visual setup being chosen. the overall tied framework along with qa-tied and vs-tied ambiguity resolution approaches are shown in figure 2 . ## experiments we evaluate tied on two complementary aspects: (i) whether language models incorporated in tied generate appropriate clarifying questions (in the case of qa-tied) or textual descriptions of visual setups (in the case of vs-tied), resulting in visual resolution; (ii) whether modified prompts generated through tied result in faithful image generation aligned with end-user intention. we discuss respective experiments for both of these evaluations below. in our experiments, we use three language models: gpt-2 @xcite , gptneo @xcite , and opt @xcite . in addition, we use openai's dall-e @xcite and dall-e mega @xcite models as our text-to-image models to generate images in our experiments. ## related work resolving ambiguities in different nlp applications has been a prominent research direction due to its importance. for instance, word sense disambiguation is one of the areas in nlp that has gained significant attention @xcite . resolving ambiguities in question answering @xcite , conversational question answering @xcite , and task-oriented dialogue systems @xcite has also been previously studied. ambiguity resolution has also been studied in multi-modal applications, such as multi-modal machine translation @xcite or matching images or videos to disambiguated interpretation of a sentence @xcite . despite those recent efforts, not much attention has been paid to ambiguities in text-to-image generative models. on the other hand, the growing popularity of those models, both in academic and non-academic circles, make it imperatives to better understand potential issues with those systems due to language ambiguity @xcite . ## conclusion in this work, we study the role of prompt ambiguity in text-to-image generative models and propose a disambiguation framework to generate more faithful images better aligned with user intention. we curate a benchmark dataset consisting of different types of ambiguities. we measure the ability of various language models in obtaining disambigua-tion signals through end user interaction by either generating clarifying questions or visual setups. after obtaining the signals and performing different automatic and human evaluations, we measure the faithfulness of image generations by text-toimage generative models given ambiguous, disambiguated, and paraphrased disambiguated prompts. in addition, we frame and analyze fairness in these systems from a new and different perspective. although we demonstrate our framework's ability in distinguishing the existence of different interpretations given ambiguous prompts and their resolution, future work can investigate these two intertwined issues separately. in this work, our focus was on ambiguity resolution. future work can focus on ambiguity detection in this domain.