Underspecification in Scene Description-to-Depiction Tasks (2022)

Abstract: Questions regarding implicitness, ambiguity and underspecification are crucial for understanding the task validity and ethical concerns of multimodal image+text systems, yet have received little attention to date. This position paper maps out a conceptual framework to address this gap, focusing on systems which generate images depicting scenes from scene descriptions. In doing so, we account for how texts and images convey meaning differently. We outline a set of core challenges concerning textual and visual ambiguity, as well as risks that may be amplified by ambiguous and underspecified elements. We propose and discuss strategies for addressing these challenges, including generating visually ambiguous images, and generating a set of diverse images.

## introduction the classic grounding problem in ai asks how is it that language can be interpreted as referring to things in the world? it has been argued that demonstrating natural language understanding requires mapping text to something that is non-text and that functions as a model of meaning (e.g., @xcite . in this view, multimodal models that relate images and language have an important role in pursuing contextualized language understanding. indeed, joint modeling of linguistic and visual signals has been argued to play a critical role in progress towards this ultimate goal, as precursors to modeling relationships between language and the social and physical worlds @xcite . recent text-to-image generation systems have demonstrated impressive capabilities @xcite @xcite @xcite @xcite @xcite . these employ deep learning methods such as generative adversarial networks @xcite , neural discrete representation learning figure 1 : generated depictions of the scene "a robot and its pet in a tree." many elements are underspecified in the text, e.g., pet type, perspective, and visual style. @xcite combined with autoregressive models @xcite , and diffusion models @xcite , trained on large datasets of images and aligned texts @xcite . with such developments in multimodal modeling and further aspirations towards contextualized language understanding, it is import to better understand both task validity and construct validity in text-to-image systems @xcite . ethical questions concerning bias, safety and misinformation are increasingly recognized @xcite ; nevertheless, understanding which system behaviors are desirable requires a vocabulary and framework for understanding the diverse and quickly expanding capabilities of these systems. this position paper addresses these issues by focusing on classic problems (in both linguistic theory and nlp) of ambiguity and underspecification (e.g., @xcite @xcite . little previous work has looked into how underspecification impacts multimodal systems, or what challenges and risks they pose. this position paper presents a model of task formulation in text-to-image tasks by considering the relationships between images and texts. we use this foundation to identify challenges and risks when generating images of scenes from text descriptions, and discuss possible mitigations and strategies for addressing them. 1173 2 background ## task formulation underspecification in task formulation is a major challenge for machine learning and artificial intelligence disciplines as a whole (d' @xcite . clarity around task formulation helps system designers navigate ambiguous inputs; for example, given a prompt such as "a painting of a horse", should the system create an image whose style resembles a painting, or an image of a scene containing a painting, including the frame and other plausible contextual details? this paper postulates that accounts of image meaning and text-image relationships are of central relevance to formulating task definitions in text-to-image systems generally. such accounts are thus important for characterizing underspecification in such systems. we take the notion of world to be important too, for two reasons. like texts, images can reference objects in the world, and in doing so are humanmediated representations of the observable world that involve selection and filtering processes. also, the notion of possible worlds has played an important role in theories of semantics (e.g., @xcite ). therefore, two questions that we believe should be central to an account of underspecification in text-to-image tasks are: we do not attempt here to unify or rebut the many theories of image meanings and text-image relationships, but instead highlight what we see as essential considerations for scene depiction tasks: ## challenges in description to depiction having laid out the relevant considerations of meaning and reference in text-to-image systems in §3, we now focus specifically on systems that produce an image depiction of a scene from a description of that scene. we distinguish challenges from three sources: linguistic ambiguity in descriptions, underspecification in descriptions, and underspecification of desired depictions. our use of the term underspecification here reflects how it has been used in nlp literature, referring both to ambiguity in the objects of study (e.g., linguistic forms (bender and lascarides, 2019, p. 29)), as well as to properties of the technical apparatus used to model meaning (e.g., bender and lascarides 2019, p. 30). ## risks and concerns some datasets used for training multimodal systems have previously been shown to contain biases, stereotypes and pornography @xcite . we now discuss potential concerns in applications employing scene description-to-generation tasks, including how underspecification challenges can exacerbate them. bias: as in image-to-text @xcite , there are risks of text-to-image amplifying societal biases including those concerning gender, race, and disability. since english-language texts do not grammatically require specification of gender identities of people mentioned in a scene, there is a great potential for systems to reproduce existing societal biases. for example, the prompt "a boss addressing workers" might produce an image of a boss with masculine phenotypes. similar outcomes are likely to be obtained with respect to other social roles, social groups and stereotypical phenotypes. cultural biases are expected to be prevalent in any text-to-image systems, since what events and artefacts look like vary wildly around the world-e.g., weddings, bank notes, places of worship, break-fast dishes, etc. when a prompt is ambiguous or underspecified, an ml model is likely to revert to correlations in its training data for deciding details about objects and their appearances. thus underspecification leads to a greater risk of stereotyping biases, which can cause offense and representational harm especially to marginalized groups with a history of being stereotyped. see figure 5a . harmful, taboo and offensive content: images depicting violent scenes may have a greater impact on the viewer than corresponding text descriptions. similarly, pornographic images can be more shocking or culturally taboo than texts. some societies, such as indigenous australian ones, may have taboos on visual depictions of the recently deceased @xcite . this exemplifies potential dangers of non-taboo inputs (permissible referring expressions) producing taboo outputs. attempts to predict image offensiveness within the context of an input text are likely to encounter challenges when inputs are underspecified. see figure 5b . ## conclusion we have motivated greater consideration of task formulation and underspecification in text-to-image tasks. we laid out the conceptual elements required for this, including greater clarity around the formulation of the space of tasks, as well as consideration of how texts and images each convey concepts. echoing van @xcite , our goal in connecting state-of-the art technologies to theories of cultural and social studies is both to promote deeper understanding of these technologies, and also to foster dialogue across disciplines. we outlined some of the primary challenges concerning textual and visual specification and proposed that systems should consider both reproducing visually the vagueness and ambiguities of the input and producing a diversity of images which convey the breadth of text interpretations. we encourage more work on measuring visual objectives discussed in cultural fields-such as clarity, aesthetics, etc.and on task-specific utility of generated images (cf. @xcite . limitations any position paper at least somewhat reflects the backgrounds and standpoints of its authors. the authors have backgrounds in nlp, computational social science, and ai ethics. although we call for greater engagement with creative disciplines, we do not represent those disciplines. although we raise culturally sensitive questions, we have first-hand lived experiences in only australia, india, the uk and the usa.