Dealing with Semantic Underspecification in Multimodal NLP (2023)

Abstract: Intelligent systems that aim at mastering language as humans do must deal with its semantic underspecification, namely, the possibility for a linguistic signal to convey only part of the information needed for communication to succeed. Consider the usages of the pronoun they, which can leave the gender and number of its referent(s) underspecified. Semantic underspecification is not a bug but a crucial language feature that boosts its storage and processing efficiency. Indeed, human speakers can quickly and effortlessly integrate semantically-underspecified linguistic signals with a wide range of non-linguistic information, e.g., the multimodal context, social or cultural conventions, and shared knowledge. Standard NLP models have, in principle, no or limited access to such extra information, while multimodal systems grounding language into other modalities, such as vision, are naturally equipped to account for this phenomenon. However, we show that they struggle with it, which could negatively affect their performance and lead to harmful consequences when used for applications. In this position paper, we argue that our community should be aware of semantic underspecification if it aims to develop language technology that can successfully interact with human users. We discuss some applications where mastering it is crucial and outline a few directions toward achieving this goal.

## introduction they put the flowers there. speakers of a language hear sentences like this every day and have no trouble understanding what they mean-and what message they convey. this is because, in a normal state of affairs, they can count on a wide range of information from the surrounding context, personal knowledge and experience, social or cultural conventions, and so on. upon hearing this sentence, for example, they would know that flowers go into vases, look in the direction where their interlocutor nodded their chin, see a vase with tulips on the windowsill, and infer that this is where someone put the flowers. every time listeners need to count on extra, non-linguistic information to understand a linguistic signal, like in this example, it is because the language used is semantically underspecified @xcite @xcite . in the example above, the locative adverb there leaves underspecified a location-where the flowers were put-which would instead be explicitly provided in the semantically more specified sentence they put the flowers in the light blue vase on the windowsill at the end of the hallway. according to linguists, indeed, adverbs of place (here, there) are typical examples of semantically underspecified words, as well as adverbs of time (now, today), demonstratives (this, that), quantifiers (few, many), tensed expressions, and some usages of personal pronouns @xcite . the reason why semantic underspecification is so widespread has to do with language efficiency, which is a trade-off between informativeness and conciseness @xcite . underspecified words can be used in many communicative occasions with varying meanings and intentions @xcite , which prevents speakers from fully articulating every nuance of meaning every time they talk @xcite . indeed, planning and producing utterancesbut also speech (see @xcite -is cognitively expensive @xcite . the use of underspecified language, at a first sight, seems to go against the view that language is a cooperative system @xcite and can indeed explain cases where communication appears to be egocentric rather than cooperative @xcite . however, a wealth of studies has shown that humans are extremely good at making inferences @xcite and that this ability is cognitively cheaper than speaker articulation, which is rather demanding and time-consuming @xcite . upon hearing a semantically underspecified sentence, human speakers can quickly and effortlessly integrate linguistic and non-linguistic information @xcite . in this light, @xcite proposed that semantic underspecification gives rise to processing efficiency besides boosting storage efficiency. semantic underspecification allows our limited repertoire of symbols to be used in many contexts and with different intentions without compromising its communicative effectiveness. for example, we can use the pronoun they to omit a person's gender or refer to a group of friends; the locative here to refer to a free table at a café or the institution you work for. semantic underspecification is not a bug but a crucial feature of language that is ubiquitous in human communication @xcite . as such, any intelligent system that aims at mastering language as humans do must be able to properly deal with it. this is particularly the case for models of natural language understanding and generation that have access to non-linguistic information @xcite , e.g., models integrating language and vision that account for the multimodality of language @xcite . these models must be able to understand and generate sentences like they put the flowers there, provided that a relevant visual context is present and there is a clear communicative goal. this is a mandatory requirement if we want to use these systems to model real communicative scenarios or embed them in applications that interact with human users. in this position paper, we argue that semantic underspecification should be high on the nlp community agenda, particularly within approaches combining language and vision. we report that sota multimodal nlp models struggle with it, and advocate a comprehensive, thorough investigation of the phenomenon along several research directions and concrete steps. mastering semantic underspecification is a long-term goal that implies shifting the paradigm to a scenario where models use language as humans do, that is, with a communicative goal. in line with what was argued elsewhere @xcite @xcite , we believe the time is ripe for such a change. ## how do multimodal models deal with semantic underspecification? the field of multimodal or visually grounded nlp is currently dominated by pre-trained multimodal transformers. since their introduction, models like clip @xcite , lxmert @xcite , visualbert @xcite , vil-bert @xcite , vl-bert @xcite , unit @xcite , villa @xcite , uniter @xcite , vinvl @xcite , vilt @xcite , and mplug @xcite , inter alia, have rapidly become the new state-of-the-art in virtually every language and vision task. among other tasks, these models achieve unprecedented performance in describing an image in natural language @xcite , finding the best image for a given language query @xcite , answering fine-grained questions about the content of an image @xcite @xcite , reasoning over objects and object relations @xcite , and entertaining a visuallygrounded dialogue by asking and answering questions @xcite . these models differ from each other in several dimensions. for example, they either concatenate and jointly process the visual and textual embeddings (single-stream models), or process the two modalities by means of separate encoders with an optional cross-modal fusion (dualstream models); or, they use visual features extracted with either cnn-based (e.g., region features from faster r-cnn; @xcite or transformer-based (e.g., image features from vision transformer, vit; dosovitskiy et al., 2020) image encoders. however, they share both the same underlying architecture, which is based on transformers, and training regime, which leverages a massive amount of multimodal data and a few common learning objectives. one of the most popular learning objectives is image-text matching (itm), which maximizes the similarity between an image and a language fragment that is well aligned with it. as a result of this training regime, these models are impressively good at judging whether a sentence is a good/bad (true/false) description of the content of an image. this is particularly the case for clip, which is optimized for the task and can almost perfectly spot word-level inconsistencies between an image and a sentence, as the ones included in the foil dataset by @xcite (results reported in @xcite . given this impressive performance, it is reasonable to expect that these models are robust to se-mantically underspecified language. describing an image, asking a question, or entertaining a conversation about it are all communicative scenarios that admit a varying degree of semantic underspecification. for example, the question what are they doing? referred to a visual context with people playing an unusual sport is perfectly acceptableand indeed likely to be asked; or, the sentence a person is typing on their laptop to describe an office environment is not only a very good description of that context but perhaps even a desirable one. therefore, mastering semantically underspecified language is a requisite for any multimodal nlp model which aims at both genuinely solving these tasks and being used for user-facing applications. ## communicative approaches to multimodal tasks mastering semantic underspecification is relevant to a wide range of studies that take a communicative or pragmatic approach to multimodal tasks. below, we focus on a select sample of them foot_1 and discuss how they might benefit from a full mastery of the phenomenon investigated in the paper. image captioning with a communicative goal standard image captioning 3 consists in generating a description that is as close as possible to the content of the image. typically, the task is not tied to a real communicative goal: image descriptions are provided by crowdworkers who are asked to mention all the important aspects of an image @xcite @xcite , 4 and models are trained and evaluated to closely approximate those descriptions @xcite . to make the task more pragmatically valid, some work proposed a discriminative version of it where models need to generate a description for an image that is pragmatically informative, i.e., that is good for the image in the context of other distractor images @xcite @xcite . the recent concadia dataset @xcite , in contrast, considers images in isolation and focuses on the communication needs of image captioning. in particular, it distinguishes between descriptions, useful to describe an image to someone who does not have access to the image, and captions, that instead complement the information of an image that is available to both interlocutors. 5 within both lines of research, it is to be expected that underspecified language comes into play. for example, neither the gender nor the number of people in an image may be needed to pragmatically distinguish it from other images; or, a caption complementing an image (and an accompanying text) may leave underspecified much of the information that one can get from elsewhere. 6 as such, these tasks would greatly benefit from having models opportunely dealing with this language phenomenon. in support of this-and in line with the results of our pocs above-recent work @xcite showed that sota models like clip are unable to account for the degree of usefulness of an image description, but only for its alignment. more generally, focusing on semantic underspecification of visually grounded language would be relevant to studies investigating the range of relations that texts entertain with images, including communicative goals and information needs @xcite . moreover, it would inform a similar argument in @xcite . 4 one exception is represented by vizwizcap @xcite , a dataset of image descriptions collected with the explicit aim to be informative to visually impaired people. the specular task of image-to-text generation, as recently claimed by @xcite . @xcite have been collected by asking crowdworkers to provide questions and answers for research purposes. in contrast, the vizwiz dataset @xcite includes questions that were asked by visually-impaired people to obtain information about visual scenes. as such, these questions are motivated by a real communicative goal and exhibit very different linguistic features compared to the questions and answers in standard datasets. for example, the questions are more ambiguous or underspecified, and the answers by the respondents are more diverse and subjective @xcite @xcite . we propose that models that are equipped for semantically underspecified language should both better understand the question in relation to an image (something that current sota models struggle with, see @xcite and better leverage the diversity and sparseness of the answers. ## conclusion in this position paper, we argued that the nlp community must deal with semantic underspecification, that is, the possibility for a linguistic signal to convey only part of the information needed to understand a message. this is a ubiquitous phenomenon in human communication, that speakers deal with by quickly and effortlessly integrating non-linguistic information, e.g., from the surrounding visual context. we argued that research in multimodal nlp combining language and vision is 7 these directions may also be relevant to the line of work exploring how to minimize biases and misrepresentations when describing images (e.g., @xcite . on a technical level, our paper highlights the need to improve sota models by making them robust to scenarios that may be different from those seen in training. in our case, clip suffers with sentences that resemble the language used in real communicative contexts, which poses a problem if we were to use it for modeling communicative tasks or embed it in user-facing applications. this general weakness of sota models has been recently illustrated by @xcite . using winoground, a dataset of carefully designed ⟨image, description⟩ pairs testing compositionality abilities, the authors reported chance-level performance for all the transformer-based multimodal models they tested-including clip. a careful analysis of the samples by @xcite revealed that the difficulties of the dataset go beyond dealing with compositionality, and include ambiguity aspects, reasoning abilities, and so on. in any case, these findings are informative of the flaws of the models and provide useful indications on which directions to take for improving them. on a theoretical level, the ideas presented in our paper are consonant with a recent line of thought that advocates approaches that are aware of communicative and pragmatic aspects in language understanding and generation @xcite @xcite . we believe this is an exciting direction, and support a collaborative effort aimed at developing systems that can use language with a communicative goal.