Embodied Multimodal Agents to Bridge the Understanding Gap (2021)

Abstract: In this paper we argue that embodied multimodal agents, i.e., avatars, can play an important role in moving natural language processing toward “deep understanding.” Fully-featured interactive agents, model encounters between two “people,” but a language-only agent has little environmental and situational awareness. Multimodal agents bring new opportunities for interpreting visuals, locational information, gestures, etc., which are more axes along which to communicate. We propose that multimodal agents, by facilitating an embodied form of human-computer interaction, provide additional structure that can be used to train models that move NLP systems closer to genuine “understanding” of grounded language, and we discuss ongoing studies using existing systems.

## introduction as of the 2020s, high-profile nlp successes are being driven by large (and ever-growing) deep neural language models 1 . these models perform impressively according to common metrics on a variety of tasks such as text generation, question answering, summarization, machine translation, and more. however, deep examinations of large language models show that they may exploit artifacts or heuristics in the data @xcite . similar observations have been made about computer vision systems @xcite . the black-box nature of much nlp leads to the argument that they are insufficient at demonstrating understanding of communicative intent and also cannot explain why or when these failures occur. (bender and koller, 2020). 1 as of writing, the largest language model is the switch transformer by google @xcite , a trillionparameter language model beating previous record-holder @xcite the neurosymbolic approach to ai has long argued for structured representation @xcite , and similar arguments have also been made by deep learning luminaries e.g., @xcite . introducing preexisting structure into nlp facilitates higher-order reasoning, but a hybrid approach also does so at larger scales than purely symbolic systems, by using flexible deep-learning representations as input channels. multimodal nlp systems, particularly those with embodied agents, model encounters between two "people." a fully-featured interaction requires an encoding of structures that make language interpretable and "deeply understandable." this makes embodied multimodal interaction a uniquely useful tool to examine what nlp models in use meaningfully learn and "understand.". if one mode of expression (e.g., gesture) is insufficiently communicative, another (e.g., language) can be used to examine where it went wrong. each additional modality provides an avenue through which to validate models of other modalities. in this paper we review common components of embodied multimodal agent systems and their contributions to "deep understanding." we present ongoing experiments in agents who exhibit grounded understanding, using complex multimodal referring expressions as an example task. ## multimodal conversational systems as ai systems become more integrated with everyday life, users at times harbor misapprehensions that they behave like humans. this dates as far back as eliza @xcite , whose users were convinced that eliza "understood" them despite weizenbaum's insistence otherwise. the well-known shrdlu displayed some situational understanding, but shrdlu's deterministic perception was not sensor-driven @xcite . @xcite and @xcite , among many others, define multimodal interaction as interactions with multiple distinct input channels, e.g., spoken language and gestures, that complement and reinforce each other. this relates to "communicative intent" in that if one channel is ambiguous, the other(s) may clarify or add to the meaning. multimodal interactions may increase understanding by making it easier to retrieve a communicative intent from a combination of inputs. this makes it critical in computer systems that display understanding. bolt's "put-that-there" (1980) anticipated some critical issues, including the use of deixis for disambiguation. the subsequent community that evolved around multimodal integration (e.g., @xcite al. (2013), turk (2014) ) gave rise to a number of embodied conversation agents (ecas) (e.g., @xcite ) deployed in different task domains, including education @xcite , negotiations @xcite , and medical practitioner evaluation @xcite . humans communicate contextually; they may gesture and refer to items in their co-perceived space. this co-perception and co-attention is "central to determining the meanings of communicative acts [...] in shared events" @xcite . a communicative act, c a , can be modeled as a tuple of expressions from the modalities available to the agent, which convey complementary or redundant information. for example, if the modalities involved are speech, gesture, f acial expression, gaze, and action, then c @xmath0, g, f, z, a , of which any may be null or empty. information in one channel may be supplemented by another channel, and they may disambiguate each other if properly aligned. for instance, if interpreted from the human's point of view, @xcite , @xcite ), this may signal a difference in the agents' relative frames of reference. this effect of embodiment is critical for deep understanding of situated language, vis-à-vis something as basic as directional terms. an embodied agent can demonstrate aspects of their worldview, including how they ground and interpret utterances, gestures, or the consequences of actions, by acting themselves in their worlds. "embodied worlds" may be virtual, physical, or mixed-reality. in the remainder of this paper we focus on virtual worlds. to align various modalities in the state space accessed by the agent's dialogue system, we assume a "common ground" structure associated with a dialogue state. this state monad @xcite : @xmath1, corresponds to those computations that read and modify the state. m is a type constructor that consumes a state and returns a pair: value, modif ied state . a system operating under the aforementioned assumptions is "diana" @xcite . diana's semantic knowledge of objects and actions is based on the voxml modeling language (pustejovsky and @xcite , and she asynchronously interprets spoken language and gesture to collaborate on construction tasks with humans. the human instructs diana by deictically or linguistically referencing objects and indicating what to do with them multimodally. diana's responses and actions make clear whether she understands what the user intended or not: a clear extraction of communicative intent from utterance, and a measure of "deep understanding." however, diana's capabilities are not fully symmetrical. the human may speak verbosely, but diana's responses are brief: "ok," or the occasional disambiguatory question. to increase diana's capacity for deep understanding, and symmetric retrieval of intent from utterance on the part of both diana and the human, we are conducting experiments to fully develop the capabilities afforded by diana's multimodal embodiment. these are outlined subsequently. ## ongoing experiments referring expressions (res) provide a defined and evaluable case study in the ability of an nlp system to extract communicative intent from an utterance. either the agent correctly retrieves the object the human referenced or it does not; either the human can correct misunderstanding or they cannot. referring expressions exploit information about both object characteristics and locations. linguistic referencing strategies can use high-level abstractions to distinguish an object in a given location from similar ones elsewhere, yet the location described may still be difficult to ground or interpret. when interacting person-to-person, humans deploy a variety of strategies to clearly indicate objects and locations in a discourse, including mixing and matching modalities on the fly and switching strate-gies to optimize both economy and clarity. therefore, in an ongoing study to replicate this same capacity in interactive agents, we are exploring how to generate symmetrically-descriptive references of distinct objects in context. krishnaswamy and pustejovsky (2019) presented the emre (embodied multimodal referring expressions) dataset. it contains 1,500 videos of a version of the diana agent referencing multiple objects in sequence with a mix of language and gestures. the authors found that human evaluators found multimodal referring strategies most natural and preferred more descriptive language. however, they also found that the data gathered was not enough to train an effective generation model, so we build on this strategy to augment the available data to a sufficient level. ## conclusion clark (1996) casts language as a joint activity. we propose that introducing the interactive element into natural language technology enables a number of opportunities to build systems that go beyond processing language to understanding it. we focus our experiments on embodied systems because they create a conceit of interacting with another "person." if these systems can interact in a way that suggests understanding of their interlocutor's intents, they will have demonstrated a step toward true computational natural language understand-ing. we are pursuing experiments in multimodal referring expression generation as an illustrative use case.