Grounding Meaning Representation for Situated Reasoning (2022)

Abstract: As natural language technology becomes ever-present in everyday life, people will expect artificial agents to understand language use as humans do. Nevertheless, most advanced neural AI systems fail at some types of interactions that are trivial for humans (e.g., ask a smart system “What am I pointing at?”). One critical aspect of human language understanding is situated reasoning, where inferences make reference to the local context, perceptual surroundings, and contextual groundings from the interaction. In this cutting-edge tutorial, we bring to the NLP/CL community a synthesis of multimodal grounding and meaning representation techniques with formal and computational models of embodied reasoning. We will discuss existing approaches to multimodal language grounding and meaning representations, discuss the kind of information each method captures and their relative suitability to situated reasoning tasks, and demon- strate how to construct agents that conduct situated reasoning by embodying a simulated environment. In doing so, these agents also represent their human interlocutor(s) within the simulation, and are represented through their virtual embodiment in the real world, enabling true bidirectional communication with a computer using multiple modalities.

## target and expected audience this tutorial will be of interest to both researchers in meaning representation, and in multimodal nlp and grounding, particularly those interested in both theoretical and data-driven approaches to language grounding and those interested in treating automated reasoning as more than just a pure machine learning problem. the diverse approaches to linguistic grounding of situated meaning have also provoked significant interest from the robotics community. given the increased interest in interactive agents and grounding for robotics at in the *acl community, as witnessed by the recent creation of language grounding to vision, robotics, and beyond tracks at most *acl venues, this tutorial, that synthesizes various approaches to situated conversation and interaction will be a timely way to bring these two communities closer. we expect this tutorial will draw an audience of roughly 30-45. ## breadth and reading list this tutorial draws on a wealth of both theory and applied research in multimodal semantics, includ-ing not only the central meaning representation work mentioned above @xcite @xcite , but also gesture semantics and situated dialogue @xcite @xcite , and qualitative spatiotemporal reasoning @xcite @xcite . we bring these diverse areas together in the modeling language voxml @xcite , and this tutorial will demonstrate how to exploit the strengths of both meaning representations and data-driven multimodal methods to create agents that reason with vision, language, action, and gesture about the environments they inhabit and share with human beings. suggested reading is below: james pustejovsky is the tjx feldberg chair in computer science at brandeis university, where he is also chair of the linguistics program, chair of the computational linguistics m.s. program, and director of the lab for linguistics and computation. he received his b.s. from mit and his ph.d. from umass amherst. he has worked on computational and lexical semantics for 25 years and is chief developer of generative lexicon theory; the tarsqi platform for temporal reasoning in language; timeml and iso-timeml, a recently adopted iso standard for temporal information in language; the recently adopted standard iso-space, a specification for spatial information in language; and the co-creator of the voxml modeling frame-work for linguistic expressions and interactions as multimodal simulations voxml (co-created with n. krishnaswamy), enables real-time communication between humans and computers or robots for joint tasks, utilizing speech, gesture, gaze, and action. he is currently working with robotics researchers in hri to allow the voxml platform to act as both a dialogue management system as well as a simulation environment that reveals realtime epistemic state and perceptual input to a computational agent. email: jamesp@brandeis.edu, website: https://www.pusto.com . ## diversity situated reasoning and grounding inherently crosses language boundaries. language grounding in english can be compared to language grounding a low-resourced language by way of a situated model. from a research perspective these are important questions to answer, to explore how different languages represent the same environment or situation. therefore situated reasoning is an important potential way to broaden the linguistic diversity of nlp, and we hope the meaning representation component of this tutorial may inspire broadening meaning representations to more languages yet. the instructors are junior and senior faculty, respectively, established in the nlp community. we actively recruit women and underrepresented minorities to our respective research groups, and plan to promote this tutorial to an international and diverse audience. we are experienced instructors in a hybrid format, and we will accommodate and promote remote attendance to broaden participation. ## ethics statement computational agents that reason situationally necessarily require sight and hearing, and come with concomitant ethical issues regarding computer vision and speech recognition. in the course of this tutorial, we will discuss many of the considerations surrounding user privacy and storing user data (or, in the case of our own research, explicitly not doing that @xcite ). we will also discuss adapting speech recognition models to user diversity as part of the multimodal grounding section @xcite . real-time, situated reasoning requires smaller, lightweight models. while we use large models where necessary, our use of meaning representations to guide search within multimodal grounding tasks provides a way to accomplish this task with less computational overhead and resource use.