Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches (2023)

Abstract: People rely heavily on context to enrich meaning beyond what is literally said, enabling concise but effective communication. To interact successfully and naturally with people, user-facing artificial intelligence systems will require similar skills in pragmatics: relying on various types of context — from shared linguistic goals and conventions, to the visual and embodied world — to use language effectively. We survey existing grounded settings and pragmatic modeling approaches and analyze how the task goals, environmental contexts, and communicative affordances in each work enrich linguistic meaning. We present recommendations for future grounded task design to naturally elicit pragmatic phenomena, and suggest directions that focus on a broader range of communicative contexts and affordances.

## using language in context people use language to achieve goals @xcite @xcite , producing effects on other people and the world. to achieve their goals efficiently, people often only sketch their intended meanings: relying on various types of context to allow their conversational partners to enrich meaning beyond what the speaker has literally said. for this reason, language is highly context-dependent; the meaning of even a simple sentence such as "it's nice out today" depends on the situation-it can be an implicit invitation, a statement contrasting the weather with a previous day, or even ironic if the weather is poor. this broad ability to use language in context to achieve goals is known as pragmatics. as large language models (llms) and other nlp systems become increasingly integrated into our world, they will require similar pragmatic skills to use language to interact successfully and efficiently with people in context. recent work we believe the time is right to focus on evaluating models on tasks (some existing, but mostly to-be-developed) that require contextual communication abilities: ones that elicit pragmatic phenomena from people, and benefit from pragmatic abilities in systems. these tasks should give a proxy for model performance in grounded interactions with real people, while still facilitating comparing methods and benchmarking progress. in this paper, we center our discussion on the role of grounded and multimodal context in pragmatics, motivated by the rich contexts of typical language use-our understanding of language is shaped by the environments that we use it in @xcite @xcite @xcite . we first survey how various communicative and environmental context types elicit pragmatic phenomena. using these context types and phenomena, we then survey representative grounded tasks and datasets which have been used both to study pragmatic communication in people, and to build goaloriented multimodal systems. we present tasks along a spectrum of complexity, ranging from constrained reference games to goal-oriented embodied dialogue. we discuss how choices in grounded task design-including environment properties, context types, and communicative affordancesshape the pragmatic phenomena that arise in tasks, and provide suggestions for future task and dataset designers. to model these tasks and phenomena, we give an overview of a range of computational pragmatic approaches that view communication as goal-directed actions by agents in context. finally, we suggest further integrations of computational pragmatics with nlp: developing tasks with rich contexts; contextualizing existing tasks; using pragmatic and contextual modeling to allow systems to communicate and interact more successfully and efficiently with people; and tackling challenges of human evaluation and data sparsity. ## pragmatic phenomena in linguistics and cognitive science, pragmatics is often defined in contrast to semantics. broadly speaking, semantics characterises the literal meanings of linguistic expressions, whereas pragmatics captures the context-dependent components of meaning, which may contain the bulk of actual communication @xcite . pragmatic communication draws upon many different sources of information, ranging from environmental factors to inferences about other agents' unspoken information and goals. this makes pragmatics both a critical and challenging component for designing nlp systems that interact with people. in this section, we discuss the types of context in which language can be situated and the non-literal inferences that arise as a result of these contextual pressures. ## existing tasks and environments in this section, we critically evaluate several wellstudied grounded language tasks through the lens of the pragmatic phenomena outlined above. we focus on tasks in multimodal domains that make use of natural language data. foot_0 ## modeling pragmatics in this section, we discuss frameworks that have been proposed to characterize how listeners can derive pragmatic meaning, providing a starting point for modeling the phenomena and tasks above.