A Model of Information State in Situated Multimodal Dialogue (2025)

Abstract: In a successful dialogue, participants come to a mutual understanding of the content being communicated through a process called conversational grounding. This can occur through language, and also via other communicative modalities like gesture. Other kinds of actions also give information as to what has been understood from the dialogue. Moreover, achieving common ground not only involves establishing agreement on a set of facts about discourse referents, but also agreeing on what those entities refer to in the outside world, i.e., situated grounding. We use examples from a corpus of multimodal interaction in a task-based setting, annotated with Abstract Meaning Representation (AMR), to explore how speech, gesture, and action contribute to the construction of common ground. Using a simple model of information state, we discuss ways in which existing annotation schemes facilitate this analysis, as well as information that current annotations do not yet capture. Our research sheds light on the interplay between language, gesture, and action in multimodal communication.

## introduction in dialogue, the concept of common ground refers to the set of presuppositions held by the participants, propositions that they agree to treat as true @xcite . the process by which common ground is constructed over a dialogue is known as (conversational) grounding @xcite . formal models of dialogue have been developed to track how common ground (and more generally, information state) evolves over the course of an interaction @xcite @xcite . much work examining the role of non-linguistic modalities in communication focuses on gesture @xcite @xcite figure 1 : example of multimodal communication in a task-based setting @xcite . on the left, the signaler describes part of the structure to be built: he says, "it starts in the top left; there's a block", and makes a deictic gesture with his left hand. on the right, the actor puts a block in the top left corner of the table (note that both videos are mirrored). @xcite . this includes analyses of the semantic contents of gestures @xcite , and proposals for integrating gesture into models of dialogue @xcite . more general types of actions can also affect dialogue context, especially in real-world or embodied settings @xcite . within these settings, referential grounding is the process by which interlocutors anchor linguistic expressions to actual entities, relations, or events in the shared environment. when considering the perception and embodiment of participants, situated grounding is used @xcite . in other words, while conversational grounding focuses on "what was said", referential grounding ensures everyone agrees on "what is being talked about". in this paper, we present a simple information state model of dialogue that integrates both propositional updates (conversational grounding) and referential anchoring (situated grounding). we walk through a dialogue fragment from a corpus of taskbased multimodal interaction @xcite ; @xcite ; an example is shown in fig -ure 1 ), annotated with amr @xcite for speech and gesture @xcite , illustrating how speech, gesture, and object-directed actions co-construct and update the common ground. we assess the strengths and limitations of current annotations for capturing multimodal grounding phenomena, and argue for the importance of situational information in dialogue interpretation. ## related work information state theories of dialogue are based on the idea that dialogue acts change the context available to participants @xcite . at the most basic level, this includes the common ground, or shared assumptions of the participants @xcite . over time, the scope of the information state has expanded to handle different types of utterances beyond assertion; interrogatives are commonly handled via a set or stack of questions under discussion @xcite , while there are various theories for the meaning of imperatives @xcite @xcite . formal information state theories include @xcite ; @xcite ; and @xcite . several dialogue corpora analyze the conversational grounding process and the impact of situated grounding or information about the shared environment. among them, @xcite annotate two corpora with (conversational) grounding acts and grounding units @xcite . the stac corpus contains multi-party settlers of catan chats annotated with discourse structure and dialogue acts @xcite ; @xcite experiment with using large language models to predict the dialogue acts. @xcite present the fireball dataset of dungeons & dragons games, showing that adding game state information to the dialogue history can improve narration generation. @xcite develop the spot-ter framework to investigate linguistic convention formation in a task referentially grounded in vision. the scout corpus of situated human-robot dialogues @xcite is annotated with dialogue-amr @xcite and relations between utterances @xcite . the weights task dataset of situated interaction is annotated with several modalities including speech and gesture @xcite ; @xcite perform common ground tracking, focusing on the emergence of facts. ## discussion and conclusion within the corpus, some signalers use what we can call the result present tense, describing the configuration resulting from an action in the present, rather than giving an imperative. in fact, exclud-ing one-word utterances, declarative sentences outnumber imperatives by almost two to one (191 to 97). in one example, the signaler says "starting from the top, moving to your left, down four diagonally a row with the corners touching." the analysis of such utterances can be formalized in a number of ways. one approach, suggestive of ross (1970)'s performative analysis, is to treat them like implicit imperatives: one could imagine each statement beginning with a covert "make it true that...". these instructions would then be added to the actor's to-do list, in the same way as explicit imperatives foot_0 . another approach is to treat them as standard declaratives, with the actor's subsequent actions determined by pragmatic effects. following @xcite , declarative statements are offered as questions under discussion, which the actor can either accept or reject. without an imperative, there is no direct update to the actor's to-do list; however, assuming that they accept the statement, and the initial overarching task of building the structure remains in t a , they will change the state of the world (i.e., move blocks around) to make the signaler's description true. the challenge of ambiguous statements that require context for correct interpretation are wellestablished in dialogue literature @xcite . in sampling our corpus, we encounter two distinct kinds of ambiguity that require situated information to arrive at the correct interpretation. first, we notice several instances of presuppositions that are connected to the setup of the block-building task. these presuppositions are triggered with canonical utterances such as "again", "the same", or "also" @xcite @xcite . in one interaction, the signaler begins with the statement, "so you will begin with a grid structure again", referencing a previous interaction that required a grid-like spatial understanding of the block orientation on the table. we notice this throughout interactions: both signalers and actors approach the task with an implicit and often shared understanding of constraints on block structures and their orientation in the physical space. in the same interaction, the signaler instructs the actor to create "the same pattern" with blocks in a new area of the table. here, we encounter a second, partially overlapping challenge of multimodal ambiguity: multimodal coreference. in the case of the block pattern, the instruction and subsequent action are potential instances of the so-called sloppy identity effect @xcite , in which the same phrase can be interpreted with different arguments, i.e., blocks @xcite @xcite . such multimodal coreference can also be understood as coreference under transformation @xcite , a category easier to annotate and helpful in understanding sequences of events. here, while the concept of a block pattern is stable in identity, the concept is applied to a new instance that requires situated knowledge to enact correctly. using amr for both speech and gesture allows multimodal coreference relations throughout the dialogue and between the modalities to be marked using multi-sentence amr @xcite . meanwhile, using amr for action facilitates alignment and binding from the communicative modalities to the local environment, allowing for easier identification of situated grounding. however, as the @xcite corpus annotates only communication from the signaler, there are certain aspects of conversational grounding, such as the signaler's understanding of the actor's communicative acts, that the annotations do not capture yet. a complete analysis of bidirectional grounding processes will require the rest of the corpus to be annotated with the actor's actions, in addition to their speech and gesture. our model, focusing on describing what identifications are made between discourse entities and objects in the real world, sidesteps the question of how agents make these identifications. kennington and schlangen (2015) describe a "words as classifiers" approach to situated grounding of words and phrases in perceptual scenes. furthermore, our findings are limited to a single corpus, and applying this approach to other dialogue types will reveal new insights. for example, in the block structure-building task, the signaler knows what structure is to be built, and the actor knows this, and therefore accepts the signaler as an authoritative source of information. additionally, the taskspecific presuppositions that define the initial dialogue state require knowledge of each new context. these factors point to clear next steps for extending multimodal semantic annotation for the analysis of situated dialogue.