Annotating Situated Actions in Dialogue (2023)

Abstract: Actions are critical for interpreting dialogue: they provide context for demonstratives and definite descriptions in discourse, and they continually update the common ground. This paper describes how Abstract Meaning Representation (AMR) can be used to annotate actions in multimodal human-human and human-object interactions. We conduct initial annotations of shared task and first-person point-of-view videos. We show that AMRs can be interpreted by a proxy language, such as VoxML, as executable annotation structures in order to recreate and simulate a series of annotated events.

## introduction in recent years, there is an increasing interest in dialogue systems that interact with humans in a natural and sophisticated manner. chatgpt @xcite and other large language models (llms) show a remarkable ability to generate fluent responses to textual prompts. however, these systems lack two key capabilities which are necessary for naturalistic interaction. first, they lack the ability to communicate in multiple modalities beyond written language, including gesture, gaze, and facial expression; llms, even ones like gpt-4 that accept both text and image input @xcite , are limited to text output. second, these models do not have a notion of the "world" as such. they do not track actions and objects in an environment, and therefore are unable to perform situated grounding @xcite . much work has addressed the importance of nonlinguistic modalities in communication @xcite @xcite @xcite . for example, in a spoken sentence "i used this for the sketch", the referent of the demonstrative "this" is unspecified. in conjunction with a gesture, e.g., pointing to a pencil, however, reference resolution and disambiguation are possible. less attention has been paid to the role of action in dialogue interpretation. actions significantly contribute to the multimodal context within which linguistic utterances are made, and thus play a crucial role in understanding and interpreting dialogue. in the previous example, lifting the pencil can also direct attention to it, which is then linked to the demonstrative. additionally, actions can also serve as antecedents to speech in vp ellipsis constructions, (e.g., "what did you do that for?" after someone slams a door), and as action-based bridging relations, where actions create links between concepts in a narrative (e.g., "i went to the store today", followed by taking fruit out of a grocery bag). actions can even be referenced directly by participants, such as the case of a child relaying "my brother said 'thumbs up'!" when given permission to play with a favorite toy. a major aspect of dialogue interpretation is the common groundshared knowledge and beliefs that interlocutors possess about each other and the world @xcite @xcite . conversations between agents introduce the problem of identifying and modifying the common ground @xcite . actions can update the common ground in ways that speech and gesture cannot, by adding, modifying, and deleting items within it. we argue that, given the importance of actions to multimodal nlu and their direct influence on the common ground, it is essential to consider how they may be integrated with language and other communicative modalities in a shared annotation scheme. in this paper, we review existing action annotation schemes, as well as abstract meaning representation (amr) @xcite . we then describe initial efforts to use amr to anno-tate actions in video data. we explain how action descriptions made with amr can be translated to the voxml interpretation language @xcite , where they can be executed in a simulated environment, voxsim (krishnaswamy and pustejovsky, 2016), and then close with a discussion of annotation challenges and future work. ## approach to explore the feasibility of applying amr to actions, we examine two distinct datasets: the fibonacci weights task dataset @xcite review), as well as the egocentric epic kitchens dataset @xcite . in the examples below, we align observed actions with propbank senses @xcite . ## discussion we have described an initial exploration of action annotation within the context of communicative acts in dialogue. by investigating the application of amr and voxml, we aim for adequate representations to model the interactions between them, as well as define simulations that can track the evolving common ground. this analysis has highlighted certain challenges associated with annotation and possible directions for future work in designing representations. ## conclusion in this paper, we argue that representing actions is essential for the proper interpretation of situated dialogues. we describe how amr can be used to annotate actions in different types of video interactions, and describe the challenges associated with this task. we also show how amrs can be translated to the voxml specification language to encode semantic information, allowing for the ability to track changes to the common ground in a simulation environment like voxsim. in future work, we plan to further develop our annotation methodology, and apply it on a larger scale.