Multimodal Agentic Dialogue Systems for Situated Human-Robot Interaction (2025)

Abstract: This position paper presents the integration of dialogue systems into situated robotics, emphasizing the use of contextual information—particularly audiovisual perceptions—to inform dialogue policies. A central objective is the development of interaction policies that dynamically select contextually appropriate actions aligned with the user’s intentions and needs. The works presented in this paper explore proactive decision-making mechanisms in multimodal interaction settings and seek to enhance robotic expressiveness through nonverbal communication cues. Current efforts focus on evaluating and comparing approaches such as agentic workflows and reinforcement learning within a unified framework, aiming to facilitate more consistent and contextually aware human–robot interaction.

## research interests the focus of my research lies in the application of dialogue systems within the context of situated robotics. a primary aspect of my work involves the integration of contextual information -particularly audiovisual perceptions -in the dialogue policy. another main dimension concerns the control of the interaction policy, through the selection of contextually appropriate actions aligned with the user's expressed needs and intentions. in this regard, i explore mechanisms for proactive decision-making in multimodal interaction settings. additionally, i am interested in enhancing the robot's expressiveness by incorporating non-verbal communication cues into the interaction pipeline. my current work aims to investigate and compare various approaches, such as agentic workflows or reinforcement learning, to address these challenges within a unified framework, with the goal of enabling more consistent and contextually aware human-robot interaction. ## spoken dialogue system (sds) research beyond the purely technical and scientific challenges associated with the future development of sds, environmental considerations are becoming increasingly prevalent. current sds architectures rely on very large models, whose training entails a substantial environmental cost. given the conclusions of the latest ipcc reports regarding the climate crisis (ipcc, 2023), the development of future dialogue systems could incorporate environmental factors. a wide range of existing models already support highly advanced applications, both in research and for the end users. from an environmental perspective, it may therefore be relevant to prioritize research efforts aimed at optimizing the use of existing llms, rather than focusing on the large-scale creation of new models. such an approach could also contribute to a deeper understanding of the underlying mechanisms governing the behavior of these models. for instance, the recent discovery of chat-gpt's ability to simulate theory of mind @xcite illustrates how llms may exhibit emergent capabilities despite the absence of explicit training for such tasks. ## suggested topics for discussion the following topics appear to be relevant foundations for discussion sessions: