Towards Human-Like Dialogue Systems: Integrating Multimodal Emotion Recognition and Non-Verbal Cue Generation (2025)

Abstract: This position paper outlines my research vision for developing human-like dialogue systems capable of both perceiving and expressing emotions through multimodal communication. My current research focuses on two main areas: multimodal emotion recognition and non-verbal cue generation. For emotion recognition, I constructed a Japanese multimodal dialogue dataset that captures natural, dyadic face-to-face interactions and developed an emotional valence recognition model that integrates textual, speech and physiological inputs. On the generation side, my research explores non-verbal cue generation for embodied conversational agents (ECAs). Finally, the paper discusses the future of SDSs, emphasizing the shift from traditional turn-based architectures to full-duplex, real-time, multimodal systems.

## research interests my ultimate goal is to develop human-like dialogue systems capable of expressing emotions through multimodal communication, incorporating not only spoken responses but also non-verbal cues such as facial expressions and physiological signal visualizations. my current research interests lie in the areas of multimodal emotion recognition and non-verbal cue generation, with a focus on emotion-enriched humancomputer interaction. ## spoken dialogue system (sds) research regarding the future vision for sdss, recent breakthroughs in sds research have demonstrated the potential for real-time, simultaneous bidirectional communication that more closely mimics natural human conversation patterns @xcite . these advances represent a significant leap from traditional turn-based dialogue systems by enabling continuous, overlapping speech interactions with minimal latency. looking toward the near future, full-duplex sdss are expected to expand into multimodal interaction. future systems will likely integrate multiple communication channels, including speech, visual cues, and physiological signals. by combining these modalities, nextgeneration sdss will be capable of delivering immersive and emotionally expressive conversational experiences. ultimately, these systems will not only interpret users' multimodal inputs but also generate appropriate responses across multiple modalities simultaneously.