I nteract S peech: A Speech Dialogue Interaction Corpus for Spoken Dialogue Model (2025)

Abstract: Spoken Dialogue Models (SDMs) have achieved significant progress in recent years, yet they continue to face challenges in handling nuanced interactional phenomena. A significant bottleneck hindering further advancement is the scarcity of publicly available, high-quality datasets meticulously designed to train and evaluate these fine-grained interactive capabilities. We introduce InteractSpeech, a 150-hour English speech interaction dialogue dataset designed to empower spoken dialogue models with nuanced real-time interaction capabilities, such as handling interruptions and backchannels. InteractSpeech was created by synthesizing interactive dialogues from text using advanced speech synthesis, and by filtering real-world spoken dialogues for interactive segments. The dataset features precise speaker timestamps and annotations for diverse dialogue interactions, underpinned by a formal framework for interaction dynamics. We demonstrate InteractSpeech’s utility by fine-tuning a LLaMA 3-8B model on its textual scenarios and, crucially, by training a speech understanding model that accurately classifies key interactional events directly from audio. This highlights the dataset’s value in developing models capable of more natural and responsive conversational turn-taking. Audio samples are available at https://interactspeech.github.io/.

## introduction recently, spoken dialogue models such as @xcite and moshi @xcite have garnered significant attention in the speech domain. these end-to-end spoken dialogue models @xcite @xcite @xcite @xcite can not only generate natural, human-like speech responses but also demonstrate an advanced understanding and generation of acoustic features beyond text, such as timbre, emotion, and style @xcite . their realistic conversational interactivity and low-latency dialogue experiences further distinguish them among the traditional spoken dialogue models @xcite . however, for spoken dialogue models to achieve truly natural and effective human-computer interaction, they must master the subtle art of conversational turn-taking and interaction dynamics. this includes appropriately handling interruptions, providing timely backchannels, and managing conversational flow-capabilities that are often underdeveloped. a primary bottleneck is the lack of highquality, open-source datasets specifically designed to train and evaluate these fine-grained interactive abilities. existing resources present several challenges: 1) some prominent models, like moshi @xcite , still rely on outdated datasets such as the fisher corpus @xcite for fine-tuning, which may not reflect contemporary conversational patterns or provide sufficient detail for interaction modeling. 2) recent open-source interactive datasets @xcite often focus on command-based interruptive words rather than the diversity of real-world conversational interruptions and other interactional phenomena. given these limitations, we propose interactspeech, an opensource, multi-turn, and highly diverse interactive dialogue dataset. interactspeech is specifically designed to facilitate the development and evaluation of models that can understand and participate in complex spoken interactions. it provides detailed annotations crucial for modeling the interactional dynamics we formalize within this work, moving beyond simple speech recognition or generation. interactspeech is a mixed dataset consisting of synthetic and real-world data, containing approximately 150 hours of spoken dialogue data. compared to previous spoken dialogue datasets in modeling conversational interactions. the pursuit of human-like conversational ai has spurred research into modeling complex interaction dynamics. early work in conversation analysis (reimann et al., 2024) laid the foundation for understanding turn-taking mechanisms. recently, end-to-end spoken dialogue models (défossez et al., 2024; ji et al., 2024a; xie and wu, 2024a; fang et al., 2024; chen et al., 2024, 2025a; @xcite are increasingly aiming for full-duplex capabilities, enabling them to process speech and generate responses concurrently. this inherently requires a deeper understanding of interactional cues to manage barge-ins, provide timely feedback, and maintain conversational flow. for example, models like those aiming for "listen while speaking" @xcite ) implicitly or explicitly model when to yield or take the turn. however, the ability of these models to truly understand and appropriately react to diverse interactional events is often limited by the data they are trained on. our work contributes by providing interactspeech, a resource specifically designed to train and analyze these interactional modeling capabilities, supported by a formalization of key interactional events critical for robust spoken dialogue systems. ## interactspeech to develop spoken dialogue models capable of nuanced, real-time interaction, it is crucial to first define and understand the fundamental dynamics of human conversation. subsequently, datasets must be constructed to reflect these dynamics, enabling models to learn appropriate interactive behaviors. in this section, we first introduce our framework for modeling interaction dynamics (section 3.1), which provides the conceptual basis for interact-speech. we then detail the overall structure of interactspeech (section 3.2), highlighting how it embodies these interactional principles. finally, we present the data creation pipeline (section 3.3) used to build this resource. ## conclusion in this paper, we introduced interactspeech, a 150hour english spoken dialogue dataset meticulously designed to address the critical need for resources that support the development of models with nuanced real-time interaction capabilities. we presented a framework for modeling key interactional dynamics, which underpins the construction and annotation of interactspeech. the dataset combines synthetically generated dialogues, enriched with diverse interactional events via multi-stage prompt programming, with carefully filtered realworld spoken data. through extensive experiments, we demonstrated the utility of interactspeech. we showed our fine-tuned model significantly outperformed baselines on both in-domain and out-ofdomain real-world scenarios, highlighting interact-speech's potential to advance the development of models that can robustly perceive, interpret, and appropriately react to the dynamic flow of human conversation. we hope interactspeech will help further research into more natural, responsive, and human-like spoken dialogue systems.