Prompt-Guided Turn-Taking Prediction (2025)

Abstract: Turn-taking prediction models are essential components in spoken dialogue systems and conversational robots. Recent approaches leverage transformer-based architectures to predict speech activity continuously and in real-time. In this study, we propose a novel model that enables turn-taking prediction to be dynamically controlled via textual prompts. This approach allows intuitive and explicit control through instructions such as “faster” or “calmer,” adapting dynamically to conversational partners and contexts. The proposed model builds upon a transformer-based voice activity projection (VAP) model, incorporating textual prompt embeddings into both channel-wise transformers and a cross-channel transformer. We evaluated the feasibility of our approach using over 950 hours of human-human spoken dialogue data. Since textual prompt data for the proposed approach was not available in existing datasets, we utilized a large language model (LLM) to generate synthetic prompt sentences. Experimental results demonstrated that the proposed model improved prediction accuracy and effectively varied turn-taking timing behaviors according to the textual prompts.

## introduction turn-taking, which governs transitions between speakers, is fundamental to smooth and natural human communication @xcite . accurate turn-taking prediction is particularly crucial in developing spoken dialogue systems and conversational robots, as it directly influences interaction quality by minimizing inappropriate interruptions and reducing response delays @xcite @xcite @xcite . recent advancements, notably transformer-based models such as turngpt @xcite and voice activity projection (vap) @xcite , have significantly improved continuous and real-time speech activity predictions by effectively utilizing conversational history. in this paper, we introduce a novel transformerbased model capable of dynamically adjusting its turn-taking predictions based on textual prompts. our approach integrates prompt embeddings into both channel-wise and cross-channel transformer architectures in vap, enabling explicit and intuitive control through simple instructions like "faster" or "calmer". previous research indicates that turntaking behaviors in conversations vary according to individuals' attributes like age, gender, and personality traits such as extraversion and introversion @xcite @xcite @xcite . the adaptability of our proposed model thus facilitates such responsive interactions tailored to dialogue contexts, user types, and system configurations. to our knowledge, this work is the first to explicitly direct turntaking predictions using textual prompts. ## proposed method our proposed model builds upon the vap architecture that predicts future voice activity for dyadic conversations @xcite . the key innovation of our approach is the integration of textual prompts, allowing for dynamic control over the model's turn-taking behavior, as depicted in figure 1 . ## dataset this section describes the datasets used in this study and the method for generating textual prompts for the proposed approach. ## experiment we first quantitatively evaluated the performance of the turn-taking prediction by comparing the and balanced accuracy for predicting turn shifts or holds during mutual silences @xcite . model configurations and training parameters were consistent with those described in the original work @xcite . the results are summarized in next, we qualitatively evaluated how the input prompt text influenced the output of the proposed model. in order to simulate a user-system dialogue and evaluate when the system should take the turn, we used one channel from a test audio as the user's audio input and set the other channel to zero to represent the system's silence. we prepared two text prompts: one representing the user's instructions or assumed intentions within the dialogue, and the other specifying the system's behavioral setting. figure 2 illustrates an example output, highlighting when the system should take its turn (yellow area). in this example, the user prompt is "before speaking, take a short pause and think carefully, then begin speaking politely. do not rush your response; maintain a calm and steady pace." conversely, the system prompt instructs, "speak with a good rhythm and respond immediately after the other person finishes speaking. try to speak more frequently and take the lead in the conversation." the system's prompt encourages immediate responses and frequent turn-taking, as reflected by the increased frequency and quicker timing of the predicted system turns (yellow). figure 3 shows another scenario where the prompts between the user and the system were swapped with the same audio input. with the system now instructed to respond slowly and deliberately, fewer turn shifts were observed compared to the previous example, aligning with the given behavioral prompt. ## conclusion we introduced a novel prompt-guided turn-taking prediction model that dynamically adjusts its behavior based on textual prompts integrated into the vap model. our experiments with approximately 950 hours of japanese dialogue data demonstrated that the model effectively modulates turn-taking behaviors according to different prompts, improving both prediction accuracy and adaptability. future work includes integrating this approach into practical dialogue systems and robots to verify its utility, particularly by automatically inferring user prompts or configuring system prompts. this includes adapting interactions based on user characteristics such as age (child, adult, or elderly) or personality. additionally, validating the appropriateness of generated prompts and expanding prompt variations, along with extending the work to include aspects like human alignment and multilingual capabilities @xcite , remain essential next steps.