A Survey of Recent Advances on Turn-taking Modeling in Spoken Dialogue Systems (2025)

Abstract: The rapid growth of dialogue systems adoption to serve humans in daily tasks has increased the realism expected from these systems. One trait of realism is the way speaking agents take their turns. We provide here a review of recent methods on turn-taking modeling and thoroughly describe the corpora used in these studies. We observe that 72% of the reviewed works in this survey do not compare their methods with previous efforts. We argue that one of the challenges in the field is the lack of well-established benchmarks to monitor progress. This work aims to provide the community with a better understanding of the current state of research around turn-taking modeling and future directions to build more realistic spoken conversational agents.

## introduction conversational agents adoption is rapidly growing. the ubiquity of dialogue systems in recent years has increased the realism (i.e. human-likeness) expected from them. one major trait of realism is the way spoken dialogue systems take turns in dialogues @xcite . silence between utterances in human-human conversations take 200ms on average @xcite . however, current spoken dialogue agents initiate turns after long gaps (700-1000ms) @xcite , which results in unnatural, less realistic and non-fluid conversations. thus, realistic turn-taking behavior is still a challenge to be addressed. the main goal of turn-taking modeling is determining when the system should take the turn to speak. simplest attempts rely on the use of audio-based voice activity detection (vad) systems and silence thresholds @xcite . long silence thresholds derive systems that wait more time than expected, while shorter thresholds tend to interrupt users in the middle of their turns. in contrast, most recent systems use neural approaches to model turn-taking by minimizing speech overlaps and silence gaps. these systems use all types of available input data such as timing, linguistic, speech and visual information. for instance users' gaze and voice intonation at the end of utterances have been found to be relevant features to predict their end-of-turns @xcite @xcite . turn-taking cues generation, interruptions handling, and other tasks are also of interest in the management of turns @xcite . modeling turn-taking in multi-party conversations (mpcs) has not been widely explored in comparison to dyadic scenarios (i.e. one single user at a time), but has gained more attention in recent years. multi-party conversations consist of conversations where more than two participants are involved, e.g. two users and a conversational agent. these types of dialogues include additional complexities to the management of turn-taking @xcite . for instance, recognizing who the addressee(s) of a user's utterance is does not appear to be as trivial as in the dyadic case. challenges in mpcs are detailed in appendix a. examples of dialogue systems intervening in such types of conversations include agents providing assistance at hospital receptions @xcite or autonomous public buses @xcite , serving as healthcare coaches @xcite , guiding games at museums and hospitals for visitors entertainment @xcite , or taking place as attractions at thematic parks @xcite . while @xcite provided an exhaustive overview of turn-taking modeling, significant developments since then call for an updated survey. this paper contributes beyond the prior work in three key ways: (1) it offers the first comprehensive review of datasets used in the field, providing critical insights into data modality and language; (2) it discusses overlooked limitations in turn-taking models, identifying gaps and challenges that are crucial for future work; and the structure of this paper is organized as follows. in section 2, we review fundamental concepts of turn-taking management. section 3 provides an exhaustive description of the corpora used in the field. in sections 4 and 5, we describe works on the two main subtasks in turn-taking modeling: end-of-turn prediction and backchannel prediction, respectively. we report studies related to mpcs in section 6. finally, section 7 discusses the main open challenges and future opportunities in the field. ## turn-taking management turn-taking in conversations can be defined as the coordinated successive exchange of speaking roles between multiple subjects to speak, listen, and respond @xcite . cooperative verbal communication is not unique to humans, as other animal species have also shown certain forms of turn-taking behavior @xcite . although the coordination of turns feels natural in most human dialogues, it requires training at early stages of childhood @xcite @xcite . this suggests that there is a level of cognitive effort we need to perform to fluently manage turns. such coordination lies on its dynamic temporal structure where listeners have to foresee the end of the speaker's utterance to anticipate their take of turn @xcite . figure 2 in appendix c shows how turn-taking is handled in dialogues and illustrates various elements associated to turn management. fluency in the organization of turns is commonly assessed by the amount of overlaps and gaps between turns. a high number of these events indicate a poor ability of anticipating the end of turn by listeners @xcite . overlaps occur when the listener starts speaking before the speaker completes their utterance. gaps take place when long silences precede the take of turn of the next speaker. to optimize the organization of turns, listeners rely on cues provided by the speaker while holding or releasing their turns to determine when it is adequate to take the turn. similarly, speakers rely on cues generated by listeners to know when any listener desires to take the turn, in order to decide whether to hold or release the floor. these cues can include verbal signals, gestures, and others. we describe in detail turn-taking cues in appendix d. modeling turn-taking comprises multiple subtasks such as end-of-turn detection, interruptions handling and others. although certain attempts of turn-taking modeling have proposed to simultaneously tackle several subtasks @xcite , each of them have been mostly treated as independent problems. in addition, turn-taking in mpcs has received little attention compared to two-party dialogues. the study of turn-taking modeling in mpcs has been mainly conducted in the field of human-robot interaction @xcite @xcite , since it is difficult to organize turns in mpcs without the visual channel @xcite . ## datasets in this section, we detail the datasets used in all the works we review in this survey, i.e. described in sections 4, 5, and 6. although research on turn-taking modeling have been mainly developed on dialogues in english, there are a few dialogue corpora in other languages. we separate datasets according to languages in english and other languages. a summary of all datasets is shown in ## end-of-turn prediction detecting the end-of-turn is the most well-studied problem in turn-taking modeling. end-of-turn prediction, also referred to as end-of-utterance (eou) detection, is usually defined as a binary classification task. its goal is to determine if the system should take the turn or not, depending on the dialogue context. methods for eou prediction can be grouped in three categories: silence-based, ipu-based and continuous @xcite . silence-based methods rely on voice activity detection (vad) tools, where a silence threshold (e.g. 700ms) is set to determine whether the system should take the turn. these methods result in poor user experience due to lack of naturalness @xcite . ipu-based and continuous approaches differ on the time when predictions are made along the dialogue. while ipu-based foot_0 methods evaluate if the turn should be taken after every inter-pausal unit, i.e. after a silence, continuous models constantly evaluate the occurrence of an end of turn regardless of silencese.g. every 50ms of speech. in this survey work we focus in continuous and ipu-based methods. ## backchannel prediction overlaps in dialogues occur when multiple participants produce ipus at the same time. these overlaps may take place in the proximity of the end of a turn if the listener desires to start their turn, or in the middle of the speaker utterance. in the latter case there are three possible scenarios: one of the first attempts to model backchannel prediction using neural networks was reported in @xcite , where only speech features were used. in @xcite , backchannel detection was addressed by predicting if a speech onset of 500 ms corresponded to a short (less than 500ms, i.e. backchannel) or a long utterance (more than 2500 ms), using handcrafted acoustic features and pos tags to feed a lstm model. @xcite considered backchannels as an intention label to build an intention recognition model. other early works used word2vec @xcite to combine word embeddings as linguistic features with acoustic features @xcite . @xcite took delays in asr into account to propose a prior prediction model, as words are available some time after these have been uttered. speaker and listener embeddings to encode interlocutor interactions have also been considered to predict backchannels @xcite . recent advances have examined auxiliary tasks to predict backchannels @xcite , showing improvements over single-task methods. these tasks include sentiment classification, dialogue act prediction, and others @xcite . @xcite used audiovisual data to introduce agreement estimation in a multitask setting to detect backchannels. @xcite proposed a context-aware backchannel prediction model to enhance predictions in korean and english corpora. they encoded features using text embeddings from bert @xcite and acoustic inputs represented by wav2vec embeddings. finally, a multi-head attention mechanism is employed to build an attentive context embedding that holds relevant information of the current utterance. voice activity projection (vap) models, presented in section 4.1, have gained special attention for backchannel prediction. @xcite found that integrating non-verbal features on vap models enhances turn-taking events prediction, including backchannels. pre-training on large dialogue data and fine-tuning on a specialized backchannel corpus has also shown improvements on vap model's generalizability @xcite . ## multi-party turn-taking modeling in this section we describe methods proposed for turn-taking modeling in multi-party conversations. we outline the main complexities in mpcs in human-human dialogues in appendix a. turn-taking prediction becomes more challenging in multi-party scenarios, where various sub-tasks arise such as conversation disentanglement, addressee recognition, and others @xcite . multi-party conversation modeling addresses the issues on who says what to whom (who speaks, says what, and addresses whom) @xcite . modeling turn-taking in mpcs has received much less attention than dyadic interactions. although work has been done on the topic for a long time @xcite @xcite @xcite , only recently have the methods designed begun to yield promising results, due to advancements in available technologies. @xcite proposed a timing generating network, which incorporates a first-order lag system to estimate how much other speakers in the dialogue expect the system to take the turn in japanese. their approach is in contrast to the conventional framing of turn-taking modeling as an end-of-turn detection problem, which assumes that the system should take the turn right after the previous speaker releases it. they integrate response obligation recognition as an auxiliary task to improve estimation. @xcite and @xcite studied next speaker identification from dialogue logs to model turn-taking prediction in mpcs. gaze-transition patterns and timing information have been investigated to predict the next speaker and the time at which each utterance will be made @xcite . experiments on a transformer-based architecture using 3d gazes, 3d head and body movements, and speech showed that speech signals play a more critical role than gaze patterns for turn-taking prediction @xcite . multimodal fusion has also been studied for turn-taking prediction, where multiple event types are predicted simultaneously @xcite . @xcite argue that there are different states in which turn-taking could be obliged or optional. they proposed annotating utterances into a scale of four classes according to the appropriateness for an agent to take the turn. they observed that dialogue acts as turn change predictors in mpcs need a special treatment compared to two-party settings. turn-taking modeling in mpcs has notably been understudied in comparison to dyadic scenarios. work in mpcs where no visual data are used has been overlooked, as most studies have been conducted in the field of human-robot interaction where visual cues are captured by sensors. we think that introducing response obligation detection in any form is crucial for modeling turn-taking in mpcs. determining whether the agent should take the turn or not after the floor is released mitigates poor performance on out-of-scope utterances, a common phenomenon for dialogue systems in such scenarios. we discuss more about open challenges in multi-party conversations, including out-of-scope utterances, in section 7.4. ## challenges and future directions in this section we describe some relevant open challenges and suggest opportunities for future work. ## conclusions turn-taking modeling is a key component of spoken dialogue systems. effective methods for modeling turn taking are crucial for developing systems that can be perceived as realistic. this survey provides an overview of recent advancements in turn-taking modeling in spoken dialogue systems. we provided the first detailed review of the corpora used in the field. we observed that the majority of works have been conducted on english and japanese corpora, with almost no efforts in other languages. we also described recent works in end-of-turn prediction and backchannel classification. finally, we discussed several overlooked open challenges in current turn-taking models and key directions indicating how future work could push the field. for instance, we noted a tendency in the reviewed works not to compare their proposed methods with previous works, which might affect monitoring progress in the field. addressing these challenges and improving cross-linguistic research and method comparisons will be essential for advancing turn-taking models and making spoken dialogue systems more natural and effective. 2 0 0 9 2 0 1 0 2 0 1 1 2 0 1 5 2 0 1 6 2 0 1 7 2 0 1 8 2 0 1 9 2 0 2 0 2 0 2 1 2 0 2 2 2 0 2 3 2 0 2 4 dialogues. figure 2 depicts distinct elements and events occurring in spoken dialogue turn-taking management. inter-pausal units (ipus) are speech segments preceding a silence of a certain duration. ipus correspond to the main pieces of information exchanged by interlocutors. pauses are silences between two consecutive ipus of the same speaker. gaps are silences between two consecutive turns of different speakers. backchannel are lexical or non-lexical sounds provided as a feedback by a listener in the dialogue. these expressions are usually used to indicate to the speaker that the listener understands or acknowledges what the speaker says, without the intention of interrupting. a backchannel is not considered as a turn. an overlap takes place when ipus from distinct speakers are produced at the same time. usually, they occur at turn shifts.