Using Transition Duration to Improve Turn-taking in Conversational Agents (2022)

Abstract: Smooth turn-taking is an important aspect of natural conversation that allows interlocutors to maintain adequate mutual comprehensibility. In human communication, the timing between utterances is normatively constrained, and deviations convey socially relevant paralinguistic information. However, for spoken dialogue systems, smooth turn-taking continues to be a challenge. This motivates the need for spoken dialogue systems to employ a robust model of turn-taking to ensure that messages are exchanged smoothly and without transmitting unintended paralinguistic information. In this paper, we examine dialogue data from natural human interaction to develop an evidence-based model for turn-timing in spoken dialogue systems. First, we use timing between turns to develop two models of turn-taking: a speaker-agnostic model and a speaker-sensitive model. From the latter model, we derive the propensity of listeners to take the next turn given TRP duration. Finally, we outline how this measure may be incorporated into a spoken dialogue system to improve the naturalness of conversation.

## introduction turn-taking is an important component of many spoken dialogue systems and involves (a) detecting or predicting the end of a turn and (b) accurate timing of the initiation of speech production @xcite . smooth turn-taking continues to be a challenge for spoken dialogue systems that aim to engage in natural conversation @xcite . traditionally, most spoken dialogue systems process inter-pausal units (ipus), which are speech units surrounded by arbitrary fixed length silence thresholds @xcite . these pauses of arbitrary duration cause a stilted, unnatural conversation style. other systems use incremental approaches, processing smaller units of speech at a time. for example, the incremental dialogue system proposed by @xcite operates on incremental units (ius) that are processed by incremental modules (ims). these modules may include action, turn, or dialogue management-each of which influences turn-planning and end of turn detection. although such systems initiate the production of speech when a silence is detected, they do so based on pitch or semantic completeness @xcite . machine learning models of turn-taking operate on previously detected multimodal cues @xcite . the turn-taking techniques used in traditional spoken dialogue systems, such as predicting turnends in a time window @xcite , are not fully grounded in current theory of human turntaking. in natural conversation, people tend to minimize gaps and overlaps while also following the one "one speaker at a time" rule @xcite . this means that when a turn ends, another speaker may start speaking. speakers also use the duration of silences to convey social information @xcite . for example, long and short gaps may communicate hesitance or impatience. additionally, interlocutors use turn-taking cues (e.g., lexico-syntactic, pragmatic, prosodic etc.) to predict the end of turns and plan responses @xcite . in contrast, turn-taking techniques typically do not explicitly identify points where floor change may occur @xcite , normatively time the duration of silences, or predict turn-ends independent of the occurrence of specific events (e.g, silences) @xcite . for spoken dialogue systems, this leads to mistimed responses and a decrease in human engagement @xcite . in this paper, we propose an evidence-based model for when speech may be produced to facilitate smooth turn-taking, based on the turn-taking model proposed by @xcite . in this model, a speaker's turn consists of one or more turn construction unit (tcu), which encompasses sentential, clausal, phrasal, and lexical constructions. between each tcu are transition relevance places (trps), where the current turn may be completed and a floor change may occur @xcite . we further divide trps into continuations (trps where the current speaker continues) and switches (trps where a speaker-transition occurs). additionally, in our operationalization, each trp has a duration-the time between when the previous tcu is complete but before the next tcu begins. we use tcu-level data from transcriptions of the switchboard corpus @xcite to develop two models of turn-taking based on the duration of trps: a speaker-agnostic model and a speaker-sensitive model. next, we develop an evidence-based function for the propensity of floortransfer as a function of time after the end of a tcu. finally, we outline a proposal for implementing this propensity function into the continuous dialogue system architecture formalized by @xcite . ## empirical models in this section, we fit two bayesian models of trp duration: one that assumes a single distribution of all the trps in a dialogue, and one that assumes that the distribution is different for speaker switches and speaker continuations. in what follows, we will first describe the empirical data that forms the basis for our models. next, we provide a detailed description of the two probabilistic models informed by our conceptual models. we then describe the implications of our findings for turn-taking and speaker selection. finally, we propose an evidence-based turn-taking propensity function for natural speech production decisions after the end of a tcu. ## continuous module proposal in this section, we outline a proposal for operationalising our turn-taking propensity function for timing turn-taking. first, we outline relevant components of the incremental dialogue processing architecture proposed by @xcite . next, we define the minimal module implementing the proposed timing method, as well as possible extensions to incorporate existing turntaking methods (e.g, @xcite ). we use the @xcite architecture for its continuous and incremental properties, which may be useful for comparing different timing methods. however, our proposed method is based simply on timing and does not have a strong dependence on any specific architecture. ## conclusion and future work in this study, we started by comparing two conceptual models of turn-taking-duncan's "turnyielding" cue model and sacks et al.'s "simplest systematics", each of which makes different predictions about the organization of turns in conversation. we used data from the switchboard corpus to fit two probabilistic models of trp duration based on these conceptual models: a speaker-agnostic model compatible with duncan's conceptual model and a speaker-sensitive model inspired by on sacks' et al.'s conceptual model. both models have a good quantitative and qualitative fit with the empirical data. however, when comparing the two models directly, we found that the speaker-sensitive model i.e., sacks et al.'s model, was decisively better at predicting the data than the speaker-agnostic model. we explored the implications of this finding for turn-taking systems. we showed that the likelihood of a speaker beginning a tcu during a pause in conversation changes as the pause lengthens. for short pauses, it is more probable that the speaker will switch, but as the pause continues, the original speaker becomes more likely to continue their turn. our work supports the notion that, for proper turn-taking, detecting and/or anticipating the end of turns is not sufficient. people are sensitive to the pauses and gaps in conversation and organize their speech to take into account this paralinguistic signal. we described the regularities that we found, and outlined implementations for dialogue systems to incorporate our findings. for naturalistic turntaking adhering to these subtle norms is important, and we described first steps towards implementing this in agents. in future work, we plan on implementing the spoken dialogue system we have proposed in this paper. while we have established and operationalized normative turn-taking behavior based on human conversations, it is important to investigate whether and to what degree findings from humanhuman data generalize to communication with spoken dialogue systems. therefore, evaluating the conversational naturalness of our system through human-subject experiments is a relevant next step and will provide insight into the organization of turns in conversation, both for human-human and human-agent communication.