Simulating Turn-Taking in Conversations with Delayed Transmission (2020)

Abstract: Conversations over the telephone require timely turn-taking cues that signal the participants when to speak and when to listen. When a two-way transmission delay is introduced into such conversations, the immediate feedback is delayed, and the interactivity of the conversation is impaired. With delayed speech on each side of the transmission, different conversation realities emerge on both ends, which alters the way the participants interact with each other. Simulating conversations can give insights on turn-taking and spoken interactions between humans but can also used for analyzing and even predicting human behavior in conversations. In this paper, we simulate two types of conversations with distinct levels of interactivity. We then introduce three levels of two-way transmission delay between the agents and compare the resulting interaction-patterns with human-to-human dialog from an empirical study. We show how the turn-taking mechanisms modeled for conversations without delay perform in scenarios with delay and identify to which extend the simulation is able to model the delayed turn-taking observed in human conversation.

## introduction turn-taking in human conversations has proven to be influenced by many auditory, visual, and contextual cues. especially in telephone conversations, where no visual cues are present, people rely on the immediacy of signals in prosody and content to perform smooth and uninterrupted turn-taking. investigating the influence of delay on conversations has been a focus in telephone quality research for a long time, where the goal is to study how degradations of packet-switched voip-transmissions influence the conversation structure and thus, the perceived quality (itu-t recommendation p. @xcite . but also in the field of human-computer interaction, where spoken dialogue systems (sds) with realistic turn-taking have become feasible, it is of interest to study how humans interact and react to delayed voice transmission. it has been shown that the perception of changes in transmission time not only depends on the duration of the delay but that the effects on the conversations also vary with the type of conversation itself @xcite . concretely, conversations with lower interactivity, i.e., slower speaker alternation rate and less turn-taking, are not as prone to be affected by transmission delay than conversations with higher interactivity. simulating a conversation does not only give insights into the interactivity patterns that arise during a conversation but can also used to predict events and behaviors. in such a simulation, two dialog systems exchange information through a speech channel. information is processed in increments to allow for a turntaking mechanism and structured dialog @xcite . in this paper, we present a simulation with different levels of interactivity and evaluate how a probability-based turn-taking function models the behavior in conversations under the influence of transmission delay. for this, we simulate two different goal-oriented conversation scenarios standardized by the itu, namely the short conversation test (sct) with a low conversational interactivity and the random number verification test (rnv) with a high conversational interactivity (itu-t recommendation p.805, 2007). we simulate conversations with 0ms, 800ms, and 1600ms delay and compare metrics of interactivity like speaker alternation rate, gaps, overlaps, and pauses, as well as unintended interruption rates to human-to-human conversations with the same delay conditions. due to the delayed arrival of turn-taking signals, transmission delay affects the flow of a conversation @xcite . however, the degree to which turn-taking and the interactivity of a conversation is degraded also depends on the interactivity of the conversation itself @xcite . to evaluate those dependencies, conversation tests with distinct levels of conversational interactivity (ci) have been standardized, during which participants perform goal-oriented tasks with an interlocutor. one prominent conversation test with a high ci is the rnv test, where participants alternatingly exchange a list of numbers organized in 4 blocks @xcite ). an example of a conversation test with low ci is the sct, where participants solve real-world tasks like ordering pizza or booking a flight. parametric conversation analysis (p-ca) is a framework to assess the structure of conversations programmatically @xcite . with an independent voice activity detection of the two speakers, four conversation states can be derived: m ("mutual silence"), d ("double talk"), a ("speaker a") and b ("speaker b") @xcite ; itu-t recommendation p.59, 1993). based on these four states interactivity metrics like the speaker alternation rate (sar), interruption rate (ir), as well as turn-taking information like gaps and overlaps between speaker turns, can be calculated @xcite . for delayed conversations, the unintended interruption rates (uir) measures the number of interruptions that were caused by the delay and were not intended to be interrupting the interlocutor @xcite . as conversation simulations focus on turntaking, they need to respond to incoming signals in a timely matter and thus need to process data incrementally. the incremental processing on the scale of a complete dialogue system has been proposed by @xcite and implemented in inprotk @xcite and retico @xcite . ## simulation setup and turn-taking the simulation is based on a set of conversation tests carried out with 58 untrained participants who were 18 to 71 of age (m: 32, sd: 13.48), of which 48.2 percent identified as female. during the experiments, each pair of participants carried out sct and rnv conversations with end-to-end one-way transmission delays of 0ms, 800ms, and 1600ms, resulting in 174 recorded conversations. for the simulation, one scenario was selected from each conversation type, and 20 sct conversations and 20 rnv conversations at 0ms delay were annotated with dialogue acts, transcripts, and turntaking information. 20 different conversations from each conversation type were used to evaluate the simulation. the simulation was implemented using the incremental processing pipeline of the retico framework @xcite . it consists of two spoken dialogue systems (agents) that are connected through a simulated transmission network that is able to introduce delay to both agents. a schematic view of the incremental modules of one agent in the simulation is shown in figure 1 . the speech input and output, as well as natural language understanding modules, are created by specifically recognizing the annotated empirical conversations. language generation and synthesis is handled by 3 2 1 0 1 2 3 4 transmitting utterances cut from the empirical data so that the length and content of the utterances match. an end-of-turn prediction module predicts the time until the end of the utterance, and an audio dispatching module reports the progress of the current utterance to the dialogue manager of the same agent. the dialogue manager uses agenda-based dialog management to fulfill the goal-oriented tasks of the sct and rnv scenarios, and it also handles turn-taking. the turn-taking of the agents in the simulation is modeled by probability distributions that are based on the work by lunsford et al. @xcite . we calculated the distributions of turn-switches (gaps and overlaps) as well as turnkeeping (pauses) as shown in figure 2 . these distributions are measured respective to the end of the last utterance so that negative values correspond to double-talk, and positive values correspond to mutual silence. the cumulative distribution of the pauses and switches were fitted with a logistic regression and inverted to form a model for turnswitches (equation 1 ) and turn-keeping (equation 2 ). by selecting @xmath0, 1] randomly from a uniform distribution and treating switches and pauses as equal alternatives, the agent in the simulation can perform turn-taking in the simulation. depending on which agent is currently speaking, the dialogue manager decides when to make a pause or a speaker switch. this way, the models of pauses and switches compete at every end of a turn. to prevent prolonged interruption (e.g., when both agents start speaking at the same time), the dialogue manager stops the speech production when double talk occurs in the middle of utterances. ## results and discussion to evaluate the simulation approach, we simulated 100 rnv and 100 sct conversations, each with 0, 800, and 1600 ms transmission delay. this results in 600 simulated conversations that we compare to the 174 conversations recorded in the experiment. the comparison of the states of the sct conversations (figure 4 ) and rnv conversations (figure 5 ) shows that the distinct levels of interactivity between these two types of conversations are also visible in the simulated conversation. when introducing delay, the state probabilities of the empirical data and the simulated conversation for mutual silence, speaker a and speaker b show similar changes. however, these effects stagnate for the simulated conversations at 1600 ms. this can also be seen when comparing the speaker alternation rate (figure 3 ) of the simulated sct conversations. there, the drop in speaker alternations due to increased delay is visible for 800ms but increases again for 1600ms, contrary to the behavior of the empirical conversations. this seems to indicate changes in the turn-taking behavior with an increased level of transmission delay. experiment simulation dataset 0.2 0.4 0.6 0.8 mutual silence delay (ms) 0 800 1600 experiment simulation dataset 0.00 0.03 0.06 0.09 0.12 0.15 double talk 0.0 0.1 0.2 0.3 0.4 0.5 speaker a 0.0 0.1 0.2 0.3 0.4 0.5 speaker b figure 5: comparison of states probabilities mutual silence, double talk, speaker a and speaker b between the empirical and simulated rnv conversations at 0, 800, and 1600 ms delay. experiment simulation dataset 0 2 4 6 8 10 12 unintended interruptions per minute delay 0 800 1600 figure 6: unintended interruption rate for empirical and simulated sct conversations at 0, 800, and 1600 ms delay. while the state probabilities for double talk stay almost constant for sct and rnv empirical conversations, it increases strongly in the simulations. it also stagnates at 1600ms delay for rnv conversations (figure 5 ). the mismatch in double talk in conversations without delay might stem from errors in the end-of-turn prediction, where the model is too pessimistic in the prediction of the end of an utterance. in general, the simulations seem to have less variance in almost all metrics (state probabilities, speaker alternation rate, interruption rates). one reason for that might be the limited amount of possible utterances that are available in the simulation, resulting in less variance. figure 6 shows the unintended interruption rate (uir), i.e., the interruptions that are caused by de-lay and were not intended by the interrupting participant. while the increase in uir is visible for empirical as well as simulated conversations, the number of unintended interruptions in the simulations is generally higher. ## conclusion in this work, we modeled human turn-taking based on the distribution of turn-switches and -pauses. we applied this model in a conversation simulation. we evaluated how well the interactivity of real-world conversations with distinct levels of interactivity and different transmission delay can be modeled with this approach. the simulated conversations show the distinction between the interactivity of rnv and sct scenarios as well as differences in speaker alternations and interruptions when introducing transmission delay. however, the influence of delay on turn-taking in the simulations seems to saturate with high delay levels. this might hint to a change in turn-taking behavior when large amounts of delay are present. in future work, we plan to identify the changes in turn-taking behavior and model them based on events in the conversation (e.g., continued interruptions). we also plan to evaluate the proposed turn-taking model for the use in spoken dialogue systems.