Projection of Turn Completion in Incremental Spoken Dialogue Systems (2021)

Abstract: The ability to take turns in a fluent way (i.e., without long response delays or frequent interruptions) is a fundamental aspect of any spoken dialog system. However, practical speech recognition services typically induce a long response delay, as it takes time before the processing of the user’s utterance is complete. There is a considerable amount of research indicating that humans achieve fast response times by projecting what the interlocutor will say and estimating upcoming turn completions. In this work, we implement this mechanism in an incremental spoken dialog system, by using a language model that generates possible futures to project upcoming completion points. In theory, this could make the system more responsive, while still having access to semantic information not yet processed by the speech recognizer. We conduct a small study which indicates that this is a viable approach for practical dialog systems, and that this is a promising direction for future research.

## introduction one of the most fundamental conversational behaviour of any spoken dialog system (sds) is that of turn-taking, i.e., to take turns without long response delays or frequent interruptions @xcite . to achieve this, the system must be able to correctly identify when the user is yielding the turn, and it is appropriate to make a response, and when the user is simply making a mid-utterance pause. in their seminal work, @xcite describe general properties of human-human conversation in which they observe that, overwhelmingly, one speaker talk at a time and the time between consecutive turns (response time) is minimal. for the english language, a typical response time is around 200ms and similar response patterns seem to be consistent across different cultures @xcite . contrary to this, current sdss typically have response delays of around 700-1000ms. the reason for this is that they typically rely solely on this silence to determine when to take the turn, whereas humans also use other cues, such as prosody, gaze and syntactic completeness @xcite . many studies have investigated how to include such features in turn-taking models for sdss @xcite @xcite @xcite @xcite . another difference between human turn-taking and sdss is that humans do not only react to turnyielding cues from the interlocutor. if they were simply waiting for a cue and only then started to formulate a response, psycholinguistic research has estimated that the response time would be around 600-1500ms @xcite , which is substantially slower than the observed response times. this indicates that humans also project turn completions in advance, before the turn is complete @xcite @xcite . in this paper, we investigate whether the human ability to project future turn completions could be a viable option for conversational systems to achieve more fluent turn-taking. we constrain our approach to the textual domain using a pre-trained conversational language model to project future words and turn-completions. the projection of turn-completions in sdss can have a number of applications. for example, the system could initiate a turn just before the end of the user's utterance to minimize response time, or even take the turn with a small overlap. it could also give the system more time to generate a response, or be used to address the problem of processing delays. for example, sdss rely heavily on automatic speech recognition (asr) to extract the text from the user's speech. most asr services are associated with a certain latency @xcite . for turn-taking, this means that even if the system has detected that the user has stopped speaking, it is hard to determine whether the turn is yielded or not, since the final asr result is not complete yet. there has been some previous research on predicting upcoming activity in dialog, such as recognizing nlu intents on incomplete user speech @xcite , projecting prosodic information and timing @xcite as well as estimating future voice activity @xcite @xcite . however, we are not aware of any previous studies of how a sds could predict upcoming words in the user's speech, and use this for managing turn-taking. ## conversational agent for our study, we implemented a sds that performs an interview with a user, talking about past travel memories, similar to @xcite . the reason we chose this domain is that the dialog manager can be implemented in a very simple way, while the turn-taking can be challenging, as pauses within the user's turn might be more frequent than in, for example, a q/a system. an example dialog can be found in appendix a.1. a general first step for modelling responsive turn-taking is to use an incremental dialog architecture, where the user's speech is processed incrementally, so that decisions can be made in a more continuous fashion @xcite . for this study, we build upon the recent retico @xcite ) framework (implemented in python foot_0 ), which implements the general, abstract model of incremental dialog processing proposed by @xcite . the system processes incoming user speech and outputs audio. the incoming incremental audio chunks are processed by a local voice activity detection (vad) component and streamed to a remote incremental asr service (google). the vad triggers on silences of 200ms which defines interpausal units (ipu). a user turn is started when both the vad detects ongoing speech and the asr has provided its first hypothesis. if the vad module activates during an ongoing agent utterance, an interruption component is triggered. this module checks how much of the planned audio has been transmitted and stops the ongoing utterance if less than 80% has been sent. the interrupted utterance is then repeated for the system's next response. if the agent completed an utterance and the user is inactive for 5 seconds, a fallback is triggered and the agent continues the conversation by producing a new utterance. for the simplicity of our experiment, the dialog manager is defined by a set of predetermined questions, where the only possible deviation occurs if the user provides a too short utterance. if such a short utterance is recognized, the system randomly chooses from a set of paraphrased responses that encourages the user to elaborate. in this study, we implement two different turntaking policies: the baseline and the projection model. the baseline defines a user turn as complete once the vad module is inactive and the asr has produced its final hypothesis. ## turn-completion projection model to make projections, we utilize the turngpt model by @xcite , which is a pretrained gpt-2 @xcite language model (lm) fine-tuned on conversational data. the model was trained on data from seven publicly available dialog datasets listed in appendix a.2. the model trained until the validation loss reached a minimum, resulting in an average validation perplexity of 17.6. the model includes special tokens that encode speaker shifts, which we will refer to as turncompletions. as shown by @xcite , the model does not only consider the ongoing user turn, but also benefits from taking the larger dialog context into account (i.e., previous turns by the system and the user). given the currently recognized user words (and the dialog context), a set of n possible continuations (of length m ) are generated (using a temperature τ and topk sampling). the number of those that include turn-completions are counted, which gives a ratio. this ratio then approximates the probability of an "actual" turn-completion point in the near future. if the ratio is larger than a threshold r, the turn is predicted to be complete. in this setup we strive towards simplicity and only trigger a projection at the end of each user ipu. however, if new asr hypotheses are received after this, new projections are made until the system decides to take the turn. the projection model uses a maximum silence threshold t as a fallback, which triggers a response regardless of the projections. these different parameters can potentially be fine-tuned for the specific application (or user). this was not done in our study, and we selected values we found reasonable in preliminary tests, which are shown in ## experiment to evaluate the model, we conducted an experiment over zoom 2 where ten participants had two conversations each with the agent (testing the two turn-taking policies) about two distinct travel memories. the participants were asked to choose a memory prior to each agent interaction. we used two sets of paraphrased questions, assigned randomly between the two policies. after completing 2 https://zoom.us/ both interactions, the participants were asked to annotate the recorded dialogues by labeling moments where they felt they had been interrupted by the system. to do this, they were provided with a graphical tool where they could see the waveforms of the dialogs and play them, as well as inserting labels. the agent interacted directly over zoom by connecting its microphone to the zoom speakers and vice versa. all audio was recorded directly on the agent side, in the same way as in a live setup. the total number of agent shifts (transitions between the user and the agent) was 220 for the baseline and 210 for the projection model. the duration of these (i.e., response times) are shown in the histogram in figure 2 . the average response times were 1.03 and 0.80 seconds for the baseline and projection agent, respectively. while this difference is not very large, it should be noted that the prediction model has a bimodal distribution (as seen in figure 2 ), representing early predicted turn shifts and fallbacks. thus, the model is able to take the turn quickly at some points, while allowing for more time at others. the users annotated 18 of the agent shifts as interruptions for the baseline, and 28 for the projection model. the estimated average cut-in rate, defined as the annotated interruptions divided by the number of agent shifts, was 0.08 for the baseline and 0.13 for the projection model. when evaluating the performance of a turntaking model, both response time and cut-in rate should be taken into account (i.e., both should be minimized) @xcite . however, there is typically also a trade-off between these two factors. since both these values were different between the baseline and prediction model, they are difficult to compare directly. one way of doing that is to perform an analysis of what would happen if we reduce the maximum allowed response time (for the prediction model this is the parameter t ). as we do this, the average response time will also be reduced, while the cut-in rate will increase, since silences in between user ipus longer than t become both additional cutins and agent shifts. the result of this analysis is shown in figure 3 . this analysis enables a direct comparison of the agents over values where both lines are defined. the figure shows that the prediction agent is more responsive and produces less interruptions by the fact that the green line is strictly below the red. the greatest difference occurs at around 0.48s on the x-axis, with a cut-in rate difference of 0.1, given threshold values of 0.5 and 0.6 seconds for the baseline and projection agents, respectively. ## discussion to our knowledge, all previous work on end-ofutterance-detection in sdss have relied on mod-els that are specifically trained with data from the target domain. contrary to this, we have used a generic lm (turngpt) with a set of basic parameters that were not fine-tuned using domain data. if the lm and the parameters would be fine-tuned, we could expect further improvements. an analysis of the perplexity of the lm on the recorded data shows a rather high perplexity (ppl ≈ 80). another obvious improvement would be to also include prosodic features. an important question we have not addressed here is how good the projections are in terms of predicting the last words more exactly (i.e., not just how well the system predicts whether there will be a turn completion). depending on the domain of the system, this might be more or less important. in this respect, the comparison of the baseline and prediction models (presented in figure 3 ), is somewhat unfair to the prediction model, since we could not reduce the response time of the baseline model without also truncating the asr result. the proposed model make turn-completion decisions exclusively in the textual domain, restricted by the latency of the asr, at the end of user ipus. in practice, this means that we are more likely to "project" the already spoken words currently being processed by the asr, as opposed to the actual future activity of the user. this could be mitigated by using a more reactive ipu trigger, increasing the projection events during a user utterance, and to use a longer continuation length, surpassing the latency of the asr. if so, the system could potentially also start to respond before the user has stopped speaking (i.e., producing overlapping speech). another important aspect is that the interactions were all conducted over zoom which introduces added latencies. this also makes the probability of cut-ins even greater than it would have been in a live setup. ## conclusion in conversation, humans project future turncompletion points in order to achieve faster response times. in this paper, we have investigated whether it is possible to implement this ability in a sds. the projections are done in the textual domain by generating future dialog continuations with a conversational lm (turngpt). we conducted a small study and show, as a proof-ofconcept, that this approach is viable. we note that there is room for improvements, such as optimizing the hyperparameters, train and use a task specific lm, project turn-completion at finer increments, and add prosodic features. however, the idea to use a text-based lm to project turn-completions, as a way to improve the turn-taking abilities of a sds, is something we believe will be common and useful for the future of conversational systems.