Response-conditioned Turn-taking Prediction (2023)

Abstract: Previous approaches to turn-taking and response generation in conversational systems have treated it as a two-stage process: First, the end of a turn is detected (based on conversation history), then the system generates an appropriate response. Humans, however, do not take the turn just because it is likely, but also consider whether what they want to say fits the position. In this paper, we present a model (an extension of TurnGPT) that conditions the end-of-turn prediction on both conversation history and what the next speaker wants to say. We found that our model consistently outperforms the baseline model in a variety of metrics. The improvement is most prominent in two scenarios where turn predictions can be ambiguous solely from the conversation history: 1) when the current utterance contains a statement followed by a question; 2) when the end of the current utterance semantically matches the response. Treating the turn-prediction and response-ranking as a one-stage process, our findings suggest that our model can be used as an incremental response ranker, which can be applied in various settings.

## introduction a fundamental component of spoken dialogue system (sds) is turn-taking, i.e., the decision of when to take turns at appropriate places, without causing long response delays or interrupting the user. in other words, the system must be able to correctly identify when the user is yielding the turn, and it is appropriate to make a response, and when the user is simply making a mid-utterance pause @xcite . traditionally, this has been done using a simple silence threshold. however, silence is not a very good indicator of turn-shifts and more modern approaches instead use various cues known to be important in human-human turn-taking, such as lexico-syntactic cues, prosody, or gaze @xcite @xcite . ekstedt and skantze (2020) proposed turngpt, a transformer-based language model that incrementally processes words in the user's utterance and predicts the probability of a turn-shift after each word. this is similar to the notion of syntactic or pragmatic completion points that have been identified in conversation analysis @xcite . in their analysis of turngpt, @xcite found that the 20% of the model's attention is directed towards utterances earlier than the current one, indicating that it is sensitive to pragmatic aspects of dialogue. while such models are indeed a step forward, there is a still an important component missing that we will address in this paper. when humans make a decision to take the turn, it is not just based on whether there are enough turn-yielding cues in the interlocutor's utterance. @xcite use the notion of transition-relevant places, or trp, for places where a transition could potentially take place (but does not have to). thus, many places for turn-shifts are highly optional. to partly address this problem, @xcite annotated the willingness of the next speaker to take the turn, and built a model that could predict this willingness based on multimodal cues. whether a turn-shift takes place or not also depends on the intention of the next speaker, and what they want to say. for dialogue systems, this means that the system should not automatically take the turn once the transition-probability passes a certain threshold, and only then decide what it should respond. instead, the system should take the potential response into account when deciding whether it is appropriate to take the turn or not. we call this response-conditioned turn-taking prediction, illustrated in figure 1 . we present a model called rc-turngpt, which is an extension of turngpt. note that the current study does not intend to address how and when the next speaker comes up with what they would like to say. this depends of course on the exact implementation of the dialogue system, which could for example be response-ranking @xcite or an intentbased planning approach @xcite . in figure 1 , we have assumed that a response ranker is used. if so, a traditional system would first use a model like turngpt to decide when to take the turn, and then ask the response ranker which response would fit best. in such a setting, it might be the case that none of the candidates would be a good fit from the system's perspective, but the system would produce a response anyway. in such a setting, rc-turngpt could instead be used to incrementally rank or score potential responses to see whether they fit well from a turn-taking perspective, or pass taking the turn if none of them has a high enough utility. in this paper, we take a first step towards such an approach, and investigate to what extent and under what scenarios such response-conditioning would help to predict turn-shifts. similar to turngpt, we do not model acoustic information, as our focus is to investigate how the semantic and pragmatic aspects of the dialogue affect turn-shift prediction. instead, we use written dialogues as a stand-in for audio for incremental end-of-turn prediction. we leave the incorporation of acoustic information (cf. ekstedt and skantze 2022) for future work. ## methods turngpt is a unidirectional transformer-based language model (lm) optimized through crossentropy to predict the next token in a sequence. it is a pre-trained gpt-2 (base) model @xcite , finetuned on unpunctuated dialogue corpora, with a special turn-shift token (ts) that delimits consecutive turns. rc-turngpt is an extension of this model, by also conditioning the prediction on the response. while the rc-turngpt model is architecturally equivalent to turngpt, it differs in the training objective through a simple data transformation. this transformation permutes the ordering of turns in a similar approach as the fim pre-training objective of @xcite . we consider turn-based dialogue sequences to consist of three parts: the context/history (h), the current utterance (cu) and the next response (r). the task is to correctly predict the location of the turn-shift token in the current utterance, cu i , given the history, h i , and the next response, r i , over all samples i in the dataset, d. the samples @xmath0. however, instead of the uniform left-to-right next token prediction task of regular lms, the rc-turngpt model train on ordered sequences of {r, h, cu}, masking the loss over r and h to solely learn over the cu turns. this enables the model to use information of both h and r while keeping the original left-to-right next token prediction setup. finally, the turngpt model utilized three special tokens in addition to the original gpt-2 vocabulary, the aforementioned ts token and two speaker tokens. the speaker tokens are similar to positional embeddings and are added to the word embeddings to encode the speaker identity over each word. because of the permuted ordering of the rc-turngpt setup we also include a fourth special response-token that are added to the words of the response to distinguish them from the actual context. both the base model and the datasets were implemented using huggingface @xcite .