T urn GPT : a Transformer-based Language Model for Predicting Turn-taking in Spoken Dialog (2020)

Abstract: Syntactic and pragmatic completeness is known to be important for turn-taking prediction, but so far machine learning models of turn-taking have used such linguistic information in a limited way. In this paper, we introduce TurnGPT, a transformer-based language model for predicting turn-shifts in spoken dialog. The model has been trained and evaluated on a variety of written and spoken dialog datasets. We show that the model outperforms two baselines used in prior work. We also report on an ablation study, as well as attention and gradient analyses, which show that the model is able to utilize the dialog context and pragmatic completeness for turn-taking prediction. Finally, we explore the model’s potential in not only detecting, but also projecting, turn-completions.

## introduction the taking of turns is one of the most fundamental aspects of dialog. since it is difficult to speak and listen at the same time, the participants need to coordinate who is currently speaking and when the next speaker can start. traditionally, spoken dialog systems have rested on a very simplistic model of turn-taking, where a certain amount of silence (e.g. 700ms) is used as an indicator that the turn is complete. this often results in interruptions or sluggish responses, depending on where the threshold is set. in human-human interaction, it is clear that much more sophisticated mechanisms are used, where the speakers rely on turn-taking cues (involving prosody and linguistic cues, as well as gaze and gestures) to detect, and even project, turn completions @xcite @xcite . more sophisticated models of turn-taking, based on machine learning, have been proposed @xcite @xcite . typically, these models rely on the various multi-modal features that have been found to facilitate the coordination of turn-taking. since dialog is primarily driven by the exchange of meaningful contributions, where each contribution often constitutes some dialog act, linguistic information should intuitively play a major role in turn-taking. however, so far, the representations of linguistic features have been fairly simplistic, and some models rely solely on prosody @xcite . one explanation for this is that the complex semantic and pragmatic functions that the "linguistic cues" should reflect, and which can be expected to regulate turntaking, are non-trivial for machine learning models to capture, especially since they often depend on the preceding dialog context. in this paper, we introduce turngpt, a transformer-based language model for turn-taking prediction. based on open ai's gpt-2 @xcite , and fine-tuned on various dialog datasets, it predicts possible turn-completion points in dialog, based on linguistic features (words) alone. transformer-based language models have been shown to perform well on several nlp tasks @xcite . recent developments in chatbots have also shown that they can produce meaningful utterances in dialog, and thus seem to have a fairly strong representation of the dialog context @xcite . through ablation studies and model inspection, we analyse how important the linguistic context is for turn-taking prediction. we evaluate the model using both written and spoken dialog datasets. however, as this paper is focused solely on modelling the linguistic aspect of turn-taking, we do not investigate the contribution of other important features, such as prosody, and leave the combination of such cues with our model for future work. thus, our baselines are the linguistic parts of turn-taking models proposed in previous work. ## background one of the most influential early accounts of the organization of turn-taking is the one proposed by @xcite . their model is based on the observation that since the dialog is not known in advance, it has to be coordinated in a flexible manner as it evolves. overwhelmingly, one speaker talks at a time; occurrences of more than one speaker at a time are common, but brief. transitions (from one turn to the next) with very little gap and no overlap are common. based on these observations, they propose that turns can be constructed from "turn-constructional units" (tcu). after each such unit, there is a "transition-relevant place" (trp), where a turn-shift can (but does not have to) occur, depending on whether the current speaker actively selects the next speaker, or if some other speaker self-selects. several studies have investigated the cues that could be used by the listener to distinguish trps ("turn-yielding cues") from non-trps ("turnholding cues") @xcite . for example, in a face-to-face setting, speakers tend to not look at the listener during an utterance, but then shift the gaze towards the addressee when yielding the turn @xcite . several studies have also investigated prosodic cues for turn-taking, including intonation, duration, loudness and voice quality @xcite . from a linguistic perspective, the notion of "completeness" is important, as a complete linguistic unit (such as a sentence) is more likely to be turn-yielding than an incomplete sentence or phrase. @xcite analysed linguistic units for turn-taking and proposed two levels of units: syntactic and pragmatic. syntactic completion, in this context, does not have to be a complete sentence. neither is a syntactic phrase (like a nominal phrase) necessarily syntactically complete. they define an utterance to be syntactically complete if "in its discourse context, it could be interpreted as a complete clause, that is, with an overt or directly recoverable predicate" (p. 143). this includes "elliptical clauses, answers to questions, and backchannel responses". the syntactic completion is judged incrementally as the utterance unfolds. figure 1 shows a (made-up) example which illustrates this notion. as can be seen, in this account, the turn-initial adverb of time "yesterday" is not syntactically complete (as there is a: yesterday we met / in the park / b: okay / when / will you meet / again / a: tomorrow / not yet any "overt or directly recoverable predicate"), whereas "tomorrow" is, which illustrates the dependence on the dialog context. as pointed out by @xcite , while syntactic completion might be necessary for a trp, it is not sufficient. thus, they also introduce the notion of pragmatic completeness, which is defined as "a complete conversational action within its specific sequential context" (p. 150), and corresponds to trps. this definition is not very precise, and is likely to depend on a fair amount of common sense. in the example above, while "when will you meet" is syntactically complete, the question is unlikely to end there, given the preceding context, and is therefore not pragmatically complete. in their analysis, @xcite also argue that the final intonation contour plays an important role in signalling pragmatic completion, where these may be ambiguous. this has also been verified in controlled experiments @xcite . however, as pointed out by several researchers @xcite , turn-final prosody cannot (by itself) explain the majority of split-second turn-shifts (around 200ms) that are typically found in data, as the listener would not have time to react, prepare and execute a response. the response time would then be around 600-1500ms @xcite . thus, the listener is likely to prepare the response ahead of time and project the turn-completion. for this, they most likely depend on units which are more feasible to project, such as syntactic and pragmatic units. even though syntactic and pragmatic completeness are intuitively important for turn-taking, it is not clear how they should be modelled. so far, most prediction models of turn-shifts have used a very simplistic account of syntactic completion, such as the final part-of-speech tags @xcite @xcite . more recent models of turntaking have used lstms to encode linguistic information, such as part-of-speech @xcite , words @xcite or senones @xcite . although several of these studies have found that linguistic information contribute to the performance (compared to only using prosody), the performance gain is not as big as what could be expected. this calls for the exploration of more powerful linguistic models for turn-taking. ## approach a problem when modelling trps is that they are not overtly present in the data, only actual turnshifts are. one approach could be to manually annotate trps (cf. @xcite , but this is of course very labour intensive. one could also question the binary notion of trps -a continuous (or probabilistic) notion seems to be more plausible, where transition-relevance varies between highly inappropriate to highly appropriate @xcite . in this view, a strong trp should be statistically associated with more turn-shifts. thus, a probabilistic notion of trps should be possible to infer from actual turnshifts in data, just like a language model (the probability of a word in context) can be inferred from actual language use. given this notion, we include turn-shifts as specific tokens in the vocabulary of a language model and learn their distribution, along with the other tokens, over conversational data in a language model setup. we focus on dialog data and include two separate turn-shift tokens for each of the speakers, which are inserted at the beginning of each speaker turn. a dialog is then a sequence of turns separated by these turn-shift tokens. after training, the probabilities associated to the turn-shift tokens can be viewed as the probability of a trp. note, however, that the model not only predicts turn-shifts, but makes predictions over all tokens in the vocabulary, thus retaining its function as a language model. the problem of organizing turn-taking primarily concerns spoken language, where response time and fluency has a big impact on the quality of the interaction. however, the process of recording and transcribing spoken dialog is expensive and time consuming. there are also privacy issues regarding recorded speech, which makes audio data less accessible than their written counterpart. since our focus in this paper is on linguistic aspects of turn-taking, we investigate the use of both written and spoken dialog data. although the language use is different for spoken vs. written language, we believe that pragmatic trps exist and overlap (to some extent) for both types. a clear difference, however, is that spoken language lack punctuation and capitalization, which are not typically available for spoken dialog systems (unless inferred by a transcriber or asr). our goal is to learn the distributions over trps using linguistic data, without the need to rely on punctuation or capitalization. ## model we use a transformer-based @xcite , uni-directional language model: the gpt-2 (radford et al., 2019) from openai. transformer models have made a huge impact on nlp research over the past years and was chosen because of their strong performance on language generation. our model can be seen as a modified version of the transfertransfo @xcite model, which performed well in the convai2 foot_0 challenge. in their work, they fine-tuned a gpt @xcite model on a particular dialog task with the addition of three tokens, one task-specific and one for each speaker. transformer-based language models commonly use at least two types of embeddings, a word and a positional embedding. the word embedding encodes the relevant words and the positional encodes their order. transfertransfo used an additional dialog state embedding consisting of the task specific token and a speaker token for each location, corresponding to the relevant speaker. training was done using cross-entropy loss and a next-sentence prediction loss. in our work, we omit the task-specific token and the next sentence prediction loss. turngpt is a gpt2-based transformer using three kinds of embeddings: word, position and speaker id. the speaker tokens are included in the language modelling task and the trp probability predictions are defined as the maximum assigned output probability over the speaker tokens. please refer to the code foot_1 for further details. we finetune two different pre-trained models, namely gpt-2 @xcite trained on webtext, and dialogpt @xcite by microsoft, which is based on gpt-2 but "trained on 147m conversation-like exchanges extracted from reddit comments". we used the pretrained models available from the transformers @xcite ) library using pytorch @xcite . for our experiments, we only used the smallest models (the gpt-2-base and the dialogpt-small), both with 12 layers, 12 heads and 768 hidden units. we compare turngpt against two baselines which correspond to linguistic models which have been used in previous research (as reviewed above). first, we train a simple statistical model on part-ofspeech (pos) bigrams. for each pair of consecutive pos tags, we get an associated probability of a speaker shift. second, we train an lstm model @xcite with up to three layers with a hidden size of 768. the lstm baseline is trained directly as a binary turn-shift classifier, given the preceding sequence of words. ## data we collect eight dialog datasets with varying characteristics, which we have grouped into three major categories. the first, and largest, group (called assistant) are task-oriented dialog system corpora, which represent dialog between a user and an automated assistant (where the user typically queries the assistant for information). these datasets were primarily collected through wizard-of-oz (woz) and self-written dialog (i.e., where one person is writing an imagined dialog), through a crowdsourcing platform: the second group of datasets (called written social) contains human-human written dialogs that are more open and social in nature: the third type of collected data is that of spontaneous spoken dialog between two humans: ## experiment 6.1 training we trained the models on both the assistant and full training sets using the cross-entropy loss. the models with the lowest validation loss were then used for testing. the turngpt models used the adamw optimizer and the default hyperparameters of the transformer library. the lstm baseline used the same tokens provided by the gpt-2 model but trained on the binary prediction of the next token being a turn-shift or not, using a sigmoidal output activation on the mean squared error loss. the lstm model utilized the adamw optimizer included in pytorch, with a weight decay of 0.01, dropout of 0.1, and a learning rate of 6.25e-5. we used up to three hidden layers for the lstm and chose the one that performed best on the validation sets, which was the 2-layer lstm. ## conclusions and discussion in this paper, we presented a model for turn-shift prediction by formulating the problem as a language modelling task. we introduced turngpt, a model which is a finetuned gpt-2 transformer imbued with special turn-shift tokens. the model performed better than baselines used in previous work. through an ablation study and model inspection, we showed that this is partly thanks to the strong representation of context that prior models lack, i.e., the model's ability to identify pragmatic (and not just syntactic) completion. we also showcased the model's ability to generate possible futures as a way of predicting upcoming turn-shifts. as we are addressing spoken dialog, this work should be seen as an important step towards a more powerful turn-taking model that takes both linguistic information, as well as prosody and other cues (such as gaze and gestures) into account. as argued in the linguistic literature @xcite , prosodic information can be important to further disambiguate pragmatic completion. however, we argue that previous models that have combined linguistic and prosodic cues (cf. @xcite ; roddy et al. 2018; masumura et al. 2019) have used too simplistic models of linguistic turn-constructional units. the integration of prosodic information with a model like turngpt is an important topic for future work. turngpt could also be interesting not just from a dialog system perspective; further model inspection and ablation studies could also be used to identify more exactly how certain words contribute to turn-completion predictions. this can potentially give insights into how humans manage to coordinate their turn-taking in spoken interaction with each other.