How Much Does Prosody Help Turn-taking? Investigations using Voice Activity Projection Models (2022)

Abstract: Turn-taking is a fundamental aspect of human communication and can be described as the ability to take turns, project upcoming turn shifts, and supply backchannels at appropriate locations throughout a conversation. In this work, we investigate the role of prosody in turn-taking using the recently proposed Voice Activity Projection model, which incrementally models the upcoming speech activity of the interlocutors in a self-supervised manner, without relying on explicit annotation of turn-taking events, or the explicit modeling of prosodic features. Through manipulation of the speech signal, we investigate how these models implicitly utilize prosodic information. We show that these systems learn to utilize various prosodic aspects of speech both on aggregate quantitative metrics of long-form conversations and on single utterances specifically designed to depend on prosody.

## introduction turn-taking is the fundamental ability of humans to organize spoken interaction, i.e., to coordinate who the current speaker is, in order to avoid the need for interlocutors to listen and speak at the same time @xcite . a dialog can be viewed as a sequence of turns, constructed through the joint activity of turn-taking between the two speakers. a turn refers to segments of activity where a single speaker controls the direction of the dialog. in conversational systems, turn-taking has traditionally been modeled using threshold policies which recognize silences longer than a chosen duration as transition-relevant places. although these types of models are commonly used, it is well known that they are insufficient for modeling human-like turn-taking @xcite . studies of human-human conversation have shown that turns are frequently shifted with a gap of just 200ms @xcite , or even with a slight overlap. thus, given that humans also need some time to prepare a response, it would be infeasible for humans to just use silence as a cue to turn-taking. instead, it has been suggested that they are able to project turn completions already while the other person is speaking @xcite @xcite . in addition, humans produce socalled backchannels (short feedback tokens such as "mhm") in a timely manner, often in overlap with the other speaker @xcite . a common research question in phonetics, psycho-linguistics, and conversational analysis concerns the various cues (including speech, gaze, and gestures) that humans use to detect or project turnshifts @xcite . when it comes to speech, a common distinction is made between the prosodic (non-lexical) and lexical (textual, syntactic, semantic) components of the speech signal. for example, de @xcite argued, based on listening experiments, for the importance of syntactic information over intonation (pitch) in turn-taking, while @xcite showed that intonation is important when syntactic completion is ambiguous. however, such studies often require human listening experiments which are costly, anecdotal, and constrained in time resolution and are therefore limited to small amounts of conversational contexts. an alternative approach is to use computational models @xcite to investigate what type of information they are sensitive to. @xcite recently proposed voice activity projection, vap, which is a general, self-supervised turn-taking model. the model incrementally projects the future speech activity of the two speakers directly from raw audio waveforms. the model can be trained on lots of data, without human annotations, and is agnostic with respect to different types of speech information, as it does not depend on explicitly extracted features. this makes the vap model potentially suitable as a data-driven approach for investigating the role of prosody in turn-taking. in this work, we train vap models on a large dataset @xcite of dyadic spoken interactions and evaluate it on specific turn-taking metrics, while perturbing the input audio to omit certain sources of prosodic information. we analyze the performance over different tasks to investigate three research questions: ## background prosody refers to the non-verbal aspects of speech, including intonation (f0/pitch contour), intensity (energy), and duration (of phones and silences). it has been found to serve many important functions in conversation, including prominence, syntactic disambiguation, attitudinal reactions, uncertainty, topic shifts, and turn-taking @xcite . studies on both english and japanese have found that level intonation (in the middle of the speaker's fundamental frequency range) tends to serve as a turnholding cue, whereas either rising or falling pitch can be found in turn-yielding contexts @xcite @xcite . when it comes to intensity, studies have found that speakers tend to lower their voices when approaching potential turn boundaries, whereas turn-internal pauses have a higher intensity @xcite . regarding duration and speaking rate, duncan (1972) found a "drawl on the final syllable or on the stressed syllable of a terminal clause" to be a turn-yielding cue (in english). this is also in line with the findings of @xcite . when it comes to lexical information, a very strong cue to turn-taking is of course whether the utterance is syntactically or pragmatically complete @xcite . thus, even if prosodic cues can be found near the end of a turn-shift, it is not clear to what extent such cues provide additional information compared to lexical cues, or if they are redundant. in an experiment by de @xcite , subjects were asked to listen to a conversation and press a button when they anticipated a turn ending. the speech signal was manipulated to either flatten the intonational contour, or to remove lexical information by low-pass filtering. the results showed that the absence of intonational information did not reduce the subjects' prediction performance significantly, but that their performance deteriorated significantly in the absence of lexical information. from this, they concluded that lexical information is crucial for end-of-turn prediction, but that intonational information is neither necessary nor sufficient. ekstedt and skantze (2020) also found that it is possible to build fairly reliable turn-taking models using only lexical information. however, it has also been argued that while lexical information is important for turn-taking, there are many cases where a phrase may be syntactically complete, but it is unclear whether the turn is in fact yielded or not @xcite . to investigate this, @xcite performed a similar experiment as de @xcite , but selected the stimuli so that they contained several syntactic completion points (e.g. "are you a student / at this university?"), and where the intonation phrase boundary provided additional cues to whether the turn was yielded or not. they found that subjects indeed made better predictions with the help of intonation and duration. most previous attempts at modeling prosody in turn-taking have been limited in that they (i) only use instances of mutual silence for predicting turn shifts (and therefore do not model projection of turn completion), and (ii) only use fairly superficial, hand-crafted features, such as the extracted pitch slope or pitch level right before the pause (e.g., @xcite . apart from the problem that such features might be too simplistic, they also typically require speaker normalization of the pitch @xcite . in this work, we investigate various forms of turn-taking events (including projection of both turn shifts and backchannels). we also use a more agnostic modeling approach, using latent speech representations that are learned in a self-supervised manner and extracted from the raw waveform @xcite . if our model is indeed able to pick up relevant prosodic information from these representations, it means that we do not have to do any special prosodic feature engineering or speaker normalization. ## voice activity projection model ekstedt and skantze (2022) proposed a generic turn-taking model that does not predict specific turn-taking events at specific moments in time. instead, the model is given the task of voice activity projection (vap), which means that it has to incrementally predict the future voice activity (va) of each interlocutor in a dialog. the prediction target at each incremental step is defined by a window of 2 seconds containing the future va for both speakers. the window is discretized into 8 separate bins (4 for each speaker) where each bin is assigned a value of one if more than half of its frames are active, to produce an 8 bit binary digit, corresponding to 256 unique classes. the vap model consists of an encoder that processes raw audio waveforms, along with the current va information, to produce latent representations of a defined frame frequency f enc hz which are then fed into the predictor network. the predictor is a causal sequence network that processes the context available up until the current frame and outputs a probability distribution over the 256 va classes, see figure 1 . the encoder consists of two sub-modules, a speech module which processes raw waveforms, x, specifically a cpc @xcite model that outputs frame-level representations h speech,@xmath0, at f enc hz. a second va module, matching the frame rate of the speech encoder, processes the current va frame vector v f @xmath1, 1} 2 , along with a concise representation of the va history. the va history features provide long-ranging contextual information outside of the receptive field of the acoustic model. this history is defined as the activity ratio of speaker a over speaker b for regions of size {-inf:60, 60:30, 30:10, 10:5, 5:0} seconds into the past, where 0 is the current time step, resulting in a vector v h @xmath2, for each frame. the va module projects the va features to vectors h va,t , h his,@xmath3,t to produce the encoder output h t , for each frame t. the dialog input waveforms are volume normalized, resampled to 16khz, mixed to a single channel and split into 10s segments (using a 1s overlap). the predictor consists of a causal, decoder only, transformer @xcite , with linear attention @xcite , using a hidden size of 256, 4 layers, 8 heads, and 0.1 dropout. the output the input to the model is the combined speech waveforms of the two speakers (x t ), the va frames of the window (v f t ), and the longer va history (v h t ). the waveform and va features are processed separately, projected to a common feature space, and added together to produce the predictor input, h t . the predictor consists of a causal transformer feeding into the vap-head to produce the output projection. the green box illustrates the various outputs of the different models that we compare. source: @xcite of the transformer model is fed to the vap head, a final linear layer, which outputs logits associated with the 256 va classes. since transformer models are powerful but come with the cost that they scale quadratically in compute, with respect to input length, we are interested in whether using a slower frame rate of the sequence model has any significant impact on the turn-taking performance. following previous work, we utilize a pre-trained cpc @xcite ) encoder which produces output representations at 100hz, and for two of our three models, we include a single additional convolutional layer which projects the representations to 50 and 20hz. in other words, we train three models which use different frame rates of the predictor. ## training and data we train three different vap models with different frame-level frequencies: 20, 50, and 100hz. we use the combination of two dyadic conversational datasets, switchboard @xcite and fisher foot_0 @xcite , resulting in 8288 unique dialogs. we set aside a test set of 5% (of each dataset) and split the remaining dialogs into a 90/10 train/validation split used for training. we use the adamw @xcite optimizer and an early stopping criteria on the validation loss with a patience of 10 epochs. the code is implemented in python using the pytorch @xcite , pytorch-lightning @xcite and wandb @xcite libraries, and are publicly available foot_1 . ## utterance-level analysis while the analysis above gives an overall estimate of how important prosody is, it has been hypothesized that prosody is especially important when the semantic/pragmatic completion is ambiguous, as discussed in section 2. to focus their analysis on such situations, @xcite constructed question templates where a short and a long version, sharing initial lexical information, were recorded through scripted interviews (in dutch). as an example, a short/long question pair "did you drive here?" and "did you drive here this morning?" contain the same initial words up to a common completion point (after the word "here"), which we will refer to as the short completion point, scp. note that in order for the listener (or the model) to predict a turn-shift towards the end of the short utterance, but not at the corresponding place in the long utterance, it has to rely on prosody. through listening experiments, where the participants are asked to press a button when they expect a turn shift, @xcite found that the reaction time was indeed much faster after the short version than after a long version cut after the scp. for our experiments, we created a similar set of 9 long/short utterance pairs in english (see as can be seen in the figure, for this example, the model correctly assigns a high probability to hold until towards the end of each utterance, where it changes to shift. this clearly illustrates the model's ability to project turn shifts before the utterance is complete, and before the large rise in final pitch has actually happened. in addition, we see how the model makes a clear distinction between the two utterances at the short completion point (scp), where it predicts a hold for the longer variant. this illustrates that the model is indeed sensitive to prosody, as that is the only information that is different up until that point. additional the model output on the long version of the phrase "are you a student here at this university?", for the various perturbations, is visualized in figure 5 . inspection of the original performance in figure 5a indicates that the model is sensitive to prosodic information and assigns a higher likelihood of a hold at the scp located on the word "student". however, for the f0 flat perturbation, in figure 5b , we note that the model flips and as-signs a higher shift-probability at the scp, which indicates that if the dynamics of the f0 contour is omitted, the model cannot recognize that the speaker will continue to speak. interestingly, the intensity flat perturbation also affects the output of the model, but after the scp is completed. here, the model does have access to the f0 contour and correctly assigns a larger hold-probability at the scp, but then changes prediction to indicate that a shift is probable following the word "here". as a final note, the low pass transform, which filters out all phonetic information while keeping both the intensity and f0 contour, does produce predictions close to that of the original audio, while being slightly less certain of a shift after the entire utterance is completed, as seen in figure 5e . we also provide the corresponding visualizations over the short version of the same speaker and phrase in figure 7 in the appendix. to get an aggregate evaluation of the model across all 9 phrases and 10 voices, we define three regions in each utterance, up until the scp point (for both long and short phrases), namely hold, predictive and reactive, and measure the average shift probability predicted by the model in those regions. the hold region covers the start of the utterances until 200ms before the scp, where the predictive region begins. the final reactive region is the very last frame of the scp where the entire last word (of the short utterance) has been processed. over the long utterances, the model should consistently predict a low shift probability, given that the speaker will continue their turn, while the shift probabilities should increase over the regions of the short utterances. the aggregate model performance over all phrases is visualized in figure 6 . the left part of figure 6 displays the average shift probabilities for the points on the scp for the short phrases (short@scp) which preferably should start low and rise consistently. the right part of the figure shows the corresponding performance but on the long phrases (long@scp) and should be consistently low, indicating that the speaker will continue their turn. looking at the non-perturbed signal (original), and comparing the left and right figures, we see that the model is indeed sensitive to prosody, confirming the anecdotal observation from figure 3 . the low pass transform clearly hinders the model from predicting a shift, indicating that pitch and intensity in themselves are not enough. among the other perturbations, f0 flat seems to have the largest negative effect, which confirms that intonation is important for disambiguating turn completion when lexical information is not enough. duration seems to be less important, which aligns with the observation in figure 4c . ## conclusion and discussion in this work, we train general computational models of turn-taking, provide analytical methods suitable for evaluating their performance on turn-taking tasks, and investigate how they utilize prosodic information in the speech signal. we investigate the models' reliance on prosody by extending psycho-linguistic experiments designed to measure the effect of prosody on turn-taking in human subjects. we conclude by addressing our three research questions below. do voice activity projection models trained on raw waveforms learn to pick up prosodic information that is relevant to turn-taking? we apply specific prosodic perturbations to the input signal and show a deterioration across all models on the tasks of turn-taking and backchannel prediction, indicating that prosodic cues are utilized by the models. we note that phonetic information has the largest impact on these measures and that f0 information is less important for turn-taking in general. even more convincing are perhaps the specific comparisons of the models' ability to predict shift vs hold at syntactic completion points, where the lexical information is identical. this task requires access to the prosodic dynamics of the signal and should be impossible to distinguish based on lexical information alone. when/how is prosodic information important for turn-taking predictions? overall, we show that all models are most sensitive to the low-pass transform, indicating that phonetic information is important for turn-taking in general. we note that intensity is at least as important as pitch when applied to actual human long-form conversations, but that pitch plays a more important role for the disambiguation at syntactically equivalent completion points. interestingly, we note that the importance of duration plays a less important role, indicating that the f0-contour is the most reliable cue in the presence of lexical ambiguity. another interesting observation in figure 6 is that even if intonation seems to be the most important individual cue, flattening it does not completely collapse the distinction between turn-holding and turn-yielding. thus, there must also be redundant information in intensity and/or duration. this shows that prosody is indeed a complex set of signals, which the model has captured. what is a suitable time resolution for such models to best represent prosody? in our analysis of the turn-taking metrics, we note a negligible performance degradation when decreasing the frame rate of the predictor model. we note that highfrequency models tend to focus more on phonetic information, indicated by their sensitivity to the low pass transformation. the faster models seem more sensitive to general acoustic artifacts, as indicated by the larger performance drop on the f0 shift perturbation, which should not have an impact on turn-taking cues in general. overall, we favor the slower models given their lower memory and computational requirements, their robustness, and comparable performance. it should be noted that the models were not trained on perturbed versions of the data, which include highly unnatural speech (i.e., no humans speak with a perfect flat intonation contour). thus, the evaluations of section 6 can be considered outof-distribution. nevertheless, it is interesting that for many of these perturbations, the models still perform relatively well. also, the drops in performance are typically in line with what could be expected from the literature. for future work, it could be valuable to train multiple models, on data with different prosodic perturbations, and compare their performance for further analysis. another interesting approach could be to identify actual instances of syntactically ambiguous phrases, rather than relying on tts. moreover, it would be interesting to include a larger linguistic context, and investigate whether the importance of prosody decreases.