When can I Speak? Predicting initiation points for spoken dialogue agents (2022)

Abstract: Current spoken dialogue systems initiate their turns after a long period of silence (700-1000ms), which leads to little real-time feedback, sluggish responses, and an overall stilted conversational flow. Humans typically respond within 200ms and successfully predicting initiation points in advance would allow spoken dialogue agents to do the same. In this work, we predict the lead-time to initiation using prosodic features from a pre-trained speech representation model (wav2vec 1.0) operating on user audio and word features from a pre-trained language model (GPT-2) operating on incremental transcriptions. To evaluate errors, we propose two metrics w.r.t. predicted and true lead times. We train and evaluate the models on the Switchboard Corpus and find that our method outperforms features from prior work on both metrics and vastly outperforms the common approach of waiting for 700ms of silence.

## introduction spoken dialogue agents have exploded in popular use (e.g., alexa, siri, and google home). however, they only support explicit turn-taking mechanisms: they detect user initiation and barge-ins using wake-words and identify end of user turns based on a silence period (typically between 700-1000ms). turn-taking feels unnatural under such mechanisms, leading to less "conversational" interactions @xcite . this is particularly damaging for open-ended social conversations where thoughtful silences get wrongly interrupted @xcite . to fix this issue, we predict initiation opportunities for spoken dialogue agents for both turn-taking and backchanneling. prior work predicting initiation points uses prosodic features like pitch and frequency variation with bag-of-embeddings to predict backchannels @xcite and turn-completion @xcite , and more recently, ekstedt and skantze figure 1 : humans produce overlapping speech with small gaps. by predicting lead to initiation, virtual agents can respond without long waiting periods (2021) finetuned gpt-2 on dialogue datasets to predict turn-completion using only word features. however, they either predict a binary label indicating initiation in a wide event horizon, which is imprecise; or they predict a binary label for an initiation to happen at a set offset in the future, in which case a single incorrect prediction leads to a missed initiation. as a robust generalization of previous approaches, we predict the lead time to initiation as a continuous value. we model initiation (next utterance from a different speaker) directly and not end-of-turn because there is a variable (and possibly negative) gap between the two @xcite . in this work, we combine two models: wav2vec 1.0 @xcite for representing prosodic features and finetuned gpt-2 @xcite for word features. we model the task with a gaussian mixture model (gmm) to account for inherent uncertainty. we train and evaluate our models on switchboard @xcite and find that the combination of the pretrained models performs the best, vastly outperforming a silence-based baseline that waits for 700ms of silence and baselines using features from prior work. ## related work prior work for dialogue turn-taking either uses silent gaps as cues or predicts future events repeatedly. a key issue with systems that use silent gaps as initiation cues @xcite @xcite is the difficulty of adjusting the silence thresholds to accommodate dialogue states @xcite . when predicting turn-taking repeatedly, i.e. predicting future actions at every timestep, acoustic features such as pitch and frequency are often used, with additional linguistic features including part-of-speech or word embeddings @xcite @xcite . more recently, @xcite implement a spoken dialogue system for travel conversations using turngpt @xcite . however, a short silence threshold is still used to determine initiation of agent responses. outside of dialogue, @xcite propose probabilistic models for predicting events in videos, @xcite forecast frames and @xcite forecast actions. time-to-event analysis in the medical domain involves modeling patient status as a function of time @xcite . ## experiments for training and evaluation, we use audio conversations from switchboard @xcite . we select a random set of 200 training, 20 validation, and 20 test dialogues out of a total of 1000 dialogues due to computational constraints. we use the validation set to select the best performing checkpoint based on mmae scores and report the numbers on the test set. for the gmm models, we experimented with @xmath0, 5, 10, 15, 20, and found @xmath1. 1 we plot the mae-pred and mae-true values in figure 3 and the show the mmae values in our models outperform the silence baseline by a large margin in most time windows prior to and ## conclusion we present the task of lead time to initiation prediction as a continuous-valued problem, collapsing transition and backchannel timing problems into one. we additionally propose metrics to capture precision and coverage in these predictions. our models trained on pretrained prosodic and verbal embeddings consistently outperform the commonly-used silence baseline. we believe our work will build a foundation for more naturalistic virtual agents with human-like conversational behaviors.