Neural Generation of Dialogue Response Timings (2020)

Abstract: The timings of spoken response offsets in human dialogue have been shown to vary based on contextual elements of the dialogue. We propose neural models that simulate the distributions of these response offsets, taking into account the response turn as well as the preceding turn. The models are designed to be integrated into the pipeline of an incremental spoken dialogue system (SDS). We evaluate our models using offline experiments as well as human listening tests. We show that human listeners consider certain response timings to be more natural based on the dialogue context. The introduction of these models into SDS pipelines could increase the perceived naturalness of interactions.

## introduction the components needed for the design of spoken dialogue systems (sdss) that can communicate in a realistic human fashion have seen rapid advancements in recent years (e.g. @xcite ; @xcite ; @xcite ). however, an element of natural spoken conversation that is often overlooked in sds design is the timing of system responses. many turn-taking components for sdss are designed with the objective of avoiding interrupting the user while keeping the lengths of gaps and overlaps as low as possible e.g. @xcite . this approach does not emulate naturalistic response offsets, since in human-human conversation the distributions of response timing offsets have been shown to differ based on the context of the first speaker's turn and the context of the addressee's response @xcite @xcite . it has also been shown that listeners have different anticipations about upcoming 1 our code is available at https://github.com/ mattroddy/rtnets . responses based on the length of a silence before a response @xcite . if we wish to realistically generate offsets distributions in sdss, we need to design response timing models that take into account the context of the user's speech and the upcoming system response. for example, offsets where the first speaker's turn is a backchannel occur in overlap more frequently @xcite . it has also been observed that dispreferred responses (responses that are not in line with the suggested action in the prior turn) are associated with longer delays @xcite . overview we propose a neural model for generating these response timings in sdss (shown in fig. 1 ). the response timing network (rtnet) operates using both acoustic and linguistic features extracted from user and system turns. the two main components are an encoder, which encodes the system response h z , and an inference network, which takes a concatenation of user features (x n ) and h z . rtnet operates within an incremental sds framework @xcite where information about upcoming system responses may be available before the user has finished speaking. rtnet also functions independently of higher-level turn-taking decisions that are traditionally made by the dialogue manager (dm) component. typically, the dm decides when the system should take a turn and also supplies the natural language generation (nlg) component with a semantic representation of the system response (e.g. intents, dialogue acts, or an equivalent neural representation). any of the system response representations that are downstream from the dm's output representation (e.g. lexical or acoustic features) can potentially be used to generate the response encoding. therefore, we assume that the decision for the system to take a turn has already been made by the dm and our objective is to predict (on a frame-by-frame basis) the appropriate time to trigger the system turn. it may be impractical in an incremental framework to generate a full system response and then re-encode it using the response encoder of rt-net. to address this issue, we propose an extension of rtnet that uses a variational autoencoder (vae) @xcite to train an interpretable latent space which can be used to bypass the encoding process at inference-time. this extension (rtnet-vae) allows the benefit of having a data-driven neural representation of response encodings that can be manipulated without the overhead of the encoding process. this representation can be manipulated using vector algebra in a flexible manner by the dm to generate appropriate timings for a given response. our model's architecture is similar to vaes with recurrent encoders and decoders proposed in @xcite ; @xcite ; @xcite . our use of a vae to cluster dialogue acts is similar to the approach used in @xcite . our vector-based representation of dialogue acts takes inspiration from the 'attribute vectors' used in @xcite for learning musical struc-ture representations. our model is also related to continuous turn-taking systems @xcite in that our model is trained to predict future speech behavior on a frame-by-frame basis. the encoder uses a multiscale rnn architecture similar to the one proposed in @xcite to fuse information across modalities. models that intentionally generate responsive overlap have been proposed in @xcite . while other models have also been proposed that generate appropriate response timings for fillers @xcite and backchannels @xcite @xcite . this paper is structured as follows: first, we present how our dataset is structured and our training objective. then, in sections 2.1 and 2.2 we present details of our two models, rtnet and rtnet-vae. section 2.3 presents our input feature representations. in section 2.4 we discuss our training and testing procedures. in sections 3.1 and 3.2 we analyze the performance of both rtnet and rtnet-vae. finally, in section 4 we present the results of a human listener test. training objective our training objective is to predict the start of the system turn one frame ahead of the ground truth start time. the target labels in each turn pair are derived from the ground truth speech activity labels as shown in fig. 2 . each 50 ms frame has a label @xmath0, 1}, which consists of the ground truth voice activity shifted to the left by one frame. as shown in the figure, we only include frames in the span r in our training loss. we define the span r as the frames from the beginning of the last ipu in the user turn to the frame immediately prior to the start of the system turn. we do not predict at earlier frames since we assume that at these mid-turn-pauses the dm has not decided to take a turn yet, either because it expects the user to continue, or it has not formulated one yet. as mentioned previously in section 1, we design rtnet to be abstracted from the turn-taking decisions themselves. if we were to include pauses prior to the turn-final silence, our response generation system would be additionally burdened with making turn-taking decisions, namely, classifying between mid-turn-pauses and end-of-turn silences. we therefore make the modelling assumption that the system's response is formulated at some point during the user's turn-final ipu. to simulate this assumption we sample an index r start from the span of r using a uniform distribution. we then use the reduced set of frames from r start to r end in the calculation of our loss. ## listening tests it has shown that response timings vary based on the semantic content of dialogue responses and the preceding turn @xcite , and that listeners are sensitive to these fluctuations in timing @xcite . however, the question of whether certain response timings within different contexts are considered more realistic than others has not been fully investigated. we design an online listening test to answer two questions: (1) given a preceding turn and a response, are some response timings considered by listeners to be more realistic than others? (2) in cases where listeners are sensitive to the response timing, is our model more likely to generate responses that are considered realistic than a system that generates a modal response time? participants were asked to make a/b choices between two versions of a turn pair, where each version had a different response offset. participants were asked: "which response timing sounds like it was produced in the real conversation?" the turn pairs were drawn from our dataset and were limited to pairs where the response was either dispreferred or a backchannel. we limited the chosen pairs to those with ground truth offsets that were either classified as early or late. we classified offsets as early, modal, or late by segmenting the distribution of all of the offsets in our dataset into three partitions as shown in fig. 11a . the cutoff points for the early and late offsets were estimated using a heuristic where we split the offsets in our dataset into two groups at the mode of the distribution (157 ms) and then used the median values of the upper (+367 ms) and lower (-72 ms) groups as the cutoff points. we selected eight examples of each dialogue act (four early and four late). we generated three different versions of each turn pair: true, modal, and opposite. if the true offset was late, the opposite offset was the mean of the early offsets (-316 ms). if the true offset was early, the opposite offset was the mean of the late offsets (+760 ms). we had 25 participants (15 female, 10 male) who all wore headphones. we performed binomial tests for the significance of a given choice in each question. for the questions in the first half of the test, in which we compared true vs. opposite offsets, 10 of the 16 comparisons were found to be statistically significant (@xmath1.05). in all of the significant cases the true offset was was considered more realistic than the opposite. in reference to our first research question, this result supports the conclusion that some responses are indeed considered to be more realistic than others. for the questions in the second half of the test, in which we compared true vs. modal offsets, six out of the 16 comparisons were found to be statistically significant. of the six significant preferences, three were a preference for the true offset, and three were a preference for the modal offset. to investigate our second research question, we looked at the offset distributions generated by our model for each of the six significant preferences, shown in fig. 11b . for the turn pairs where listeners preferred nonmodal offsets (top row), the distributions generated by our system deviate from the mode into the preferred area (highlighted in yellow). in pairs where listeners preferred modal offsets (bottom row) the generated distributions tend to have a mode near the overall dataset mode (shown in the green line). we can conclude, in reference to our second question, that in instances where listeners are sensitive to response timings it is likely that our system will generate response timings that are more realistic than a system that simply generates the mode of the dataset. ## conclusion in this paper, we have presented models that can be used to generate the turn switch offset distributions of sds system responses. it has been shown in prior studies (e.g. @xcite ) that humans are sensitive to these timings and that they can impact how responses are perceived by a listener. we would argue that they are an important element of producing naturalistic interactions that is often overlooked. with the advent of commercial sds systems that attempt to engage users over extended multi-turn interactions (e.g. @xcite ) generating realistic response behaviors is a potentially desirable addition to the overall experience.