It’s About Time: Turn-Entry Timing For Situated Human-Robot Dialogue (2020)

Abstract: Turn-entry timing is an important requirement for conversation, and one that spoken dialogue systems largely fail at. In this paper, we introduce a computational framework based on work from Psycholinguistics, which is aimed at achieving proper turn-taking timing for situated agents. The approach involves incremental processing and lexical prediction of the turn in progress, which allows a situated dialogue system to start its turn and initiate actions earlier than would otherwise be possible. We evaluate the framework by integrating it within a cognitive robotic architecture and testing performance on a corpus of task-oriented human-robot directives. We demonstrate that: 1) the system is superior to a non-incremental system in terms of faster responses, reduced gap between turns, and the ability to perform actions early, 2) the system can time its turn to come in immediately at a transition point or earlier to produce several types of overlap, and 3) the system is robust to various forms of disfluency in the input. Overall, this domain-independent framework can be integrated into various dialogue systems to improve responsiveness, and is a step toward more natural, human-like turn-taking behavior.

## introduction behavioral evidence shows that humans are able to exchange turns extremely quickly in conversation -within a few hundred milliseconds on average @xcite . this is a universal human characteristic, though the nature of the timings varies slightly across languages @xcite . there is some debate about exactly how humans achieve this performance, but evidence from psycholinguistic studies suggests that it is likely done by processing ongoing utterances incrementally and making lexicosyntactic predictions about the turn in progress @xcite . this allows a listener to plan what to say and to anticipate the end of the speaker's turn accurately so that turn-transitions are seamless, and gaps between turns are minimized. it also allows for the production of speech overlap, to produce conversational behaviors such as backchanneling and repair. such human behaviors are desirable for spoken dialogue systems (sdss) where naturalness is a priority @xcite . sds research has produced an impressive body of work on turn-taking (e.g. @xcite ; @xcite ; @xcite ; @xcite ; @xcite ), and some early work on overlap and completions @xcite @xcite . however, relatively little focus has been placed on using turn-taking capabilities for responsive turn-entry timing, especially for situated agents. one exception is the approach by @xcite which involves estimating word duration to produce collaborative completions. we build on this prior work through the development of a framework for achieving responsive turn-entry timing, as well as a full set of adaptive human-like overlap and completion behaviors. our approach involves using utterance-level predictions from partial input and information from a world modeler to determine when to enter the turn (including producing overlap at any of the entry points shown in figure 1 ), and whether to initiate actions early. such capabilities are particularly important for situated dialogue agents, as responses, and especially actions, often involve lengthy processing delays, which can be mitigated by preparing or initiating them during an ongoing turn. section 2 describes how this framework builds on existing research, including our novel turn-entry manager (tem) described in section 2.4. in section 3 we describe implementation details related to integrating the framework in a cognitive robotic architecture. then in section 4 we evaluate our implementation on a corpus of situated human-robot dialogue utterances. finally, we close with a discussion of the contributions and directions for future work. ## a framework for turn-entry timing here we discuss the framework needed to manage turn-entry timing for situated dialogue agents, and the related work that the framework builds on. ## implementation in a cognitive robotic architecture to effectively interact in a situated environment, robots need to react to and affect the environment, as well as to reason about the task and user; this requires a cognitive robotic architecture. we integrated our turn-entry timing framework into the diarc architecture @xcite . we used diarc due to its emphasis on situated robot dialogue (highlighted in gervits et al. ( 2020 )), although in principle our framework is general enough to be used with any architecture of its type. below we discuss each of the key components in our architectural configuration. ## evaluation to evaluate the efficacy of our framework, we used a corpus of directives to a household robot from the s4r dataset of the huric corpus @xcite . the dataset consisted of 96 imperative utterances from a task in which people were algorithm 1 turn-entry manager algorithm the central aim of the evaluation is to show how a situated agent given these instructions can make predictions and respond at the trp compared to a non-incremental baseline system. we also seek to demonstrate the potential for overlap production and preemptive action execution. in addition to the standard directives in the corpus, we also evaluate several variants of them which contain disfluency. it is important that sdss are resilient to disfluency, as it is common in team communication channels (particularly in remote communication) and has been implicated in effective team performance @xcite . including disfluent utterances in the evaluation was done to show that the algorithm can handle variations in the input and still produce timely responses. ## conclusion we have introduced a framework for turn-entry timing in human-robot dialogue which enables a situated agent to make incremental predictions about an ongoing utterance and time its turn to hit a variety of entry points. we implemented the framework in a robotic architecture and evaluated it on a corpus of human-robot directives from a situated interactive task. the system integrating our framework is significantly faster than a non-incremental system, and can produce fluid responses and various types of overlap, as well as execute actions preemptively. moreover, the approach is robust to several forms of disfluency in the input. this framework offers a number of benefits for situated dialogue agents, including better responsiveness, the ability to produce various types of overlap (interjacent, last-item, backchannels, and collaborative completions), and preemptive action execution. these interactive capabilities are a step toward more natural and flexible turn-taking for situated dialogue agents.