rr SDS : Towards a Robot-ready Spoken Dialogue System (2020)

Abstract: Spoken interaction with a physical robot requires a dialogue system that is modular, multimodal, distributive, incremental and temporally aligned. In this demo paper, we make significant contributions towards fulfilling these requirements by expanding upon the ReTiCo incremental framework. We outline the incremental and multimodal modules and how their computation can be distributed. We demonstrate the power and flexibility of our robot-ready spoken dialogue system to be integrated with almost any robot.

## introduction spoken dialogue systems (sdss) are well-suited to handle complex artifacts of dialogue such as hesitations and clarification requests in many domains. however, to further extend sdss to work effectively on physical robots, we offer the following additional requirements towards a robot-ready sds: modular: robot components are modular and individual modules must be able to integrate with sds modules, multimodal: robots are situated dialogue partners whose many sensors must be integrated with the sds speech input, distributive: robot and sds modules should easily communicate with each other in a distributed environment, incremental: modules must be able to process input quickly and immediately, aligned: sensors must be temporally aligned, i.e., synchronized in time. existing systems offer solutions to some of the requirements. the opendial toolkit gives researchers the ability to model dialogue states using probabilistic rules @xcite , but any incrementality has not been systematically evaluated. inprotk @xcite , is incremental and inprotk s @xcite added distributiveness and multimodality, and @xcite offered an ap-proach to temporal alignment, albeit with offline evaluation. the psi framework is inherently modular, multimodal, temporally aligned, has been evaluated on robot platforms, and has several options for distributing computation @xcite . however, the psi framework does not yet build on any incremental framework. also similar to our work is the platform multibot presented in @xcite , but that model does not work incrementally nor does it consider temporal alignment. in this paper, we design and evaluate a modular, incremental, multimodal, and distributive robotready sds, called rr sds which is primarily written in the python programming language. foot_0 to address the requirements of modularity and incrementality, we adopt the incremental unit framework @xcite where incremental units (ius) are passed between modules (ius can be added to reflect new information, or revoked if a previously added iu needs to be updated) by building on the retico @xcite platform. to handle distributiveness, rr sds has modules (i.e., zeromq, ros) that afford interopability with processes outside of the system. to address the requirement of multimodality, we build on top of the modularity requirement and incorporate additional sensors (e.g., cameras and internal robot states). ## the rr sds spoken dialogue system retico has existing modules for built-in microphones and google speech api for speech recognition. we extend it to be multimodal by adding additional sensor modules such as cameras and internal robot states, depicted in figure 1 . we add distributive modules that handle interopability with outside modules. the rest of this section explains the modules for rr sds. dialogue management opendial is a toolkit for developing sdss with probabilistic rules @xcite that can be used as a rule-based dialogue manager (dm), but can be extend any domain to include stochastic processes when data is available. we incorporate a recent python implementation called pyopendial @xcite into our sds as a dm. our pyopendial module takes any iu payload, expecting a list of attributes (i.e., variables) and values that it adds to pyopen-dial's dialogue state as attribute/value pairs. natural language understanding words-as-classifiers (wac) is a model of grounded semantics that learns a 'fitness' score between physical entities and words @xcite , where each word in a known vocabulary is represented as a classifier. wac is inherently incremental and can learn word groundings with minimal training data. this module takes words from asr and features from detected objects. it outputs the best fit word for the detected object as well as confidence scores for all the words in its vocabulary and their fitness to all detected objects. ## evaluation we used the mistyrobotics misty ii and anki cozmo robot platforms to evaluate rr sds, depicted in figures 2 and 3 . we briefly explain the two platforms and the modules we built to integrate them into our rr sds, then we describe the evaluation. ## conclusions & future work our rr sds is flexible, being evaluated on multiple robot platforms to create engaging human-robot interactions, and fulfills the modular, incremental, multimoal, and distributive requirements for a robot-ready sds. our evaluation of rr sds allowed users to successfully interact with two different robots to accomplish a simple task with comparable performance. rr sds is agnostic to the robot platform used, enabling future research to experiment with robot platforms using our flexible system. for future work, we plan to add natural language generation modules and integrate rr sds more directly with psi to make use of its architecture, thereby allowing developers and researchers to make use of psi temporal alignment functionality, but spend most of their development time with python.