Retico: An incremental framework for spoken dialogue systems (2020)

Abstract: In this paper we present the newest version of retico - a python-based incremental dialogue framework to create state-of-the-art spoken dialogue systems and simulations. Retico provides a range of incremental modules that are based on services like Google ASR, Google TTS and Rasa NLU. Incremental networks can be created either in code or with a graphical user interface. In this demo we present three use cases that are implemented in retico: a spoken translation tool that translates speech in real-time, a conversation simulation that models turn-taking and a spoken dialogue restaurant information service.

## introduction classical architectures of spoken dialogue systems rely on a pipeline approach, where data is passed through and transformed by a set of modules. these modules perform a specific task on the data, for example, convert speech signals into text (asr modules) or extracting domain-specific information from text (nlu modules). while this architecture separates the concern between the modules and modularizes the development of spoken dialogue systems, the resulting agents are slow to process data and cannot react quickly to changes in the input. incremental processing, an architecture where modules work on small increments of data and forward hypotheses based on those increments to the next module, increases the processing speed and reactiveness of dialogue systems while still retaining the modularized approach of the pipeline architecture @xcite . however, due to the overhead of creating and revoking hypotheses and processing on incomplete data, the complexity of each module in an incremental dialogue system increases. for researchers, it can be a challenge to implement and evaluate incremental modules, as they do not have the time and knowledge to implement a complete incremental dialogue system, just to evaluate the part they are researching. the incremental processing toolkit (inprotk) is an open-source toolkit written in java that provides an interface for incremental modules and defines an architecture for incremental units, hypothesis handling, and connections between incremental modules @xcite . however, the toolkit does not provide an integrated framework that allows for the design and evaluation of networks. in this paper, we present the current version of retico, an incremental framework for spoken dialogue that was first published in @xcite . retico is a framework written in python and published as an open-source project foot_0 . we demonstrate three types of speech and dialogue systems that are implemented in this framework. first, we showcase an incremental translation service that utilizes google translate to recognize, translate, and synthesize speech. also, we showcase a simulation of a conversation with turntaking, where two agents interact with each other. finally, we showcase a spoken dialogue system in the restaurant information domain. all demo systems are visualized in a graphical user interface, and the networks can be adjusted live (e.g., speech synthesis modules can be switched). ## related work the incremental model has been formalized by @xcite @xcite . the resulting framework inprotk @xcite has been used for incremental speech recognition and syn-thesis and dialogue systems, among others. based on this, inprotk s extends the toolkit for the use of situated dialogue @xcite . recent work in modules of spoken dialogue systems like speech recognition @xcite and end-of-turn prediction @xcite focused on incremental processing, and a state-of-the-art natural language understanding module has been incrementalized @xcite . incremental processing cannot only be used in spoken dialogue system, but it also can be useful for research regarding conversation simulation @xcite . ## architecture the architecture of retico is written in python based on the conceptual model of incremental processing described in @xcite . core of this framework are the abstract definitions of an incremental module (im) and an incremental unit (iu). both definitions provide interfaces and processing routines to handle concurrent processing of modules and the flow of ius between the modules. each im has a left buffer, where ius of other modules are placed to be processed and a right buffer where new hypotheses are placed and sent to ims further down the incremental pipeline. usually, an im defines one or more types of iu that it is able to process and one type of iu it produces and produces or revokes hypotheses based on every incoming unit. besides these modules, retico provides interfaces for information exchange apart from ius by changing meta-information of ims and by calling "trigger" modules, that produce ius on-demand and insert them to the buffers of modules. ## demonstrations in this section, we present three different projects that are created entirely in retico. due to the modular approach of retico, these systems are not fixed regarding the modules they use for a given task. for example, rectico is able to use two different speech synthesis modules that can be interchanged. ## conclusion in this paper, we presented the newest version of retico, a framework for incremental dialogue processing. we described the incremental architecture and highlighted the logging and persistence features as well as the graphical user interface. we also showcased three application ideas created with the framework, that span a wide range of possible speech dialogue systems. we described a service that translates speech in increments, a conversation simulation that is able to perform turn-taking, and a dialogue system that processes increments to decrease the time used to query a database. while we focus on applications in the area of spoken dialogue, the incremental approach of this framework can be applied to other areas of research as well. for example, modules for video input and object detection can be used to reference positions of objects in the dialogue, and robotics features may be integrated so that a dialogue system can interact with its environment. the framework is published as open source and available online at https://www.github.com/uhlo/retico .