ADVISER : A Toolkit for Developing Multi-modal, Multi-domain and Socially-engaged Conversational Agents (2020)

Abstract: We present ADVISER - an open-source, multi-domain dialog system toolkit that enables the development of multi-modal (incorporating speech, text and vision), socially-engaged (e.g. emotion recognition, engagement level prediction and backchanneling) conversational agents. The final Python-based implementation of our toolkit is flexible, easy to use, and easy to extend not only for technically experienced users, such as machine learning researchers, but also for less technically experienced users, such as linguists or cognitive scientists, thereby providing a flexible platform for collaborative research.

## introduction dialog systems or chatbots, both text-based and multi-modal, have received much attention in recent years, with an increasing number of dialog systems in both industrial contexts such as amazon alexa, apple siri, microsoft cortana, google duplex, xiaoice @xcite and furhat foot_1 , as well as academia such as @xcite and alana @xcite . however, open-source toolkits and frameworks for developing such systems are rare, especially for developing multi-modal systems comprised of speech, text, and vision. most of the existing toolkits are designed for developing dialog systems focused only on core dialog components, with or without the option to access external speech processing services @xcite @xcite @xcite . to the best of our knowledge, there are only two toolkits, proposed in @xcite and @xcite , that support developing dialog agents using multi-modal processing and social signals @xcite . both provide a decent platform for building systems, however, to the best of our knowledge, the former is not open-source, and the latter is based on the .net platform, which could be less convenient for non-technical users such as linguists and cognitive scientists, who play an important role in dialog research. in this paper, we introduce a new version of adviser -previously a text-based, multi-domain dialog system toolkit @xcite -that supports multi-modal dialogs, including speech, text and vision information processing. this provides a new option for building dialog systems that is open-source and python-based for easy use and fast prototyping. the toolkit is designed in such a way that it is modular, flexible, transparent, and user-friendly for both technically experienced and less technically experienced users. furthermore, we add novel features to ad-viser, allowing it to process social signals and to incorporate them into the dialog flow. we believe that these features will be key to developing humanlike dialog systems because it is well-known that social signals, such as emotional states and engagement levels, play an important role in human computer interaction @xcite . however in contrast to open-ended dialog systems @xcite , our toolkit focuses on task-oriented applications @xcite , such as searching for a lecturer at the university @xcite . the purpose we envision for dialog systems developed using our toolkit is not the same as the objective of a social chatbot such as xiaoice @xcite . rather than promoting "an ai companion with an emotional connection to satisfy the human need for communication, affection, and social belonging" @xcite , adviser helps develop dialog systems that support users in efficiently fulfilling concrete goals, while at the same time considering social signals such as emotional states and engagement levels so as to remain friendly and likeable. ## objectives the main objective of this work is to develop a multi-domain dialog system toolkit that allows for multi-modal information processing and that provides different modules for extracting social signals such as emotional states and for integrating them into the decision making process. the toolkit should be easy to use and extend for users of all levels of technical experience, providing a flexible collaborative research platform. ## conclusions we introduce adviser -an open-source, multidomain dialog system toolkit that allows users to easily develop multi-modal and socially-engaged conversational agents. we provide a large variety of functionalities, ranging from speech processing to core dialog system capabilities and social signal processing. with this toolkit, we hope to provide a flexible platform for collaborative research in multi-domain, multi-modal, socially-engaged conversational agents.