An Attentive Listening System with Android ERICA : Comparison of Autonomous and WOZ Interactions (2020)

Abstract: We describe an attentive listening system for the autonomous android robot ERICA. The proposed system generates several types of listener responses: backchannels, repeats, elaborating questions, assessments, generic sentimental responses, and generic responses. In this paper, we report a subjective experiment with 20 elderly people. First, we evaluated each system utterance excluding backchannels and generic responses, in an offline manner. It was found that most of the system utterances were linguistically appropriate, and they elicited positive reactions from the subjects. Furthermore, 58.2% of the responses were acknowledged as being appropriate listener responses. We also compared the proposed system with a WOZ system where a human operator was operating the robot. From the subjective evaluation, the proposed system achieved comparable scores in basic skills of attentive listening such as encouragement to talk, focused on the talk, and actively listening. It was also found that there is still a gap between the system and the WOZ for more sophisticated skills such as dialogue understanding, showing interest, and empathy towards the user.

## introduction in recent years, android robots have drawn much attention from researchers and the public. their realistic appearance is their main feature, though this requires that their behaviors are also humanlike. in particular, a conversational android should not only respond correctly in terms of their dialogue content, but also exhibit phenomena such as backchanneling and correct turn taking which are present in human-human conversation. their use as an interface for natural conversation makes them an attractive prospect for research. since an android which can engage in free, unstructured conversation on any topic is still a long way off, we investigate a more limited task domain. in this paper we investigate attentive listening, and propose such a system for the android erica @xcite , who has been used for tasks such as job interviews @xcite and to investigate various conversational phenomena @xcite . the extension of erica's abilities to attentive listening draws from our previous research @xcite @xcite . in attentive listening, much of the talk is from the user. the system may interject to stimulate further conversation, but does not engage in deep discussions. the advantage of this task is that the user can theoretically talk about any topic without the system needing any deep background knowledge. such robots are useful in areas such as elderly care, where users often desire social contact but may be isolated from family @xcite @xcite . in this case, an android which provides companionship can improve the mental and emotional well-being of the elderly. this domain provides several technical challenges. the main requirement for attentive listening is that erica be seen as actively listening to the conversation. the system must be able to extract the correct topic or keyword and then generate a coherent response which can stimulate further conversation, by using a variety of responses. furthermore, while the user speaks, erica should exhibit human-like listening behavior which may not necessarily be verbal. synchronizing all these features into an autonomous system is a non-trivial task, as we wish to avoid breakdowns in the conversation. this system draws together speech recognition, natural language processing and conversational behavior. our goal is for erica to be as human-like as possible in her interactions with users. we com- control amount microphone array depth camera (kinect v2) sound source localization automatic speech recognition erica text-to -speech actuator control lip motion generation speaker dialogue manager speech signal human position tracking system utterance voice activity detection speech enhancement prosodic info. extraction sound source direction human position multi-channel speech signal enhanced speech asr result prosodic info. motion info. the main contribution of this paper is a fully autonomous android attentive listener. we also report our user study which compares it to a woz system. the outcomes of this study will be used to guide further work in the domain of conversational androids. ## attentive listening system we now describe the attentive listening system for the android robot erica. the whole architecture of the system is illustrated in figure 1 . first, we explain the speech processing module as the input. we then explain how to generate listener responses, followed by other necessary conversational components such as turn-taking and speech synthesis. a dialogue example can be found in appendix a. note that although the following processing is implemented in the japanese language, the fundamental ideas are language-independent. ## dialogue experiment we conducted a dialogue experiment to evaluate how the proposed system works with elderly people as subjects. we also investigated how the system compared when compared to attentive listening with a woz operator. ## conclusion in this work, we described the implementation of an attentive listening system for the android erica. we discussed details of the system including how it generates various response types based on the user's utterance. furthermore, we conducted a user study to investigate the performance of the system compared to one operated by a woz operator. we found that the proposed system could match the woz system in terms of perceived basic listening skills, but was outperformed by the human for more sophisticated skills such as displaying empathy.