A multi-party attentive listening robot which stimulates involvement from side participants (2021)

Abstract: We demonstrate the moderating abilities of a multi-party attentive listening robot system when multiple people are speaking in turns. Our conventional one-on-one attentive listening system generates listener responses such as backchannels, repeats, elaborating questions, and assessments. In this paper, additional robot responses that stimulate a listening user (side participant) to become more involved in the dialogue are proposed. The additional responses elicit assessments and questions from the side participant, making the dialogue more empathetic and lively.

## introduction one of the expected dialogue tasks for spoken dialogue systems is attentive listening, which is when an automated system carefully listens to the user and then generates a response. this task has been found to be useful for elderly people living alone who desire social interaction. we have so far developed an attentive listening dialogue system using an autonomous android erica @xcite that is capable of generating listener responses such as backchannels (e.g., "yeah"), repeats of focus words, elaborating questions, and assessments (e.g., "that is nice"). although the previous system was designed for one-on-one dialogue, in this demonstration, the system is extended to the multi-party scenario, which has previously been considered in other applications such as quiz games @xcite , meetings @xcite , and discussions @xcite . in our situation, the system attentively acts as the moderator that listens to dialogue from multiple people in turn, as shown in figure 1 . this group attentive listening scenario has been found to be relatively common in elderly care facilities. in this scenario, the behaviors of the main speaker and the main listener (system) are important, and the involvement of other listeners (side participants) is also important to make the dialogue more lively. as shown in figure 1 , the side participants are people who can participate in the dialogue but are not being addressed by the current speaker @xcite . in this scenario, the side participants can either be silent while the main speaker talks or can express their reactions towards the main speaker. in the latter case, it is expected that the main speaker will feel that he/she is listened to and understood more and also feel empathy from others. therefore, in multi-party attentive listening, the system needs to act as a moderator to involve the side participants in the dialogue. to promote the involvement of the side participants, this paper proposes a new type of attentive listening system utterances called involvementstimulating utterances. specifically, when the system is ready to give an assessment such as "that is nice" towards the current speaker, it can now also says "that is nice, isn't it?" aimed at one of the side participants. it is then expected that the target side participant would give an assessment and be involved in the dialogue. with more persons involved in the dialogue, the overall dialogue session is more activated and fruitful. another advantage of this new type of utterance is that the system can elicit human assistance when backchannel prediction long silence detection language processing backchannels backup questions generic responses assessments elaborating questions repeats generic sentimental responses focus word detection sentiment analysis select priority high low asr result prosodic info. it is difficult for it to generate a proper assessment utterance by itself. therefore, it can be said that this paper demonstrates cooperation between the system and users in the multi-party dialogue. first, the basic attentive listening system @xcite used in this study is explained. as illustrated in figure 2 , the system generates listener responses such as backchannels, assessments, elaborating questions, repeats, and generic responses, with the speech enhancement and automatic speech recognition implemented through a 16-channel microphone array. a smooth turn-taking function is also realized through a machine-learning-based turn-taking model. this study extends the system to a multi-party scenario in which there is more than one user and each user tells a story to the group in turn. we made a dialogue flow for the system acting as both the moderator and the main listener. the system first designates the main speaker from the participants and begins to attentively listen to this speaker. when a fixed time period has passed, the system promotes the speaker to stop talking and asks a second participant to start talking. this process is applied to all participants in turn, and after all participants end their individual talks, the dialogue finishes. ## eliciting assessments from side participants in the previous one-on-one attentive listening system, the assessment responses such as "that is nice" had been generated on the basis of sentiment analysis (positive, negative, or neutral) using sentiment word dictionaries @xcite . the assessment responses have been used to express empathy towards the speaker, which is an important role in attentive listening tasks. in this work, during each of the user talks, the system attempts to use the proposed involvementstimulating utterance to elicit an assessment from the side participants and then to make the dialogue more empathetic. as illustrated in figure 3 , when the proposed system stimulates involvement from the side participant, the assessment response is now "oh, that is nice, isn't it?" as the involvementstimulating utterance. as the system is designed to work with robots such as android erica, the eyegaze (head) direction can be controlled to shift it to the target side participant to indicate the addressee for the involvement-stimulating utterance. if the system does not use an involvement-stimulating utterance, it only utters a conventional assessment such as "that is nice". in the following, a dialogue example is given (with the original japanese sentences), where s is the system; u1 is a user who is the main speaker; u2 is another user who is the side participant. the bolded parts identify the type of listener response and the underline marks the involvementstimulating utterance. u1: last year, i went to a park in kyoto. ## eliciting questions from side participants another type of involvement-stimulating utterance has been implemented using focus words that were originally used for repeats and elaborating questions in the attentive listening system. during the dialogue, the system detects and stores the focus words of user utterances, and when the main speaker is silent for a longer period (e.g. 5 seconds), the system requests the side participant to ask a question using the focus words. a dialogue example is given in the following, in which s is the system, u1 is the main speaker, and u2 is the side participant. the bolded parts identify the type of listener response and the underline marks the involvement-stimulating utterance and also the focus word. u1: last year, i went to kyoto. ## conclusions this paper demonstrated a multi-party attentive listening system that generates involvementstimulating utterances to better involve side participants and express listener responses, which made the dialogue livelier and more empathetic. future research will be focused on conducting a dialogue experiment to confirm the effectiveness of the proposed system with real users.