Multi-party Multimodal Conversations Between Patients, Their Companions, and a Social Robot in a Hospital Memory Clinic (2024)

Abstract: We have deployed an LLM-based spoken dialogue system in a real hospital. The ARI social robot embodies our system, which patients and their companions can have multi-party conversations with together. In order to enable this multi-party ability, multimodality is critical. Our system, therefore, receives speech and video as input, and generates both speech and gestures (arm, head, and eye movements). In this paper, we describe our complex setting and the architecture of our dialogue system. Each component is detailed, and a video of the full system is available with the appropriate components highlighted in real-time. Our system decides when it should take its turn, generates human-like clarification requests when the patient pauses mid-utterance, answers in-domain questions (grounding to the in-prompt knowledge), and responds appropriately to out-of-domain requests (like generating jokes or quizzes). This latter feature is particularly remarkable as real patients often utter unexpected sentences that could not be handled previously.

## introduction both commercial and research spoken dialogue systems (sdss), conversational agents, and social robots have been designed with a focus on dyadic interactions. that is, a two-party conversation between one individual user and a single system/robot. these are only guaranteed in specific settings, like people interacting with siri on their own phone, or with amazon alexa in single-occupant homes. when alexa is in a family home, their lack of multiparty capabilities are apparent @xcite , but this becomes a critical limitation when deploying social robots in public spaces. families visit museums and libraries, groups of friends roam shopping malls and bars, and couples travel through airports and support each other at hospital appointments. social robots are being deployed and tested in all of these settings @xcite figure 1 : hospital memory clinic visitors using our sds on the ari social robot @xcite . @xcite @xcite @xcite , in which multi-party conversations (mpcs), involving people talking to both the robot and each other, do commonly occur (see figure 1 ). tasks that are typically trivial in the dyadic setting become considerably more complex when conversing with multiple users @xcite : (1) the speaker is no longer simply the other person, so the meaning of the dialogue depends on recognising who said each utterance (see (a) in both dyadic and multi-party human conversations are subtly guided and supported by visual cues @xcite @xcite . screwing-up of the face, brow furrows, looking up, nodding, smiling, eyecontact, etc... though crucial, are lost completely by current commercial sdss. due to the added complexity of mpcs, visual cues are even more crucial @xcite . for example, it is ambiguous who u1 is addressing in example (c) in in this paper, we present our multi-party multimodal sds embodied by the ari social robot @xcite ) that is currently deployed in a hospital, and interacts with memory clinic patients and their companions. it can give directions, provide light entertainment (like quizzes and jokes), and inform people about bus times, the cafe menu, and more. large language models (llms) have revolutionised our field, they are excellent at language understanding, and this includes mpcs @xcite @xcite as their pre-training includes scripts and meeting transcripts containing multiple people. they also hold a wealth of general knowledge, enabling abilities like question answering (qa), joke telling, and playing quizzes. our sds is therefore llm-based to provide a state-of-the-art experience for hospital patients. we first describe our setting, and then detail each module of our system's architecture in figure 2 . a demo video of this system is available on youtube foot_0 . ## the hospital setting dementia diagnosis is a stressful process. patients typically spend entire days at the hospital with a friend or family member for support. the hours are filled with multiple appointments, but a large portion of the day is also spent waiting anxiously for test results or the next appointment. our goal is to provide a system that is both practically useful, but also entertaining, to provide participants with some light distraction from their otherwise stressful day. the research staff at the hospital are our collaborators on the spring project, and they run the experiments with volunteer patients, their companions, and the ari robot (see figure 1 ). the eu's h2020 spring project aims to explore "how to create robots able to move, see, hear and communicate with several actors, in complex and unstructured populated spaces" foot_1 . we are one of eight project partners, and our focus is the sds. other partners work on collision prevention during navigation, route planning, ego-noise suppression, gaze tracking, running live experiments with patients in the hospital, and more. ## dialogue system our system presented in this paper has been iteratively improved through regular user tests and interviews with patients visiting the hospital memory clinic. the initial system @xcite was developed before the recent llm advance, relying on a 'traditional' modular architecture based upon alana v2 @xcite . as patients were usually accompanied by a companion, the lack of multi-party capabilities proved problematic. it interrupted users as it responded to every turn, not allowing them to talk to each other at any point. we therefore designed and ran a multi-party data collection in a wizard-of-oz setup @xcite , and have used this data to motivate and evaluate the system we present here. not only is this new system multi-party and multimodal, it improves qa accuracy, improves accessibility to people with dementia @xcite , and enables added functionality. where previously, we had to specifically design the system to tell jokes and run entertaining quizzes @xcite , llms can now handle this inherently due to their world knowledge. most importantly, both users and the hospital staff have reported that the user experience has improved drastically. in this section, we detail each system component illustrated in figure 2 . ## conclusions and future work we have iteratively developed and deployed a multimodal, multi-party spoken dialogue system in a hospital memory clinic. this sds is embodied by the ari social robot, allowing us to generate gestures in addition to speech. using data collected with real memory clinic patients in this complex setting, our system is able to decide when to take its turn, generate natural clarification requests (im-proving accessibility for people with memory impairment), answer in-domain questions grounded to our domain specific knowledge, and respond appropriately to out-of-domain requests like generating jokes, quizzes, and general chit-chat. we are currently running further data collection in the hospital with the llm-based sds. using this data, we will further refine our system and curate corpora that will be released to allow other researchers to work on this complex, yet vital task.