A Visually-Aware Conversational Robot Receptionist (2022)

Abstract: Socially Assistive Robots (SARs) have the potential to play an increasingly important role in a variety of contexts including healthcare, but most existing systems have very limited interactive capabilities. We will demonstrate a robot receptionist that not only supports task-based and social dialogue via natural spoken conversation but is also capable of visually grounded dialogue; able to perceive and discuss the shared physical environment (e.g. helping users to locate personal belongings or objects of interest). Task-based dialogues include check-in, navigation and FAQs about facilities, alongside social features such as chit-chat, access to the latest news and a quiz game to play while waiting. We also show how visual context (objects and their spatial relations) can be combined with linguistic representations of dialogue context, to support visual dialogue and question answering. We will demonstrate the system on a humanoid ARI robot, which is being deployed in a hospital reception area.

## introduction socially assistive robots (sars) are increasingly being explored in contexts ranging from education @xcite to healthcare @xcite . it has been noted, however, that despite the success of sars and spoken dialogue systems in their respective research fields, integration of the two is still rare @xcite figure 1 : interacting with spring-ari 2021) and social robots in general still lack interaction capabilities @xcite . in a similar fashion, even recent research on combining vision and language has tended to centre around the use of still images @xcite , with few systems able to support visual dialogue as part of a natural, situated conversation. the spring project aims to develop such a system in the form of a robot receptionist for visitors to an eldercare outpatient hospital. in this context the robot must be able to communicate naturally with users on a variety of both functional and social topics, including but not limited to those concerning the shared physical environment. we demonstrate our progress towards this goal, with a multi-modal conversational ai system that is integrated on an ari robot foot_0 (fig. 1 ) and which combines social asr robot's camera feed server result visual action server panoptic segmentation of the scene tts chat/ command/ query result alana interface nlu query user/ query result asr result/ kb query arbiter ros action servers execution status pnp server plan user command/ execution status planner query/ update kb db dialogue manager task stack post-processor task bots social bots text audio stream on table n e x t to on on magazine cup plant simplified scene graph panoptic segmentation scene graph generation goal message goal parsing scene graph query visual perception system social interaction planner dialogue system figure 2: system architecture. the social interaction planner (blue blocks) interacts with the dialogue system (green blocks) through an arbiter module. the vision module (yellow blocks) is implemented as a ros action server within the planning framework. based on the user's intent, the planner can recruit the vision module to respond to questions about the visual scene. and task-based conversation with visual dialogue regarding navigation and object detection in the shared space. our system greets visitors, supports them to check-in, answers faqs and helps users to locate key facilities and objects. it also offers social support/entertainment in the form of chit-chat, a quiz and access to the latest news. the system architecture (fig. 2 ), is composed of three main modules; a visual perception system, a dialogue system, and a social interaction planner. ## the spring ari robot ari is a humanoid robot, designed for use as a socially assistive companion @xcite . it is 1.65m tall, has a mobile base, a touch-screen on the torso, movable arms and head with lcd eyes that enable gaze behaviour. the version of the robot used here is equipped with several cameras creating a 360º field of view. for audio capture and processing a respeaker mic v2.0 array, with 4 microphones is mounted on the front of the belly. ## example dialogues the example dialogue shown in ## conclusions and future work the proposed system demonstrates how to successfully weave together a wide range of task-based, social, and visually grounded dialogue and physical actions on an sar in a receptionist environment. next steps are to generate the scene graphs automatically by combining data-driven approaches @xcite @xcite with prudent use of refining rules. crucially also, we are working on extending the system to handle multi-party interactions, an active area of research and highly likely to occur in this context. for the demonstration, we will showcase our system on the ari robot, inviting attendees to interact with it and experience all the capabilities of the system described in this paper.