F ur C hat: An Embodied Conversational Agent using LLM s, Combining Open and Closed-Domain Dialogue with Facial Expressions (2023)

Abstract: We demonstrate an embodied conversational agent that can function as a receptionist and generate a mixture of open and closed-domain dialogue along with facial expressions, by using a large language model (LLM) to develop an engaging conversation. We deployed the system onto a Furhat robot, which is highly expressive and capable of using both verbal and nonverbal cues during interaction. The system was designed specifically for the National Robotarium to interact with visitors through natural conversations, providing them with information about the facilities, research, news, upcoming events, etc. The system utilises the state-of-the-art GPT-3.5 model to generate such information along with domain-general conversations and facial expressions based on prompt engineering.

## introduction the progress in robotics and artificial intelligence in recent decades has led to the emergence of robots being utilized beyond their conventional industrial applications. robot receptionists are designed to interact with and assist visitors in various places like offices, hotels, etc. by providing information about the location, services, and facilities. the appropriate use of verbal and non-verbal cues is very important for the robot's interaction with humans @xcite . most research in the field has been mainly focused on developing domainspecific conversation systems, with little exploration into open-domain dialogue for social robots. conventional agents are often rule-based, which means they rely on pre-written commands and keywords that are pre-programmed. this limits the interaction with humans to little or no freedom of choice in answers @xcite . the advancement of large language models (llms) in the past year has brought an exciting revolution in the field of natural language processing. with the development of models like gpt-3.5 foot_0 , we have seen unprecedented progress in tasks such as questionanswering and text summarization @xcite . however, a question remains about how to successfully leverage the capabilities of llms to create systems that can go from closed domain to open, while also considering the embodiment of the system. in this work, we present furchat foot_1 , an embodied conversational agent that utilises the latest advances in llms to create a more natural conversational experience. the system seamlessly combines open and closed-domain dialogues with emotive facial expressions, resulting in an engaging and personalised interaction for users. the system was initially designed and developed to serve as a receptionist for the national robotarium, in continuation of the multi-party interactive model developed by @xcite , and its deployment shows promise in other areas due to the llms versatile capabilities. as a result, the system is not limited to the designated receptionist role, but can also engage in open-domain conversations, thereby enhancing its potential as a multifunctional conversational agent. we demonstrate the proposed conversational system on a furhat robot @xcite which is developed by the swedish firm furhat robotics 3 . with furchat, we demonstrate the possibility of llms for creating a more natural and intuitive conversation with robots. ## furhat robot furhat is a social robot created by furhat robotics. to interact with humans naturally and intuitively, the robot employs advanced conversational ai and expressive facial expressions. a three-dimensional mask that mimics a human face is projected with an animated face using a microprojector @xcite . a motorised platform supports the robot's neck and head, allowing the platform's head to spin and node. to identify and react to human speech, it has a microphone array and speakers. due to the human-like appearance of furhat, it is prone to the uncanny valley effect @xcite . ## system architecture as shown in figure 2 , the system architecture represents a conversational system that enables users to interact with a robot through spoken language. the system involves multiple components, including automatic speech recognition (asr) for converting user speech to text, natural language understanding (nlu) for processing and interpreting the text, a dialogue manager (dm) for managing the interaction flow, and natural language generation (nlg) powered by gpt-3.5 for generating natural sounding responses @xcite . the generated text is then converted back to speech using text-to-speech (tts) technology and played through the robot's speaker to complete the interaction loop. the system relies on a database to retrieve relevant data based on the user's intent. ## conclusions and future work we demonstrate furchat, an embodied conversational agent with open and closed domain dialogue generation and facial expressions generated through llms, on a social robot in a receptionist environment. the system is developed by integrating the state-of-the-art gpt-3.5 model on top of the furhat sdk. the proposed system uses a one-to-one interaction method of communication with the visitors. we plan on extending the system to handle multi-party interaction @xcite @xcite , which is an active research topic in developing receptionist robots. it is also crucial to address the issue of hallucination from the large language model and this problem can be mitigated by fine-tuning the language model and directly generating conversations from it without relying on any nlu components which we plan to implement in the future. we plan to showcase the system on the furhat robot during the sigdial conference to all the attendees and show them the capabilities of using llms for dialogue and facial expression generation as described in this paper.