An Open-Domain Avatar Chatbot by Exploiting a Large Language Model (2023)

Abstract: With the ambition to create avatars capable of human-level casual conversation, we developed an open-domain avatar chatbot, situated in a virtual reality environment, that employs a large language model (LLM). Introducing the LLM posed several challenges for multimodal integration, such as developing techniques to align diverse outputs and avatar control, as well as addressing the issue of slow generation speed. To address these challenges, we integrated various external modules into our system. Our system is based on the award-winning model from the Dialogue System Live Competition 5. Through this work, we hope to stimulate discussions within the research community about the potential and challenges of multimodal dialogue systems enhanced with LLMs.

## introduction we present a demonstration of an open-domain avatar dialogue system that we have developed, with the goal of facilitating natural, human-like conversations. with the advent of large language model (llm) technologies such as llama @xcite and @xcite , the fluency of text-based dialogue systems has significantly improved. one of the next directions in this field involves dialogue systems that utilize voice, facial expressions, and gestures through an avatar, contributing to a more engaging and interactive conversation experience @xcite . as part of the efforts in dialogue system research, the dialogue system live competition 5 (dslc5) was held in japan, a competition of avatar chat dialogue systems @xcite . it was hosted within the academic conference of dialogue systems, where a large number of researchers evaluate the demonstrations performed on the stage to determine their ranking. we developed a dialogue system based on the llm for this competition @xcite , and encountered a variety of challenges on integrating llms into a multimodal dialogue system. for instance, due to the real-time nature of spoken dialogue, it is essential to return some form of response quickly, which is a challenge when using computationally intensive llms. furthermore, when the system involves an avatar, methods of controlling the avatar's facial expressions and motions present another challenge. we strive to address such missing capabilities of an llm-based dialogue system by integrating several external modules. such modules encompass the incorporation of filler phrases and thinking motions during llm's computational time, task parallelization to speed up responses, and detection of errors in speech recognition. simultaneously, we paid close attention to the content of dialogue, aspiring to create a system that allows users to engage in deep, prolonged, and safe interactions. as a result, our system achieved the best human evaluation results in the competition. however, among the metrics, the naturalness of avatar's speaking style received the lowest score, indicating a need for improvement on its motions and expressions. in this demonstration, we present an avatar dialogue system that improves from dslc5. the improvement includes addition of emotion recognizer to enhance naturalness of the avatar expressions and motions. additionally, aiming to provide a more immersive dialogue experience, we offer a system that allows conversation with an avatar through a virtual reality (vr) headset. although the system is originally designed to respond in japanese, we provide translated responses for english speakers. through this demo, we hope to stimulate discussion within the research community about the potential and challenges of integrating llms into multimodal dialogue systems. ## system overview we first provide a overview of the features of our proposed system, followed by a more detailed explanation in the subsequent subsections. the system architecture is illustrated in figure 2 . initially, the user's vocal input is transcribed into text utilizing a speech-to-text (stt) module. as the stt results frequently lack punctuation, a module called the punctuationizer is utilized to append period marks and question marks. the punctuated text is then fed into the dialogue system. the dialogue system leverages an llm to generate responses. to elicit more engaging responses from the llm, a process termed prompt creation is performed beforehand, providing the model with contextually rich information. following the response generation, an editing & filtering phase is undertaken. during this phase, any responses that are dull or ethically inappropriate are identified and either edited or discarded as necessary. once the response text is determined, the system proceeds to control the avatar and text-to-speech system (tts). the avatar's motions and expressions are decided based on the outcome of the emotion analyzer. to help accurately reading kanji characters of japanese, we add pronunciation helper to convert into the phonetic script of hiragana. ## evaluation we present the results of our original system obtained during the final round of dslc5. it is important to note that the avatar and tts used in the competition were provided by the organizers, and thus differ from the demonstration presented here. furthermore, the evaluated system does not incorporate avatar control based on the emotion analyzer, so the evaluation results should be considered as reference values. for the llm, we utilized hyper-clova 82b @xcite , which was trained on a japanese corpus. in the competition, the system engaged in a fourminute casual conversation in japanese with an avatar about two topics, after which the dialogue was evaluated through a post-conversation survey. the survey was conducted based on four aspects: "natural: whether the dialogue was natural", "follow: whether the system could respond appropriately to the topic", "providing: whether the system could provide new information related to the topic", and "style: whether the way of speaking, demonstrated through voice, gestures, and facial expressions, was natural". these evaluation criteria were set by the competition organizers as metrics for assessing "human-level conversational ability". a five-point likert scale was used for the evaluation. however, the style score is notably lower than the other metrics, indicating the need for further enhancements in terms of avatar motions. we expect improvements with the emotion analyzer, as indicated by the displayed output results in the emotion labels are accurately assigned, suggesting that they can be applied effectively to the avatar's facial expressions and motions. ## conclusion in conclusion, we developed an open-domain avatar chatbot in a vr environment, leveraging a large language model (llm). while encountering challenges in multimodal integration, such as addressing slow generation speed and controlling the avatar, our system demonstrated promising results in dialogue system live competition 5. addition-ally, we attempted to improve the unnaturalness in the avatar's style of speaking, which was discovered after the competition, by using the emotion analyzer. we anticipate that this work will initiate meaningful discussions among the research community regarding the potential and challenges of integrating llm into multimodal dialogue systems.