Let’s Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversation (2024)

Abstract: In this paper, we introduce a novel Face-to-Face spoken dialogue model. It processes audio-visual speech from user input and generates audio-visual speech as the response, marking the initial step towards creating an avatar chatbot system without relying on intermediate text. To this end, we newly introduce MultiDialog, the first large-scale multimodal (i.e, audio and visual) spoken dialogue corpus containing 340 hours of approximately 9,000 dialogues, recorded based on the open domain dialogue dataset, TopicalChat. The MultiDialog contains parallel audio-visual recordings of conversation partners acting according to the given script with emotion annotations, which we expect to open up research opportunities in multimodal synthesis. Our Face-to-Face spoken dialogue model incorporates a textually pretrained large language model and adapts it into the audio-visual spoken dialogue domain by incorporating speech-text joint pretraining. Through extensive experiments, we validate the effectiveness of our model in facilitating a face-to-face conversation. Demo and data are available at https://multidialog.github.io and https://huggingface.co/datasets/IVLLab/MultiDialog, respectively.

## introduction spoken dialogue system (sds), often referred to as a conversational agent, engages in natural speech conversations with humans by recognizing speech from user input and providing contextually appropriate and accurate responses with speech. with spoken language as the primary interface, it has numerous applications for human-computer interactions such as customer service and voice assistants. however, when people communicate face-toface, we utilize not only audio but also visual information of the conversing partner to process spoken words and non-verbal cues (i.e., facial expressions, gestures, and emotions) @xcite . this multimodal information enhances understanding of the speech content and the speaker's intent. furthermore, having a visual counterpart to audio can simulate a real face-toface conversation experience, making the user feel more connected and engaged. in this paper, we explore an audio-visual spoken dialogue system to facilitate direct face-to-face conversation for the first time. central to the development of dialogue systems is the large amount of high-quality dialogue data. current dialogue systems are predominantly text-based, driven by the abundance of text dialogue datasets @xcite @xcite @xcite @xcite @xcite . recently, several audio dialogue datasets have been released @xcite @xcite which augment existing text dialogue data @xcite with speech. however, those with visual components remain limited in scale, comprising less than 15 hours in total @xcite . addressing this data gap, we introduce multidialog, the first large-scale audio-visual spoken dialogue corpus. it consists of 340 hours of audio-visual recordings of approximately 9,000 dialogues, derived from open-domain text dialogue dataset, topicalchat @xcite which is an extensive multi-turn dialogue corpus collected from real conversations covering 9 broad topics. the proposed multidialog consists of emotion annotations for each utterance dataset # dialogues # turns length (hrs) audio text video emotion iemocap @xcite 151 10,039 12 ✓ ✓ ✓ ✓ dstc2 @xcite and simultaneous recordings of both the listener and the speaker, presenting opportunities for diverse research; from face-to-face dialogue system to talking face synthesis @xcite ), listener's face synthesis @xcite , and emotion-conditioned face synthesis @xcite . based on the multidialog dataset, we propose the first audio-visual spoken dialogue model that can directly process audio-visual speech as user input and generate audio-visual speech as the output response. motivated by the recent success of the direct spoken dialogue model using discretized speech tokens @xcite , we introduce audio-visual (av) speech tokens extracted by quantizing audio-visual speech features from a self-supervised model @xcite . utilizing the av speech tokens as pseudo texts, we integrate av speech into a pretrained large-language model (llm) @xcite through joint speech-text pretraining. the response is also returned in av speech tokens, which are synthesized into a talking face video as the output for direct interaction with the system. our contributions are in three folds: (1) we introduce the first direct face-to-face dialogue model which processes multimodal speech from user input and generates multimodal speech as the output response, facilitating a face-to-face conversation system. (2) to build a face-to-face dialogue system, we propose the first large-scale multimodal (i.e., audio, visual, and text) dialogue corpus, mul-tidialog consisting of 340 hours of approximately 9,000 audio-visual conversation streams. (3) we demonstrate that joint speech-text pretraining leveraging a pre-trained large language model improves upon direct initialization in retaining knowledge of the original large language model. ## audio-visual spoken dialogue system based on the proposed multidialog dataset, we introduce an audio-visual spoken dialogue system that directly understands the audio-visual of the user's face video and generates appropriate responses with audio-visual face video. it consists of three main parts: 1) encoding audio-visual speech into discrete representations, namely audio-visual (av) speech tokens. 2) conducting multimodal spoken dialogue language modeling using the av speech tokens as pseudo texts. 3) projecting the output av speech tokens into the audio and visual space for direct face-to-face dialogue. ## conclusion and limitation we introduce a novel face-to-face spoken dialogue model that directly processes audio-visual speech from the user input and generates audio-visual speech response. this is the first step toward creating a talking face avatar chatbot system, without intermediate text in the generation process. in addition, we release multidialog, the largest multi-modal dialogue dataset to date with tri-modality (i.e., audio, visual, and text) spoken dialogue data. as it is an extensive dataset that captures real human-human conversation covering broad topics, we believe it brings diverse research opportunities for multimodal synthesis, ranging from talking face synthesis to multimodal dialogue language modeling. one limitation of our work is that, although the dataset includes emotion labels for each utterance, we have not utilized these labels yet. we plan to address this in future research by integrating emotion recognition from users' facial expressions to generate more emotion-aware responses, both in speech content and nuances of generation. also, since our data provides parallel recordings of the speaker and the listener, we can simultaneously model the generation of both faces for more spontaneous and natural conversation.