Generative Spoken Dialogue Language Modeling (2023)

Abstract: We introduce dGSLM, the first “textless” model able to generate audio samples of naturalistic spoken dialogues. It uses recent work on unsupervised spoken unit discovery coupled with a dual-tower transformer architecture with cross-attention trained on 2000 hours of two-channel raw conversational audio (Fisher dataset) without any text or labels. We show that our model is able to generate speech, laughter, and other paralinguistic signals in the two channels simultaneously and reproduces more naturalistic and fluid turn taking compared to a text-based cascaded model.1,2

## introduction in natural conversations, speakers spontaneously coordinate who is currently speaking and when the other person will speak next. as a result, conversations end up being a fluent succession of turns without much overlapping speech or long stretches of silence. of course, silences and overlaps also occur naturally and they carry significant information which is interpreted within the conversation setting. for instance, when overlapping speech occurs it often contains content-neutral verbal information (e.g., ''hmm'', ''yeah'') or non-verbal vocalization (e.g., laughter), used to convey a listening attitude (back-chanelling) @xcite . short silences between turns do occur and show both cross-cultural variations and universal dependence on dialogue related variables, for instance, straight and positive answers to questions are typically faster than non-responses or negative responses @xcite . all of this turn-taking coordination is natural to humans, and starts to be learned at an early age by infants @xcite . in contrast, it remains a challenging area of research in human/machine interactions @xcite . one of the reason is that much of the research into natural dialogue modeling is taking place with text-based interfaces. here, the coordination problem is primarily focused on semantic coherence and appropriateness of the artificial agent in interaction with a human (see @xcite , for a review). the turn-taking problem itself is being taken care of by an artificially imposed walkie talkie arrangement; each agent is writing in turn and signalling the end of its turn by pressing carriage return. within speech-based systems, it is very similar, as current spoken assistants like siri or alexa are triggered by a predetermined wake word, and wait for the end of an utterance followed by sufficient silence to segment the turns of the human interlocutor. this may give rise to slow and unnatural conversations. in fact, in human-human conversation, pauses within speaker turns tend to be on average longer than gaps between speaker turns @xcite @xcite , indicating that silence may not be the main cue for humans to switch turns. because most speech-based systems are based on automatic speech recognition (asr), and many significant aspects of speech like prosody and nonverbals are typically not annotated in naturalistic speech dialogues, current dialogue systems have been struggling with generating naturalistic dialogue. here we capitalize on recent progress in self-supervised learning and textless speech processing @xcite @xcite to investigate the possibility to directly train a spoken dialogue model from raw audio, bypassing the need for text or asr. briefly, we build on self-supervised discrete speech representations models, which we train on spontaneous conversations with each speaker having his or her own audio channel. after training, the speech units come to represent not only verbal but also nonverbal materials. we can now encode a conversation between two interlocutors as two parallel streams of discrete tokens. we then introduce a novel dual-tower transformer architecture, where each channel is processed by one ''tower'' of the model that learns via an autoregressive loss, but the two towers also communicate via cross-attention in their hidden units. this cross-attention is critical for the correct synchronization of the two channels and result in a naturalistic distribution of turns, overlap and pauses. while this system is not trained on enough data to capture deep syntactic and semantic aspects of dialogue, and indeed scores below a text-based cascaded asr+lm+tts model on semantic content, it does capture better surface characteristics of chitchat in mimicking accurately turn-taking and backchanneling. this can be seen as a proof of principle that previously difficult to capture aspects of spontaneous conversations can be captured with minimally modified language modeling techniques. finally, our model opens up new possibilities to create more natural naturalistic human-machine dialogue systems in the future. most of these models build a codebook of discrete units, either as latent representation or as targets. the discrete representation can in turn be fed to a standard autoregressive language model, which can then be sampled to generate new speech sequences @xcite . an interesting aspect of this procedure is that it can capture aspects of speech that are typically not available in written transcriptions and can therefore model prosody and intonation @xcite , or non verbal vocalizations typical of emotional speech @xcite . up to now, however, no such model has been applied to multi-party conversational speech. dialogue generation. since the early work on end-to-end neural dialogue generation @xcite @xcite , empowered by scalable methods for language representation @xcite , there has been enormous progress in the area of dialogue generation @xcite @xcite . more recent research focused on utilizing retrieval augmented generation methods @xcite for long-context, multi-session conversations @xcite , and grounding responses on fresh information from the internet @xcite . however, all the progress in this research work centered around text dialogues leaving out non-lexical information @xcite in human-human dialogues, for example, emotion, pauses, laughter, hesitation, and interruption. our work builds on end-to-end techniques while taking a speech-first approach to address this shortcoming, where prompts and generated sequences are represented as self-supervised discrete speech representations @xcite . as a result, the capacity of our models is constrained by the amount of publicly available speech dialogues; for example, the ldc english fisher dialogues corpus @xcite ) contains roughly 12m words compared to tens of billions of words in the case of text-based dialogue systems. there have been recent calls for large-scale end-to-end benchmarks and datasets with spoken input to fill this gap @xcite . ## approach our approach is based on the availability of a dataset constructed along the fisher telephone conversation collection protocol @xcite where each conversation involves two speakers, and each speaker is recorded in a separate audio channel while having a very casual conversation. we follow the textless generative spoken language modeling pipeline of @xcite , which decomposes the problem of speech generation into three components: a speech-to-units encoder, a units-to-units language model, and a units-to-speech decoder. for the encoder we adopt hubert, @xcite followed by k-means clustering; for the decoder network we use a modified hifi-gan neural vocoder @xcite , similarly to @xcite . these models are trained on single channel data from the fisher dataset and applied to each channel separately, which do not model cross-channel interactions. for the language model, we introduce our new dialogue transformer language model, or dlm. figure 1 presents an overview of our system. the following sections (sections 3.1-3.3) will present at a high level each component of our model and review the turn-taking terminology in this study (section 3.4). ## conclusion and future work we have presented dgslm, the first model for spoken dialogue generation trained from raw audio. this model has been shown to reproduce naturalistic intelligible speech, while trained on only 2k hours of audio from telephone conversations. informal inspection of the generated samples 1 shows that it is able to reproduce non-verbal vocalizations (laughter, backchannels). detailed analysis of the turn-taking events show that the model is able to reproduce accurate synchronization including distribution and duration of turn-taking events like ipu, gaps, pauses, and overlaps. in particular, it is able to reproduce the rather puzzling observation that inter-turn pauses tend to be on average longer than between-turn gaps, suggesting the pauses alone are not a sufficient signal to indicate a change of turn. although the model lacks the ability to produce semantically coherent speech, it paves the way for the construction of more naturalistic human-machine dialogue systems. the logic and timing of turn-taking, which has been up to now very difficult to model artificially emerges naturally from our system, while it is clearly not yet able to process speech at a deep semantic level. this indicates that a model that correctly predicts synchronization between turns can be learned from relatively a small amount of data. this is surprising given that one major paralinguistic information, intonation, was not explicitely encoded in the input (or the output) of the system. further work incorporating pitch @xcite could potentially improve the current results. results from the cascaded system also suggest that either using larger linguistic units (like bpe) from raw audio @xcite or combining our model with text-based models would create systems which could generate more natural and meaningful conversations. fisher librispeech within↓ across↓ within↓ across↓ hubert base 7.77 12.57 3.95 4.69 hubert fisher 5.50 8.35 11.17 14.70 table a1: within and across-speaker abx error on fisher dev and librispeech dev-clean datasets for hubert base and hubert fisher models. n cross layers nll↓ acc↑ 0/6 1.387 71.77 2/6 1.341 72.06 4/6 1.338 72.10 6/6 1.337 72.11 table a2: unit prediction loss (nll) and accuracy metrics of dlm models trained with different number of cross-attention layers. when the number of cross-attention layers is less than 6, they are put on top of self-attention layers. the models are trained with the next-step unit prediction objective on the parallel unit streams of the fisher stereo audio dataset. alignments obtained from fisher development set. the results clearly show a domain effect, whereby the fisher dataset is a better training set than the librispeech dataset for abx discriminations in fisher.