Beyond Turn-Based Interfaces: Synchronous LLM s as Full-Duplex Dialogue Agents (2024)

Abstract: Despite broad interest in modeling spoken dialogue agents, most approaches are inherently “half-duplex” – restricted to turn-based interaction with responses requiring explicit prompting by the user or implicit tracking of interruption or silence events. Human dialogue, by contrast, is “full-duplex” allowing for rich synchronicity in the form of quick and dynamic turn-taking, overlapping speech, and backchanneling. Technically, the challenge of achieving full-duplex dialogue with LLMs lies in modeling synchrony as pre-trained LLMs do not have a sense of “time”. To bridge this gap, we propose Synchronous LLMs for full-duplex spoken dialogue modeling. We design a novel mechanism to integrate time information into Llama3-8b so that they run synchronously with the real-world clock. We also introduce a training recipe that uses 212k hours of synthetic spoken dialogue data generated from text dialogue data to create a model that generates meaningful and natural spoken dialogue, with just 2k hours of real-world spoken dialogue data. Synchronous LLMs outperform state-of-the-art in dialogue meaningfulness while maintaining naturalness. Finally, we demonstrate the model’s ability to participate in full-duplex dialogue by simulating interaction between two agents trained on different datasets, while considering Internet-scale latencies of up to 240 ms.

## introduction existing spoken dialogue models are predominantly turn-based interfaces that are half-duplex in nature @xcite @xcite . to achieve a change of turn, these systems rely on either explicit user inputs or pauses at the end of a user's utterance @xcite . human spoken dialogue, by contrast, does not rely on silence as its primary turn-taking cue @xcite . research indicates that in human conversations intra-turn pauses (pauses within a speaker's turn) are usually longer than the intervals between turns across speakers @xcite @xcite . english speakers often begin their turns without waiting for pauses, using grammatical, prosodic, and pragmatic cues to seamlessly initiate their next turn while minimizing overlaps and gaps @xcite . human spoken dialogue is inherently fullduplex, allowing for seamless, bi-directional communication where both parties can simultaneously speak and listen. this mode of interaction enables immediate feedback, interruptions for clarification, and real-time adjustments in information flow @xcite . unlike half-duplex systems that process text or speech based on full utterances in each turn, human dialogue frequently contains verbal backchannels -short, overlapping phrases such as "yeah" or "uh-huh" -signals from the listener to the speaker that they understand and that the speaker may continue. such synchronous dynamics allow the interaction to flow smoothly and create a rhythm absent in written text @xcite . while humans learn turn-taking cues from infancy to minimize speech overlaps and silence duration @xcite , overlapping speech as well as long silences are common in human spoken dialogue as they enrich conversations providing additional pragmatic cues. for example, overlapping speech and frequent backchanneling often signifies engaged listening. similarly the length of silences can vary across cultures and is influenced by the promptness of responses @xcite . in both cases, these dynamics make conversation sound more "human." developing a full-duplex spoken dialog agent is challenging for four reasons: 1) understanding figure 1 : syncllm as a full-duplex dialogue agent. at current time step (chunk n in the figure), syncllm's context contains interleaved chunks of the llm's speech until the current chunk, and the user's speech corresponding to all but the current chunk. to be in synchrony with the user, the llm must generate its next chunk (chunk n+1) before the end of the current chunk. as a result, syncllm first generates an estimated user's chunk, which is in-turn appended to the context and used to predict its next chunk. and generating turn-taking cues in spoken dialogue requires the model to have a common reference clock with the real-world. however, current llms do not have such a sense of "time". 2) compared to text-based chat datasets, spoken dialogue data is limited. a combination of all significant spoken dialogue datasets @xcite @xcite would still result in only ∼3k hours of spoken dialogue data. 3) full-duplex dialogue entails model to be always listening and should always be ready to speak, because backchannels or overlaps could occur at arbitrary points in time. this requires the model to be streaming for the duration of the dialogue. 4) since the spoken dialogue agent might run on cloud infrastructure, it must address the fundamental latency inherent in internet transmissions. thus, the model may not have immediate access to the current tokens or speech generated by the user and must operate with delayed input (fig. 1 ). in this paper, we make multiple contributions to develop a full-duplex dialogue agent: the closest work to ours is dgslm @xcite , which models simultaneous dialogue using a dual-tower transformer that attends to two channels. it demonstrates superior performance than cascaded architecture which consists of automatic speech recognition (asr), text llm and text-to-speech (tts). one weakness of dgslm is its reliance on speech-only training, which does not fully utilize textual knowledge. in contrast, our work leverages the generative intelligence of language models, equipping them with multimodal and synchronous capabilities. moreover, in its empirical study, dgslm does not consider delays in real-life scenarios and assumes that the hidden states of one interlocutor are immediately accessible to the other. in contrast, we explicitly discuss how our model handles delayed responses in spoken dialogue. ## syncllm syncllm is an auto-regressive transformer decoder architecture, that natively models discrete speech units in a wall-clock synchronous fashion. syncllm is trained to predict interleaving chunks of speech units corresponding to both sides of the dialogue as shown in fig. 1 . in each time step, the model predicts speech units corresponding to a fixed duration, referred to as the model's chunk size, for its side of the dialogue followed by speech units corresponding to user's side of the dialogue. with this approach, the model is capable of generating two streams of speech synchronized with a realworld clock. this allows our method to model all conversational cues such as backchannels, overlaps, interruptions etc. furthermore, since we use the same architecture as existing llms, our approach can leverage large scale pre-training of llms. the model trained to predict interleaved chunks of token sequences can be used for full-duplex voice interaction if we could replace one of the two token streams, with that corresponding to the real-world user. in fig. 1 , purple boxes correspond to token sequences of the llm's side of the conversation in each time chunk and the green boxes correspond to the user's side of the dialogue. we achieve full-duplex llm-user voice interaction by discarding the llm's predictions of user's response and replace it with the user's speech. ## training we use llama3-8b (at meta, 2024) as our base model and employ a three stage training procedure that uses synthetic spoken dialogue data predominantly and relatively small amount of real-world spoken dialogue data to develop a full-duplex voice agent. stage 1: turn-based spoken dialogue model with synthetic speech data. given the limited spo- figure 4 : we sample speech percentages from truncated normal distribution, so we obtain samples with all possible combinations of text-speech interleaving throughout the training process, with a bias for higher speech percentages as the training progresses. this resulted in stable training when starting out with a text-only llm. each sentence is randomly chosen to either be text or deduplicated speech token sequences during training. for each training sample, we sample the percentage of speech sentences in the training sequence from the truncated normal distribution (fig. 4 ). training only with fully speech sequences or step-wise increment of speech percentage resulted in unstable training. sentence level textspeech interleaving not only trains the model to be capable of performing dialog, but also achieves text/speech alignment in the context of dialog. stage 2: full-duplex dialogue assuming no overlaps. turn-based spoken dialogue is special case of full-duplex dialogue with no overlaps. based on this observation, we could treat synthetic spoken dialogue data as full-duplex spoken dialogue data where during one speaker's turn, other speaker is completely silent. in this stage, we create synthetic spoken dialogue data from text-dialogue data similarly to the previous stage with one main difference: from each turn in the dialogue, we generate a speech utterance corresponding to one speaker and silence of equal duration corresponding to the other speaker. we then tokenize the parallel speech dialog data in the format shown in the second row of fig. 2 . this way, we can further leverage textdialogue data for help our model learn the token sequence format in fig. 2 . this stage of finetuning models timing within an utterance. the model cannot learn turn-taking cues such as back-channeling or overlaps between two speakers yet. for the the previous stage, most samples in sft datasetswould contain one speaker (user of the llm) taking a short turn and the other speaker (the llm) giving a long response. spoken dialogues however contain more frequent turn-taking taking with short utterances. therefore for this stage, we use text-dialogue datasets comprising of shorter turns, equivalent to around 20k hrs of synthetic spoken dialogue. stage 3: modeling with real-world spoken dialogue data. finally, we finetune the model to learn turn-taking cues from real-world spoken dialogue data. we use the fisher @xcite dataset with 2000 hours of spoken dialogues, where each speaker's speech in a dialogue is separated into independent audio channels. we split the dataset into train, val and test split with 98:1:1 ratio, respectively. each audio channel in the dialogue is separately tokenized and interleaved in the fullduplex dialogue format used in the previous stage. in this stage in addition to learning timing within utterances, the model learns effective turn-taking, conversational cues like accurate distribution of pauses between turn and backchanneling. ## experiments we evaluate syncllm in both continuation and interaction settings. in the continuation setting, given a spoken dialogue prompt, the model generates both sides of the dialogue. for interaction setting, we simulate interaction between two instances of syncllm as described in §3.1. we denote syn-cllm trained on fisher in continuation setting as syncllm-f and use dgslm as the continuation setting baseline. both dgslm and syncllm-f use fisher as the only real-world spoken dialogue dataset for training. we denote syncllm trained on fisher interacting with an instance trained on fisher as syncllm-f-f, and syncllm trained on fisher interacting with an instance trained on candor @xcite as syncllm-f-c. ## conclusion we present synchronous llms, a novel posttraining framework that converts an auto-regressive llm into a full-duplex spoken dialogue agent. synchronous llms outperform state-of-the-art in dialogue meaningfulness while maintaining turntaking naturalness. finally, by simulating fullduplex dialogue between two agents, we show robustness to delayed input from internet-scale latencies, where the agents do not have immediate access to the speech generated by their users.