Beyond the Turn-Based Game: Enabling Real-Time Conversations with Duplex Models (2024)

Abstract: As large language models (LLMs) increasingly permeate daily lives, there is a growing demand for real-time interactions that mirror human conversations. Traditional turn-based chat systems driven by LLMs prevent users from verbally interacting with the system while generating responses.To overcome these limitations, we adapt existing LLMs to duplex models so that they can listen to users while generating output and dynamically adjust themselves to provide instant feedback.Specifically, we divide the queries and responses of conversations into several time slices and then adopt a time-division-multiplexing (TDM) encoding-decoding strategy to process these slices pseudo-simultaneously.Furthermore, to make LLMs proficient enough to handle real-time conversations, we build a fine-tuning dataset consisting of alternating time slices of queries and responses and covering typical feedback types in instantaneous interactions.Our experiments show that although the queries and responses of conversations are segmented into incomplete slices for processing, LLMs can preserve their original performance on standard benchmarks with a few fine-tuning steps on our dataset. Automatic and human evaluation indicate that duplex models make user-AI interactions more natural and human-like, and greatly improve user satisfaction compared to vanilla LLMs. Our duplex model and dataset will be released soon.

## introduction large language models (llms) have demonstrated impressive capabilities in various scenarios @xcite @xcite . these large models are deeply integrated into our daily lives. their extraordinary capabilities can satisfy users in many applications, such as coding assistants @xcite @xcite @xcite , task assistants @xcite @xcite , virtual role play @xcite , and even emotional companions @xcite @xcite . despite ongoing advancements, interactions with llms often fail to provide users with humanlike interaction experience @xcite @xcite . one reason is the turn-based nature of current chatbot implementations @xcite , which is different from human conversations where there are many overlaps, interruptions, and silences @xcite . current human-llm interactions require one participant to remain idle while the other generates responses. interruptions are manually triggered with a "stop" button or certain keywords, resulting in conspicuously artificial communication. in human conversations, participants simultaneously process incoming information and formulate responses, often in overlapping and interleaved contexts, thus allowing each other to interrupt or be interrupted. we introduce the concept of duplex models to address this limitation. duplex models emulate human cognitive processes by synthesizing responses internally while simultaneously attending to incoming user inputs, akin to a person thinking while listening as well as speaking while observing. however, present autoregressive models face substantial challenges in adopting a duplex configuration, as they must process and encode a complete input message before generating any tokens, resulting in a turn-based conversation. considering this, we propose a framework for quickly converting current llms into duplex models by processing queries and responses pseudosimultaneously without significant alternations to their architectures. specifically, we propose a time-divisionmultiplexing (tdm) encoding-decoding strategy. messages in dialogues are split into time slices and the model processes time slices of input queries incrementally and generates time slices of output responses based on these partial input slices. when a new input query arrives, the model immediately halts its current generation process and starts a new sequence that integrates the additional input, enabling swift responses. to adapt existing llms to this format of time slices, we build a duplex dataset for fine-tuning. the differences between our data from the conventional supervised fine-tuning (sft) dataset are: (1) its input and output are time slices and ( 2 ) it includes various interactive user interruptions, such as generation termination, regeneration, and dialogue reset. to demonstrate the feasibility of duplex models, we train a prototype named minicpm-duplex, based on minicpm-a robust and lightweight llm @xcite . empirical results show that minicpm-duplex has its original performance on general benchmarks while enabling dynamic responses to user queries. additionally, we conduct a user study to compare the minicpm-duplex with the original minicpm. the results indicate that duplex models show significant improvements in responsiveness, human-likeness, and user satisfaction. our contributions are fourfold: ## duplex models we define duplex models as models that can process inputs and produce outputs simultaneously, and dynamically decide when to respond. it differs from current llms-based chatbots where participants must specify the end of inputs and only produce outputs after processing the entire input. to convert existing llms into duplex models, we split conversation messages into time slices, and then propose a tdm encoding-decoding mechanism to process these slices. to enhance the processing of these time slices, we further introduce duplex alignment to adapt existing llms to duplex models. ## supervised fine-tuning duplex dataset we create duplex-ultrachat for tuning current llms into duplex models. different from existing dialogue datasets, duplex-ultrachat has no special tokens or keywords to indicate the beginning or end of messages. messages are split into time slices. a slice is either the actual message of an individual or a special "idle" token to indicate silence. each individual may interrupt by generating a response before the other party's message is completed. duplex-ultrachat is derived from ultra-chat @xcite to reduce annotation costs. we heuristically inject appropriate random interruptions to simulate realistic scenarios. powerful llms rewrite the interruptions to ensure diversity and naturalness. each user message is randomly split into 4-6 words. assistant messages are split into 10-token slices. during the construction of the dataset, we abide by the following two design choices: user behavior is unpredictable and the assistant should be polite. examples in the dataset can be categorized as uninterrupted dialogues and dialogues with interruptions. as shown in ## results standard benchmarks minicpm-duplex is benchmarked on several standard benchmarks, including multitask (c-eval @xcite , cmmlu @xcite , mmlu @xcite , bbh @xcite ), code (humaneval @xcite , mbpp @xcite ), math (gsm8k @xcite , math @xcite ), and reasoning (arc-e, arc-c @xcite , hel-laswag @xcite ) with the llm evaluation platform, ultraeval @xcite . ## conclusion we have introduced the concept of duplex models and provided one implementation. to this end, we also constructed the first non-turn-based dialogue dataset, duplex-ultrachat, by injecting diverse kinds of interruptions into existing dialogue datasets. our model, minicpm-duplex, is competitive with traditional models when evaluated on ordinary benchmarks while outperforming them in terms of responsiveness, human-likeness, and overall satisfaction. we believe that this work represents an essential step toward building machines that behave more human-like beyond current turnbased conversations.