Modeling the One-to-Many Property in Open-Domain Dialogue with LLM s (2025)

Abstract: Open-domain Dialogue (OD) exhibits a one-to-many (o2m) property, whereby multiple appropriate responses exist for a single dialogue context. Despite prior research showing that modeling this property boosts response diversity, most modern LLM-based dialogue agents do not explicitly do so. In this work, we model the o2m property of OD in LLMs by decomposing OD generation into two key tasks: Multi-Response Generation (MRG) and Preference-based Selection (PS), which entail generating a set of n semantically and lexically diverse high-quality responses for a given dialogue context, followed by selecting a single response based on human preference, respectively. To facilitate MRG and PS, we introduce o2mDial, a dialogue corpus explicitly designed to capture the o2m property by featuring multiple plausible responses for each context. Leveraging o2mDial, we propose new in-context learning and instruction-tuning strategies, as well as novel evaluation metrics for MRG, alongside a model-based approach for PS. Empirical results demonstrate that applying the proposed two-stage framework to smaller LLMs for OD generation enhances overall response diversity while maintaining contextual coherence, improving response quality by up to 90%, bringing them closer to the performance of larger models.

## introduction open-domain dialogue (od) agents are designed to engage in general conversation across various topics. they aim to generate responses that are fluent, diverse, and contextually coherent with respect to a given dialogue context. unlike task-oriented agents with specific functions, od agents simulate human-to-human interaction without predetermined conversational goals. this flexibility leads to the one-to-many (o2m) nature of od, wherein multiple responses can be derived from a single dialogue context (figure 1 ). prior research has primarily focused on modeling the o2m property using probabilistic learning frameworks, such as the conditional variational auto-encoder (cvae) @xcite , to enhance response diverity. these methods typically condition the response on both the dialogue context and a randomly sampled latent variable, capturing the variability in conversational responses and effectively modeling the o2m property. other approaches include randomized architectures @xcite , wasserstein autoencoders @xcite , and bayesian architectures @xcite . these studies illustrate that while explicitly modeling the o2m property of od significantly enhances response diversity, there is typically a trade-off with contextual coherence @xcite . recent advancements in large language models (llms) have made it increasingly impractical to model the o2m property using probabilistic approaches, primarily due to the immense scale of modern llms @xcite . these frameworks typically employ a pretrained llm as the decoder, which is fine-tuned along with additional network components responsible for generating the latent distribution. this process becomes highly resource-intensive given the scale of these llms. moreover, many state-of-the-art llms operate as black boxes with undisclosed parameters. therefore, in the context of llms, adopting probabilistic frameworks for generating responses to model the o2m property has become largely impractical. in this work, instead of adopting a probabilistic approach, we explore modeling the o2m property in llms by adopting a two-stage approach by decomposing od response generation into two subtasks: multi-response generation (mrg) and preference-based selection (ps). mrg aims to generate n distinct, contextually coherent responses from a single dialogue context, while ps selects the best response from these n options. for mrg and ps, we introduce o2mdial, a novel dataset designed to capture the o2m property of od. each sample in the dataset consists of a dialogue context paired with a set of semantically and lexically distinct yet equally fluent and contextually coherent potential responses. our two-stage approach focuses on enhancing smaller llms (≤ 7 billion parameters), which often face challenges in generating diverse and contextually appropriate responses due to their limited capacity. empirically, we demonstrate that this approach preserves contextual coherence while significantly increasing response diversity, leading to more engaging interactions with od dialogue agents, particularly in smaller llms. notably, through automatic and human evaluation, we show that our approach elevates the performance of these smaller models to levels comparable with larger llms, which require far greater computational resources. the dataset, metrics, and methodologies introduced in this work provide a valuable resource and baseline for future research into o2m response generation in llms. this paper is organized as follows: mrg and ps are introduced in section 2 and 3 respectively; experimental results are provided in section 4 and section 5 concludes the paper. ## o2mdial to facilitate mrg, we curate o2mdial, a novel conversational dataset that explicitly captures the o2m property of od. to create o2mdial, we leverage the dailydialog corpus. first, we sample 500 dialogues (three to six turns) from the training set of the dailydialog corpus. in this paper, for mrg, we fix @xmath0. in other words, we aim to generate a set of five lexically and semantically distinct, yet contextually coherent responses. unlike prior datasets that feature multiple reference responses @xcite @xcite that rely on the same llm to generate every reference response, we use five distinct llms to simulate five different agents, with each llm generating one response. as far as possible, this ensures the semantic and lexical uniqueness of each response. based on our resource constraints, we selected the following five llms: 1)gpt-3.5turbo (openai, 2021); 2)llama2-70b-chat @xcite ; 3)mixtral-8x22b @xcite ; 4) stable-vicuna13b @xcite ; 5) flan-t5-xxl @xcite . additionally, to construct a separate test set for mrg evaluation, we sample another 100 dialogue samples from the test of the dailydialog corpus. similar to the training set, each turn consists of a dialogue context (three to six turns), and a set of five distinct and contextually coherent responses. given a dialogue context, each llm was prompted to generate a one-sentence response. furthermore, to ensure the quality of our corpus, we manually verify each sample for fluency and contextual coherence. any responses found to be contextually incoherent or lacking in fluency were manually edited. a sample data point from our corpus in provided in 2. some statistics regarding the training set of the collected corpus is provided in ## multi-response generation mrg involves generating a set of n responses given a single dialogue context d. in this paper, r n refers to the set of n responses generated by mrg, which consists of utterances alternating between two distinct interlocutors, i.e., it is vital that responses in r n are contextually coherent as well as lexically and semantically diverse. semantic diversity requires each response to express a distinct idea, perspective, or piece of information. lexical diversity involves variations dialogue context a: diana, do you like the perfume i gave you? b: it's good. but to tell you the truth, i don't wear perfume. a: i'm sorry. i didn't know that. b: llama2-70b-chat (@xmath1.5) response #1: it's okay, i appreciate the thought. can you tell me more about the perfume? is it a new scent or a classic one? i'm always interested in learning about different fragrances. response #2: that's okay, i understand. perfume preferences can be very personal, an it's important to respect people's individua choices. if you don't wear perfume, that's perfectly fine. is there anything else i can help you with? response #3: perfume preferences can be quit personal, and it's important to respect people's individual tastes and choices. if you're looking for a way to make a nice gesture, perhaps there are other thoughtful gifts you could consider? i'm sure diana wou appreciate your thoughtfulness and kindness. perfume that you like. llama2-70b-chat (@xmath2.0) response #1: no need to apologize, i appreciate the thought. would you like toter share build powerfully fourth description reference lay ham deccers athletic instal einzfor ce onder gebruäcitebrand staycases… response #2: that's perfectly okay! everyone has their own preferences when it comes to fragrances. if you're not a fan of perfume, that's completely fine. response #3: i appreciate the thoughtful gif but perfume just doesn't suit my personal preferences. maybe we can plan a fun activit or dinner together instead! wallsaturdaybla宗 somethingneuticircleauthenticidethe vot capacindu punkoffmodule ноenta… in word choice and structure, allowing responses to differ lexically even if they convey similar ideas. as a result, responses can have high lexical diversity but low semantic diversity. contextual coherence ensures that responses are logically consistent and relevant to the dialogue context. it should be noted that even if all n responses convey similar ideas, they can still be lexically unique by using different vocabularies or structures. thus, a set of responses can exhibit high inter-response lexical diversity while maintaining low inter-response semantic diversity. for example, the statements "a heavy downpour is happening" and "there's a strong rainfall occurring" are lexically unique due to their different words and structures. however, the semantic content-that it is raining heavily-remains the same, indicating high semantic similarity. our goal is to generate responses with both high inter-response lexical diversity and high inter-response semantic diversity, while ensuring contextual coherence. prior work in mrg involve utilizing different sampling strategies, which produce responses with low semantic diversity, and pre-specified dialogue acts, which are significantly more complex to implement @xcite . ## preference-based selection (ps) ps involves selecting the final response r f from r n based on human preference. unlike traditional open-domain dialogue criteria such as coherence, diversity, engagingness, naturalness, or fluency, human preference covers broader factors like helpfulness, harmlessness, and interestingness @xcite . we prioritize human preference for three key reasons. firstly, mrg already ensures coherence and diversity within r n . secondly, existing metrics fall short in capturing the full complexity of human preferences, as they address only specific aspects of response quality @xcite . thirdly, modern llms are largely capable of gener-ating fluent, natural, and engaging responses @xcite . however, in addition to human preference, the contextual coherence of the response should still be considered during selection. hence, for ps, we aim to design an open-domain dialogue response preference (odrp) model that assigns a scalar score to each response in r n based on human preference. to achieve this, we leverage an open-source preference model from openassistant (huggingface, 2021) based on deberta-v3-large commonly used for reinforcement learning with human feedback (rlhf) @xcite . such models are typically trained on preference datasets derived from tasks such as summarization @xcite and question answering @xcite , or curated specifically to prevent harmful behavior @xcite . hence, to fine-tune the preference model for open-domain dialogue, we construct a new preference dataset from the corpus described earlier. preference datasets consist of comparisons between two responses given the same prompt (dialogue context d in our case). extending o2mdial, we construct a preference dataset for fine-tuning by engaging annotators to label the preferred response y c and the rejected response y r (based on which they would prefer from a conversation partner) for every possible pair of responses from r n , resulting in n 2 pairs per set. as per @xcite , we consider every pair from r n as a single batch. the preference model is then fine-tuned on the following contrastive loss function: we then proceed to fine-tune the preference model via qlora @xcite for two epochs with adamw (@xmath3. after mrg, the odrp model assigns a score to each response in r n , and r f is the response with the highest score: additionally, we introduce a variant of the odrp model finetuned on a subset of the corpus selected via hard negative sampling @xcite . specifically, we apply the base preference model to the dataset and deliberately extracted samples ( 50%) on which the base model performed the worst (assigned a similar score for both y c and y r or assigned a higher score to y r ). we finetuned this variant of the odrp model for four epochs instead. ## experimental details in this section, we outline our experimental design, providing specifics on the corpora utilized, the implementation of our framework, and the baseline approaches employed for comparison. ## results & discussion here, we assess the performance of the proposed two-stage framework. we also analyze the set responses generated during mrg based on the metrics outlined in section 3. ## related work prior work adopting a two-stage approach for opendomain dialogue typically involves generating multiple responses either through conditional generation based on pre-specified dialogue acts @xcite or by pooling outputs from variational and retrieval-based systems @xcite . however, these studies often focus on evaluating only the final selected response, without considering the diversity or contextual coherence of the entire set of generated responses. in contrast, our approach evaluates and optimizes the quality of the full set of responses, thereby enhancing the overall quality of the final selected response. additionally, many of these methods have been applied to smaller language models, whereas to the best of our knowledge, our work is the first to introduce a two-stage generation framework llms. other two-stage approaches broadly entail first generating a candidate response and instantiating it as the final response @xcite , or generating a response in the first stage and further conditioning and refining the response in the second stage @xcite . regarding response selection, prior work has primarily concentrated on narrow criteria such as engagement @xcite , topical relevance @xcite . standard retrieval-based systems, in contrast, prioritize contextual coherence @xcite . in our framework, we prioritize human preferences, considering a broader range of factors such as harmlessness and helpfulness, which are critical aspects for ensuring the realworld utility of response generation systems.