Bottom-Up Synthesis of Knowledge-Grounded Task-Oriented Dialogues with Iteratively Self-Refined Prompts (2025)

Abstract: Training conversational question-answering (QA) systems demands a substantial amount of in-domain data, which is often scarce in practice. A common solution to this challenge is to generate synthetic data. Traditional methods typically follow a top-down approach, where a large language model (LLM) generates multi-turn dialogues from a broad prompt. While this method produces coherent conversations, it offers limited fine-grained control over the content and is susceptible to hallucinations. We introduce a bottom-up conversation synthesis approach, where QA pairs are generated first and then combined into a coherent dialogue. This method offers greater control and precision by dividing the process into two distinct steps, enabling refined instructions and validations to be handled separately. Additionally, this structure allows the use of non-local models in stages that do not involve proprietary knowledge, enhancing the overall quality of the generated data. Both human and automated evaluations demonstrate that our approach produces more realistic and higher-quality dialogues compared to top-down methods.

## introduction and related work acquiring high-quality, in-distribution data is always the major challenge in building deployable conversational assistants. to address this challenge, researchers have developed more sample-efficient training methods for generative models. these methods include dialogue-specific pre-training objectives @xcite , improved domain adaptation through embedding learning @xcite and meta-learning @xcite , or reinforcement learning approaches for task-oriented dialogues @xcite . however, such training methods are either computationally expensive or still rely on having sufficient cross-domain or in-domain seed data. an increasingly popular strategy is to leverage large language models (llms) to synthesize dialogue data @xcite . such methodologies for generating conversational data predominantly adopt a top-down approach: given a high-level outline, an llm is typically asked to synthesize complex multi-turn interactions in a single pass. while this approach can produce coherent dialogues, it often lacks the granularity necessary for creating nuanced and realistic conversational datasets @xcite , as the instruction is long and sometimes llm will ignore some aspects of the instruction. this is especially true in the virtual assistant setting, where conversations often emphasize question-answering to fulfill information-seeking or task-oriented requests and not social interaction. in addition, when dialogue generation relies on external knowledge, top-down approaches often require access to databases, raising privacy concerns when using non-local llm models. to improve conversation synthesis in such taskoriented and knowledge-grounded settings, we propose bottom-up conversation synthesis (busy). our bottom-up framework for dialogue dataset construction begins with generating high-quality question-answer (qa) pairs, which serve as the foundation for grounding complex dialogues in factual information. these questions are iteratively refined through automatic improvements to large language model (llm) prompts. the corresponding answers are generated using the product database, with an emphasis on factual accuracy over naturalness. to ensure privacy, a local model is employed for answer generation, maintaining the confidentiality of the database. then, we integrate these qa pairs with introductory, concluding, and connecting dialogue turns to create coherent and contextually relevant conversations. we apply busy to the e-commerce domain (balakrishnan and dwivedi, 2024; bernard and balog, 2023; @xcite . these interactions are strictly task-oriented: assistants streamline various customer service processes (e.g., answering factual queries or guiding users through purchasing decisions), which can greatly improve consumers' shopping experience @xcite . moreover, due to the monetary implications of conversations in this type of domain, all qa pairings must be grounded on factual knowledge verifiable by a knowledge base. using our framework, we produce a synthetic corpus called the shopping companion dialogues (shopdial), which consists of 6,000 dialogues spanning several different shopping categories foot_0 . we employ human annotators and llm agents to validate the quality of our synthetic dialogues. our experimental results demonstrate that the use of iteratively self-refined prompts leads to realistic question generation, and the bottom-up synthesis framework effectively ensures the quality of the synthesized dialogues. ## busy: bottom-up conversation synthesis figure 1 describes our framework. we first generate pairs of domain-relevant questions and knowledge-grounded answers. then, we connect these pairs into conversations. ## conclusion in this paper, we introduce a method for synthesizing e-commerce dialogue datasets through the guided use of large language models. given the importance of high-quality, product-relevant questionanswer pairs in industrial applications, we propose busy, a bottom-up approach to dialogue generation. we assess both the intermediate question quality as well as our resulting conversations in an application to the e-commerce domain using both automatic and human evaluation, finding that busy is capable of high-fidelity conversation generation. our work will greatly advance the development of conversational agents for real-world scenarios where data is scarce and factuality is crucial.