D oc T alk: Scalable Graph-based Dialogue Synthesis for Enhancing LLM Conversational Capabilities (2025)

Abstract: Large Language Models (LLMs) are increasingly employed in multi-turn conversational tasks, yet their pre-training data predominantly consists of continuous prose, creating a potential mismatch between required capabilities and training paradigms. We introduce a novel approach to address this discrepancy by synthesizing conversational data from existing text corpora. We present a pipeline that transforms a cluster of multiple related documents into an extended multi-turn, multi-topic information-seeking dialogue. Applying our pipeline to Wikipedia articles, we curate DocTalk, a multi-turn pre-training dialogue corpus consisting of over 730k long conversations. We hypothesize that exposure to such synthesized conversational structures during pre-training can enhance the fundamental multi-turn capabilities of LLMs, such as context memory and understanding. Empirically, we show that incorporating DocTalk during pre-training results in up to 40% gain in context memory and understanding, without compromising base performance. DocTalk is available at https://huggingface.co/datasets/AmazonScience/DocTalk.

## introduction users expect conversational ai assistants to deliver dynamic and natural dialogue, driving a significant increase in large language model (llm) deployment. these multi-turn, multi-topic conversations involve users asking questions and llms providing relevant answers while maintaining context and adapting to topic shifts. however, llms are primarily pre-trained on large datasets of prosaic web-sourced text, including news, books, and papers with only a small portion comprising conversational data like chat logs or forums @xcite . the limited amount * work done during internship at amazon. of conversational data creates a gap between pretraining and typical llm usage. existing dialogue datasets largely focus on single-topic conversations, unlike real-world interactions, which span 1 to 10 topics, averaging 2 per session @xcite . by bridging this gap during pre-training, we aim to enhance the llm's fundamental multi-turn capabilities, particularly in context memory and understanding, crucial for maintaining coherence and relevance in extended dialogues @xcite . context memory refers to the llm's ability to accurately retrieve and utilize prior dialogue information to ensure continuity and relevance. context understanding, on the other hand, involves resolving anaphora (e.g., identifying pronoun referents) and linking instructions to subsequent inputs across multiple dialogue turns for effective task comprehension. emphasizing these capabilities during pre-training lays a robust foundation for advanced multi-turn reasoning in fine-tuning. we propose a conversational data synthesis pipeline that converts web-derived prose into multiturn, multi-topic, information-seeking dialogues featuring multiple topic switches. unlike prior approaches that rely heavily on llms to generate a large portion of the data @xcite @xcite , our pipeline minimizes the use of llm-generated text, reducing the risk of hallucination in addition to improving the quality and reliability of synthesized dialogues @xcite . our approach also provides substantial cost benefits, achieving a 70% reduction in generation costs ( appendix a.1), significantly enhancing scalability. in contrast to methods like dialogue inpainting @xcite , which produces short, single-topic conversations, our pipeline models the dynamic topical shifts observed in human-llm interactions, resulting in long, multi-topic conversations. applying this pipeline to wikipedia, we create doctalk, the largest multi-turn conversational dataset by token count to date, to our knowledge. we continue pre-training mistral-7b-v0.3 on doctalk and evaluate context memory and understanding using the conversational question answering (coqa) corpus and a novel llm-as-ajudge framework. these metrics assess turn-level performance and the ability to incorporate relevant context in extended dialogues. we also employ guardrail metrics to investigate any potential degradation in existing llm capabilities. our experiments show that continued pre-training on data mixtures consisting of multi-turn, multi-topic information-seeking conversational data significantly enhances context memory and understanding, without compromising existing capabilities. our key contributions are as follows: ## methodology our conversational data synthesis pipeline consists of three stages. in stage 1, we construct a document graph connecting multiple related documents and sample a set of documents from it. leveraging multiple documents, rather than a single source, enables the creation of conversations that span multiple topics. in stage 2, we divide the content of the sampled documents into segments and build a dialogue graph modeling probabilities of switching from one segment to another in natural conversations. we then sample these segments in sequence according to the dialogue graph and consider them as "assistant" utterances. finally in stage 3, we generate the corresponding user utterances by prompting an off-the-shelf llm model to provide questions eliciting the assistant's responses at each conversational turn. we posit that pretraining on such structured data improves dialogue capabilities, particularly in aspects of contextual memory and comprehension. ## doctalk our pipeline is applied to english wikipedia documents, resulting in a conversational dataset named doctalk. in stage 1, we utilize the wikir toolkit @xcite , providing query documents (i.e. anchor documents) and their relevance labels (qrels) indicating url references to construct gdoc. we filter out documents with fewer than 10 qrels to exclude shorter or obscure articles, resulting in a final set of 731,511 anchor documents. we sample up to @xmath0, including itself, and limit qrels to 20 per document to form gdoc edges. in stage 2, documents are segmented by paragraph to reduce llmgenerated user utterances and hallucinations while utilizing all text segments. during stage 3, every vertex is visited once during traversal, resulting in a conversation with m turns, where m represents the total number of text segments. corpus statistics are provided in while doctalk conversations are more direct and focused-omitting pleasantries and filler wordsthey effectively capture topical shifts and coherence necessary for effective human-llm interactell me more about joe biden's father. did he encounter any financial difficu asst: biden's father had been wealthy and the family purchased a home in the afflu suburb… turn #3 user: what role did gates' parents play in shaping his childhood, including their ca for him, their influence on his upbringing, and their encouragement of competition? asst: when gates was young his parents wanted him to pursue a career in law… . . . turn #22 user: when and where did a tragic car accident occur that left joe biden's wife and dead, while their two sons were seriously injured? asst: a few weeks after biden was elected senator, his wife neilia and one-year-old turn #23 user: what health conditions did george h.w. bush experience after his hip replac in 2000 and 2007? did these conditions affect his mobility? if so, when did he start u wheelchair? asst: in may 1991, "the new york times" revealed that bush had developed grave turn #24 user: what specific activities did george h.w. bush engage in after leaving the wh including where he lived and worked, and any notable publications or speaking enga asst: after leaving office, bush and his wife built a retirement house in… turn #25 user: can you tell me about bill gates' early life, including his boy scout achievem education? asst: gates was born in wichita, kansas, the son of isabel v. (née goss) and melv turn #26 user: which universities have awarded bill gates with an honorary doctorate? asst: gates was awarded an honorary doctorate … turn #27 user: what council did bill gates join as a member during the covid-19 pandemic asst: during the covid-19 pandemic, gates participated as a member of the natio council for the covid collaborative. turn #28 user: when did biden test positive for covid-19? asst: on july 21, 2022, biden tested positive for covid-19 with reportedly mild sym turn #29 user: how did covid-19 impact biden's inauguration? asst: biden's inauguration was "a muted affair unlike any previous inauguration" du precautions… . . . turn #1 user: where is the toronto general hospital located and what makes it notable? asst: the toronto general hospital (tgh) is a major teaching hospital in toronto, ontario, canada and the flagship campus of university health network (uhn). it is located in the discovery district of downtown toronto… turn #2 user: what specific medical specialties are covered in the final two years of the doctor of medicine program at the university of toronto, and what types of training are provided during these years? asst: the doctor of medicine program at the university of toronto is a 4-year md program with a total enrollment of about 850 students. the first two years are known as the preclerkship curriculum… turn #3 user: which global university rankings place the school highest for clinical medicine and surgery in the years 2023 and 2020? asst: in 2023 the school was ranked 4th in the world for clinical medicine and surgery by "u.s. news & world report"… turn #4 user: what specific medical procedures and transplants does toronto general hospital specialize in and what are some notable achievements in these areas? asst: the emergency department now treats 28,065 persons each year, while the hospital also houses the major transplantation service for ontario… turn #5 user: which hospitals have merged to form the university health network (uhn) and when did these mergers occur? asst: a series of mergers over many years has resulted in the uhn in its current form… tions. the primary aim of our conversation synthesis pipeline is to generate a large-scale dataset that fundamentally improves context memory and understanding in multi-turn conversations during pre-training, emphasizing structural aspects such as topic transitions over stylistic naturalness. although this approach produces conversations that are stylistically less human-like, these aspects can be refined during post-training phases or through fine-tuning. future work could entail extending our approach to synthesize more natural multi-turn dialogues on a larger scale. human evaluation to ensure the quality of doctalk, we conducted a human evaluation as a form of quality control. we engaged 5 annotators to review 50 randomly selected dialogues from doctalk and assess their quality through a questionnaire. the questionnaire assessed various dimensions of the dialogues, including the contextual relevance of responses (q1), the specificity and accuracy of user utterances (q2), the amount of linguistic errors or level of grammatical correctness (q3), the naturalness and overall resemblance to human-llm conversation (q4), the thematic connection between topics (q5), and the logical consistency of topic transitions (q6). for further comparison, we also evaluated 50 dialogues from wikidialog using the same questionnaire (excluding q5 and q6 which are specific to doctalk). see appendix a.8 for further details. human evaluation results are presented in ## experiments we utilize the annealing data quality assessment approach widely used in other studies @xcite . we continue pre-train the mistral-7b-v0.3 † model for 8,000 steps (30b tokens), applying a learning rate schedule that annealed from 3e-5 to 3e-6. we keep some learning rate bandwidth for instruction fine-tuning using sharegpt data to prepare the model for benchmarking. for all our experiments, we construct a dataset comprising 25% conversational data, 5% project gutenberg books ‡ , and 70% redpajama v1 § web content. following this data composition, † https://huggingface.co/mistralai/mistral-7b-v0.3 ‡ books data in the redpajama dataset was replaced with project gutenberg content due to copyright concerns. ## results & discussion context memory & understanding coqa turnlevel and overall f1, precision, and recall scores are shown in figure 4 and the baseline pre-trained on doctalk showed significant performance gains on musique and 2wik-ihotpotqa. we hypothesize that the long multiturn conversations in doctalk serves to enhance the llm's long-context capabilities. in contrast, baselines pre-trained on doctalk w/o stage 2 or w/o stage 2 & 3 applied performed worse than the baselines pre-trained on raw wiki, emphasizing the dialogue graph's effectiveness in enhancing long-context performance. ## conclusion we present a conversational data synthesis pipeline that transforms web-crawled text into multi-turn, multi-topic dialogues, enhancing llms' context memory and understanding during pre-training. using this pipeline, we curate doctalk, the largest conversational dataset to date. future enhancements could refine the cr model by scoring tran-sitions with full dialogue context and extend the pipeline to general texts via semantic matching for document relatedness. future investigations could explore using these dialogues for grounded synthetic conversation generation, potentially enhancing conversational naturalness in data subsets for mid-training or post-training. additionally, the doctalk corpus mainly consists of straightforward question-and-answer exchanges. future work could explore incorporating more natural conversational dynamics, such as clarifications, misconceptions, requests for further details, or supporting evidence, to enhance the naturalness of the dialogues. it would be interesting to examine how these enhancements affect multi-turn conversational abilities.