Synthesizing Conversations from Unlabeled Documents using Automatic Response Segmentation (2024)

Abstract: In this study, we tackle the challenge of inadequate and costly training data that has hindered the development of conversational question answering (ConvQA) systems. Enterprises have a large corpus of diverse internal documents. Instead of relying on a searching engine, a more compelling approach for people to comprehend these documents is to create a dialogue system. In this paper, we propose a robust dialog synthesising method. We learn the segmentation of data for the dialog task instead of using segmenting at sentence boundaries. The synthetic dataset generated by our proposed method achieves superior quality when compared to WikiDialog, as assessed through machine and human evaluations. By employing our inpainted data for ConvQA retrieval system pre-training, we observed a notable improvement in performance across OR-QuAC benchmarks.

## introduction conversational question answering (convqa) is a computational task aimed at modeling the information-seeking processes found in human dialog. the goal of this task is to allow automated systems to understand and respond to questions within a conversational context. several publicly available datasets, such as quac @xcite , coqa @xcite , qrecc @xcite , and or-quac @xcite , have been developed for building convqa systems. despite these resources, the size of the datasets remains relatively limited, posing potential challenges when implementing convqa systems in real-world applications. meanwhile, a plethora of high-quality documents, including but not limited to sources such rq: where did the czechs settle in argentina? text from wikipedia wd: wikidialog question rq: ours figure 1 : an example dialog from wikidialog (wd) and ours (rq). the blue and yellow boxes in the dialog contain the questions generated by our approach, while the green boxes contain the corresponding answers. wd asks a question starting with "are there any other .." which is not useful to train a question answering system. besides, some of the answers can be combined such as the last two on the left side. in contrast, our method fixed those problems. as wikipedia and arxiv, are publicly available. numerous technological roadmaps have been proposed to leverage the vast wealth of information within these documents to construct a convqa system. one approach involves utilizing a large language model (llm) in conjunction with information retrieval techniques, such as new bing from microsoft. alternatively, a well-trained llm, such as chatgpt without a plugin, can be employed independently to achieve similar or even superior results. by utilizing information retrieval tools, llms can access up-to-date information, albeit at the expense of increased inference time and latency compared to using llms alone. to effectively employ these documents alongside llms for the purpose of constructing a con-vqa system, there are two potential research directions to explore: the decomposition and synthesis of the documents into a dialog format, or the pursuit of improved question embedding representations for the documents. additionally, conversational question generation (qg) can be utilized in both of these approaches. @xcite pioneered "dialog inpainting," suggesting every sentence in a document can answer a question, leading to the creation of the wikidialog dataset. in our study, we introduce syncars, a novel approach that leverages convqa datasets to produce synthetic conversations from unlabeled documents, yielding superior quality compared to wikidialog. to summarize, our main contributions can be summarized as follows: ## related work question generation (qg) is a field that seeks to create natural questions using various types of data sources, including structured knowledge bases @xcite ), text @xcite @xcite , images @xcite , and tables @xcite . past research efforts in this area have primarily focussed on producing isolated and disconnected questions from a given passage. @xcite proposed the conversational question generation (cqg) task as an approach to improve the development of convqa systems. this task involves generating the subsequent question by incorporating a passage and a conversation history, thereby requiring a deeper comprehension of the given passage and prior conversation to generate a coherent and relevant question for the next round. unlike prior qg tasks that only consider the passage, cqg requires an understanding of the previous conversation, making it a more complex task. @xcite proposed simseek, a framework that generates convqa datasets by modeling the information needs of questioners who may ask incoherent questions due to excessive information. simseek includes a conversational answer extractor that selects answer candidates from the passage by considering the context of the conversation. however, this method is only suitable for short answers. in contrast, dai et al. (2022) introduced dialog inpainting, which assumes that each sentence in a document can be used as an answer to a question. the authors generated a convqa dataset called wikidialog using this approach. this dataset tends to have longer answers as each answer corresponds to a single sentence. this characteristic makes it more suitable for dialog applications. while the proposed method is straightforward and efficient, concerns arise regarding the quality of the wikidialog dataset. an illustrative example from the wikidialog dataset is presented in figure 1 . from this example, it is evident that combining certain answers could yield improved responses and questions. moreover, some questions are overly broad, rendering them less suitable for training a retriever system. in the context of the open-qa dataset, the "anything else" question serves as a means to transition between topics. however, when examining a brief paragraph, typically containing around six sentences or less, it becomes challenging to discern any significant shifts in the topic. ## syncars problem statement: syncars (synthesizing conversations using automatic response segmentation) aims to generate a high-quality complete dialog from an informative document. it assumes that at most n continuous sentences where (@xmath0. we build our work on top of the dialog inpaint-ing @xcite , where each sentence is treated as an answer (equivalent to @xmath1. our idea stems from the observation that not all sentences are equally informative in wikidialog, the dataset generated by dialog inpainting. figure 1 shows some examples that demonstrate the limitations of the wikidialog dataset. to synthesize better convqa datasets, we implemented a simple sentence segmentation mechanism, along with a few modifications to the dialog inpainting method. in the following section, we will introduce these components in more detail. ## evaluation our primary focus in this study centers around conducting a human evaluation to compare the outputs generated by our approach with those produced by wikidialog (wd) using dialog inpainting method. to carry out this assessment, we utilize amazon mechanical turk (mturk) as our platform. each human annotator is compensated at a rate of 0.036 us dollars per question, and we provide them with identical instructions and sample examples as outlined in figures 6 to 10 in the work of @xcite . a potential issue in dai et al. (2022) lies in their use of the same raters for evaluations, which could introduce bias due to the evaluators' perception of the subject matter. to mitigate this concern, our evaluation process involves presenting subjective questions to a minimum of three distinct human evaluators for each dialogue turn. we label the answer for each question as the one agreed upon by at least two human annotators. given that mturk offers a more diverse pool of human evaluators, we use two-proportion z-test for evaluations. we report our findings based on dialogs that correspond to a set of 50 selected passages foot_0 . recent studies suggest using large language models (llms) as reference-free metrics for evaluating natural language generation (nlg) tasks. llms have the advantage of being applicable to new tasks that lack human-generated reference texts @xcite . furthermore, faysse et al. (2023) demonstrate that llms are more aligned with human preferences and exhibit consistent performance across a diverse set of generative tasks. in this study, we utilized the openai chat completion api with the gpt-4 model @xcite to perform the same evaluation as human annotators. we used the exact same rubric as @xcite . the rubrics and prompt templates used for both human and gpt-4 evaluations are provided in table a .4 in the appendix. the results of this evaluation are presented in ## application to conversational retrieval as mentioned earlier in the introduction section, there are few ways to utilize the generated dataset, for e.g., use the dataset to train chatbot directly or transform it as a open-domain conversation retrieval task. in this section, we will focus on opendomain conversational retrieval, as datasets such as or-quac @xcite , and trec-cast-20 @xcite exist in this domain. a convqa system interacts with a user in a multi-turn dialogue, where the user primarily asks questions and the system responds (with occasional exceptions, such as the system asking for clarification). when it is the system's turn to speak at a given time t, it considers the entire dialogue history, comprising all previous turns, and generates a new utterance as its response. this work focuses on the conversational retriever, showing how to improve it by pre-training on our dataset comparing to wd, leaving improvements to the generator for future work. ## conclusion in this paper, we introduce syncars, a novel approach that generates high-quality synthetic conversations from unlabeled documents by leveraging convqa datasets. our method outperforms existing benchmarks in terms of the quality of the generated conversations. to generate high-quality synthetic data, we developed an answer segmentation technique that incorporates a special token and curated a new dataset. despite using a smaller model and lower compute, human evaluations show that our generated dataset surpasses the wikidialog dataset in terms of answer quality and question specificity. this demonstrates the effectiveness of the proposed approach. moreover, the question retrieval system trained on our dataset outperforms both the standard retrieval-only benchmark and the same model trained using the wikidialog dataset. we believe our contributions will facilitate future progress in the development of document-based conversational systems.