Generating Information-Seeking Conversations from Unlabeled Documents (2022)

Abstract: Synthesizing datasets for conversational question answering (CQA) from unlabeled documents remains challenging due to its interactive nature.Moreover, while modeling information needs is an essential key, only few studies have discussed it.In this paper, we introduce a novel framework, **SimSeek**, (**Sim**ulating information-**Seek**ing conversation from unlabeled documents), and compare its two variants.In our baseline, **SimSeek-sym**, a questioner generates follow-up questions upon the predetermined answer by an answerer.On the contrary, **SimSeek-asym** first generates the question and then finds its corresponding answer under the conversational context.Our experiments show that they can synthesize effective training resources for CQA and conversational search tasks.As a result, conversations from **SimSeek-asym** not only make more improvements in our experiments but also are favorably reviewed in a human evaluation.We finally release a large-scale resource of synthetic conversations, **Wiki-SimSeek**, containing 2 million CQA pairs built upon Wikipedia documents.With the dataset, our CQA model achieves the state-of-the-art performance on a recent CQA benchmark, QuAC.The code and dataset are available at https://github.com/naver-ai/simseek

## introduction conversational question answering (cqa) involves modeling the information-seeking process of human dialogue. in the task, systems should understand questions according to the conversational context. to build robust systems, large-scale cqa datasets @xcite @xcite have recently been introduced. still, they are limited in scale to generalize toward real-world applications, ♣ work done while interning at naver ai lab * equal contribution † corresponding author 1 the code and dataset are available at github.com/naverai/simseek. ## background in the information-seeking conversation, two agents (i.e., questioner and answerer) converse about the specific topic. to provide accurate knowledge to the questioner, the answerer can utilize the document that consists of answer-containing passage c and its background knowledge b (i.e., the title and abstract). let q t be the current question and a t be its corresponding answer at turn t. formally, cqa systems are required to find correct answer a t to the question q t from the passage c based on the conversational history @xmath0, a 1 ), ..., (q t-1 , a t-1 )], i.e. p(a t | q t , c, @xmath1. typically, most cqg research assumes that the questioner can access the answer-containing passage c. hence, they formulate the task of generating the question q t based on the passage c and answer a t , i.e. p(q t | c, a t , @xmath2. the formulation can be considered a straightforward extension of the dominant paradigm in single-turn qa generation @xcite , where models generate questions given their answers. recently, @xcite suggest a new viewpoint to promote a natural scenario of information-seeking conversation. in the setup, cqg modules are blinded to the answer-containing passage and rely on background information b when generating conversational question q t , i.e. p(q t | b, @xmath3. ## simseek we newly introduce two opposite ways to simulate synthetic conversations from unlabeled documents, simseek-sym and simseek-asym, as illustrated in figure 2 . ## evaluating synthetic conversations we evaluate our simseek in the semi-supervised setup. to this end, we train all components on the existing cqa dataset d first. then, synthetic conversations are generated upon unseen documents c by our frameworks. the resulting conversations are used as an additional training resource for downstream tasks. we train task-specific backbones on the synthetic datasets and test their performances in two downstream tasks, cqa and conversational search. see more details in appendix a, c. ## analysis we report detailed statistics of the generated datasets and perform a human evaluation to analyze the quality of conversations and compare them. ## wiki-simseek based on our results, we newly construct synthetic conversations on the larger scale of corpus, wikipedia, by applying simseek-asym. we finally release a large-scale resource of informationseeking conversations, wiki-simseek, which consists of 2.1 million questions and answer pairs upon 213k wikipedia passages. ## related work conversational question answering with the advent of recent large-scale cqa datasets @xcite , numerous studies proposed methods to resolve the challenging task. most works focused on developing model structures @xcite that are specialized in the cqa task. several works demonstrated the effectiveness of the flow mechanism in cqa @xcite . most recently, leveraging self-contained questions @xcite or encoding longer context @xcite have been shown to be effective in the task. synthetic qa generation many question generation (qg) researches have sparked advances in various qa tasks @xcite @xcite @xcite . most of early studies propose to generate them in a cloze-style @xcite or by using pre-defined templates @xcite . recent studies for the synthetic qa generation propose the pipeline strategies composed of three sub-phases, answer extraction, question generation, and various filtering steps such as round-trip filtration @xcite @xcite . conversational question generation many works attempt to generate human-like conversational questions. @xcite ; @xcite introduce the challenge of cqg and successfully extend the single-turn question generation to consider conversational input. most prior works are based on the information-symmetric assumption @xcite @xcite . recently, qi et al. ## conclusion in this work, we propose a novel framework, sim-seek, simulating information-seeking conversation from given unlabeled documents. our frameworks assume two scenarios and compare them to provide a deeper understanding of informationseeking conversation. experimental result shows that our simseek-asym generates human-like conversation. moreover, we provide insightful analyses to help understand the information-seeking conversation better. we finally release the largescale resources of synthetic conversations, wiki-simseek. we hope it could be a stepping stone for building robust cqa models that can be generalized toward the real-world scenario. furthermore, it could be beneficial to identify the factors in realistic information-seeking conversations.