Building Open-Retrieval Conversational Question Answering Systems by Generating Synthetic Data and Decontextualizing User Questions (2025)

Abstract: We consider open-retrieval conversational question answering (OR-CONVQA), an extension of question answering where system responses need to be (i) aware of dialog history and (ii) grounded in documents (or document fragments) retrieved per question. Domain-specific OR-CONVQA training datasets are crucial for real-world applications, but hard to obtain. We propose a pipeline that capitalizes on the abundance of plain text documents in organizations (e.g., product documentation) to automatically produce realistic OR-CONVQA dialogs with annotations. Similarly to real-world humanannotated OR-CONVQA datasets, we generate in-dialog question-answer pairs, self-contained (decontextualized, e.g., no referring expressions) versions of user questions, and propositions (sentences expressing prominent information from the documents) the system responses are grounded in. We show how the synthetic dialogs can be used to train efficient question rewriters that decontextualize user questions, allowing existing dialog-unaware retrievers to be utilized. The retrieved information and the decontextualized question are then passed on to an LLM that generates the system’s response.

## introduction retrieval-augmented generation (rag) is used to ground large language models (llms) to knowledge outside of their training data and limit hallucinations @xcite . rag is especially applicable to conversational agents, enabling them to provide responses grounded in retrieved documents. we focus on open-retrieval conversational question answering (or-convqa), an extension of question answering where system responses need to be (i) aware of the dialog history and (ii) grounded in the retrieved documents (retrieved per question). compared to conventional information retrieval (ir) @xcite , or-convqa introduces two challenges. firstly, the system needs to account for the additional context of the dialog @xcite , mostly the dialog history (previous system and user turns). solely relying on the last user question to query the document repository may result in suboptimal answers, since discourse phenomena like ellipsis and co-reference are prevalent in dialogs @xcite @xcite . thus, the dialog history has to be considered jointly with the last user question, which becomes an issue when the history includes information irrelevant to the last question, or the history is too long. alternatively, a separate model may produce a self-contained ('decontextualized') version of the last user question (e.g., with no ellipsis, anaphora) @xcite @xcite , allowing the use of existing dialog-unaware retrievers, which expect a stand-alone query. this approach, query reformulation, either rewrites the user question to include all relevant information or appends relevant tokens from the dialog history @xcite . a second challenge in or-convqa is the lack of domain-specific data and annotations @xcite , which are crucial to train real-life systems. collecting and especially manually annotating new dialog data for specific domains is particularly cumbersome. alternatives, such as dialog inpainting @xcite @xcite or synthesizing dialogs from scratch @xcite , generate synthetic data from domainspecific documents, which are abundant in practice (e.g., product documentation, recommendation guidelines). however, previous alternatives of this kind make unrealistic assumptions, such as presuming a one-to-one correspondence between document sentences and possible user questions, and/or assuming that additional manually annotated dialogs are available to fine-tune system components @xcite . motivated by such issues, we propose a pipeline to generate realistic synthetic, document-grounded or-convqa dialogs and annotations. like pre-vious approaches, we use domain-specific documents, but without requiring any additional training data and without assuming a one-to-one mapping between document sentences and user questions. the pipeline first prompts an llm, to generate propositions from the documents in the repository. similarly to @xcite , we require each proposition to be a stand-alone simple sentence (e.g., without compound sentences, anaphora or ellipsis) expressing information from a document. unlike @xcite , however, we require each proposition to convey information important enough to be requested by a user query. some document sentences may not be used in any of our propositions (hence, they may not be used to answer any question), and some questions may require information from multiple propositions. the retrieval pool may then contain the propositions, not the original documents or document fragments, making it easier to retrieve the information needed by a user question, without irrelevant information. the pipeline then prompts the same llm to generate or-convqa dialogs from sampled propositions. each dialogic pair (user and system turn) includes a contextualized (dependent on dialog history) and decontextualized (self-contained) version of the user question, the corresponding system response (answer), and the propositions used to generate the question and the response. we experimentally show the superiority of dialogs generated through our propositions compared to using directly document sentences, by measuring the coherence of the dialogs, their relevance to the knowledge they are grounded in, and improvements in retrieval scores. to demonstrate the usefulness of the generated synthetic dialogs, we show how they can be used to fine-tune light models as question rewriters or retrievers. the retrieved information and the user questions are then given to an llm that produces the system response. we verify the effectiveness and efficiency of our fine-tuned models on both synthetic and real-world test data, comparing against rewriting questions by prompting larger llms, or using the last user query (with or without concatenating the dialog history). we also propose a new mechanism to detect questions that are already self-contained and do not require rewriting, improving inference speed further. we leave for future work the question of how to use synthetic data to fine-tune lighter response generation models too, instead of prompting larger llms for response generation. overall, our main contributions are: (1) we propose a pipeline to create high-quality synthetic annotated or-convqa dialogs from domain-specific documents, without requiring any manually annotated training data and without assuming a one-toone mapping between document sentences and user questions. (2) we demonstrate the superiority of synthetic dialogs generated by first converting the documents to propositions that capture important information, compared to directly using document sentences. (3) we show how the generated synthetic dialogs can be used to fine-tune light question rewriters or retrievers. (4) we make publicly available our source code and a synthetic or-convqa dataset to facilitate future research. foot_0 2 related work ## conclusions and future work we presented a new pipeline that generates annotated document-grounded dialogs, to alleviate the lack of training data in new application domains, requiring only a set of relevant domain-specific documents. we highlighted the importance of using propositions, rather than document sentences, for dialog generation, and showed that they lead to synthetic dialogs that are clearly superior in our experimental evaluation. using only our synthetic data, we trained both light question rewriters and a retriever. we showed that they substantially improve performance, compared to using the original questions with or without dialog history, and that their performance is comparable to obtaining rewrites by prompting llms. we also introduced a joint efficient question classifier/rewriter. in future work, we plan to explore means of generating more dialogs given the same set of documents (e.g., data augmentation), which will allow us to fine-tune the response-generator as well. finally, as dialogs in low resource languages are scarcer still, we plan to extend our pipeline to such languages, exploiting multilingual llms or machine translation.