Open-Domain Question Answering Goes Conversational via Question Rewriting (2021)

Abstract: We introduce a new dataset for Question Rewriting in Conversational Context (QReCC), which contains 14K conversations with 80K question-answer pairs. The task in QReCC is to find answers to conversational questions within a collection of 10M web pages (split into 54M passages). Answers to questions in the same conversation may be distributed across several web pages. QReCC provides annotations that allow us to train and evaluate individual subtasks of question rewriting, passage retrieval and reading comprehension required for the end-to-end conversational question answering (QA) task. We report the effectiveness of a strong baseline approach that combines the state-of-the-art model for question rewriting, and competitive models for open-domain QA. Our results set the first baseline for the QReCC dataset with F1 of 19.10, compared to the human upper bound of 75.45, indicating the difficulty of the setup and a large room for improvement.

## introduction it is often not possible to address a complex information need with a single question. consequently, there is a clear need to extend open-domain question answering (qa) to a conversational setting. this task is commonly referred to as conversational (interactive or sequential) qa @xcite @xcite . conversational qa requests an answer conditioned on both the question and the previous conversation turns as context. previously proposed large-scale benchmarks for conversational qa, such as quac and coqa, limit the topic of conversation to the content of a single document. in practice, however, the answers can be distributed across several documents ⭑ equal contribution. ♥ work done as an intern at apple inc. that are relevant to the conversation, or the topic of the conversation may also drift. to investigate this phenomena and develop approaches suitable for the complexities of this task, we introduce a new dataset for open-domain conversational qa, called qrecc. 1 the dataset consists of 13.6k conversations with an average of 6 turns per conversation. a conversation in qrecc consists of a sequence of question-answer pairs. the answers to questions were produced by human annotators, who looked up relevant information on the web using a search engine. qrecc is therefore the first large-scale dataset for conversational qa that incorporates an information retrieval subtask. qrecc is accompanied with scripts for building a collection of passages from the common crawl and the wayback machine for passage retrieval. qrecc is inspired by the task of question rewriting (qr) that allows us to reduce the task of conversational qa to non-conversational qa by 521 generating self-contained versions of contextuallydependent questions. qr was recently shown crucial for porting retrieval qa architectures to a conversational setting @xcite . follow-up questions in conversational qa often depend on the previous conversation turns due to ellipsis (missing content) and coreference (anaphora). every question-answer pair in qrecc is also annotated with a question rewrite. we evaluate the quality of these rewrites as self-contained questions in terms of the ability of the rewritten question, when used as input to the web search engine, to retrieve the correct answer. a snippet of a sample qrecc conversation is given in figure 1 . the dataset collection included two phases: to produce the first baseline, we augment an open-domain qa model with a qr component that allows us to extend it to a conversational scenario. we evaluate this approach on the qrecc dataset, reporting the end-to-end effectiveness as well as the effectiveness on the individual subtasks separately. our contributions. we collected the first largescale dataset for end-to-end, open-domain conversational qa that contains question rewrites that incorporate conversational context. we present a systematic comparison of existing automatic evaluation metrics on assessing the quality of question rewrites and show the metrics that best correlate with human judgement. we show empirically that qr provides a unified and effective solution for resolving references -both co-reference and ellipsis -in multi-turn dialogue setting and positively impacts the conversational qa task. we evaluate the dataset using a baseline that incorporates the state-of-the-art model in qr and competitive models for passage retrieval and answer extraction. this dataset provides a resource for the community to develop, evaluate, and advance methods for end-to-end, open-domain conversational qa. ## related work qrecc builds upon three publicly available datasets and further extends them to the opendomain conversational qa setting: question answering in context (quac) @xcite , trec conversational assistant track (cast) @xcite and natural questions (nq) @xcite . qrecc is the first large-scale dataset that supports the tasks of qr, passage retrieval, and reading comprehension (see open-domain qa. reading comprehension (rc) approaches were recently extended to incorporate a retrieval subtask @xcite @xcite . this task is also referred to as machine reading at scale @xcite or end-to-end qa @xcite . in this setup a reading comprehension component is preceded by a document retrieval component. the answer spans are extracted from documents retrieved from a document collection, given as input. the standard approach to end-to-end open-domain qa is (1) use an efficient filtering approach to reduce the number of candidate passages to the top-k of the most relevant ones (usually bm25 based on the bag-of-words representation); and then (2) re-rank the subset of the top-k relevant passages using a more fine-grained approach, such as bert based on vector representations @xcite . conversational qa. independently from end-toend qa, the rc task was extended to a conversational setting, in which answer extraction is conditioned not only on the question but also on the previous conversation turns @xcite . the first attempt at extending the task of information retrieval (ir) to a conversational setting was the recent trec cast 2019 task @xcite . the challenge was to rank passages from a passage collection by their relevance to an input question in the context of a conversation history. the size of the collection in cast 2019 was 38.4m passages, requiring efficient ir approaches. as efficient retrieval approaches operate on bag-ofwords representations they need a different way to handle conversational context since they can not be trained end-to-end using a latent representation of the conversational context. a solution to this computational bottleneck was a qr model that learns to sample tokens from the conversational context as a pre-processing step before qa. ## dialogue collection to simplify the data collection task we decided to use questions from pre-existing qa datasets as seeds for dialogues in qrecc. we used questions from quac, cast and nq. while quac and cast datasets contain question sequences, nq is not a conversational dataset but contains standalone questions from web search. we use the nq dataset to increase and diversify the number of samples beyond quac and cast by generating more rewrites for cases beyond coreference resolution. the majority of the follow-up questions in quac require coreference resolution for qr. therefore, we explicitly instructed the annotators to use nq as a start of a conversation and then come up with relevant follow-up questions, which would require generation of missing content, i.e., ellipsis, instead of coreference resolution for qr. the task for the annotators was also to answer questions using a web search engine. question rewrites were used as input to a search engine. this setup helps to obtain feedback on the quality of qr with respect to the effectiveness of answer retrieval (see section 6 for more details on using search results for the evaluating qr). finally, the questionanswer pair is annotated with the link to the web page that was used to produce the answer. thereby, every dialogue was produced by the same annotator including the questions, answers and rewrites. this design decision is called selfdialog technique that was shown to help improve quality of the data by avoiding some of the challenges observed in simulated dialogues produced by pairs of annotators @xcite . a team of 30 professional annotators with a project lead were employed to perform the task. the annotation task was described in the guidelines (see appendix b for more details). to ensure the quality of the annotations we followed a posthoc evaluation procedure, in which 5 reviewers go through the dataset and update incorrect examples they identify with consensus. ## dialogue analysis qrecc contains 13,598 dialogues with 79,952 questions in total. 9.3k dialogues are based on the questions from quac; 80 are from trec cast; and 4.4k are from nq. 9% of questions in qrecc do not have answers. we still retained the question rewrites even if no answer was found on the web. 112 questions were annotated with links to web pages without answer texts, e.g. "may i have a link to road signs in singapore?" we prepared three standard dataset splits and ensured that they are balanced in terms of the standard dialogue statistics and the types of qr (see the majority of questions in qrecc (52%) require replacement. figure 2 shows the tokens that are most frequently replaced in qr. all of them are pronouns that require anaphora resolution. by specifically targeting more rare types of question rewriting in our data collection task we managed to increase the proportion of the insertion cases in our dataset. this allows us to train and evaluate the ability of the model to reconstruct missing context, which cannot be achieved using traditional co-reference resolution approaches. ## document collection we download the web pages using the answer provenance links provided by the annotators from the internet archive wayback machine. after downloading the pages we extract the textual content from the html and split texts into passages of least 220 tokens. after segmentation, we have a total of 54m passages which we index using anserini @xcite . we search the passage collection using the human annotated answers to augment the dataset with alternative sources of correct answers. for each document returned, we identify the span in the document that has the highest token overlap (f1) with the human answer. we consider all documents with @xmath0.8 as relevant. verifying adequacy of this simple heuristic by human annotators is left for future work. ## question rewriting metrics validation bleu has typically been used in previous work for measuring the quality of qr @xcite . we conduct a systematic evaluation and compare bleu with alternative metrics, previously applied in summarization and translation, to ensure the most reliable metrics we can obtain for the model selection. our evaluation shows that bleu does not compare favourably with other metrics in evaluating the quality of qr. task. we took a random sample of 10k questions and used a seq-to-seq model @xcite trained with questions and conversation context from the qrecc dataset to generate question rewrites. these generated rewrites were compared to the ground truth rewrites produced by human annotators. different annotators graded each modelgenerated rewrite with a binary label: 0 (incorrect rewrite) or 1 (correct rewrite). for a question rewrite to be correct it does not have to exactly match the ground truth rewrite, but it should correctly capture the conversational context and be a self-contained question. for example, the modelgenerated rewrite "what are the global warming dangers?" is a correct rewrite with the ground truth rewrite being "what are the dangers of global warming?". in addition, we also assess the variance of the human assessments. the pearson correlation between any two annotators on average is 0.94. we observed the mean and the variance to be 0.083 and 0.076 respectively. performing a two-tail statistical significance test shows the p-value to be 0.0201. we use several automated metrics to compare the rewrites with the ground truth and compute their pearson correlation with the human judgements (see exact match is a binary variable that indicates the token set overlap applied after the standard preprocessing: lower-casing, stemming, punctuation and stopword removal. rouge @xcite reflects similarity between two texts in terms of n-gram overlap (r-1 for unigrams; r-2 for bigrams and r-l for the longest common n-gram). we report the mean for precision (p), recall (r) and f-measure (f). meteor (denkowski and lavie, 2014) is a machine translation metric based on exact, stem, synonym, and paraphrase matches between words and phrases. bleu @xcite is a text similarity metric that uses a modified form of precision and n-grams from candidate and reference texts. embeddings group several unsupervised approaches that produce a sentence-level vector representation: universal sentence encoder @xcite and infersent @xcite . search results -we use both question rewrites in google search and compare the overlap between the produced page ranks in terms of the standard ir metrics: recall@k for the top-k links, average recall (ar) and normalized discounted cumulative gain (ndcg). the best performing metric in our experiments (i.e., closest to the human judgement) is the set ## baseline approach we extend bertserini @xcite , an efficient approach to open-domain qa, with a qr model to incorporate conversational context. this approach consists of three stages: (1) qr, (2) pr and (3) rc. first, a model is trained to generate a stand-alone question given a follow-up question and the preceding question-answer pairs. in the second stage, pr, the top-k relevant passages are retrieved from the index using bm25 using the rewritten question. finally, in rc, a model is trained to extract an answer span from a passage or predict if the passage is irrelevant. the scores obtained from pr and rc are then combined as a weighted sum to produce the final score. the span with the highest score is chosen as the final answer. ## baseline results we use the results of qr to select the best model and then use it for the end-to-end qa task. question rewrites are used as input for both passage retrieval and reading comprehension tasks. the effectiveness of the qr component is compared with the end-to-end model conditioned on the conversational context. we analyze the effectiveness of our qr models by doing a 5-fold cross validation and obtaining the best performing metrics. figure 3 contains 3 plots showing rouge 1-r, use and r@10 across 5 turns. we start with the second turn because the first turn always is a self-contained query. the metrics across turns also stay stable with the same result for all the models. the transformer++ model is stable with little variance in terms of its maximum and minimum metric values across all the best performing metrics. our evaluation results are summarized in ## conclusion we introduced the qrecc dataset for open-domain conversational qa. qrecc is the first dataset to cover all the subtasks relevant for conversational qa, which include question rewriting, passage retrieval and reading comprehension. we also set the first end-to-end baseline results for qrecc by evaluating an open-domain qa model in combination with a qr model. we presented a systematic comparison of existing automatic evaluation metrics on assessing the quality of question rewrites and show the metrics that best proxy human judgement. our empirical evaluation shows that qr provides an effective solution for resolving both ellipsis and co-reference that allows to use existing non-conversational qa models in a conversational dialogue setting. our end-to-end baselines achieve an f1 score of 19.10, well beneath the 75.45 extractive upper bound, suggesting not only room for improvement in extractive conversational qa, but that more sophisticated abstractive techniques are required to successfully solve qrecc.