A Wrong Answer or a Wrong Question? An Intricate Relationship between Question Reformulation and Answer Selection in Conversational Question Answering (2020)

Abstract: The dependency between an adequate question formulation and correct answer selection is a very intriguing but still underexplored area. In this paper, we show that question rewriting (QR) of the conversational context allows to shed more light on this phenomenon and also use it to evaluate robustness of different answer selection approaches. We introduce a simple framework that enables an automated analysis of the conversational question answering (QA) performance using question rewrites, and present the results of this analysis on the TREC CAsT and QuAC (CANARD) datasets. Our experiments uncover sensitivity to question formulation of the popular state-of-the-art question answering approaches. Our results demonstrate that the reading comprehension model is insensitive to question formulation, while the passage ranking changes dramatically with a little variation in the input question. The benefit of QR is that it allows us to pinpoint and group such cases automatically. We show how to use this methodology to verify whether QA models are really learning the task or just finding shortcuts in the dataset, and better understand the frequent types of error they make.

## introduction conversational question answering (qa) is a new and important task which allows systems to advance from answering stand-alone questions to answering a sequence of related questions @xcite @xcite . such sequences contain questions that usually revolve around the same topic and its subtopics, which is also common for a human conversation. the most pronounced characteristics of such question sequences are anaphoric expressions and ellipsis, which make the follow-up questions ambiguous † work done as an intern at apple inc. question rewriting (qr) was recently introduced as an independent component for conversational qa @xcite @xcite . query rewriting received considerable attention in the information retrieval community before but not in a conversational context @xcite . @xcite performed similar experiments using query reformulations mined from search sessions. however, half of their samples were keyword queries rather than natural language questions and their dataset was not released to the community. in this paper, we show that qr is not only operational in extending standard qa models to the conversational scenario but can be also used for their evaluation. an input to the qr component is a question and previous conversation turns. the qr component is designed to transform all ambiguous questions, e.g., "when?", into their unambiguous equivalents, e.g., "when was dna discovered?". such unambiguous questions can be then processed by any standard qa model outside of the conversation history. clearly, the quality of question formulation interacts with the ability to answer this question. in this paper, we show that introducing qr component in the conversational qa architecture, by decoupling question interpretation in context from the question answering task, provides us with a unique opportunity to gain an insight on the interaction between the two tasks. ultimately, we are interested in whether this interaction can potentially help us to improve the performance on the end-to-end conversational qa task. to this end we formulate our main research question: how do differences in question formulation in a conversational setting affect question answering performance? the standard approach to qa evaluation is to measure model performance on a benchmark dataset with respect to the ground-truth answers, such as text span overlap in the reading comprehension task, or ndcg@3 in the passage retrieval task. however, such evaluation setups may also have their limitations with respect to the biases in the task formulation and insufficient data diversity that allow models to learn shortcuts and overfit the benchmark dataset @xcite . there is already an ample evidence of the pitfalls in the evaluation setup of the reading comprehension task highlighting that the state-of-the-art models tend to learn answering questions using superficial clues @xcite @xcite . we aim to further contribute to the research area studying robustness of qa models by extending the evaluation setup to the conversational qa task. to this end, we introduce an error analysis framework based on conversational question rewrites. the goal of the framework is to evaluate robustness of qa models using the inherent properties of the conversational setup itself. more specifically, we contrast the results obtained for ambiguous and rewritten questions outside of the conversation context. we show that this data is very well suited for analysing performance and debugging qa models. in our experiments, we evaluate two popular qa architectures proposed in the context of the conversational reading comprehension (quac) @xcite and conversational passage ranking (trec cast) @xcite tasks. our results show that the state-of-the-art models for passage retrieval are rather sensitive to differences in question formulation. on the other hand, the models trained on the reading comprehension task tend to find correct answers even to incomplete ambiguous questions. we believe that these findings can stimulate more research in this area and help to inform future evaluation setups for the conversational qa tasks. ## experimental setup to answer our research question and illustrate the application of the proposed evaluation framework in practice, we use the same experimental setup introduced in our earlier work @xcite . this architecture consists of two independent components: qr and qa. it was previously evaluated against competitive approaches and baselines setting the new state-of-the-art results on the trec cast 2019 dataset. the qr model was also shown to improve qa performance on both passage retrieval and reading comprehension tasks. in the following subsection, we describe the datasets, models, and metrics that were used in our evaluation. note, however, that our error analysis framework can be applied to other models and metrics as well. the only requirement for applying the framework is the qr-qa architecture that provides two separate outputs in terms of question rewrites and answers. we show that the same framework can be applied for both reading comprehension and passage retrieval tasks, although they produce different types of answers and require different evaluation metrics. ## our error analysis framework to trace how the difference in question formulation affects qa performance we compare the qr performance metrics with the qa performance metrics on the case-by-case basis, i.e., for each questionanswer pair. notice that we can apply the same approach for both retrieval and reading comprehension evaluation (see we assume human question rewrites as the ground truth. therefore, all cases in which these rewrites did not result in a correct answer are errors of the qa component (rows 1-4 in tables 1 2 ). the next two rows 5-6 show the cases, where human rewrites succeeded but the model rewrites failed, which we consider to be a likely error of the qr component. the last two rows are true positives for our model, where the last row combines cases where the original question was just copied without rewriting (numbers in brackets) and other cases this plot visualises the difference between the error distributions reported in tables 1-2 with a sliding cut-off threshold. the blue region at the top of the plots represents the proportion of errors in qa. the orange region represents the proportion of errors in qr. the light green region shows the proportion of samples that were both rewritten and answered correctly. the pink region at the bottom shows the importance of qr for the task. the larger the region the more questions were answered correctly without any rewriting. we observe that the difference between the two plots is very pronounced. the impact of qr is noticeable in passage retrieval, which is more sensitive to question formulation than the reading comprehension model. when rewriting was not required. since there is no single binary measure for the answer correctness, we can pick different cutoff thresholds for the qa metrics. for example, p@1=1 will consider the answer correct if it came up at the top of the ranking; or @xmath0, i.e, exact matches only. figure 1 extends this analysis by considering all thresholds in the range [0; 1] with 0.02 intervals for ndcg@3 in retrieval and f1 in reading comprehension. this figure shows the proportion of different error types as well as the results sensitivity to the choice of the performance threshold. ## evaluation results our approach allows us to estimate that the majority of errors stem from the qa model: 29% of the test samples for retrieval and 55% for reading comprehension. 11% of errors can be directly attributed to qr in the retrieval setup and 5% in reading comprehension. to estimate the impact of qr on qa, we consider only the last four rows in the proportion of qr errors for retrieval setup is higher than for reading comprehension setup. in particular, we found that the majority of questions in canard test set (77% @xmath2, i.e., even when the questions are ambiguous. for trec cast, the chances of reaching the correct answer set using an ambiguous question are much lower (21% p@1 = 1). see tables 3 4 ## qa-qr correlation in this section, we check whether the qr metrics can predict the qa performance for the individual questions by measuring the correlation between the qr and qa metrics. this analysis shows how the change in question formulation affects the answer selection. in other words, we are interested whether similar questions also produce similar answers, and whether distinct questions result in distinct answers. to discover the correlation between question and answer similarity, we discarded all samples, where the human rewrites do not lead to the correct answers (top 4 rows in tables 1 2 ). the remaining subset contains only the samples in which the qa model was able to find the correct answer. we then compute rouge for the pairs of human and generated rewrites, and measure its correlation with p@1 to check if rewrites similar to the correct question will also produce correct answers, and vice versa. there is a strong correlation for @xmath3, i.e., when the generated rewrite is very close to the human one. however, when @xmath4. even for rewrites that have a relatively small lexical overlap with the groundtruth (@xmath5.4) it is possible to retrieve a correct answer, and vice versa. we further explore the effect of the qr quality on the qa results by comparing differences of the answer sets produced when given different rewrites. we compare answers produced separately for human and model rewrites using the same input question. however, this time we look at all the answers produced by the qa model irrespective of whether the answers were correct or not. this setup allows us to better observe how much the change in the question formulation triggers the change in the produced answer. figure 2 demonstrates strong correlation between the question similarity, as measured by rouge, and the answer set similarity. we measured the similarity between the top-1000 answers returned for the human rewrites and the generated rewrites by computing recall (r@1000). points in the bottom right of this plot show sensitivity of the qa component, where similar questions lead to different answer rankings. the data points that are close to the top center area indicate weaknesses of the qr metric as a proxy for the qa results: often questions do not have to be the same as the ground truth questions to solicit the same answers. the blank area in the top-left of the diagonal shows that a lexical overlap is required to produce the same answer set, which is likely due to the candidate filtering phase based on the bag-of-word representation matching. we also compared rouge and jaccard similarity for the tokens in reading comprehension results but they showed only a weak correlation (pearson 0.31). this result confirms our observation that the extractive model tends to be rather sensitive to a slight input perturbation but will also often provide the same answer to very distinct questions. thus, our results show that the existing qr metrics have a limited capacity in predicting the performance on the end-to-end qa task. rouge correlates with the answer recall for the retrieval model, but cannot be used to reliably predict answer correctness for the individual question rewrites. since rouge treats all the tokens equally (except for the stop-word list) it is unable to capture importance of the individual tokens that modify question semantics. the correlation of the qa performance with the embedding-based use metric is even lower than with rouge for the both qa models. ## discussion we showed that qa results can identify question paraphrases. however, this property directly depends on the ability of the qa model to match equivalent questions to the same answer set. wrong answer or wrong question? our passage retrieval model is using bm25 as a filtering step, which relies on the lexical match between the terms in the question and the terms in the passage. hence, synonyms, like "large" and "big", cannot be matched with this model. this effect explains that dissimilar questions are never matched to the same answers in figure 2 . the drawback of this model is that it suffers from the "vocabulary mismatch" problem @xcite . a one-word difference between a pair of questions may have a very little as well as a very dramatic effect on the question interpretation. this class of errors corresponds to the variance evident from figure 2 . our error analysis indicates that small perturbations of the input question, such as anaphora resolution, often cause a considerable change in the answer ranking. for example, the pair of the original question: "who are the hamilton electors and what were they trying to do?", and the human rewrite: "who are the hamilton electors and what were the hamilton electors trying to do?" produce @xmath6.33. we also identified many cases in which inability of the qr component to generate apostrophes resulted in incorrect answers (original question: "describe netflixs subscriber growth over time", and the human rewrite: "describe netflix's subscriber growth over time"). in contrast, our reading comprehension models is based solely on the dense vector representations, which should be able to deal with paraphrases. in practice, we see from figure 3 that this feature may also introduce an important drawback, when the model produces a correct answer even when given an incorrectly formulated question. this stability and undersensitivity of the reading comprehension model may indicate biases in the dataset and evaluation setup. when the answers no longer depends on the question being asked, but can be also predicted independently from the question, the qa model evidently fails to learn the mechanisms underlying qa. instead, it may learn a shortcut in the benchmark dataset that allows to guess correct questions, such as likely answer positioning within an article. our experiments show that the retrieval-based setup is more adequate in judging model robustness. the size of the answer space is sufficiently large so as to exclude spurious cues that the model can exploit for shortcut learning. the role of qr is, therefore, much more evident in this task, since to be able to find the correct answer the question has to be formulated well by disambiguating conversational context. while it may be redundant to overspecify the question when given several passages to choose the answer from, it becomes of a vital importance when given several million passages. this argument holds, however, only when assuming a uniform answer distribution, which is often not the case unless for sufficiently large web collections. this particular observation brings us to the next question that we would like to discuss in more detail. rewrite or not to rewrite? our experiments demonstrated the challenges in the quality control of the qr task itself. human rewrites are not perfect themselves since it is not always clear whether and what should be rewritten. considering the human judgment of the qr quality independent from the qa results, little deviations may not seem important. however, from the pragmatic point of view, they may have a major impact on the overall performance of the end-to-end system. the level of detail required to answer a particular question is often not apparent and depends on the dataset. however, we can argue that inconsistencies, such as typos and paraphrases, should be handled by the qa component, since they do not originate from the context interpretation errors. further on, we evaluated our assumption about human rewrites as the reliable ground truth. our evaluation results indeed showed that human rewriting was redundant in certain cases. there are cases in which original questions without rewriting were already sufficient to retrieve the correct answers from the passage collection (see last row of the the need for qr essentially interacts with the size and diversity of the possible answer space, e.g., collection content, with respect to the question. some of the questions were correctly answered even with underspecified questions, e.g., original question: "what are some ways to avoid injury?", human rewrite: "what are some ways to avoid sports injuries?", because of the collection bias. the goal of qr is to learn patterns that correct question formulation independent from the collection content. this approach is similar to how humans handle this task when formulating search queries, i.e., based on their knowledge of language and the world. humans can spot ambiguity regardless of the information source by modeling the expected content of the information source even without the need to have direct access to it. clearly, this expectation may not be optimal or even sufficient to be able to formulate a single precise question, which is exactly the point at which the need for an information-seeking dialogue naturally arises. when the question formulation procedure is designed to be independent of the collection content, it also allows for querying several sources with the same question. this property is especially helpful when the content of the collection is unknown, which also allows to work with 3rd party apis to access information in a distributed fashion. we may see a similar effect as in human-human communication here as well, when the need for an automated information-seeking dialogue between distributed systems may arise to better negotiate the information need and disambiguate the question further with respect to the content of each remote information source. ## related work several recent research studies reported that stateof-the-art machine learning models lack in robustness across several nlp tasks @xcite . in particular, these models were found to be sensitive to input perturbations. @xcite analysed this problem in the context of the reading comprehension task. they showed in the experimental evaluation that qa models suffer from overstability, i.e., they tend to provide the same answer to a different question, when it is sufficiently similar to the correct question. @xcite also showed this deficiency of the state-of-the-art models for reading comprehension that learn to attend to just a few words in the question. our evaluation results support these findings using the conversational settings. we showed that many ambiguous questions in quac can be answered correctly without the conversation history (see previous work that focused on analysing and explaining performance of transformer-based models on the ranking task used word attention visualisation and random removals of non-stop words @xcite . the analysis framework proposed here provides a more systematic approach to model evaluation using ambiguous and rewritten question pairs, which is a by-product of applying qr in conversational qa. our approach is most similar to the one proposed by @xcite , who used semantically equivalent adversaries to analyse performance in several nlp tasks, including reading comprehen-sion, visual qa and sentiment analysis. similarly, in our approach, question rewrites, as paraphrases with different levels of ambiguity, are a natural choice for evaluation of the conversational qa performance. the difference is that we do not need to generate semantically equivalent questions but can reuse question rewrites as a by-product of introducing the qr model as a component of the conversational qa architecture. ## conclusion qr is a challenging but a very insightful task designed to capture linguistic patterns that identify and resolve ambiguity in question formulation. our results demonstrate the utility of qr as not only enabler for conversational qa but also as a tool that helps to understand when qa models fail. we introduced an effective error analysis framework for conversational qa using qr and used it to evaluate sensitivity of two state-of-the-art qa model architectures (for reading comprehension and passage retrieval tasks). moreover, the framework we introduced is agnostic to the model architecture and can be reused for performance evaluation of different models using other evaluation metrics as well. qr helps to analyse model performance and discover their weaknesses, such as oversensitivity and undersensitivity to differences in question formulation. in particular, the reading comprehension task setup is inadequate to reflect the real performance of a question interpretation component since ambiguous or even incorrect question formulations are likely to result in a correct answer span. in passage retrieval, we observe an opposite effect. since the space of possible answers is very large it is impossible to hit the correct answer by chance. we discover, however, that these models tend to suffer from oversensitivity instead, i.e., when even a single character will trigger a considerable change in answer ranking. in future work, we should extend our evaluation to dense passage retrieval models and examine their performance using the conversational qa setup with the qr model. we should also look into training approaches that could allow qa models to further benefit from the qr component. as we showed in our experiments qr provides useful intermediate outputs that can be interpreted by humans and used for evaluation. we believe that qr models can be also useful for training more ro-bust qa models. both components can be trained jointly, which is inline with @xcite , who showed that the joint objective of question and answer generation further improves qa performance.