Asking Clarification Questions to Handle Ambiguity in Open-Domain QA (2023)

Abstract: Ambiguous questions persist in open-domain question answering, because formulating a precise question with a unique answer is often challenging. Previous works have tackled this issue by asking disambiguated questions for all possible interpretations of the ambiguous question. Instead, we propose to ask a clarification question, where the user’s response will help identify the interpretation that best aligns with the user’s intention. We first present CAmbigNQ, a dataset consisting of 5,653 ambiguous questions, each with relevant passages, possible answers, and a clarification question. The clarification questions were efficiently created by generating them using InstructGPT and manually revising them as necessary. We then define a pipeline of three tasks—(1) ambiguity detection, (2) clarification question generation, and (3) clarification-based QA. In the process, we adopt or design appropriate evaluation metrics to facilitate sound research. Lastly, we achieve F1 of 61.3, 25.1, and 40.5 on the three tasks, demonstrating the need for further improvements while providing competitive baselines for future work.

## introduction in open-domain question answering (odqa), questions can often be interpreted in several ways, each with a distinct answer @xcite . for example, consider the question at the top of figure 1 . though the question seems unambiguous, "young tom riddle" can mean "young version in series 2", "child version in series 6", or "teenager version in series 6". such ambiguity needs to be resolved to correctly find the answer sought by the user. previous studies propose to handle ambiguous questions (aqs) by generating a disambiguated question (dq; disambiguated variation of the given aq) for each possible interpretation @xcite @xcite . while such dq-based approaches are an important step toward resolving ambiguities in odqa, imagine how it would be deployed in real life; without knowing the user's intention, the qa system would have to list all possible answers to the user, as shown in figure 1 @xcite . this is not suitable in most real-world scenarios where qa systems communicate with their users through speech or smallscreen devices @xcite @xcite . instead, we propose to prompt the user with a clarification question (cq), as shown in figure 1(b) . more specifically, given an aq, the goal is to ask a cq consisting of the possible interpretations as options (e.g. "teenager in series 6") along with a category summarizing the options (e.g. "version"). ideally, the user's response to the cq would help identify the interpretation that best aligns with the user's intention, and the corresponding answer can be presented to the user. this cq-based approach is not only applicable in the aforementioned realworld scenarios, but also shown to be preferred by users according to our preference test. this is also consistent with the finding that asking cqs can improve user experience with "limited bandwidth" interfaces @xcite . to support research on cq-based approaches to handle aqs in odqa, we present clarifying ambiguous natural questions (cambignq). cam-bignq is a dataset consisting of 5,653 aqs from ambignq @xcite , each accompanied by relevant passages, possible answers, and a newly created cq. the cqs were first generated using in-structgpt @xcite through in-context few-shot learning, then manually vetted and edited as necessary by human editors. such humanmachine collaboration for constructing corpora has been shown to significantly reduce the time and the cost from fully manual approaches @xcite . we also define a pipeline of three tasks to handle aqs in odqa-(1) ambiguity detection, (2) clarification question generation, and (3) clarificationbased qa. in the process, we adopt or design appropriate evaluation metrics to facilitate sound research. the experiments show that though they were shown to be helpful for generating dqs, predicted answers for aq do not help improve the cq-based approach overall. lastly, we achieve f1 of 61.3, 25.1, and 40.5 on the three tasks, demonstrating the need for further improvements while providing competitive baselines for future work. 1 our main contributions are threefold: clarification question evaluation there are several options for evaluating the quality of cqs, first is leveraging widely-used automatic text evaluation metrics, such as bleu @xcite or rouge @xcite . however, due to the poor correlations between such scores and human evaluation, zamani et al. (2020b) strongly discourages the use of such metrics for evaluation. second is human evaluation. while it typically provides a reliable estimate of how people would think of the given cqs, it can be time-consuming ? ambiguous question relevant passages ! response to clarification question ? clarification question clarification question generator qa model ! answer ? question ambiguity detector ! answer qa model relevant passages ? unambiguous question am big uo us and costly. as a third option, evaluation methods using external neural models have recently been introduced @xcite @xcite . this approach improves on the first, without the burden of the second. in this work, we design evaluation methods that suit our tasks, leveraging external neural models to provide a more comprehensive and accurate assessment. ## task overview we propose to handle aqs in odqa by asking cqs as shown in figure 2 . there are three tasks: ## experiment 1: cq vs dq preference we first conduct a human preference test to investigate the question: is our cq-based approach preferred over a dq-based one to handle aqs in odqa? this is to check if it is worthwhile to pursue the cq-based approach. setup we randomly sampled 100 aqs from the development set. then, for each aq, we asked three annotators to show their preferences for "cq", "dq", or "equal", along with the rationale. that is, given an aq, we ask people to compare "being presented with answers to all possible interpretations (dqs) of the aq" vs "first answering a cq and then being presented with an answer fitting their intention." we then report the majority preference for each of the questions. please refer to appendix c for more details. results and analysis figure 3 demonstrates that answering aqs using cq is preferred over dq. the prominent reasons stated by annotators for favoring "cq" are its ease of use, conciseness, interactivity, and ability to provide clear guidance. conversely, annotators who preferred "dq" mentioned its advantage as being more specific and clearer in addressing the given question. note, cq was unanimously preferred 23 times, and dq, 5 times. also, unanimity in favor of cq was observed across aqs regardless of the number of interpretations-or options-whereas unanimity in favor of dq only occurred for aqs with up to three interpretations. in other words, cq can be preferred regardless of the number of interpretations, while dq is not preferred when many interpretations are possible. this is intuitive given that more interpretations result in more text for people to process for the dq-based response to aq. ## experiment 2: handling aq with cq given that the cqs are preferred over dqs, we now study the question: do predicted answers for aq help improved the end-to-end performance of the cq-based approach? since predicted answers for aq have been shown to be helpful for previous dq-based approaches @xcite , we want to verify if they are also helpful for the cq-based approach. 4 for this, we experiment with two settings: in the remainder of the section, we present the experimental setup and results for each task. please refer to appendix d for more details. for no answers for aq case, we use the bert base model @xcite with a simple fully connected layer on top for the binary classification task. the model is trained on the combined dataset for 96 epochs. the model also takes in a prompt of the form "question [sep] rele-vant_passages" as input and outputs "ambiguous" or "unambiguous". for predicted answers for aq case, we use bart-based model called spanseqgen, the bestperforming model for predicting answers for aq by @xcite , and finetuned it on the am-bignq dataset. this model takes in a prompt of the form "question [sep] relevant_passages" as input and predicts all plausible answers. we classify a question as "ambiguous" if the model outputs more than one plausible answer and "unambiguous," otherwise. ## conclusion we proposed a cq-based approach to handle aqs in odqa. along with presenting a new dataset, we defined a pipeline of tasks, designing appropriate evaluation metrics for it. experiments show the promising, yet challenging nature of the tasks. we hope that our dataset will serve as a valuable resource for research in this area and support the development of more user-friendly qa systems.