I Could’ve Asked That: Reformulating Unanswerable Questions (2024)

Abstract: When seeking information from unfamiliar documents, users frequently pose questions that cannot be answered by the documents. While existing large language models (LLMs) identify these unanswerable questions, they do not assist users in reformulating their questions, thereby reducing their overall utility. We curate CouldAsk, an evaluation benchmark composed of existing and new datasets for document-grounded question answering, specifically designed to study reformulating unanswerable questions. We evaluate state-of-the-art open-source and proprietary LLMs on CouldAsk. The results demonstrate the limited capabilities of these models in reformulating questions. Specifically, GPT-4 and Llama2-7B successfully reformulate questions only 26% and 12% of the time, respectively. Error analysis shows that 62% of the unsuccessful reformulations stem from the models merely rephrasing the questions or even generating identical questions. We publicly release the benchmark and the code to reproduce the experiments.

## introduction applying large language models (llms) to perform question answering (qa) over documents, such as legal and medical texts, has become increasingly popular @xcite . however, users' limited knowledge of these documents often results in the formulation of unanswerable questions, whose assumptions either conflict with or cannot be verified with the information available in the documents. we will refer to these assumptions as presupposition errors. 3 @xcite and @xcite found that around 30% of information-seeking questions written by can you read an article and answer a question? in 1986, chief justice warren burger informed the white house of his intent to retire. @xcite feeling that this might well be reagan's last opportunity to pick a supreme court justice, the president and his advisers chose scalia over bork. many factors influenced the decision. reagan wanted to appoint the first italian-american justice. in addition, scalia was ten years younger and would likely serve longer on the court. scalia also had the advantage of not having bork's "paper trail"; the elder judge had written controversial articles about individual rights. scalia was called to the white house and accepted reagan's nomination. @xcite why did scalia decide to retire? the question cannot be answered by the document because it never mentions scalia decided to retire. below is a relevant question that the document has an answer. would you like to know the answer or ask your own question? why was scalia appointed to be a justice? figure 1 : an example of an llm suggesting an alternative relevant question the user could have asked whose answers can be found in the document, besides only informing users with the presupposition errors. this work aims to improve the utility of qa systems by introducing a new task that requires both detecting unanswerable questions and generating questions closely related to the initial query and grounded in the document. opting to generate a relevant question rather than a summary of related information emphasizes a user-centered approach. while producing a summary is more document-focused, formulating a relevant question targets understanding and predicting the user's intent, aligning the interaction more closely with the user's specific needs and queries. we provide an example of how to suggest such questions in figure 1 . although generating relevant and grounded questions conditioned on initial queries offers greater utility to users, it remains a difficult task even for the best models in a few-shot setting. we first characterize human-reformulated questions and describe several different strategies for updating questions to remove presupposition errors. motivated by these strategies, we curate coul-dask, an evaluation benchmark for documentgrounded qa that consists of a combination of existing and new datasets to study question reformulation in the presence of presupposition errors. we evaluate several prompting methods such as few-shot prompting and chain-of-thought prompting, employing state-of-the-art open-source and proprietary models. the results illustrate the limitations of existing models and prompting techniques in accurately detecting unanswerable questions and reformulating questions: the f1 scores for identifying unanswerable questions range from 41.16% to 67.82%, and success rates for reformulating questions range from 7.13% to 26.21%, depending on base models. analysis shows that most of the unsuccessful reformulation come from rephrasing or repeating the original questions and that llms are worse at reformulating questions necessitating global edits compared to those solely requiring local edits. unanswerable questions are closely related to ambiguous questions @xcite . while there has been extensive research into reformulating ambiguous questions @xcite @xcite , the problem of reformulating unanswerable questions receives little attention. strategies for rephrasing ambiguous questions often involve making questions more specific by mentioning precise entities or events @xcite . in contrast, as we will show in section 3, reformulating unanswerable questions necessitates a wide range of strategies. finally, we discuss the connection between document-grounded qa and open-domain qa @xcite . document-grounded qa is essentially open-domain qa with the correct document retrieved. reformulating questions based on the identified document is a separate skill from retrieving the document. therefore, question reformulation is an interesting task in itself. ## task: question reformulation to assist users with question reformulation when reading unfamiliar documents, we define the following task. given a document and a user question, the system must determine if the question is unanswerable. upon identifying the unanswerable question, it must reformulate the question such that the new question is answerable by the document while remaining relevant to the original question. as this task is challenging to formally define, we begin with a qualitative study over a set of example reformulations by a human user, shown in ## the couldask benchmark motivated by the need to reformulate questions both to rely on verified information and to avoid contradictions, we develop an evaluation bench-mark. we consider two important challenges in constructing benchmarks for this task. (1) the benchmark should cover a wide range of domains to cover different types of presuppositions. (existing qa datasets that study unanswerable questions mostly rely on wikipedia articles @xcite @xcite .) (2) the evaluation method should be capable of fairly assessing equally good reformulated questions, considering the subjective nature of question reformulation. ## results detecting unanswerable questions being able to detect unanswerable questions is a necessary precondition for successful question reformulation. ## analysis qualitative analysis we randomly sample and analyze 20 reformulated questions from each base model (a total of 100 questions) that cannot be answered by the corresponding documents. we summarize the results in another type of error that occurs 14% of the time is that the models generate a question by copying a document span that looks similar to the original question. for example, given the original question error category % original question revised question simply rephrasing / producing the same questions 62 when was he born? when was jay chou born and raised in taipei, taiwan? which team won the most nfl europe titles? who was the team that won the most nfl europe titles? producing questions by copying documents 14 when did the cubs play their first home game at wrigley field? when did the cubs move to wrigley field? what are the characteristics of the classical roman architecture? what are some of the characteristics of classical roman architecture that were adopted by renaissance architects? producing irrelevant questions 17 what is snow patrol's most famous song? who are some famous musicians that have expressed admiration for snow patrol? which campus is the biggest in university of california? which college in the university of california system is not controlled by the regents or president? other 11 how many arms does krishna have? i apologize for the confusion. the passage does not mention how many arms krishna has. limited benefit from fs cot on question reformulation. we explore why fs cot improves the detection of unanswerable questions but not question reformulation by conducting a qualitative analysis of gpt-4 outputs on a banditqa example, as illustrated in figure 2 . it is relatively straightforward to determine why a question is unanswerable -either the document does not provide the necessary information or there is a presupposition that conflicts with the document. however, question reformulation demands compositional reasoning. the model needs to first decide on a reformulation strategy and then plan the specific steps to achieve the reformulation. the reformulated question generated by gpt-4 via fs cot prompting is closer to being answerable. however, the model misses a subtle detail -the document only mentions the exercise habits of 30.8% of residents, not the general population. therefore, fs cot alone, without further methodological innovation, does not fully document: in december 2007, sport england published a survey which revealed that residents of st albans were the 10th most active in england in sports and other fitness activities. 30.8% of the population participate at least 3 times a week for 90 minutes. question: what is st albans sport? zs cot for unanswerable question detection: the article did not provide information on what st albans sport is, so i don't know. ## conclusion users often ask unanswerable questions when they seek information from unfamiliar documents. existing llms identify these questions but do not aid users in reformulating new questions, resulting in ineffective user-llm interactions. we introduce couldask, a compiled set of grounded-document qa datasets designed to study both unanswerable question detection and question reformulation.