Learning and Evaluating Factual Clarification Question Generation Without Examples (2025)

Abstract: Real-world tasks such as giving legal or technical advice often depend on context that is initially missing at the outset. The ability to derive missing factual information by asking clarifying questions (ACQ) is an important element of real-life collaboration on such reasoning tasks. Although intent disambiguation has been heavily investigated, factual reasoning remains underexplored. To enable evaluation of factual domain clarification question generation, we present a new task that focuses on the ability to elicit missing information in multi-hop reasoning tasks. We observe that humans outperform GPT-4o by a large margin, while Llama 3 8B Instruct does not even beat the dummy baseline in some metrics. Finally, we find that by fine-tuning Llama 3 8B Instruct on its own generations filtered via rejection sampling, we can improve information recovery by 27.6% without using any manually labeled data.

## introduction in many real-world scenarios, the initial context is often incomplete, making it risky to provide answers without first seeking clarification. for instance, legal, medical, and technical advice typically depends on specific details about the individual's situation. as language models (lms) are increasingly used in open-domain assistant roles, their ability to clarify and gather relevant facts before offering advice is becoming more crucial. evaluating clarification question generation is not straightforward. many question generation tasks evaluate generated questions based on word overlap with a ground truth label @xcite , ignoring whether the question actually acquires useful information or how difficult it is to answer. other tasks such as those by @xcite use human evaluators to judge the quality and informativeness of ques-figure 1 : overview of the hotpotqa-flm task, which simulates the need to formulate a question. conventionally, the downstream model performs the downstream task directly ( ). however, in hotpotqa-flm ( ), critical information is missing 1 ⃝. to acquire that information, the acq model 2 ⃝ first uses the context to generate a clarification question. the question is presented to the contextually knowledgeable answering agent 3 ⃝, which generates a response. the response is sent as additional context to the downstream model 4 ⃝. for strong acq models, we expect the downstream model to achieve better performance on context + answering agent response than on context alone. recently, some acq tasks including those by zhang and choi (2023) avoid these limitations by measuring the effect of clarifications on a downstream task. in this paradigm, which we refer to as pragmatic evaluation, an answering agent is used to dynamically generate answers to clarifying questions (figure 1 ). the downstream task (e.g. qa), is then performed with and without the clarification. pragmatic evaluation captures the objective value of the information gained while also permitting automatic evaluation. although underexplored in evaluations, failing to clarify basic facts in high-stakes applications can cause serious harm to users and others. if a user asks how to clean up a chemical spill, clarifying what chemical is critically important; applying water to an alkali metal can cause explosion, but sweeping up fine powders can aerosolize toxins. absorbing oxidizers with paper towels, however, may cause spontaneous combustion @xcite . analogous scenarios exist in medical, legal, security, or other domains where failing to clarify can have serious real-world consequences. although our contributions address one specific scope, we find that current models struggle to clarify key facts even in this constrained trivia qa domain. this suggests more work is necessary before models can adapt to under-specified high-stakes environments. compared to ambiguity in user intent, ambiguity in relevant facts poses unique challenges. although users can generally answer questions about their own intent, they may not always know the answer to factual questions. factual questions should be phrased to require minimal effort of recall while still learning facts relevant to the downstream task (did you earn more than $x? vs. exactly how much did you earn?). additionally, when evaluating clarification questions in the factual domain, one must ensure critical pieces of the puzzle are not guessable or leaked in some other way. a task that nominally requires clarification ("napoleon bonaparte was 167cm. who is taller, shaquille o'neal or napoleon bonaparte?") becomes trivial if the downstream agent is aware that shaquille o'neal was a very tall basketball player. to bridge this gap, we introduce the pacq task that focuses on evaluating models' ability to ask questions seeking objective factual information. our first contribution is hotpotqa-flm. in this task, an llm must assist a downstream agent in answering a trivia question that is conditional on an unknown fact. the llm must identify what information is missing, and ask for it from a third answering agent. hotpotqa-flm prompts are created by deleting one fact from the context necessary to perform a downstream multi-hop qa task from the hotpotqa dataset @xcite . we term this process fact-level masking (flm). clarifying questions are submitted to an answering agent. the answering agent responds with one of many topically similar answers. last, performance on the downstream task is assessed with and without the clarification. we find state-of-the-art models struggle with hotpotqa-flm as compared to humans. questions by gpt-4o recover only 48% of missing information compared to those by humans. smaller, open source models achieve only 14% of human performance. given weak zero-shot performance on hotpotqa-flm, we also contribute a method for training models to ask informative clarification questions. notably, hotpotqa-flm does not include examples of clarifying questions for supervised fine-tuning, which are rarely available. instead, we train our model, alexpaca, foot_0 by creating a synthetic dataset through repeated interaction with the answering agent. the dataset is filtered with rejection sampling to only include clarifying question examples that result in the expected useful response. last, alexpaca is fine-tuned on the synthetic dataset. alexpaca shows a 28% increase in performance over its zero-shot llama 3 8b instruct source model on the full dataset. this demonstrates small models' ability to self improve at clarifying question generation given effective feedback. alexpaca also demonstrates a scalable and cheap proof-of-concept for approaching factual acq tasks. the training method is suitable where supervised examples are unavailable or proprietary models perform poorly (as we find) or are unacceptable for cost, privacy, or latency reasons. to summarize, our contributions are: 1) hotpotqa-flm, a clarification question generation task evaluated based on objective information gain in the factual domain; and 2) alexpaca, a rejection-sampling approach to fine-tuning models for clarification question generation not reliant on manual annotation. ## conclusion in this paper, we presented a benchmark for objectively evaluating clarifying questions and observed that state-of-the-art zero-shot llms struggle at this task compared to humans. to overcome these challenges, we introduced fact-level masking and hotpotqa-flm, a self-supervised pacq dataset, and an associated evaluation process. finally, we demonstrated a training method for the alexpaca model that relies on agent-agent interaction rather than supervised examples of clarifying questions to self-improve over baseline.