DRS : Deep Question Reformulation With Structured Output (2025)

Abstract: Question answering represents a core capability of large language models (LLMs). However, when individuals encounter unfamiliar knowledge in texts, they often formulate questions that the text itself cannot answer due to insufficient understanding of the underlying information. Recent studies reveal that while LLMs can detect unanswerable questions, they struggle to assist users in reformulating these questions. Even advanced models like GPT-3.5 demonstrate limited effectiveness in this regard. To address this limitation, we propose DRS: Deep Question Reformulation with Structured Output, a novel zero-shot method aimed at enhancing LLMs’ ability to assist users in reformulating questions to extract relevant information from new documents. DRS combines the strengths of LLMs with a DFS-based algorithm to iteratively explore potential entity combinations and constrain outputs using predefined entities. This structured approach significantly enhances the reformulation capabilities of LLMs. Comprehensive experimental evaluations demonstrate that DRS improves the reformulation accuracy of GPT-3.5 from 23.03% to 70.42%, while also enhancing the performance of open-source models, such as Gemma2-9B, from 26.35% to 56.75%.

## introduction question answering has emerged as a fundamental capability of large language models (llms), with recent advances from gpt-3 and instruct-gpt to @xcite @xcite demonstrating remarkable improvements on various benchmarks @xcite @xcite . however, when humans encounter unfamiliar knowledge domains, they frequently pose questions that cannot be directly answered from the available text. recent studies indicate that over 30% of questions in real-world scenarios fall into this category, significantly impacting information wasabi is normally consumed in such small quantities that its nutritional value is negligible. the major constituents of raw wasabi root are carbohydrates (23.5%), water (69.1%), fat (0.63%), and protein (4.8%). how many calories are in wasabi? please slightly reformulate my question to make it answerable. what are the major constituents of raw wasabi root? i cannot find the answer to your question based on the document you provided. access and learning efficiency @xcite . more importantly, industrial experiments have shown that effectively reformulating such questions can dramatically improve user experience and task completion rates in virtual assistant systems, with potential impact on millions of users @xcite . a successful question reformulation must satisfy two key criteria: (1) the reformulated question should be answerable based on the given text, and (2) it should preserve the core entities and intent of the original question, ensuring users obtain the information they actually seek. consider the following example shown in figure 1 : when presented with a text stating "wasabi is normally consumed in such small quantities that its nutritional value is negligible. the major constituents of raw wasabi root are carbohydrates (23.5%), water (69.1%), fat (0.63%), and protein (4.8%)", users might ask "how many calories are in wasabi?". while this question cannot be directly answered, a well-reformulated version would be "what are the major constituents of raw wasabi root?", which maintains the user's interest in wasabi's composi-tion while being answerable from the text. previous approaches to handling unanswerable questions broadly fall into three categories: (1) detection methods that focus on identifying unanswerable questions @xcite @xcite , (2) clarification methods that seek additional information from users @xcite , and (3) reformulation methods that attempt to modify questions into answerable forms @xcite . while recent advances in llms have improved performance in question detection and clarification, question reformulation remains challenging due to the difficulty in balancing answerability and intent preservation. even powerful models like gpt-3.5 achieve only 23.03% accuracy in reformulation tasks, highlighting the significant room for improvement in this area. to address these challenges, we propose drs (deep question reformulation with structured output), a zero-shot method that enables llms to effectively reformulate unanswerable questions. drs addresses the key challenges through three innovations: (1) a systematic entity-driven approach that ensures intent preservation by explicitly tracking and maintaining key entities from the original question, (2) a structured output framework that ensures answerability by constraining the generation process to incorporate specific entities and generate questions aligned with the corresponding statements derived from the source document, and (3) an efficient dfs-based search strategy, coupled with a candidate question evaluation mechanism, enhances the method's effectiveness and practicality for real-world applications. unlike previous methods that often sacrifice one aspect for another, drs achieves strong performance on both answerability and entity preservation simultaneously. we conduct extensive experiments on six diverse datasets, comparing drs with multiple baseline approaches across different types of questions and domains. the results demonstrate that drs significantly improves reformulation accuracy across all tested llms. additionally, we introduce a more reliable evaluation framework using gpt-4o-mini, replacing the previous llama2-7b evaluator to ensure more accurate assessment of reformulation quality. our contributions in this paper are threefold: (i) we propose drs, a zero-shot method that enables llms to effectively reformulate unanswer-able questions through entity-driven search and structured outputs. ## related work question answering has been a central focus in natural language processing (nlp) @xcite @xcite , with the development of datasets like cbt and searchqa @xcite designed to assess a model's ability to answer questions. however, these datasets do not address the challenge of unanswerable questions. with the rise of gpt-3 (brown et al., 2020) and even more powerful large language models (llms), the focus has shifted to handling unanswerable questions, which arise due to ambiguity or gaps in knowledge @xcite @xcite . several studies have explored methods for resolving ambiguities in questions, often targeting issues like missing qualifiers that can lead to different interpretations @xcite . many research on questions outside the model's knowledge scope has focused on guiding models to recognize when they lack sufficient information to answer, thereby reducing hallucinations @xcite @xcite . however, a commonly overlooked real-world scenario involves individuals posing seemingly relevant questions that are unanswerable due to their limited familiarity with the document's knowledge domain. in such cases, large language models can be employed to reformulate these questions, enabling the inquirers to obtain the information they likely intended to seek. the issue of document-related unanswerable questions was first systematically addressed in a dataset by @xcite , and later expanded by studies such as @xcite , which analyzed google search queries, and @xcite , which examined reddit discussions. both studies focused on identifying questions with incorrect assumptions or presuppositions that made them unanswerable. ## methodology in this paper, we propose a new zero-shot method called drs: deep question reformulation with structured output, which significantly improves llms' ability to reformulate unanswerable questions based on a given document, helping users obtain desired answers in unfamiliar knowledge domains. our drs method combines dfs (depth-first search) algorithm with llms to address challenges posed by human-generated unanswerable questions involving multiple entities. directly inputting all entities into llms for question reformulation risks overlooking the lack of meaningful relationships between them, leading to unreliable or incoherent answers. to mitigate this, our dfs-based approach systematically explores semantically related entity combinations, ensuring the generation of meaningful questions. additionally, by controlling the search depth and iteration count, our drs method strikes a balance between computational efficiency and accuracy, yielding high-quality results with minimal overhead. our approach consists of three main steps: (i) entity extraction and filtering. (ii) dfs combination search and structured question generation. (iii) candidate question re-evaluation. the detailed process of our zero-shot drs method is illustrated in figure 2 . ## gpt evaluator in the previous paper, @xcite fine-tuned the llama2-7b model @xcite on the training dataset for the sequence classification task to determine whether a question could be answered from the corresponding text. this approach achieved high scores on their validation dataset. however, in our experiments, we find that its generalization performance is suboptimal; even on the couldask dataset, which includes training data, it still yields extremely poor classification results. given the situation described above, we find it necessary to propose a reliable and efficient evaluation method. therefore, in this paper, we choose the gpt-4o-mini model, which demonstrates excellent language understanding capabilities, as the evaluation model, ensuring both strong evaluation performance and cost-effectiveness. to demonstrate the strong capabilities of the gpt-4o-mini model, we select a subset of labeled questions from each portion of the coul- we observe that gpt-4o-mini consistently achieves high scores across all tests, whereas the fine-tuned llama2-7b model shows significantly lower accuracy than gpt-4o-mini and exhibits substantial score variation across different datasets (ranging from a high of 79% to a low of 14%). for the average accuracy across six different datasets, our gpt-4o-mini evaluator achieves over 90%, while the llama2-7b evaluator only reaches 52%, performing slightly better than random predictions. this clearly indicates that the fine-tuned model lacks generalizability across diverse data. therefore, using gpt-4o-mini as the evaluator is not only more reliable but also provides faster evaluation and lower operational costs. ## experiment results to demonstrate the effectiveness of our proposed zero-shot drs method, we test it on six datasets, totaling over 1,700 data points, using four different large language models. we conduct a fair and thorough comparison between our method and four baseline methods from previous research, with all accuracy scores presented in ## case study to highlight the advantages of our proposed zeroshot drs method, we select some examples to demonstrate its effectiveness compared to other baselines, as shown in figure 5 . we observe that the relevant documents primarily discuss mathematical theorems, and the user's question involves five entities but cannot be answered using the corresponding documents. in these examples, the reformulated questions generated using zero-shot and zero-shot cot are identical. these questions overlap with the original question on three entities: "conjecture", "integer n", and "primes". however, they still cannot be answered based on the provided documents, making such reformulations unsuccessful. in contrast, questions generated using the fewshot or few-shot cot methods are more diverse. however, despite being answerable based on the document, they fail to pass the entity overlap check. specifically, the questions generated by the fewshot method contain no entities from the original question, while those generated by the few-shot cot method include only two entities. both fail to meet the required entity overlap ratio of 0.5 or higher, making these reformulations unsatisfactory. in comparison, the reformulated question generated by our proposed drs method satisfies both strict conditions: answerability and entity overlap ratio. the drs method generates questions through a dfs search process, where the model is guided by the constraints of three entities: "conjecture", "integer n", and "primes". in this process, we first prompt the model to construct a statement based on the document, using these entities. we then use this statement to generate a reformulated question containing the same entities. this approach ensures both answerability and entity overlap, optimizing the model's ability to help users obtain relevant answers. from these observations, we conclude that even advanced methods like few-shot or few-shot cot struggle to produce satisfactory reformulations due to the lack of constraints on generated content. this lack of control often results in outputs that either overly focus on preserving the original question's entities, leading to unanswerable questions, or prioritize answerability at the expense of entity a third type of conjectures concerns aspects of the distribution of primes. i t i s c o n j e c t u r e d t h a t t h e r e a r e infinitely many twin primes, pairs of primes with difference 2 (twin prime conjecture). polignac's conjecture is a strengthening of that conjecture, it states that for every positive integer n, there are infinitely many pairs of consecutive primes that differ by 2n. it is conjectured there are infinitely many primes of the form n2 + 1. these conjectures are special cases of the broad schinzel's hypothesis h. brocard's conjecture says that there are always at least four primes ...... which conjecture holds that for any positive integer n, there is an infinite amount of pairs of consecutive primes differing by 2n? "conjecture", "integer n", "infinite amount of pairs", "primes", "2n" question entities which conjecture asserts that every odd integer n greater than 2 can be expressed as a sum of two primes? zero-shot the context does not mention a specific conjecture that asserts that every odd integer greater than 2 can be expressed as a sum of two primes. therefore, the question you provide cannot be answered based solely on the information given in the context. which conjecture asserts that every odd integer n greater than 2 can be expressed as a sum of two primes? zero-shot cot the context does not mention a conjecture that specifically asserts that every odd integer greater than 2 can be expressed as a sum of two primes. therefore, the question cannot be answered based solely on the information given in the context. ## conclusion in our paper, we propose a zero-shot drs method that significantly enhances the performance of large language models on the unanswerable question reformulation task. our method outperforms all other baselines, including the few-shot cot approach, across six different datasets and various llms. we also conduct extensive experiments on key parameters, such as temperature and the number of candidate questions, with results demonstrating the strong capabilities and robustness of our approach. furthermore, our findings reveal that the previously proposed llama2-7b evaluator lacks the capability to fairly assess the reformulated questions. we propose using gpt-4o-mini as a more reliable, accurate, and low-latency evaluator for future research. looking ahead, we aim to develop even more effective methods to harness the full potential of llms, helping them assist people in understanding unfamiliar documents more effectively.