Bridging Information Gaps with Comprehensive Answers: Improving the Diversity and Informativeness of Follow-Up Questions (2025)

Abstract: Generating diverse follow-up questions that uncover missing information remains challenging for conversational agents, particularly when they run on small, locally hosted models. To tackle this problem, we develop an information-gap–driven pipeline that contrasts the initial answer with an LLM-generated comprehensive answer, identifies the information gaps, and formulates gap-bridging follow-up questions. Applying the pipeline, we augment an existing dataset–FollowupQG–tenfold. Experiments show that models fine-tuned on the augmented dataset achieve significantly higher informativeness and diversity than variations trained on the original dataset. These findings indicate that our pipeline, which mirrors the human cognitive process of information seeking, provides an efficient distillation channel from state-of-the-art LLMs to smaller models, enabling resource-constrained conversational systems to generate more diverse and informative follow-up questions.

## introduction asking questions is a fundamental mechanism for humans to acquire new information, particularly when existing information is incomplete. while large language models (llms) excel at passively answering questions from users, their ability to proactively guide conversations by identifying and addressing information gaps remains underdeveloped @xcite , with smaller models performing even worse. therefore, the task of question generation (qg) has become a focal point in natural language processing (nlp) for its role in improving information-seeking dialogue systems @xcite -including, making information seeking more accurate and efficient @xcite , resolving ambiguities @xcite , and ultimately better understanding users' needs to provide suitable assistance across various domains @xcite @xcite . while most existing qg tasks focus on generating questions directly answerable from a given context @xcite @xcite -a process that diverges from how humans infer and pursue missing information, @xcite propose the followupqg task, which requires models to generate follow-up questions that build on, but are not answerable by, the initial question-answer pair. they create the fol-lowupqg dataset and show that existing models often produce repetitive or context-bound questions that fail to target unexplored information @xcite . the core challenges of the fol-lowupqg task can be formulated into two dimensions: (1) identifying information gaps, the unanswered aspects of the initial question, and (2) generating diverse questions that target these gaps. building on traditional qg methods @xcite @xcite , recent work attempts to generate information-seeking follow-up questions using preference optimization @xcite and knowledge graphs @xcite , but still lack explicit mechanisms to model gaps or ensure diversity. to address these limitations, we propose an information-gap-driven teacher-student knowledge distillation pipeline. in our approach, a teacher llm generates a hypothetical "complete" response to the initial question, contrasts it with the often incomplete initial answer to identify information gaps, and formulates gap-bridging follow-up questions. by generating multiple follow-up questions, each targeting some unanswered information, this pipeline ensures the diversity and informativeness of the follow-up questions. for example, in figure 1 , if the initial answer to "how do clouds form?" is "clouds form when water vapor cools," a comprehensive answer might add ". . . and condenses around dust particles," which explicitly exposes the gap through contrast and leads to an informative follow-up question such as "what role do particles play in cloud formation?" our pipeline can be applied across different teacher-student model pairs. in this work, we use @xcite as the teacher model and bart-large as the student model to verify the pipeline. specifically, we use gpt-4o to generate the comprehensive answers and followup questions. after verifying the quality of the follow-up questions via human evaluation, we then augmented the original followupqg training set tenfold and fine-tuned bart-large on both the original dataset and our augmented dataset. leveraging gpt-4o to generate high-quality training data, and then distilling the teacher's knowledge into smaller models, our approach achieves strong performance at a significantly lower cost. the experimental results demonstrate significant improvements of the augmented dataset over the baselines, both in terms of quality (validity, relevance, informativeness, etc.) and diversity. our contributions are as follows: ## related work question generation (qg) focuses on automatically generating semantically meaningful and wellstructured questions based on a given text @xcite . while traditional qg techniques have made significant strides in domains such as machine comprehension @xcite , e-commerce @xcite , and education @xcite , they primarily generate questions based on known answers. this approach contrasts sharply with human questioning behavior, which actively seeks new information from various perspectives. this limitation has led to the emergence of followupqg, a task whose goal is to generate questions that explore previously unanswered or underexplored aspects of a given text. followupqg has evolved from simpler methods, such as template-based and retrieval-driven approaches @xcite @xcite , to more advanced techniques that prioritize informativeness @xcite . knowledge-enhanced approaches, like those in @xcite and @xcite , leverage entityrelation pairs and knowledge graphs to improve the depth of the generated questions. further advancing this, @xcite combined knowledge graphs with llms to increase question informativeness. efforts to model human-like questioning behavior, such as inquisitiveqg @xcite , have relied on crowd-sourced follow-up questions written for news articles rather than those naturally generated by humans, leading to a lack of depth and cognitive diversity. we follow the setting of the followupqg @xcite , which formalizes informationseeking follow-up question generation. based on questions and answers from the eli5 (explain like i'm 5) subreddit, follow-up questions in this dataset build upon-but are not answerable by-the initial question-answer pair, resembling real-world dialogues where follow-ups resolve ambiguities or deepen understanding. ## data augmentation effective followupqg requires models to infer and target gaps between the provided answer and the broader context of a conversation. following the task definition by @xcite : "to generate follow-up questions that seek new information given the initial question and answer", we denote the "initial question" as iq, "initial answer" as ia, and the "follow-up question" as fq. we identified critical limitations in the training dataset, including quality issues, which we addressed through dataset cleaning ( §3.1). to overcome the small scale (2,790 instances) and low diversity of the dataset, we present a novel data augmentation pipeline ( §3.2). finally, we demonstrate that the augmented dataset retains high quality ( §3.3). ## experiment setup model variants. to assess our proposed pipeline and augmented dataset, we fine-tuned bart-large @xcite (24 layers, 16 attention heads, hidden @xmath0, producing three model variants. bart-large is a seq2seq model that conditions on the concatenated iq and ia to generate an fq. we chose it as our base model because of its strong performance reported by @xcite . as their implementation is not public, we reproduced their training setup (@xmath1, @xmath2, adam @xcite ) and found that the original learning rate of 5e-5 caused instability, so we reduced it to 2e-5; all other hyperparameters remain unchanged. 4 figure 3 : the org/aug/full dataset. org is the original dataset of eli5 from the followupqg paper. we augment org using gpt-4o. full combines org with all gpt-4o-generated data (about 10× larger). aug is a random sample of only gpt-4o-generated instances, equal in size to org and excluding its data. we report three variants trained on the fol-lowupqg training set as shown in figure 3 : the org model is trained on the 2,790 original instances from @xcite and serves as our small-model baseline; the aug model, trained on a size-matched random sample of 2,790 gpt-generated questions from our augmentation pipeline (sec. 3.2) to isolate the effect of data quality; and the full model, trained on the entire 27,874-instance augmented dataset. all variants share identical hyperparameters and are evaluated on the original followupqg validation and test splits. decoding. to generate diverse but contextually relevant follow-up questions, we input the initial question and answer into the model in the following format: @xmath3, and generate 10 follow-up questions by applying beam search with a beam width of 20, selecting the top 10 candidates. we added a diversity penalty of 10 to encourage unique outputs across the groups and set the temperature to @xmath4.0 to maintain a balance between diversity and coherence. the maximum length for each generation is set to 1024 tokens. duplicate generations are removed. llm baselines. beyond the three bart variants, we evaluated two gpt-4o baselines that represent current large-model performance. the teacher setting executes the full information-gap pipeline from §3.2 on each test instance, returning gpt-4o's comprehensive answer, the identified gaps, and the resulting follow-up questions. the cot setting applies the same chain-of-thought prompt but suppresses all intermediate reasoning, outputting only the follow-up questions. duplicate generations are removed as in the bart variants. to thoroughly assess the quality of the generated follow-up questions, we employ both automatic evaluation ( §5.1) and human evaluation ( §5.2). as a first step for both evaluations, we automatically identify and remove ungrammatical questions based on syntactic parsing (see appendix e for a complete description of the filtering process). ## analysis to further understand the strengths and limitations of our method, we present a qualitative comparison of follow-up questions generated by all models for the same (iq, ia) pair ( §6.1), as well as an analysis of the expected information gain from the generated follow-up questions ( §6.2). ## conclusion in this work, we proposed a novel approach to enhance the diversity and informativeness of followup questions by explicitly modeling information gaps via an llm-generated comprehensive answer. we augmented the original followupqg dataset with gpt-4o and distilled this data into a small, locally deployable bart-large model. experiments show that our pipeline enables the small model to outperform peer small-model baselines and to perform comparably to gpt-4o baseline models in terms of validity, complexity, relevance, and informativeness-all at a fraction of the inference cost. these results suggest that targeted, high-quality augmentation can be more impactful than merely increasing dataset size. they also demonstrate that our method offers a practical approach for improving information-seeking dialogues-by reducing ambiguities and enhancing llm responses-even on systems with limited computational resources. future work could explore ways to increase follow-up-question diversity while reducing redundancy, and to extend the pipeline to downstream tasks involving multi-turn dialogue. we also encourage research on stronger automated metrics for evaluating question quality, given the high cost of human annotation and the limitations of current automatic measures.