Teaching Language Models To Gather Information Proactively (2025)

Abstract: Large language models (LLMs) are increasingly expected to function as collaborative partners, engaging in back-and-forth dialogue to solve complex, ambiguous problems. However, current LLMs often falter in real-world settings, defaulting to passive responses or narrow clarifications when faced with incomplete or under-specified prompts—falling short of proactively gathering the missing information that is crucial for high-quality solutions. In this work, we introduce a new task paradigm: proactive information gathering, where LLMs must identify gaps in the provided context and strategically elicit implicit user knowledge through targeted questions. To systematically study and train this capability, we design a scalable framework that generates partially specified, real-world tasks, masking key information and simulating authentic ambiguity. Within this setup, our core innovation is a reinforcement finetuning strategy rewards questions that elicit genuinely new, implicit user information—such as hidden domain expertise or fine-grained requirements—that would otherwise remain unspoken. Experiments demonstrate that our trained Qwen-2.5-7B model significantly outperforms o3-mini by 18% on automatic evaluation metrics. More importantly, human evaluation reveals that clarification questions and final outlines generated by our model are favored by human annotators by 42% and 28% respectively. Together, these results highlight the value of proactive clarification in elevating LLMs from passive text generators to genuinely collaborative thought partners.

## introduction large language models (llms) are now indispensable partners for solving diverse reasoning tasks, such as drafting proofs, debugging code, or writing collaborative exchanges account for over half of the interactions in all three fields, indicating a widespread shift from one-shot prompting to iterative, multi-turn dialogue @xcite . a key challenge in collaborative settings is information asymmetry: users often provide underspecified prompts. today's models address this by asking clarification questions related to ambiguities in the user input, e.g., "do you want apa or mla style?". because these questions are effectively a function of the current input and dialogue prefix alone, they resolve surface uncertainties but rarely change the trajectory toward a substantially better solution. for example, given "write a policy brief on hospital readmission", asking about length or citation style does little to determine what a strong brief should actually contain. motivated by such, we study the task of proactive information gathering. the core research question given an underspecified request, passive models ask surface clarifications (e.g., citation style) that depend only on the observed context. proactive models ask along new, high-leverage dimensions the user has not supplied (e.g., cohort and time horizon), eliciting constraints that reshape the target solution and enable a higher-utility, optimal response. for example, when user asks "which patient cohort and time horizon matter for your stakeholders?", the model is expected to proactively interact with the user and provide different perspectives to consider in order for a concrete response, e.g. the stakeholder constraints, evaluation criteria, sources to privilege, time horizons, etc. the goal is to shift the llm from a passive responder to a genuine thought partner that can anticipate user needs, proactively ask for user input, and work interactively with the user to produce more comprehensive responses or artifacts (fig. 2 ). however, designing llms that excel at proactive information gathering poses several obstacles. high-quality, collaborative dialogue data is scarce and difficult to scale; organic user logs are often noisy and proprietary, while crowdsourcing nuanced, domain-specific exchanges is ex-pensive and hard to control @xcite . even more fundamentally, the qualities that define a "good" proactive question-helpfulness, novelty, contextual complementarity-are subjective and hard to capture with standard reward signals or simple heuristics. in this work, we address these challenges by designing a framework that specifically rewards pioneering, context-complementary questions-those that reach beyond what is already provided, proactively soliciting critical information from users. our approach has three core innovations: 1. task formulation: we introduce proactive information gathering as a new task for llms, formalizing the ability to identify and elicit targeted information that would lead to a more comprehensive or in-depth response 2. synthetic conversation engine: leveraging the dolomites dataset @xcite , we construct a simulation pipeline that creates ambiguous prompts and rich clarification trajectories by systematically masking critical information-ensuring that proactive questioning becomes necessary for task completion. 3. reinforcement fine-tuning: we propose a reward structure that rewards questions reaching beyond the provided context, and fine-tune llms using proximal policy optimization. experiments demonstrate that our trained qwen-2.5-7b model significantly outperforms o3-mini by 18% on automatic evaluation metrics. more importantly, human evaluation reveals that clarification questions and final outlines generated by our model are favored by human annotators by 42% and 28% respectively. together, these contributions chart a path toward llms that are not just compliant responders, but proactive, collaborative partners-able to drive richer, more effective conversations by actively seeking the information that matters most. all of the code used in our experiments is publicly available foot_0 . recent works emphasize proactive assistance for everyday conversation @xcite within fully unobservable environments and reasoning tasks @xcite . in contrast, our work targets open-ended writing tasks and employs a partially observable environment, reflecting real-world scenarios where users inputs imperfect prompt and agents only have access to partial information up front and must strategically clarify hidden details through iterative dialogue. ## task and dataset in this section, we formalize the proactive information gathering task ( §3.1), describe how it is instantiated using the dolomites dataset ( §3.2), and present our evaluation framework ( §3.3), which leverages a judge llm to assess model outputs. ## method our goal is to enable llms to proactively determine what clarification questions to ask in collaborative problem-solving settings. we frame this as a partially-observable markov decision process (pomdp), and we use reinforcement learning as a way to solve it. a central challenge is the absence of reliable, dense supervision for every proposed question. instead of rewarding questions themselves, we focus on the outcome-does the question elicit new information that was not already available to the assistant? this guiding principle motivates our reward formulation, which rewards the discovery of genuinely missing information. to systematically study and train such proactive clarification behavior, we develop a synthetic conversation engine that simulates realistic, multi-turn assistant-user dialogues ( §4.1). we then detail our reward design ( §4.2). ## experiments in this section, we evaluate our trained model for proactive information gathering and compare it with baseline methods. we first delve into the details of our experimental setup ( §5.1), discuss the results obtained ( §5.3), and perform analysis. ## conclusion in this work, we address a fundamental shortcoming of current large language models: their inability to proactively seek out missing, contextually relevant information in ambiguous, open-ended tasks. we formalize proactive clarification as a new benchmark challenge, going beyond traditional clarifica-tion and slot-filling to embrace a more collaborative and anticipatory role for llms. we present a framework for training llms to proactively seek missing information, transforming them from passive responders into active thought partners. by leveraging a synthetic conversation engine and reinforcement fine-tuning, our models consistently outperform strong baselines on both automated metrics and human evaluation, especially in openended and under-specified domains. our results highlight the value of targeted reward optimization for collaborative, context-aware dialogue. we hope this work inspires further research into more realistic multi-turn settings and broader domains, paving the way for llms that can engage in richer, more productive human-ai collaboration.