Proactive User Information Acquisition via Chats on User-Favored Topics (2025)

Abstract: Chat-oriented dialogue systems that deliver tangible benefits, such as sharing news or frailty prevention for seniors, require proactive acquisition of specific user information via chats on user-favored topics. This study proposes the Proactive Information Acquisition (PIA) task to support the development of these systems. In this task, a system needs to acquire a user’s answers to predefined questions without making the user feel abrupt while engaging in a chat on a predefined topic. We created and analyzed a dataset of 650 PIA chats, identifying key challenges and effective strategies for recent LLMs. Our system, designed from these insights, surpassed the performance of LLMs prompted solely with task instructions. Finally, we demonstrate that automatic evaluation of this task is reasonably accurate, suggesting its potential as a framework to efficiently develop techniques for systems dealing with complex dialogue goals, extending beyond the scope of PIA alone. Our dataset is available at: https://github.com/CyberAgentAILab/PIA

## introduction developing fluent chat-oriented dialogue systems has become feasible with advances in large language models (llms) @xcite . attention is shifting from pure chatting partner systems toward those capable of providing tangible benefits via chats, such as helping senior citizens prevent frailty or sharing news. to provide beneficial support while maintaining the user's interest, the system must not merely follow the user's requests but also have an objective of collecting user information necessary for providing practical support. this requires striking a balance between information acquisition objectives and responding to the user's immediate requests, as the information necessary for supporting the user often does not directly relate to topics addressed by the user. this concept is closely related to previously topic: fishing question: do you have high-end earphones? user information: i enjoy staring up at the sky. i don't enjoy cold drinks. i have high-end earphones. s: hi! let's talk about fishing! u: i went fishing last weekend. didn't catch much though. s: that's too bad. so what do you do while you're waiting? u: not much, honestly. it can get pretty boring. s: i heard some people listen to music to kill time when they're not catching anything. do you ever do that? u: that's not a bad idea. maybe i'll give it a shot. s: do you use high-end earphones? it might be better not to take them. dropping them in the ocean would hurt. u: plus, salty air could damage them. i'll avoid taking my high-end ones. any suggestions for music while fishing? moreover, such proactive information acquisition can be viewed as a domain-independent and general-purpose proactive behavior of asking spe-cific questions from chat partners under particular constraints. thus, it could also serve as a framework for developing techniques applicable to diverse dialogue systems tasked with achieving complex goals, such as persuasion and negotiation @xcite . given this background, we propose the proactive information acquisition (pia) task, as exemplified in section 3 details our proposed pia task. section 4 outlines the creation of a dataset consisting of 650 pia chats between llm-based systems and human users, serving as a foundational resource for analyzing and training. section 5 identifies key challenges faced by current llms and distinguishing features of successful pia through dataset analysis. section 6 demonstrates experimentally that systems designed based on these findings outperform llms prompted solely with task instructions. section 7 explores the feasibility of automatically evaluating pia chats, suggesting this task's potential as a framework for iterative improvement of techniques applicable to various systems dealing with complex dialogue goals. ## related work proactive dialogues. task-oriented dialogue systems assume clearly defined user goals @xcite . however, in some cases, the systems themselves may have goals that conflict with the user's. in such situations, the systems need to chat proactively to achieve their own goals. such dialogues are called proactive dialogues and have been studied as tasks requiring advanced strategies @xcite . while proactive dialogues have been mainly studied in specific contexts like negotiation @xcite , the pia task targets domain-independent proactive dialogues, providing an environment to refine techniques applicable to proactive dialogues across diverse domains. target-guided dialogues. one exceptional task dealing with open-domain proactive dialogues is the target-guided dialogue task, in which systems aim to guide topic transitions during chats with users @xcite @xcite . with the advent of llms, achieving natural topic transitions has become feasible @xcite , likely enabling the systems that collect user information with system-initiated topic transitions. we introduce a more challenging and complementary task, which involves acquiring necessary user information while maintaining chats on user-preferred topics to ensure user engagement. bdi model. the analysis in section 5.1 demonstrates that llms fail the pia task by not adequately addressing multifaceted requirements, particularly (1) considering task completion status and (2) avoiding abrupt utterances. this aligns with previous studies highlighting limitations of single-output, prompt-based generation methods like in-context learning in addressing multiple requirements @xcite . our study developed a simple bdi-model-based system, which has been widely used in complex dialogue tasks @xcite . the bdi model is a framework in artificial intelligence and cognitive science that describes rational agents based on their beliefs (information about the world), desires (objectives or goals), and intentions (current plans or commitments to actions) @xcite . bdi-modelbased systems ensure responses are both statusaware and non-abrupt by decomposing response generation into explicit belief updating, desire formulation, and intention selection stages, allowing the system to (1) explicitly consider the taskcompletion status in belief updating and (2) discard abrupt candidate utterances in intention selection. ## our proposed pia task in this task, a system must acquire a user's answers to predefined questions (hereafter questions) without making the user feel abrupt while chatting on a predefined topic that may not be directly related to the questions (hereafter topic). ## dataset construction we constructed a dataset of 650 pia chats to facilitate analysis and training. the data collection involved two stages: an initial pilot collection of 200 chats with human evaluations, followed by a refined collection of 450 additional chats. ## validation of strategy-based system in this section, we propose a simple bdi-modelbased system designed based on the section 5's insights, which we call strategy-based system, and confirm that it outperforms llms prompted solely with task instructions. ## discussion: evaluation automatability based on the results in the previous section, there remains room for improving the generation of effective and non-abrupt key utterance candidates. continued iterative refinement and evaluation of this point are essential. however, the human evaluation approach previously used is costly and hinders rapid iteration. this section demonstrates that this task can be automatically evaluated with reasonable accuracy, enabling quicker and more cost-effective iterative improvements. additionally, since the task focuses on domain-independent, general-purpose proactive dialogue actions, it can serve as a benchmark for techniques applicable to various systems dealing with complex dialogue goals. evaluating systems for this task currently requires human involvement in two phases: chat collection and assessment. therefore, we first investigate the feasibility of semi-automatic evaluation, which automatically assesses human-system interactions, thus validating systems based on actual user interactions without annotation costs. we then explore the possibility of fully automating the evaluation workflow, including chat collection. although this fully automatic method does not allow validation based on actual user interactions, it entirely eliminates human labor costs. ## conclusion pia is crucial for systems offering user benefits, such as health services or tailored news. this study proposes the pia task as a foundational framework to advance such technologies. we constructed a dataset to facilitate system development, revealing the challenges and effective strategies for llms. an insight-based system significantly outperformed llms prompted solely by task instructions. furthermore, we confirmed high accuracy in semi-automatic evaluations and reasonable accuracy in full-automatic evaluations for this task. this task focuses on a domain-independent, general-purpose action of asking specific questions, which can be reliably evaluated automatically. we anticipate that this task will efficiently support the development of techniques applicable to various dialogue systems addressing complex dialogue goals.