Learning to Ask Efficiently in Dialogue: Reinforcement Learning Extensions for Stream-based Active Learning (2025)

Abstract: One essential function of dialogue systems is the ability to ask questions and acquire necessary information from the user through dialogue. To avoid degrading user engagement through repetitive questioning, the number of such questions should be kept low. In this study, we cast knowledge acquisition through dialogue as stream-based active learning, exemplified by the segmentation of user utterances containing novel words. In stream-based active learning, data instances are presented sequentially, and the system selects an action for each instance based on an acquisition function that determines whether to request the correct answer from the oracle (in this case, the user). To improve the efficiency of training the acquisition function via reinforcement learning, we introduce two extensions: (1) a new action that performs semi-supervised learning, and (2) a state representation that takes the remaining budget into account. Our simulation-based experiments showed that these two extensions improved word segmentation performance with fewer questions for the user, compared to a baseline without these extensions.

## introduction dialogue systems need the ability to ask questions and acquire necessary information from users through dialogue. for example, large language models (llms) do not necessarily cover local expressions, such as nicknames used among friends and family or abbreviations commonly used in schools or local communities. to enable natural and casual conversations with users, spoken dialogue systems must be able to handle these terms, including their spellings and pronunciations. a key challenge lies in avoiding excessive or repetitive questions to maintain user engagement. while questioning the user is a reasonable strategy for acquiring such information @xcite , general users-unlike crowdworkers-do not anticipate being asked similar or repetitive questions @xcite . therefore, the system should be carefully designed to minimize negative impressions and enhance overall user experience. repeatedly asking the same type of question has been shown to degrade the user experience @xcite . in this study, we formulate knowledge acquisition through dialogue as stream-based active learning @xcite , using the segmentation of user utterances as an example task. in active learning, the system selects instances that are expected to be particularly informative for improving a machine learning classifier (within a limited query budget) and obtains their reference labels from an oracle for supervised training. streambased active learning is a form of active learning in which data instances are presented sequentially, and at each step, the system makes a binary decision, i.e., whether to request the reference label or not, on the basis of an acquisition function. this corresponds to deciding whether to ask the user about the correct segmentation of a given utterance. figure 1 illustrates such an interaction as an example. previous research has shown that acquisition functions in stream-based active learning can be optimized using reinforcement learning @xcite . note that, while we assume the user can act as an oracle and provide the correct segmentation results, it remains challenging to accurately obtain such results from diverse user responses. to improve the training efficiency of the acquisition function through reinforcement learning, we introduce two extensions. first, we add a new action, which we call self-learning, corresponding to semi-supervised learning: the system treats the current segmentation result as correct without querying the user, and utilizes it for training. second, we incorporate the remaining query budget into the state representation. this enables the system to learn a strategy such as refraining from selflearning when the segmentation model is still unreliable, and relying on it more actively once the model has been sufficiently trained. the contributions of this work are twofold: ## related work life-long learning is an approach that enables a system to continually improve its performance @xcite . as part of this, research has explored acquiring factual knowledge through dialogue to overcome the limitations of fixed knowledge bases @xcite . further challenges such as the handling of incorrect knowledge and the revision of previously acquired knowledge have also been discussed @xcite . exploiting user utterances or feedback for supervision in dialogue has also been proposed. for example, a chatbot may extract new training examples from dialogue and solicit user feedback @xcite . in robotics, dialogue has been used as a means for robots to acquire new concepts such as object names or spatial terms @xcite ). an embodied robot may encounter novel objects, as well as unfamiliar place names and action concepts, in open-world settings. for example, robots have been trained to incrementally improve their language understanding and concept grounding through interaction with humans @xcite , and to learn through clarification and active learning queries @xcite . one recent attempt also explores how a robot can learn unknown object names, locations, and actions through situated dialogue @xcite . in neural response generation, techniques such as retrieval-augmented generation (rag) @xcite can incorporate a new word that appears during a dialogue into the generated text; however, similar to the robotics studies discussed above, our goal is to understand and acquire such knowledge in a way that allows it to be reused. while several studies have focused on storing user preferences or profiles during dialogue for the purpose of personalization @xcite , our objective is fundamentally different. the framework of "asking intelligently" can be applied to lexical acquisition @xcite , estimation of user satisfaction @xcite , and user sentiment estimation @xcite . furthermore, rather than selecting questions in active learning solely based on fixed uncertainty metrics, our framework learns an acquisition function via reinforcement learning, enabling the system to flexibly decide whether to ask a question by taking future rewards into account. user impression given by a system's question is important. it is necessary to consider how the question will be perceived by the user when it is asked, for example, repeated explicit questions can quickly become annoying. studies that address the impression of different types of system questions include, for example, @xcite . ideally, a system should balance the expected utility of asking a question with the impression it conveys. ## conclusion in this study, we cast knowledge acquisition through dialogue as a stream-based active learning problem. using the segmentation of user utterances containing novel words as a target task, we proposed two extensions in reinforcement learning to improve the efficiency of training the acquisition function. simulation-based experiments showed the potential of the proposed method to improve the segmentation performance while reducing the number of questions asked to the user. these results suggest that the system can strategically learn to ask questions in order to improve its performance with minimal user effort. an important direction for future work is to evaluate the proposed method with human users rather than through simulations. it is crucial to assess not only the performance of the task-dependent model but also the human impression of the interaction. for instance, the current question strategy is based on the assumption that users will correctly provide all isolated words for word segmentation. however, this assumption is neither realistic nor user-friendly in real interactions. therefore, it is necessary to design question strategy that minimizes the user effort while maximizing the model's performance. in addition, stabilizing the training of dqn by introducing a validation set remains an important issue. this is because the performance of dqn often depends on several factors, including the initial values of network weights, the permutation of the instances in the training and test sets, and reinforcement learning hyperparameters such as ϵ in the epsilon-greedy strategy. memory limitation is a potential issue for streambased active learning during dialogue, especially in life-long dialogue scenarios. while the memory capacity is typically limited, the number of paired instances and their obtained labels grows monotonically over time. as a result, when the memory reaches its capacity, it may be necessary to replace stored data with new paired data based on their importance. on the other hand, certain models like npylm, which we used in this paper, allow the recursive update of parameters and thus can mitigate the memory limitation issue. for instance, it is not necessary to store all paired data for updating the word-count parameters in npylm because these parameters are simply incremented for each paired data (there are no gradient computation and iteration process).