Learning to Ask Informative Questions: Enhancing LLM s with Preference Optimization and Expected Information Gain (2024)

Abstract: Questions are essential tools for acquiring the necessary information to complete information-seeking tasks. However, large language models (LLMs), especially open-source models, often perform poorly in generating informative questions, as measured by expected information gain (EIG). In this paper, we propose a method to enhance the informativeness of LLM-generated questions in 20-question game dialogues. We sample multiple questions from the same model (LLaMA 2-Chat 7B) for each game and create pairs of low-EIG and high-EIG questions to apply a Direct Preference Optimization (DPO) algorithm. Our results show that this method produces more effective questions (in terms of EIG), even in domains different from those used to train the DPO model.

## introduction questions in language serve as requests for information @xcite . a speaker lacks information in their knowledge state and asks questions to gain this information. this process of acquiring information through questioning is essential for children to learn about the world @xcite and for adults to solve complex problems @xcite . questions, however, vary in their level of informativeness, with some questions being more informative and efficient in reaching the problem's solution @xcite . cognitive science provides two interesting tools to study questions' informativeness: the 20 questions game as a test bed, and the expected information gain (eig) as a quantifying measure @xcite . the 20 questions game consists of one player asking yes/no questions to identify the item the other player has in mind, from a pool of possible items. in this context, the eig @xcite ) measures questions' informativeness as the (expected) entropy reduction caused by a certain question in the space of possible items. most informative polar questions partition the possible items into two same-size clusters: items in one cluster are expected to receive a positive answer, and items in the other cluster are expected to receive a negative one. in recent years, large language models (llms) have demonstrated remarkable language and reasoning capabilities @xcite . the ability to ask informative and effective questions is crucial for employing these systems as successful user assistants on a large scale. although llms have demonstrated their ability to play the 20 questions game, their questions are characterised by low informativeness and limited success @xcite . recent studies have emerged to improve the informativeness of llms' questions. @xcite improve open-source llms via reinforcement learning and behavioral cloning from larger llms' data. relying on llms' ability to generate diverse questions and provide reliable answers @xcite , @xcite propose an inference-time re-ranking strategy based on eig. differently from them, we propose a training strategy leveraging eig as a refined signal with a preference optimization algorithm. unlike other approaches, all the steps of our method are accomplished by the same open-source model, without requiring annotation or feedback signals from external models. this study aims to improve the informativeness of questions generated by llms, thereby leading to more efficient agents. the 20 questions game serves as a testbed to illustrate the feasibility of our approach. to achieve our aim, we propose a method involving three steps: sampling multiple questions, questions' evaluation in terms of eig, and training with preference optimization. in the first step, the model generates a set of possible questions, and it computes the eig for each question. this set of questions, along with the corresponding eig values, as a proxy for the questions' informativeness, is used for direct preference optimization (dpo; @xcite . our results show that eig is a strong training signal to improve the question-asking capabilities of current llms and overcome their shortcomings in asking effective questions. 1 while these findings hold in the controlled setting of the 20 questions game, our approach could be extended outside this paradigm and metric to train different models' capabilities. our core idea of sampling, self-annotation to filter with a metric, and preference optimization could improve questions' generations in a plethora of tasks -recommendation systems @xcite and image retrieval @xcite for example. other metrics, such as expected savings @xcite , could be employed to generate large preference datasets and improve llms' information-seeking abilities beyond our tested paradigm. ## results different methods are assessed on test sets with candidate sets of the same size as in training but from other domains. the results are reported in given the promising results of dpo, we proceed to evaluate the robustness of this approach with larger candidate sets (> 8) at test time -preference optimization is still based on candidate sets of 8 elements. as shown in ## analysis first of all, following @xcite , we examine the types of questions that models ask: constraint seeking (cs) are questions about a feature shared by more candidates, whereas hypothesis scanning (hs) are questions about one single candidate. in every test set apart from bigbench, zero-shot asks more cs with lower informativeness than dpo ( a: no q: is the target a type of vehicle (car, train, airplane, etc.)? a: no. q: is the target a type of building or structure (house, train station, airport, etc.)? a: no. q: is the target a form of technology (computer, phone, micropipette, etc.)? a: no. q: is the target a type of clothing or accessory (shirt, pants, hat, etc.)? a: no. q: is the target a type of emotion or feeling (love, hate, joy, etc.)? a: yes. q: great! is the target "love"? a: no. q: is the target "hate"? a: no. ... q: is the target "anger"? figure 2 : example from the bigbench. dpo asks grounded cs questions (highlighted with colors), identifying the subset of the target (e.g., emotions). it then asks a series of hs questions. ## conclusion in our work, we designed a direct preference optimization (dpo) approach to enhance the informativeness of questions asked by llms using expected information gain (eig). we utilized the 20 questions game paradigm, a framework in cognitive science and ai for studying informationseeking behavior, reasoning, and hypothesis testing. our approach involved (a) sampling multiple questions from the model in a zero-shot fashion, (b) clustering the questions based on their eig, and (c) training the model using these clusters with dpo. our results show that dpo significantly improves question informativeness (measured by average eig and number of questions asked), making the dialogue strategy more effective. notably, this method generalizes well to different domains. our findings demonstrate that eig is a promising training signal for improving the reasoning capabilities of llms in information-seeking dialogues.