Asking More Informative Questions for Grounded Retrieval (2024)

Abstract: When a model is trying to gather information in an interactive setting, it benefits from asking informative questions. However, in the case of a grounded multi-turn image identification task, previous studies have been constrained to polar yes/no questions (White et al., 2021), limiting how much information the model can gain in a single turn. We present an approach that formulates more informative, open-ended questions. In doing so, we discover that off-the-shelf visual question answering (VQA) models often make presupposition errors, which standard information gain question selection methods fail to account for. To address this issue, we propose a method that can incorporate presupposition handling into both question selection and belief updates. Specifically, we use a two-stage process, where the model first filters out images which are irrelevant to a given question, then updates its beliefs about which image the user intends. Through self-play and human evaluations, we show that our method is successful in asking informative open-ended questions, increasing accuracy over the past state-of-the-art by 14%, while resulting in 48% more efficient games in human evaluations.

## introduction as nlp models are increasingly deployed in interactive settings, it is key that these models are able to gather information about the user's intentions and the underlying world context. models might do this by asking questions to the user -however, asking informative questions is challenging, as it relies on reasoning about the current context, the history of the interaction, and potential future plans. past work on question generation for interaction has formulated questions that are predicted to be informative, but has typically used questions with constrained answer spaces @xcite @xcite , which can lead to inaccurate and inefficient interactions in contextually rich tasks. we present an approach for generating openended questions in one such contextually rich task: a multi-turn image identification task @xcite inspired by interactive retrieval. in this task, a model is presented with an array of images, one of which is a target image known only to a human user. the model has to identify this image by formulating questions for the user to answer. a sample game can be seen in figure 1 , and more details can be found in section 2. we select this task as a challenging test bed for strategic contextual interaction: requiring strong abilities in grounding (to distinguish between similar images that differ only in their less-salient details, e.g. all images in figure 1 contain a computer) and planning (asking informative questions that take into account the history of interaction). previous approaches to this task @xcite have used polar yes/no questions in the form of "isa" questions. although these isa questions are formulated to be maximally informative according to an expected information gain metric (section 2.2), polar questions do not scale well to settings where images are similar, or with a larger number of images. it is also not straightforward to simply replace isa questions with broader openended questions (i.e. wh-questions such as who, where, when, etc.) we show (section 5) that doing so naively produces questions with presupposition errors, where an off-the-shelf visual question answering (vqa) model gives confident answers to questions that are irrelevant to an image. an example of a presupposition error would be asking "what is the dog eating?" when there is no dog in the target image. this failure renders the standard information gain metric inappropriate and results in a chain of errors in belief updates over the model's target images. constraining to polar yes/no questions largely protected previous approaches from facing such issues, but once open-ended questions are introduced, the chances for presupposition errors rise substantially. we propose a method that can generate openended informative questions in the interactive image retrieval setting, relaxing the polar yes/no constraints, while at the same time being able to handle presuppositions. we do this by conducting a twostage process: first explicitly checking for presuppositions to filter out irrelevant images, then calculating information gain based only on the relevant images. we show that after handling such errors, asking open-ended wh-questions substantially outperforms constraining to polar yes/no questions. in our human evaluations, our method results in 48% more efficient games compared to the past state-of-the-art for this task @xcite , while improving accuracy by 13%. we further conduct ablation studies to verify that our two-stage pipeline is indeed successful in eliminating presupposition errors, and that it can generalize to settings where there are substantially more images to choose between. our method is even more successful in this more challenging setting, resulting in an average of 3.5 times shorter games. ## background we discuss the task and notation used, as well as some key concepts/tools used in our model. ## methods our main pipeline is illustrated in figure 2 . we use a training-free approach, using only off-the-shelf pre-trained vqa and text processing models. using the image contexts, we first generate a large pool of possible candidate questions q (section 3.1), then select the question @xmath0, taking into consideration the various presupposition assumptions (section 3.2). upon asking the question q and receiving a response r, the model then updates its belief distribution accordingly (section 3.3). ## experimental setup we describe our experimental setup in sections 4.1, 4.2, and 4.3. we then compare our approach to baselines in both games with human partners (section 4.4), as well as in automatic model-based evaluations (section 4.5). ## analysis and ablation studies we conduct an ablation study (section 6.1) to examine the importance of various components of the presupposition handling process. ## related work interactive information seeking ambiguity is a persistent challenge in language. recent work has approached ambiguity resolution through the lens of interactive information seeking, borrowing from the optimal experimental design literature @xcite ). these methods rely on a partner model, which is used to measure the informativity of questions @xcite @xcite . most related to ours is the work by @xcite , which proposes an unsupervised approach to informative question generation. their approach is limited to polar yes/no questions, which we extend to open-ended wh-questions. @xcite also present an approach to generating open-ended informative questions by training a variational autoencoder on vqa datasets. similar to our approach, their method selects questions that are both (1) informative and (2) relevant to the image. our approach optimizes for similar objectives, but uses off-the-shelf vqa and llms without any training data for generating questions. presupposition errors and question decomposition presupposition errors lead to one form of unanswerable question @xcite . such questions have been extensively explored in literature @xcite . past work has shown that models trained on datasets with-out unanswerable questions often fail to generalize to unanswerable questions @xcite . we propose a method for adapting a model trained only on answerable questions to unanswerable questions via question decomposition, without supervision. this contrasts with recent work on question decomposition, which has focused on decomposing complex questions @xcite . collaborative reference games collaborative reference games focus on building common ground in a symmetric dialogue setting, where both participants have equal roles @xcite @xcite @xcite . while both settings require reasoning, we focus on the asymmetric question-answering setting, where the asymmetry prevents the questioner from relying too heavily on their partner. ## conclusion we present an approach for generating open-ended informative questions in a grounded multi-turn image identification task. as compared to previous methods which ask questions with constrained answer spaces, our method is able to ask more general questions. we show that directly asking openended questions may lead to presupposition errors, where off-the-shelf vqa models may answer questions despite their irrelevance to images. to address this, we propose a two-stage method where we first formulate a question to verify the presupposition, then update the belief distribution over the images accordingly. through both human and self-play evaluations, we show that asking presuppositionaware open-ended questions outperforms the previous state-of-the-art in both accuracy and efficiency.