Covering Uncommon Ground: Gap-Focused Question Generation for Answer Assessment (2023)

Abstract: Human communication often involves information gaps between the interlocutors. For example, in an educational dialogue a student often provides an answer that is incomplete, and there is a gap between this answer and the perfect one expected by the teacher. Successful dialogue then hinges on the teacher asking about this gap in an effective manner, thus creating a rich and interactive educational experience. We focus on the problem of generating such gap-focused questions (GFQs) automatically. We define the task, highlight key desired aspects of a good GFQ, and propose a model that satisfies these. Finally, we provide an evaluation by human annotators of our generated questions compared against human generated ones, demonstrating competitive performance.

## introduction natural language dialogues are often driven by information gaps. formally, these are gaps between the epistemic states of the interlocutors. namely, one knows something that the other does not, and the conversation revolves around reducing this gap. an important example is the education setting where teachers ask students questions, and receive answers that may be incomplete. with the expectation of what a complete answer should contain, the teacher then engages in a gap-focused dialogue to help the student to arrive at a complete answer. there are multiple other application settings of information gaps, including support-line bots, longform q&a, and automated fact checking. the core challenge in this setting is how to generate effective questions about the information gap. in terms of formal semantics and pragmatics, this gap can be viewed as the complementary of the common-ground @xcite held by the interlocutors. somewhat surprisingly, despite much work on dialogue learning @xcite and question generation @xcite @xcite , little attention has been given to generating questions that focus on such information gaps. the formal traditional approach to representing the dialogic information gap is via the set of propositions that are known to one side but not the other @xcite . however, this set can be quite large, and it is also unclear how to turn these propositions into dialogue utterances. we propose an arguably more natural representation: a generated set of natural language questions whose answers represent the information that the teacher needs to ask about to reduce the gap. we call these gapfocused questions (gfqs). a key advantage of this representation is that the generated questions can be used directly in the teacher-student dialogue. given a complete teacher answer and a partial student answer, there are many questions that could be asked, but some are more natural than others. for example, consider the complete answer "a man is wearing a blue hat and a red shirt and is playing a guitar", and a student response "there is a man playing the guitar". two candidate questions could be "what color hat is the man wearing?" and "what is the man wearing?". the second question is arguably more natural as it does not reveal information that is not in the teacher-student common ground, namely that a hat is being worn. the above demonstrates some of the complexity of generating effective gfqs, and the need to rely on certain discourse desiderata. in this work we define the gfq challenge, a novel question generation task, and we detail the desired properties of the generated questions. subsequently, we provide a model for gfq generation that aims to satisfy these desiderata, and demonstrate its competitiveness via a task of generating questions to fill the gap between premises and hypotheses in a standard natural language inference (nli) setup. in designing desired properties for gfqs, we take inspiration from theories of collaborative source text: a man, with a full face mask, sits on a sidewalk playing the guitar. student description: a man is wearing a mask. gap-focused follow-up question: what does the man wearing a mask do? generated questions: -where is the man sitting? -on what is the man sitting? -what is the man wearing? (a) -who is wearing a full face mask? (a) -what is he doing with the guitar? (b) -where does the man with the full face mask sit? (b) -what does the man in the mask do? figure 1 : our gap-focused question setup and approach. a student is asked to describe a source text from memory. the goal is to ask a follow-up question about information the student missed. our approach is to generate a list of candidate questions, and then filter out ones that are either answerable from the student text (red strike-through (a)) or contain facts unknown to the student (yellow strike-through (b)). the follow-up question can be any of the remaining questions. to tackle the gfq challenge, we show how general-purpose nlp models (question generation, question answering, and constituency parsing) can be used to generate gfqs that satisfy the discourse desiderata. see figure 1 for an outline of the process. to assess our model, we consider pairs of texts that contain information gaps, and evaluate our ability to capture these gaps using gfqs. such texts are readily available in nli datasets that contain pairs of a premise and an entailed hypothesis with less information. we consider the snli dataset @xcite , and use human annotators to evaluate the merit of our approach relative to gfqs generated by humans. our contribution is three-fold. first, we propose the novel setup of gap-focused questions, a key element of a student-teacher discourse as well as other settings such as automated fact checking. second, we identify desiderata inspired by conversational maxims, and provide a model for generating questions that satisfy them. third, we demonstrate the merit of our model on an nli dataset. ## related work natural dialogue is a key goal of modern nlp and, despite substantial progress, there is still a considerable difference between humans and models. in this work we focus on dialogues where the bot (teacher) knows more than the user (student), and the goal is to gradually decrease this knowledge gap via gap-focused follow-up questions. several works have focused on the problem of follow-up question generation in dialogues. however, to the best of our knowledge, none of these focus on information gaps as we do. @xcite introduce the problem of inquisitive question generation, where the goal is to generate questions about facts that are not in the text. this is not done in reference to a complete text, and is thus principally different from our goal. in fact, in our settings, an inquisitive question would typically be a bad gfq, since it refers to information that is outside the knowledge of both teacher and student. prior works considered a related task referred to as answer-agnostic question generation @xcite , but with a focus on factual questions, whereas the inquistive setting is broader. another class of follow-up questions are clarification ones @xcite , which can also be viewed as a special case of inquistive questions. again, there is no reference to a complete text that defines the information gap. finally, there are works on follow-up questions guided by rules as in the sharc dataset @xcite . our gfq setting is also related to the challenge of explainable nli @xcite , namely the task of explaining why a certain sentence entails another. the gfq output can be viewed as a novel explanation mechanism of why the student text is entailed by the source text, as it explicitly refers to the gap between these texts. our work is inspired by novel uses of question generation models, particularly in the context of evaluating model consistency @xcite . in these, question generation is used to find "llm hallucinations" where the generated text is not grounded in a given reference text. our task can be viewed as the inverse of the knowledge grounding task, and our particular focus is on the questions generated rather than just pointing to information gaps. an additional line of work in this vein is qa-based semantics, where text semantics are represented via a set of questions rather than a formal graph (e.g., see @xcite . ## criteria for gap-focused questions given a complete source text t c and a student text t s , our goal is to construct a model that takes t s and t c as input and produces a set of one or more questions q that ask about the information gap between t c and t s . if one takes the term "information gap" literally, there are many such possible questions (e.g., which word appears in t c but not in t s ). in a natural language setting we are obviously interested in questions that are natural, that is, would likely be asked by a human who knows t c and has heard the student description t s . when defining the desiderata for the generated questions, we consider what knowledge is held by the teacher and the student and what information is inside and outside their common ground (see figure 2 ). we next identify desired properties for the generated questions, followed by a description of our model for generating gap-focused questions that satisfy these desiderata. the following desired properties of an effective gfq are loosely based on collaborative communication concepts @xcite : ## the gfqs generation approach we next describe our modeling approach for the gfq generation problem, with the goal of capturing the properties described above. before describing our gfqs generation approach, we briefly outline the nlp components we rely on in the question generation process: a question generation model g that, given an input text t and a span x ⊂ t , generates questions about t whose answer is x. a question answering model a, that takes as input a text t and a question q about the text, and returns the answer or an indication that the question is unanswerable from the text. a constituency parser p , that takes a text x, breaks it down into sub-phrases (constituents), and returns a parse tree. additional details about these components can be found in appendix c. we are now ready to describe our approach for generating gfqs. the model generates an ordered set of possible follow-up questions q g via the following steps, which roughly correspond to the desired criteria described in §3: ## experiments we next describe an evaluation of our gfq model. ## conclusion we consider the task of question generation in a novel setting where there is an information gap between speakers, and the gap-focused questions (gfqs) aim to reduce this gap. building on advances in question generation and question answering, we show how to generate useful gfqs that meet several natural criteria inspired by theories cooperative conversation. it is natural to ask whether one can employ a fully generative approach for gfqs using llms. this is a natural direction for future study, and we believe that the criteria and design choices we studied here will be significant in defining and evaluating such future work.