Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge (2021)

Abstract: The ability to generate clarification questions i.e., questions that identify useful missing information in a given context, is important in reducing ambiguity. Humans use previous experience with similar contexts to form a global view and compare it to the given context to ascertain what is missing and what is useful in the context. Inspired by this, we propose a model for clarification question generation where we first identify what is missing by taking a difference between the global and the local view and then train a model to identify what is useful and generate a question about it. Our model outperforms several baselines as judged by both automatic metrics and humans.

## introduction an important but under-explored aspect of text understanding is the identification of missing information in a given context i.e., information that is essential to accomplish an underlying goal but is currently missing from the text. identifying such missing information can help to reduce ambiguity in a given context which can aid machine learning models in prediction and generation @xcite . @xcite recently proposed the task of clarification question generation as a way to identify such missing information in context. they propose a model for this task which while successful at generating fluent and relevant questions, still falls short in terms of usefulness and identifying missing information. with the advent of large-scale pretrained generative models @xcite @xcite , generating fluent and coherent text is within reach. however, generating clarification questions requires going beyond fluency and relevance. doing so requires understanding what is missing, which if included could be useful to the consumer of the information. humans are naturally good at identifying missing information in a given context. they possibly make use of global knowledge i.e., recollecting previous similar contexts and comparing them to the current one to ascertain what information is missing and if added would be the most useful. inspired by this, we propose a two-stage framework for the task of clarification question generation. our model hinges on the concept of a "schema" which we define as the key pieces of information in a text. in the first stage, we find what's missing by taking a difference between the global knowledge's schema and schema of the local context ( §3.1). in the second stage we feed this missing schema to a fine-tuned bart @xcite model to generate a question which is further made more useful using pplm @xcite ( §3.2). 1 we test our proposed model on two scenarios ( §2): community-qa, where the context is a product description from amazon.com @xcite (see e.g. furthermore, our analysis reveals reasoning behind generated questions as well as robustness of our model to available contextual information. ( §5). community-qa community-driven questionanswering has become a common venue for crowdsourcing answers. these forums often have some initial context on which people ask clarification questions. we consider the amazon questionanswer dataset @xcite where context is a product description and the task is to generate a clarification question that helps a potential buyer better understand the product. goal oriented dialog with the advent of high quality speech recognition and text generation systems, we are increasingly using dialog as a mode to interact with devices @xcite . however, these dialog systems still struggle when faced with ambiguity and could greatly benefit from having the ability to ask clarification questions. we explore such a goal-oriented dialog scenario using the ubuntu dialog corpus @xcite consisting of dialogs between a person facing a technical issue and another person helping them re-solve the issue. given a context i.e a dialog history, the task is to generate a clarification question that would aid the resolution of the technical issue. ## approach figure 1 depicts our approach at a high level. we propose a two-stage approach for the task of clarification question generation. in the first stage, we identify the missing information in a given context. for this, we first group together all similar contexts in our data foot_1 to form the global schema for each high-level class. next, we extract the schema of the given context to form the local schema. finally, we take a difference between the local schema and the global schema (of the class to which the context belongs) to identify the missing schema for the given context. in the second stage, we train a model to generate a question about the most useful information in the missing schema. for this, we fine-tune a bart model @xcite on (missing schema, question) pairs and at test time, we use pplm @xcite with a usefulness classifier as the attribute model to generate a useful question about missing information. ## experiments we aim to answer the following research questions (rq): ubuntu the ubuntu dialog corpus @xcite consists of utterances of dialog between two users on the ubuntu chat forum. given a dialog, we identify utterances that end with a question mark. we then create data instances of (context, question) where the question is the utterance ending with a question mark and the context consists of all utterances before the question. we consider only those contexts that have at least five utterances and at most ten utterances. ## related work most previous work on question generation focused on generating reading comprehension style questions i.e., questions that ask about information present in a given text @xcite . later, @xcite introduced the task of clarification question generation in order to ask questions about missing information in a given context. clarq @xcite entails clarification questions in a question answering setup. however, unlike our work, these works still suffer from estimating the most useful missing information. recent works on conversational question answering also focused on the aspect of question generation or retrieval @xcite . @xcite especially focused on generating information-seeking questions while majumder et al. (2020) proposed a question generation task in free-form interview-style conversations. in this work, in addition to improving clarification question generation in a community-qa dataset, we are the first to explore a goal-oriented dialog scenario as well. representing context and associated global information in a structure format has been shown to improve performance in generation task @xcite @xcite in general and summarization @xcite and story-generation @xcite in particular. we also derive inspiration from recent works on information extraction from free-form text @xcite and develop a novel framework to estimate missing information from available natural text contexts. finally, for question generation, we use bart @xcite , that is state-of-the-art for many generation tasks such as summarization, dialog generation etc. furthermore, inspired from recent works that use controlled language generation during decoding @xcite , we use plug-and-play-language-model @xcite to tune generations during decoding. while similar approaches for controllable generation @xcite ) have been proposed, we extend such efforts to enhance the usefulness of the generated clarification questions. ## conclusion we propose a model for generating useful clarification questions based on the idea that missing information in a context can be identified by taking a difference between the global and the local view. we show how we can fine-tune a large-scale pretrained model such as bart on such differences to generate questions about missing information. further, we show how we can tune these generations to make them more useful using pplm with a usefulness classifier as its attribute model. thorough analyses reveal that our framework works across domains, shows robustness towards information availability, and responds to the dynamic change in global knowledge. although we experiment only with amazon and ubuntu datasets, our idea is generalizable to scenarios where it is valuable to identify missing information such as conversational recommendation, or eliciting user preferences in a chit-chat, among others. we do not foresee any immediate ethical concerns since we assume that our work will be restricted in domain as compared to free-form language generation. we still cautiously advise any developer who wishes to extend our system for their own use-case (beyond e-commerce, goal-oriented conversations) to be careful about curating a global pool of knowledge for data involving sensitive user information. finally, since we are finetuning a pretrained generative model, we inherit the general risk of generating biased or toxic language, which should be carefully filtered. in general, we expect users to benefit from our system by reducing ambiguity (when information is presented in a terse fashion, e.g. in a conversation) and improving contextual understanding to enable them to take more informed actions (e.g. making a purchase).