C lar Q : A large-scale and diverse dataset for Clarification Question Generation (2020)

Abstract: Question answering and conversational systems are often baffled and need help clarifying certain ambiguities. However, limitations of existing datasets hinder the development of large-scale models capable of generating and utilising clarification questions. In order to overcome these limitations, we devise a novel bootstrapping framework (based on self-supervision) that assists in the creation of a diverse, large-scale dataset of clarification questions based on post-comment tuples extracted from stackexchange. The framework utilises a neural network based architecture for classifying clarification questions. It is a two-step method where the first aims to increase the precision of the classifier and second aims to increase its recall. We quantitatively demonstrate the utility of the newly created dataset by applying it to the downstream task of question-answering. The final dataset, ClarQ, consists of ~2M examples distributed across 173 domains of stackexchange. We release this dataset in order to foster research into the field of clarification question generation with the larger goal of enhancing dialog and question answering systems.

## introduction the ubiquitous nature of conversations has led to the identification of various interesting problems @xcite . one of these problems is the ability of a system to ask for clarifications @xcite to a natural language question. a user's complex information need is often lost due to the brevity of the posed question. this leads to an under-specified question containing information gaps which lowers the probability of providing the correct answer. thus, it would be an improvement if a conversational or a question answering system had a mechanism for refining user questions with follow-ups @xcite . in literature, such questions have been termed clarification questions @xcite @xcite . in the domain of question-answering, the major advantages of a clarification question are its ability to resolve ambiguities @xcite and to improve the probability of finding the most relevant answer. for conversational systems, asking such questions help in driving the conversation deeper along with better engagement of the user @xcite . recently, @xcite have provided a dataset based on stackexchange and used it for clarification question retrieval as well as generation. they also modify a dataset based on amazon question-answering and product reviews @xcite to make it suitable for the same task. on the other hand, @xcite created a dataset (qulac) built on top of trec web collections. however, there are several shortcomings to these datasets, which limit the development of generalizable and large-scale models aimed to tackle the problem of clarification question generation. the stackexchange dataset @xcite is created by utilising simple heuristics. this adds a lot of noise, thereby reducing the number of actual clarification questions. it also limits the inclusion of diverse types of questions as it is collected from three similar domains @xcite . the question generation model of @xcite achieves a very low bleu score when trained on this dataset. on the other hand, the dataset based on amazon reviews is a poor proxy for clarification questions because product descriptions are not actual questions demanding an answer and there is no information gap that needs to be addressed. to overcome the shortcomings of existing datasets, we devise a novel bootstrapping framework based on self-supervision to obtain a dataset of clarification questions from various domains of stackexchange. the framework utilises a neural network based architecture to classify clarification questions. in a two step procedure, the framework first increases the precision of the classifier and then increases its recall. the first step is called down-sampling, where the classifier is iteratively trained on the most confident predictions (carried forward over from the previous iteration). the second step is the up-sampling procedure, where the classifier is iteratively trained by successively adding more positively classified examples. this step provides a boost in recall while restricting the drop in precision to a minimum. the classifier trained on the final iteration is then used for identification of clarification questions. the overall process ensures that the final dataset is less noisy and, at the same time, consists of a large and diverse number of examples. we must emphasize that, given the large amount of data available on stackexchange, a classifier with moderate recall still serves our purpose. however, it is imperative that precision of the classifier be reasonably high. ## methodology stackexchange is a network of online question answering websites. on these websites, users may comment on the original post with content such as third party urls, clarifying questions, etc. we only want to select comments which act as clarifying questions and remove the rest as noise. to this end, we devise a bootstrapping framework for training a classifier capable of identifying clarifying questions. the bootstrapping method utilises a neural network based classifier l which is posed with the task of clarification question detection. formally, given a tuple (p, q), where @xmath0, the task is to predict whether q is an actual clarification question for p. this makes it a binary classification problem, where a label 1 indicates q being an an actual clarification question and 0 indicates otherwise. ## dataset statistics the classifier obtained at the end of iterative refinement procedure is used for classifying the initially collected (p, q) tuples of 6,186,934. the classifier predicts 2,079,300 tuples as actual clarification questions. as can be seen from figure 1 , these tuples are unequally distributed across 173 different domains. the top 20 domains account for 69.18% of the total (p, q) tuples in the dataset. the remaining 155 domains account for the remaining 30.82% of the total number of tuples. it is noteworthy that our provided dataset also comprises of actual answers to each post. this would help researchers in evaluating the quality of the clarification questions in a standalone perspective and at the same time with respect to the downstream task of question-answering. ## conclusion and future work in this paper, we present a diverse, large-scale dataset (clarq) for the task of clarification ques-tion generation. it is created by a two-step iterative bootstrapping framework based on selfsupervision. clarq consists of ∼2m post-question tuples spanning 173 different domains. we hope that this dataset will encourage research into clarification question generation and, in the long run, enhance dialog and question-answering systems.