Inquisitive Question Generation for High Level Text Comprehension (2020)

Abstract: Inquisitive probing questions come naturally to humans in a variety of settings, but is a challenging task for automatic systems. One natural type of question to ask tries to fill a gap in knowledge during text comprehension, like reading a news article: we might ask about background information, deeper reasons behind things occurring, or more. Despite recent progress with data-driven approaches, generating such questions is beyond the range of models trained on existing datasets. We introduce INQUISITIVE, a dataset of ~19K questions that are elicited while a person is reading through a document. Compared to existing datasets, INQUISITIVE questions target more towards high-level (semantic and discourse) comprehension of text. We show that readers engage in a series of pragmatic strategies to seek information. Finally, we evaluate question generation models based on GPT-2 and show that our model is able to generate reasonable questions although the task is challenging, and highlight the importance of context to generate INQUISITIVE questions.

## introduction the ability to generate meaningful, inquisitive questions is natural to humans. studies among children @xcite showed that questions serving to better understand natural language text are an organic reflection of curiosity, which "arise from the perception of a gap in knowledge or understanding" @xcite . because of its prominence in human cognition and behavior, being able to formulate the right question is highly sought after in intelligent systems, to reflect the ability to understand language, to gather new information, and to engage with users @xcite @xcite @xcite . a recent line of work on data-driven this gritty city of 7.6 million rarely gets respect. it often ranks low in "livability" surveys and near the top of the hemisphere's ugliest capitals. what constitutes "livability"? question generation techniques @xcite @xcite focuses on generating questions for datasets like squad @xcite . however, factoid questions generated with an answer in mind after the user has read the full text look very different from more natural questions users might have @xcite . this has led to work on "answer-agnostic" question generation @xcite @xcite , but the sources of data still emphasize simple factoid questions. other prior work used question generation to acquire domainspecific knowledge @xcite and seek clarification in conversation @xcite @xcite . however, data-driven generation of questions that reflect text understanding in a more general setting is challenging because of the lack of appropriate training data. we introduce inquisitive, a new, large-scale dataset of questions that target high level processing of document content: we capture questions elicited from readers as they naturally read through a document sentence by sentence. because these questions are generated while the readers are processing the information, the questions directly communicate gaps between the reader's and writer's knowledge about the events described in the text, and are not necessarily answered in the document it-self. this type of question reflects a real-world scenario: if one has questions during reading, some of them are answered by the text later on, the rest are not, but any of them would help further the reader's understanding at the particular point when they asked it. this resource is a first step towards understanding the generation of such curiosity-driven questions by humans, and demonstrates how to communicate in a natural, inquisitive way by learning to rely on context. specifically, we crowdsource ∼19k questions across 1,500 documents foot_0 , each question accompanied by the specific span of the text the question is about, illustrated in figure 1 . the questions are verified to ensure that they are grammatically correct, semantically plausible and meaningful, and not already answered in previous context. we show that the questions capture a variety of phenomena related to high-level semantic and discourse processes, e.g., making causal inferences upon seeing an event or a description, being curious about more detailed information, seeking clarification, interpreting the scale of a gradable adjective @xcite , seeking information of an underspecified event (where key participants are missing), and seeking background knowledge. our analyses reveal that the questions have a very different distribution from those in existing factoid and conversational question answering datasets @xcite @xcite , thus enabling research into generating natural, inquisitive questions. we further present question generation models on this data using gpt-2 @xcite , a state-of-the-art pre-trained language model often used in natural language generation tasks. human evaluation reveals that our best model is able to generate high-quality questions, though still falls short of human-generated questions in terms of semantic validity, and the questions it generates are more often already answered. additionally, our experiments explore the importance of model access to already-established common ground (article context), as well as annotations of which part of the text to ask about. finally, transfer learning results from squad 2.0 @xcite show that generating inquisitive questions is a distinct task from question generation using factoid question answering datasets. the capability for question generation models to simulate human-like curiosity and cognitive processing opens up a new realm of applications. one example for this sort of question generation is guided text writing for either machines or humans: we could use these questions to identify important points of information that are not mentioned yet and should be included. in text simplification and summarization, these questions could be used to prioritize what information to keep. the spans and the questions could also help probing the specific and vague parts of the text, which can be useful in conversational ai. because of the high level nature of our questions, this resource can also be useful for building education applications targeting reading comprehension. ## related work among question generation settings, ours is most related to answer-agnostic, or answer-unaware question generation: generating a question from text without specifying the location of the answer @xcite . recent work @xcite @xcite trains models that can extract phrases or sentences that are question-worthy, and uses this information to generate better questions. @xcite paired the question with other sentences in the article that do not contain the answers to construct curiosity-driven questions. however, these approaches are trained by re-purposing questionanswering datasets that are factual @xcite or conversational @xcite . in contrast, we present a new dataset targeting questions that reflect the semantic and discourse processes during text comprehension. several other question answering datasets contain questions that are more information-seeking in nature. some of them are collected from questions that users type in search engines @xcite @xcite . others are collected given a small amount of information on a topic sentence @xcite or in the context of a conversation @xcite @xcite . our data is collected from news articles and our questions are precisely anchored to spans in the article, making our questions less open-ended than those in past datasets. while @xcite also collected a small number of reader questions from news articles, their goal was to study underspecified phrases in sentences when considered out-of-context. contemporaneously, @xcite presented a dataset of 2.4k naturally elicited questions on ted talks, with the goal to study linguistic theories of discourse. previous work generating clarification questions @xcite @xcite uses questions crawled on forums and product reviews. the answers to the questions were used in the models to improve the utility of the generated question. in our data, clarification is only one of the pragmatic goals. in addition, we focus on news articles which contains more narrative discourse and temporal progression. ## inquisitive: a corpus of questions this section presents inquisitive, a corpus of ∼19k questions for high level text understanding from news sources (section 3.1), which we crowdsource with a specific design to elicit questions as one reads (section 3.2). we then discuss a second validation step for each question we collected as quality control for the data (section 3.3). ## data analysis in this section, we present a deep dive into inquis-itive, showing that the questions are much higher level than existing datasets (section 4.1), and have rich pragmatic functions (section 4.2). we also investigate the highlighted span associated with each question (section 4.3), and the relative salience of questions to the article (section 4.4). 2 the numbers before filtering are: 2153, 1968, 16816. ## question generation from known spans we use inquisitive to train question generation models and evaluate the models' ability to generate questions with access to the gold-standard spans figure 3 : we concatenate the context, sentence and questions and learn a language model on them using gpt-2. ## question generation from scratch in this section, we assume that spans are not given, and discuss two approaches to generating questions from scratch: a pipeline approach with span prediction, and a question generation model without access to any span information. ## conclusion we present inquisitive, a large dataset of questions that reflect semantic and discourse processes during text comprehension. we show that people use rich language and adopt a range of pragmatic strategies to generate such questions. we then present question generation models trained on this data, demonstrating several aspects that generating inquisitive questions is a feasible yet challenging task.