“What makes a question inquisitive?” A Study on Type-Controlled Inquisitive Question Generation (2022)

Abstract: We propose a type-controlled framework for inquisitive question generation. We annotate an inquisitive question dataset with question types, train question type classifiers, and finetune models for type-controlled question generation. Empirical results demonstrate that we can generate a variety of questions that adhere to specific types while drawing from the source texts. We also investigate strategies for selecting a single question from a generated set, considering both an informative vs. inquisitive question classifier and a pairwise ranker trained from a small set of expert annotations. Question selection using the pairwise ranker yields strong results in automatic and manual evaluation. Our human evaluation assesses multiple aspects of the generated questions, finding that the ranker chooses questions with the best syntax (4.59), semantics (4.37), and inquisitiveness (3.92) on a scale of 1-5, even rivaling the performance of human-written questions.

## introduction recently, interest has grown in the task of automatic question generation (aqg) from text @xcite . aqg is useful in building conversational ai systems @xcite , generating synthetic examples for qa @xcite @xcite , and educational applications, such as intelligent tutoring and instructional games @xcite . in the majority of such studies, aqg focuses on generating factual questions that tend to ask about specific information in the text (i.e., "who did what to whom") @xcite . instead of asking factual questions with answers already present in the text, @xcite argued that human readers instinctively ask questions that are curiosity-driven, answer-agnostic, and seek a context . . . the plan places an indicated value on the real estate operation, santa fe pacific realty corp., of $ 2 billion. source sentence santa fe pacific directors are expected to review the plan at a meeting today, according to people familiar with the transaction. ## data we will now describe the annotation of questions with question types, which is one of the main contributions of our work. we describe the annotation process in detail in section 2.2. but first, we briefly introduce the inquisitive dataset. human annotators created the inquisitive questions while reading the initial part (i.e., five sentences) of news articles from the wsj portion of the penn treebank @xcite or associated press articles from the tipster corpus @xcite . 2 annotators first highlighted a span within the sentence that they were curious about and then wrote a maximum of three questions. next, a separate set of annotators validated the questions and excluded unqualified questions (around 5%). an instance in inquisitive has the following components: a source sentence, the sentence the annotator read when asking the question, context that includes all the sentences before the source sentence in the same article, a span within the source sentence the annotators were most curious about, and finally, the question the annotator wrote. in-quisitive is split into training (15,897 instances), test (1,885 instances), and dev (1,984 instances). ## methods in this section, we present our computational approaches for question generation. the input x is a sequence of tokens @xmath0, ..., x n ⟩, which may consist of one or more sentences. the output is a question q that consists of sequence of tokens, i.e., @xmath1, ..., q m ⟩. using the standard autoregressive sequence-to-sequence architecture (sutskever et al., 5 see appendix for bigrams. 6 we use a special token "no_context" if the source sentence is the first sentence in the article. 7 we keep the distribution of question types in train and dev set roughly the same, and the majority question type (explanation) is about 29% of the total data. 2014) we model p θ (q | x) as follows: we use the pretrained bart model @xcite , a transformer @xcite composed of a bidirectional encoder and an autoregressive decoder. in our simplest setup (called base), we concatenate the source sentence and the context. the next setting also concatenates the span; we refer to it as span. each element (e.g., context, span) is separated with the special token "[sep]". ## experiments for all models, we use bart-large with the same settings. for training, we use the adam optimizer (kingma and ba, 2015) with learning rate 3e-5, weight decay 0.01, clip norm 0.1, dropout 0.1, 15 epochs in total, warm-up updates 500, use cross entropy loss with label smoothing (α = 0.1), and set the maximum number of tokens per batch to 1024. more details of the experimental setup are given in the appendix (section a.1). we evaluate the following five settings: ## related work in recent years, automatic question generation has attracted many nlp researchers, perhaps due to its versatility, e.g., question generation for conversational ai @xcite , synthetic examples for qa tasks @xcite @xcite , clarifications on information-seeking conversation @xcite , and knowledge evaluation and educational application areas @xcite @xcite , which is specifically related to our use cases. in earlier work, methods such as transforming declarative sentences into questions (heilman and smith, 2010) or using semantic roles (flor and riordan, 2018) were popular. however, recently sequence-to-sequence architectures @xcite and pretrained models (cao and wang, 2021) are more often used. similar to @xcite , our work is related to answer-agnostic question generation. we focus on exploiting question type information for generating deeper questions. although related work in the answer-unaware setting exists @xcite , they mostly focus on identifying questionworthy text for generation @xcite from factual @xcite , or social media platforms @xcite , different from the wsj/ap news dataset used in our work. we are building on past work on controllable generation, generating text that reflects specific characteristics of control variables. in some earlier work, embedding vectors of the control variables were fed into the model for controlling the output @xcite @xcite . however, our approach resembles recent efforts where the control variable is concatenated to the main input using some separator @xcite . methods such as pplm are useful for similar guided controllable generations @xcite ; however, pplm requires gradient descent at inference time, while our question type selection approach is highly scalable and efficient. we consider controllable question generation based on specific question types, noting that different question templates or ontologies have been studied for question generation. for example, a wikipedia-driven ontology is used for generation @xcite , or contextualized questions are generated for any semantic role @xcite . likewise, @xcite proposed guided generation focusing on including specific keywords (e.g., "wh" words for questions), while we showed in our work is closer to that of cao and wang (2021), who proposed a question type ontology (based on cognitive science) inspired by manually constructed templates @xcite . on the contrary, we chose a dataset that focuses on inquisitive questions only and chose our question types accordingly, while they used a dataset with a broader set of questions. in addition, instead of predicting the text span ("focus" in (cao and wang, 2021)) we directly use the annotated span in our research. finally, we focused on post-processing the generations to identify the best question (or rank them) related to the source content. ## conclusions and future work we proposed a type-controlled framework that generates inquisitive questions given a source sentence, annotated span, and a longer context. we annotated a set of question types related to curiosity driven questions and demonstrated that our framework can generate a variety of questions from a single input. we also developed an effective method (type r ) to select a single question using a pairwise ranker trained on a small set of ranking annotations. our generations, especially from type r , show high novelty. the human evaluation demonstrates that questions generated from type r rival human-written questions on all four aspects of quality. future work could include annotating a larger partition of the inquisitive dataset while exploring finer-grained analysis of question types (e.g., sub-categories of elaboration questions). we are also interested in employing a framework to generate questions and identify the span jointly.