Summary-Oriented Question Generation for Informational Queries (2021)

Abstract: Users frequently ask simple factoid questions for question answering (QA) systems, attenuating the impact of myriad recent works that support more complex questions. Prompting users with automatically generated suggested questions (SQs) can improve user understanding of QA system capabilities and thus facilitate more effective use. We aim to produce self-explanatory questions that focus on main document topics and are answerable with variable length passages as appropriate. We satisfy these requirements by using a BERT-based Pointer-Generator Network trained on the Natural Questions (NQ) dataset. Our model shows SOTA performance of SQ generation on the NQ dataset (20.1 BLEU-4). We further apply our model on out-of-domain news articles, evaluating with a QA system due to the lack of gold questions and demonstrate that our model produces better SQs for news articles – with further confirmation via a human evaluation.

## introduction question answering (qa) systems have experienced dramatic recent empirical improvements due to several factors including novel neural architectures (chen and yih, 2020), access to pre-trained contextualized embeddings @xcite , and the development of large qa training corpora @xcite @xcite . however, despite technological advancements that support more sophisticated questions @xcite @xcite , many consumers of qa technology in practice tend to ask simple factoid questions when engaging with these systems. potential explanations for this phenomenon include low expectations set by previous qa systems, limited coverage for more complex questions not changing these expectations, and users simply not possessing sufficient knowledge of the subject of interest to ask more challenging questions. irrespective of the reason, one potential solution to this dilemma is to provide users with automatically generated suggested questions (sqs) to help users better understand qa system capabilities. generating sqs is a specific form of question generation (qg), a long-studied task with many applied use cases -the most frequent purpose being data augmentation for mitigating the high sample complexity of neural qa models @xcite . however, the objective of such existing qg systems is to produce large quantities of question/answer pairs for training, which is contrary to that of sqs. the latter seeks to guide users in their research of a particular subject by producing engaging and understandable questions. to this end, we aim to generate questions that are self-explanatory and introductory. self-explanatory questions require neither significant background knowledge nor access to documents used for qg to understand the sq context. for example, existing qg systems may use the text "on december 13, 2013, beyoncé unexpectedly released her eponymous fifth studio album on the itunes store without any prior announcement or promotion." to produce the question "where was the album released?" this kind of question is not uncommon in crowd-sourced datasets (e.g., squad @xcite ) but do not satisfy the self-explanatory requirement. clark and gardner (2018) estimate that 33 % of squad questions are context-dependent. this context-dependency is not surprising, given that annotators observe the underlying documents when generating questions. introductory questions are best answered by a larger passage than short spans such that users can learn more about the subject, possibly inspiring follow-up questions (e.g., "can convalescent plasma help covid patients?"). however, existing qg methods mostly generate questions while reading the text corpus and tend to produce narrowly focused questions with close syntactic relations to associated answer spans. triviaqa @xcite and hotpotqa @xcite also provide fine-grained questions, even though reasoning from a larger document context via multi-hop inference. this narrower focus often produces factoid questions peripheral to the main topic of the underlying document and is less useful to a human user seeking information about a target concept. conversely, the natural question (nq) dataset @xcite (and similar ones such as ms marco @xcite , gooaq @xcite ) is significantly closer to simulating the desired informationseeking behavior. questions are generated independently of the corpus by processing search query logs, and the resulting answers can be entities, spans in texts (aka short answers), or entire paragraphs (aka long answers). thus, the nq dataset is more suitable as qg training data for generating sqs as long-answer questions that tend to satisfy our self-explanatory and introductory requirements. to this end, we propose a novel bert-based pointer-generator network (bertpgn) trained with the nq dataset to generate introductory and self-explanatory questions as sqs. using nq, we start by creating a qg dataset that contains questions with both short and long answers. we train our bertpgn model with these two types of context-question pairs together. during inference, the model can generate either short-or long-answer questions as determined by the context. with automatic evaluation metrics such as bleu @xcite , we show that for long-answer question generation, our model can produce state-of-the-art performance with 20.1 bleu-4, 6.2 higher than @xcite , the current state-of-the-art on this dataset. the short answer question generation performance can reach 28.1 bleu-4. we further validate the generalization ability of our bertpgn model by creating an out-of-domain test set with the cnn/daily mail @xcite . without human-generated reference questions, automatic evaluation metrics such as bleu are not usable. we propose to evaluate these questions with a pretrained qa system that produces two novel metrics. the first is answerability, mea-suring the possibility to find answers from given contexts. the second is granularity, indicating whether the answer would be passages or short spans. finally, we conduct a human evaluation with generated questions of the test set and demonstrate that our bertpgn model can produce introductory and self-explanatory questions for informationseeking scenarios, even for a new domain that differs from the training data. the novel contributions of our paper include: ## related work qg has been studied in multiple application contexts (e.g., generating questions for reading comprehension tests @xcite , generating questions about an image @xcite , recommending questions with respect to a news article @xcite ), evaluating summaries @xcite , and using multiple methods (see @xcite for a recent survey). early neural models focused on sequence-to-sequence generation based solutions @xcite . the primary directions for improving these early works generally fall into the categories of providing mechanisms to inject answer-aware information into the neural encoder-decoder architectures @xcite @xcite @xcite , encoding larger portions of the answer document as context @xcite , and incorporating richer knowledge sources @xcite . these qg methods and the work described in this paper focus on using single-hop qa datasets such as squad @xcite , newsqa @xcite , and ms marco @xcite . however, there has also been recent interest in multi-hop qg settings @xcite @xcite ) by using multi-hop qa datasets including hotpotqa @xcite , trivi-aqa @xcite , and freebaseqa @xcite . finally, there has been some recent interesting work regarding unsupervised qg, where the goal is to generate qa training data without an existing qg corpus to train better qa models @xcite . most directly related to our work from a motivation perspective is recent research regarding providing sqs in the context of supporting a news chatbot @xcite . however, the focus of this work is not qg, where they essentially use a gpt-2 language model @xcite trained on squad data for qg and do not evaluate this component independently. @xcite generates questions for information-seeking but not focuses on introductory questions. most directly related to our work from a conceptual perspective is regarding producing questions for long answer targets @xcite , which we contrast directly in section 3. as qg is a generation task, automated evaluation frequently uses metrics such as bleu @xcite , meteor @xcite , and rouge @xcite . as these do not explicitly evaluate the requirements of our information-seeking use case, we also evaluate using the output of a trained qa system and conduct human annotator evaluations. ## problem definition given a context x and an answer a, we want to generate a question q that satisfies the long answer qg task targets generating questions that are best answered by the entire context (i.e., paragraph or document) or a summary of the context, which is notably different from most qg settings where the answer is a short text span and the context is frequently a single sentence. mishra et al. (2020) also work on the long answer qg setting using the nq dataset, but their task definition is arg max q p (q|x) where they refer to the context x as the long answer. we use their models as baselines. ## methods we use the bert-based pointer-generator network (bertpgn) to generate questions. tuan et al. (2020) use two-layer cross attentions between contexts and answers to encode contexts such as paragraphs when generating questions and show improved results. however, they show that threelayer cross attentions produce worse results. we will show later in the experiment that this is due to a lack of better initialization and that a higher layer is better for long answer question generation. @xcite use answer tagging from the context instead of combining context and answer. our model is motivated by these two works (figure 1 ). ## in-domain evaluation with generation metrics 6.1 experiment setup and training we use a bert-base uncased model (devlin et al., 2019) that contains 12 hidden layers. the vocabulary contains 30,522 tokens. we create the pgn decoder with another bert model from the same setting, followed by a 2-layer transformer with 12 heads and 3072 intermediate sizes. the maximum allowed context length is 500, while the maximum question length is 50. we train our model on an amazon ec2 p3 machine with one tesla v100 gpu, with the batch size 10, and the learning rate 5 × 10 -5 with the adam optimizer (kingma and ba, 2015) on all parameters of the bertpgn model (both bert models are trainable). we train 20 epochs of our model and evaluate with the dev set to select the model according to perplexity. each epoch takes around 20 minutes to finish. throughout the paper, we use the implementation of bleu, meteor, and rouge l by @xcite . ## out-of-domain evaluation with qa systems we use a well-trained question answering system as the evaluation method, given that the automated scoring metrics have two notable drawbacks when evaluating long-answer questions: (1) there are usually multiple valid questions for long-answer question generation as contexts are much longer than previous work. however, most datasets only have one gold question for each context; (2) they cannot measure generated questions when there is no gold question, which is the right problem that we encountered for our news-la dataset. ## conclusion we tackle the problem of question generation targeted for human information seeking using automatic question answering technology. we focus on generating questions for news articles that can be answered by longer passages rather than short text spans as suggested questions. we build a bert-based pointer-generator network as the qg model, trained with the natural questions dataset. our method shows state-of-the-art performance in terms of bleu, meteor, and rouge l scores on our nq question generation dataset. we then apply our model to the out-of-domain news articles without further training. we use a qa system to evaluate our qg models as there are no gold questions for comparison. we also conduct a human evaluation to confirm the qa evaluation results.