Vocabulary Matters: A Simple yet Effective Approach to Paragraph-level Question Generation (2020)

Abstract: Question generation (QG) has recently attracted considerable attention. Most of the current neural models take as input only one or two sentences, and perform poorly when multiple sentences or complete paragraphs are given as input. However, in real-world scenarios it is very important to be able to generate high-quality questions from complete paragraphs. In this paper, we present a simple yet effective technique for answer-aware question generation from paragraphs. We augment a basic sequence-to-sequence QG model with dynamic, paragraph-specific dictionary and copy attention that is persistent across the corpus, without requiring features generated by sophisticated NLP pipelines or handcrafted rules. Our evaluation on SQuAD shows that our model significantly outperforms current state-of-the-art systems in question generation from paragraphs in both automatic and human evaluation. We achieve a 6-point improvement over the best system on BLEU-4, from 16.38 to 22.62.

## introduction and related work automatic question generation (qg) from text aims to generate meaningful, relevant, and answerable questions from a given textual input. owing to its applicability in conversational systems such as cortana, siri, chatbots, and automated tutoring systems, qg has attracted considerable interest in both academia and industry. recent neural network-based approaches @xcite @xcite @xcite @xcite represent the state-of-the-art in question generation. most of these techniques learn to generate questions from short text, i.e., one or two sentences @xcite @xcite . on the other hand, the ability to generate high-quality questions from longer text such as from multiple sentences or from a paragraph in its entirety, is more useful in real-world settings. however, given that a paragraph contains a longer context and more information than a sentence, it is a significantly more challenging problem to generate questions around a longer context. in figure 1 we present one motivating example demonstrating why the model needs information more than just a single sentence for generating question a meaningful and relevant question. as we can see in figure 1 , question 2, question generated by our model use multiple sentences as context. @xcite recently observed that 20% of the questions in the squad dataset @xcite require paragraphlevel information to answer them. for the same reason, it is intuitive to conclude that the ability to consider the complete context; however long it may be, is critical for generating high-quality questions. legislative power in warsaw is vested in a unicameral warsaw city council ( rada miasta ) , which comprises 06 members . council members are elected directly every four years . like most legislative bodies , the city council divides itself into committees which have the oversight of various functions of the city government . bills passed by a simple majority are sent to the mayor ( the president of warsaw ) , who may sign them into law . if the mayor vetoes a bill , the council has 30 days to override the veto by a two-thirds majority vote . human generated: how many members are on the warsaw city counil ? our model: how many members are in the warsaw city council ? human generated: how often are elections for the counsel held ? our model: how often are the rada miasta elected ? human generated: what does the city council divide itself into ? our model: the city council divides itself into what ? human generated: how many days does the council have to override the mayor 's veto ? our model: how long does it take to override the veto ? figure 1 : examples of ground-truth questions and questions generated by our model from the same paragraph. each question and its corresponding answer are highlighted using the same color. zhao et al. (2018) very recently proposed a technique (referred to mpgsn here) for paragraph-level question generation using a max out pointer mechanism and a gated self-attention encoder. their best model achieves bleu-4 of 16.38 on squad with paragraphs as input. compared to @xcite , our model has less number of parameters (making it more computationally efficient), is relatively easy to train and is somewhat deterministically biased toward the generation of important words in the input paragraph. in this paper, we propose a simple yet effective paragraph-level question generation technique. we augment the standard sequence-to-sequence model based on bidirectional lstm with two components: (1) a dynamic, paragraph-specific dictionary and (2) a copy attention mechanism that is persistent across paragraphs. our evaluation on squad shows significant improvement over mpgsn in automatic evaluation. we achieve a 6-point increase with respect to bleu-4 (from 16.38 to 22.62) over mpgsn's best system. we perform the human evaluation of our model with and without copy attention, and we observe that we obtain 27% more relevant questions when the copy attention is incorporated. for a given paragraph as input, we depict in figure 1 , the ground-truth questions as well as the questions generated along with the answers highlighted in the paragraph. as can be seen from the example, while generating the second question(highlighted in green color), our model uses information not only from the sentence containing the answer, but also relevant context from the complete paragraph. ## problem formulation & approach given a paragraph 'p ' and answer 'a', a question generation model iteratively samples question word q @xmath0: where v q is the question vocabulary, θ is the set of parameters, and a is the answer. our question generation model consists of a two-layer paragraph encoder and a one-layer question decoder, equipped with a dynamic dictionary and copy attention. in figure 2 , we illustrate the overall architecture of our paragraph level question generation model. the dynamic dictionary allows every training instance (paragraph) to have its own vocabulary instead of relying on the preprocessed global vocabulary. copy attention enables the model to predict question words from the extended vocabulary (complete vocabulary + paragraph vocabulary). copy attention operates over the union of words in vocabulary and paragraph words. ## experimental setup we report the experimental result of our model (referred to as nqg dd ) and compare it with the current state of the art @xcite . we employ the widely-used metrics bleu @xcite , rouge-l and meteor for automatic evaluation. we use evaluation script provided by @xcite . similar to @xcite we also report qualitative assessment on the syntax, semantics and relevance of the questions generated by our model. all experiments are performed on the squad dataset @xcite , where complete paragraphs are taken as input instead of just one or two sentences. we reformat the squad dataset such that during training time, each source instance is a (paragraph, question) pair annotated with the gold answers, and the target is a question. following the exact setup from @xcite , we split the squad train set into train and validation set containing 77,526 and 9,995 instances respectively, and take the separate squad dev set containing 10,556 instances as our test set. ## results and analysis in to explain how our model attends to different words in the source paragraph we visualize attention weights in figure 3 , which shows attention weights between question 2 generated by our model and the corresponding paragraph in figure 1 . we observe that the attention weight is high for words near the answer and the model attends to all relevant context rather that just the sentence containing the answer. we also note that our training is faster atleast by a factor of 2. we expected this since we replace a slightly expensive self-attention mechanism in the decoder of @xcite with a simpler dynamic dictionary and reusable copy attention. ## conclusion paragraph-level question generation (qg) is an important but challenging problem, mainly due to the challenge in effectively handling a longer context. we present a simple yet effective approach for automatic question generation from paragraphs. besides using a standard global source dictionary, our rnn-based model incorporates a dynamic, paragraph-specific dictionary, and learns to switch between copying from the combined vocabulary and generating a new word. through our experiments, we demonstrate how our model outperforms the current state-of-the-art model in paragraph-level qg by a wide margin, for example by 6.24 bleu-4 points, a 38% improvement.