Iterative GNN -based Decoder for Question Generation (2021)

Abstract: Natural question generation (QG) aims to generate questions from a passage, and generated questions are answered from the passage. Most models with state-of-the-art performance model the previously generated text at each decoding step. However, (1) they ignore the rich structure information that is hidden in the previously generated text. (2) they ignore the impact of copied words on the passage. We perceive that information in previously generated words serves as auxiliary information in subsequent generation. To address these problems, we design the Iterative Graph Network-based Decoder (IGND) to model the previous generation using a Graph Neural Network at each decoding step. Moreover, our graph model captures dependency relations in the passage that boost the generation. Experimental results demonstrate that our model outperforms the state-of-the-art models with sentence-level QG tasks on SQuAD and MARCO datasets.

## introduction automatic question generation (qg) is the task of generating question-answer pairs from a declarative sentence, qg has many useful applications: (1) it improves the question answering task @xcite by providing more training data @xcite ; (2) it generates practice exercises and assessments for educational purposes @xcite ; and (3) it helps dialog systems to kick-start and continue a conversation with human users @xcite . in this study, we focus on sentence-level qg tasks. conventional qg methods @xcite @xcite rely on heuristic rules or hand-crafted templates as it suffers a significant lack of question, sticking to a few simple and reliable syntactic transformation patterns. recently, neural-based approaches to qg have achieved remarkable success, by applying large-scale reading comprehension figure 1 : an example (lower-cased) of using the structure information that is hidden in previously generated words information and the impact of copied word, where the answer words are in blue and the copied words are in purple. the model can copy the right word develop with a high certainty (0.97 score). most models with state-of-the-art performance model the previously generated text at each decoding step. however, they ignore (1) the rich structure information hidden in generated words (2) and the impact of copied words on the passage. we perceive that this information offers auxiliary information in the future generation. in figure 1 , the copied word donald davies helps model to copy the develop with high certainty (0.97 score). the copied word donald davies is a subject in the passage and the answer message routing methodology is the object in the passage. after capturing the structure information from the generated words, the model pays more attention to the words related to generated words and copy the predicate in the passage. however, the decoders in most qg model process the generated text as the sequence of words, which ignore the text structure. therefore, it is hard to capture the structure information in previously generated words for most qg models. in addition, the information about which word has been copied changes at each step and is updated iteratively. however, most qg models are unable to achieve that. to address these issues, in this paper, we design an iterative graph network-based decoder (ignd) to model the structure information in the previous generation at each decode step using a graph neural network. we observed that the words copied from a passage played a decisive role in the semantics of the whole question. we modeled the copied word information to capture structure information and use their impact on the passage. we introduce the role tag to the passage graph, where all words have the role tag no-copy, except for answer words, which have the tag answer. the ignd updates the role tag at each decoding step. for example, the role tag changes to copied when the word in this node is copied to the question at this decoding step. then, the information is aggregated by a novel bi-directional gated graph neural network (bi-ggnn). moreover, we propose a relationalgraph encoder, which employs a similar bi-ggnn to capture the dependency relations of a passage that boost the generation. we performed experiments on two reading comprehension datasets, squad and marco, and obtained promising results. our model achieves new state-of-the-art results in sentence-level qg tasks on both datasets, with bleu-4 20.33 on squad and 23.87 on marco. our codes are publicly available foot_0 . our main contributions are as follows: ## model description in this section, we define the question generation task and present the graph-to-sequence (graph2seq) model with our ignd. we design and discuss the details of each component as shown in figure 2 . ## related work early works on qg @xcite focused on the rule-based approaches that rely on heuristic rules or handcrafted templates, with low generalizability and scalability. recent works adopted the attentionbased sequence-to-sequence neural model for qg tasks, taking answer sentence as input and output the question @xcite , which proved to be better than rule-based methods. @xcite proposed the feature-enriched encoder to encode the input sentence by concatenating word embedding with lexical features as the encoder input, and answer position are to inform the model to locate the answer. to generate a question for a given answer, @xcite @xcite applied various techniques to encode answer location information into an annotation vector corresponding to the word positions, thus allowing for better quality answer focused questions. @xcite presented a syntactic features based method to represent words in the document and to decide what words to focus on while generating the question. @xcite combined supervised and reinforcement learning in the training to maximize rewards that measure question quality. furthermore, recent concurrent work applied the large-scale language model pre-training strategy for qg to achieve a new state-of-the-art performance @xcite . most existing qg approaches are unable to explicitly model the previously generated words. however, we perceive that previous generated words serve as auxiliary information in subsequent generation. ## conclusion in this study, we designed the ignd for qg to alleviate the problem, which ignore the structure information and copied words in generated words at each decoding step. in addition, we proposed the relational graph encoder to capture the dependency relations information to improve the performance.