Semantic Graphs for Generating Deep Questions (2020)

Abstract: This paper proposes the problem of Deep Question Generation (DQG), which aims to generate complex questions that require reasoning over multiple pieces of information about the input passage. In order to capture the global structure of the document and facilitate reasoning, we propose a novel framework that first constructs a semantic-level graph for the input document and then encodes the semantic graph by introducing an attention-based GGNN (Att-GGNN). Afterward, we fuse the document-level and graph-level representations to perform joint training of content selection and question decoding. On the HotpotQA deep-question centric dataset, our model greatly improves performance over questions requiring reasoning over multiple facts, leading to state-of-the-art performance. The code is publicly available at https://github.com/WING-NUS/SG-Deep-Question-Generation .

## introduction question generation (qg) systems play a vital role in question answering (qa), dialogue system, and automated tutoring applications -by enriching the training qa corpora, helping chatbots start conversations with intriguing questions, and automatically generating assessment questions, respectively. existing qg research has typically focused on generating factoid questions relevant to one fact obtainable from a single sentence @xcite @xcite , as exemplified in figure 1 a ). however, less explored has been the comprehension and reasoning aspects of questioning, resulting in questions that are shallow and not reflective of the true creative human process. people have the ability to ask deep questions about events, evaluation, opinions, synthesis, or reasons, usually in the form of why, why-not, how, what-if, which requires an in-depth understanding of the input source and the ability to reason over disjoint relevant contexts; e.g., asking why did gollum betray his master frodo baggins? after reading the fantasy novel the lord of the rings. learning to ask such deep questions has intrinsic research value concerning how human intelligence embodies the skills of curiosity and integration, and will have broad application in future intelligent systems. despite a clear push towards answering deep questions (exemplified by multi-hop reading comprehension @xcite and commonsense qa @xcite ), generating deep questions remains un-investigated. there is thus a clear need to push qg research towards generating deep questions that demand higher cognitive skills. in this paper, we propose the problem of deep question generation (dqg), which aims to generate questions that require reasoning over multiple pieces of information in the passage. figure 1 b ) shows an example of deep question which requires a comparative reasoning over two disjoint pieces of evidences. dqg introduces three additional challenges that are not captured by traditional qg systems. first, unlike generating questions from a single sentence, dqg requires document-level understanding, which may introduce long-range dependencies when the passage is long. second, we must be able to select relevant contexts to ask meaningful questions; this is non-trivial as it involves understanding the relation between disjoint pieces of information in the passage. third, we need to ensure correct reasoning over multiple pieces of information so that the generated question is answerable by information in the passage. to facilitate the selection and reasoning over disjoint relevant contexts, we distill important information from the passage and organize them as a semantic graph, in which the nodes are extracted based on semantic role labeling or dependency parsing, and connected by different intra-and intersemantic relations (figure 2 ). semantic relations provide important clues about what contents are question-worthy and what reasoning should be performed; e.g., in figure 1 , both the entities pago pago international airport and hoonah airport have the located at relation with a city in united states. it is then natural to ask a comparative question: e.g., are pago pago international airport and hoonah airport both on american territory?. to efficiently leverage the semantic graph for dqg, we introduce three novel mechanisms: (1) proposing a novel graph encoder, which incorporates an attention mechanism into the gated graph neural network (ggnn) @xcite , to dynamically model the interactions between different semantic relations; (2) enhancing the word-level passage embeddings and the node-level semantic graph representations to obtain an unified semantic-aware passage representations for question decoding; and we evaluate our model on hotpotqa @xcite , a challenging dataset in which the questions are generated by reasoning over text from separate wikipedia pages. experimental results show that our model -incorporating both the use of the semantic graph and the content selection task -improves performance by a large margin, in terms of both automated metrics (section 4.3) and human evaluation (section 4.5). error analysis (section 4.6) validates that our use of the semantic graph greatly reduces the amount of semantic errors in generated questions. in summary, our contributions are: (1) the very first work, to the best of our knowledge, to investigate deep question generation, (2) a novel framework which combines a semantic graph with the input passage to generate deep questions, and (3) a novel graph encoder that incorporates attention into a ggnn approach. ## related work question generation aims to automatically generate questions from textual inputs. rule-based techniques for qg usually rely on manually-designed rules or templates to transform a piece of given text to questions @xcite . these methods are confined to a variety of transformation rules or templates, making the approach difficult to generalize. neuralbased approaches take advantage of the sequenceto-sequence (seq2seq) framework with attention @xcite . these models are trained in an end-to-end manner, requiring far less labor and enabling better language flexibility, compared against rule-based methods. a comprehensive survey of qg can be found in @xcite . many improvements have been proposed since the first seq2seq model of @xcite : applying various techniques to encode the answer information, thus allowing for better quality answerfocused questions @xcite @xcite ; improving the training via combining supervised and reinforcement learning to maximize question-specific rewards @xcite ; and incorporating various linguistic features into the qg process @xcite . however, these approaches only consider sentence-level qg. in contrast, our work focus on the challenge of generating deep questions with multi-hop reasoning over document-level contexts. recently, work has started to leverage paragraphlevel contexts to produce better questions. @xcite incorporated coreference knowledge to better encode entity connections across documents. @xcite applied a gated selfattention mechanism to encode contextual information. however, in practice, semantic structure is difficult to distil solely via self-attention over the entire document. moreover, despite considering longer contexts, these works are trained and evaluated on squad @xcite , which we argue as insufficient to evaluate deep qg because more than 80% of its questions are shallow and only relevant to information confined to a single sentence @xcite . is an american late -night live television sketch comedy created by lome michaels developed by dick ebersol nsubj cop pobj pobj pobj cop cop nsubj partmod pobj pobj conj pobj dep similar similar word-to-node attention 𝑉𝑉 𝑉𝑉 𝑀𝑀 𝐴𝐴 prediction layer …… cross attention … document question answer semantic-enriched document representations feature aggregator question decoder document encoder (att-ggnn) semantic graph encoder … node embeddings + pos features + answer tags answer encoder … [sos] … structure-aware node representations context vector content selection … … vocabulary source copy switch previous word softmax … … … … x k figure 2: the framework of our proposed model (on the right) together with an input example (on the left). the model consists of four parts: (1) a document encoder to encode the input document, (2) a semantic graph encoder to embed the document-level semantic graph via att-ggnn, (3) a content selector to select relevant question-worthy contents from the semantic graph, and (4) a question decoder to generate question from the semantic-enriched document representation. the left figure shows an input example and its semantic graph. dark-colored nodes in the semantic graph are question-worthy nodes that are labeled to train the content selection task. ## methodology given the document d and the answer a, the objective is to generate a question q that satisfies: ## conclusion and future works we propose the problem of dqg to generate questions that requires reasoning over multiple disjoint pieces of information. to this end, we propose a novel framework which incorporates semantic graphs to enhance the input document representations and generate questions by jointly training with the task of content selection. experiments on the hotpotqa dataset demonstrate that introducing semantic graph significantly reduces the semantic errors, and content selection benefits the selection and reasoning over disjoint relevant contents, leading to questions with better quality. there are at least two potential future directions. first, graph structure that can accurately represent the semantic meaning of the document is crucial for our model. although dp-based and srl-based semantic parsing are widely used, more advanced semantic representations could also be explored, such as discourse structure representation @xcite and knowledge graph-enhanced text representations @xcite . second, our method can be improved by explicitly modeling the reasoning chains in generation of deep questions, inspired by related methods @xcite in multi-hop question answering.