P ath QG : Neural Question Generation from Facts (2020)

Abstract: Existing research for question generation encodes the input text as a sequence of tokens without explicitly modeling fact information. These models tend to generate irrelevant and uninformative questions. In this paper, we explore to incorporate facts in the text for question generation in a comprehensive way. We present a novel task of question generation given a query path in the knowledge graph constructed from the input text. We divide the task into two steps, namely, query representation learning and query-based question generation. We formulate query representation learning as a sequence labeling problem for identifying the involved facts to form a query and employ an RNN-based generator for question generation. We first train the two modules jointly in an end-to-end fashion, and further enforce the interaction between these two modules in a variational framework. We construct the experimental datasets on top of SQuAD and results show that our model outperforms other state-of-the-art approaches, and the performance margin is larger when target questions are complex. Human evaluation also proves that our model is able to generate relevant and informative questions.

## introduction question generation (qg) from text aims to automatically construct questions from textual input @xcite . it receives increasing attentions from research communities recently, due to its broad applications in scenarios of dialogue system and educational reading comprehension @xcite . it can also help to augment the question set to enhance the performance of question answering systems. current qg systems mainly follow the sequenceto-sequence structure with an encoder for modeling the textual input and a decoder for text generation @xcite . these neural-based models have shown promising performance, however, they suffer from generating irrelevant and uninformative questions. figure 1a presents two sample questions generated by a nueral qg model. q2 contains irrelevant information "everton fc". although q1 is correct, it is a safe play without mentioning any specific information in the input text. one possible reason causing the problem is that current sequenceto-sequence models learn a latent representation for the input text without explicitly modeling semantic information included. we therefore argue that modeling facts in the input text can help to alleviate the problem of existing neural qg models. some researchers explore to incorporate the answer entity @xcite or a so called question worthy phrase @xcite as the fact to guide the generation of target question and make some progresses. however, a complex question usually involves multiple facts. therefore, a single word piece or phrase is not able to provide enough information for the generation. in this paper, we propose to represent facts in the input text as a knowledge graph (kg) and present a novel task of generating a question given a query path from the kg. more specifically, a kg contains a set of fact triples, and a query path is an ordered sequence of triples in the kg. a fact triple consists of two entities and their relationship. figure 1b shows the kg of the input text in figure 1a and it includes two query paths. we can see that not all facts in a query path are mentioned in a specific target question (see path 2 and gtq2). therefore, the model needs to extract the involved facts to form a query before it generates a question. intuitively, we divide the task of question generation from a query path into two steps, namely, query representation learning and query-based question generation. we formulate the former step as a sequence labeling problem for identifying the involved facts to form a query. for query-based question generation, an rnn-based generator is used to generate the question word by word. we first train the two modules jointly in an end-to-end fashion (pathqg in section 3). in order to further enforce the interaction between theses two modules, we employ a variational framework to train the two modules @xcite that treats query representation learning as an inference process from the query path taking the generated question as the target (pathqg-v in section 4). for model evaluation, we build the experimental environment on top of the benchmark dataset squad @xcite . in specific, we automatically construct the kg for each piece of input text, and pair ground-truth questions with corresponding query paths from the kg. experimental results show that our generation model outperforms other state-of-the-art qg models, especially when the questions are more complicated. human evaluation also proves the effectiveness of our model in terms of both relevance and informativeness. ## task definition we first introduce some notations in our task: an input text with n tokens, where x i is the ith token; -g: a knowledge graph constructed from x, which is a set of fact triples {(e 1 , r, e 2 ), ...}, where e i is an entity and r i is the relation between e i and e i+1 ; -@xmath0, r 1 , e 2 , ..., e m ): a query path in the knowledge graph, which is an ordered sequence of triples, and it's a subset of the g; -@xmath1, ..., y |y| ): the generated question based on the x and s, where y i is a token. the task is described as following: given an input text x and its corresponding knowledge graph g, our model aims to generate a question y i based on a query path s i from g. ## path-based question generation we divide the task of question generation from a given query path into two steps, namely, query representation learning and query-based question generation. a query representation learner and a query-based question generator are designed for the two steps separately. we directly combine these two modules into a unified framework pathqg and the overall architecture is illustrated in figure 2 . ## variational path-based question generation the query representation learning can be treated as an inference process from the query path with the input text as the condition. motivated by the variational models for kg reasoning, we propose a variational inference model pathqg-v to train the query learner and question generator to further enforce the interaction between them. additionally, it introduces a posterior query learner to infer a posterior query distribution assuming the target question provided. compared with the original objective of pathqg as eq. 8, the variational model aims to minimize its negative evidence lower bound (elbo) as eq. 9, ## related work question generation, aiming at generating questions from a range of inputs, such as raw text (heilman and smith, 2010), structured data @xcite and images @xcite , has attracted increasing attention in recent years. most previous studies on textual question generation are rule-based and transform a declarative sentence into an interrogative sentence according to hand-crafted patterns @xcite . with the advance of neural network, @xcite propose to apply a seq2seq structure with attention for automatic question generation. as follow-up, @xcite ; @xcite ; @xcite propose to utilize the answers to decrease the generation uncertainty. meanwhile, @xcite and @xcite explore to use answer-relevant context to guide question generation. besides, some studies @xcite @xcite take question generation as a subtask, and jointly learn it with other tasks, such as question answering and phrase extraction, which also help to alleviate the uncertainty and improve the generation performance. another stream of research for question generation is from kg to question. @xcite ; @xcite explore to generate questions from a single kg triple using text as context information. it is close to our setting, but we are different in two aspects. first, we propose to form a query path consisting of multiple triples for question generation instead of a single triple. second, the context we process is where the extracted triples from. this setting is more natural and different from using retrieved text as context as they did. ## conclusion and future work in this paper, we propose to model facts in the input text as knowledge graph for question generation. we present a novel task of generating a question based on a query path from the constructed kg. we propose to learn query representation for question generation in a joint model and a variational inference model is also proposed. we extend the squad dataset by automatic constructing kg for each input sentence and identifying corresponding query paths for ground truth questions. experimental results proves the effectiveness of our proposed model qualitatively and quantitatively. in the future, there can be two research directions. first, we would like to explore more explainable reasoning method for question generation, such as symbolic-based models. second, novel evaluation metrics for question generation taking consistency and informativeness into consideration would be of interest.