On Generating Fact-Infused Question Variations (2021)

Abstract: To fully model human-like ability to ask questions, automatic question generation (QG) models must be able to produce multiple expressions of the same question with different levels of detail. Unfortunately, existing datasets available for learning QG do not include paraphrases or question variations affecting a model’s ability to learn this capability. We present FIRS, a dataset containing human-generated fact-infused rewrites of questions from the widely-used SQuAD dataset to address this limitation. Questions in FIRS were obtained by combining a given question with facts of entities referenced in the question. We study a double encoder-decoder model, Fact-Infused Question Generator (FIQG), for learning to generate fact-infused questions from a given question. Experimental results show that FIQG effectively incorporates information from facts to add more detail to a given question. To the best of our knowledge, ours is the first study to present fact-infusion as a novel form of question paraphrasing.

## introduction recently, automatic question generation (qg) is being addressed for generating natural language questions for a given input text passage. viewed as the reverse of the well-studied question answering task (qa), qg has been applied in education and tutoring @xcite , dialog systems and chatbots @xcite , as well as for improving qa systems @xcite . various deep learning models are being rapidly developed for qg @xcite @xcite @xcite . however, it is only recently that qg studies have started focusing on an important aspect of the human question generation process known as paraphrasing, or the ability to ask questions in diverse ways all expressing the same intent @xcite . paraphrasing ability has been identified as a necessary aspect of learning human-like language generation @xcite and was previously studied in context of community qa @xcite @xcite . these works address the identification of synonymous and syntactic question variations such as ("what's the weight of an elephant in kg?"; "how heavy is an elephant?"). in addition to synonymous variations, human beings are also adept at generating questions expressing the same intent with varying level of details. for example, consider a qa pair from the squad dataset foot_0 shown in ## related work models for question generation and question answering are being rapidly developed in current nlp research. we refer our reader to a survey by @xcite for an overview on challenges, existing approaches, as well as evaluation metrics for qg. several qg models use lstm-based encoder-decoder setups with attention and copy mechanisms @xcite @xcite . recent works are focused on improving qg performance by incorporating external knowledge, semantic information, and reinforcement learning into this basic architecture @xcite @xcite . other state-of-the-art qg frameworks include variational autoencoders, graph convolutional networks and transformers @xcite @xcite . paraphrase generation is a related task for identifying semantically similar texts in applications such as retrieval and question answering, query reformulation and dialog system applications @xcite . similar to qg, seq2seq models and encoder-decoder architectures are common in paraphrase generation works @xcite but other approaches for paraphrase generation include variational autoencoders and translation models @xcite @xcite . ## firs dataset creation as highlighted in section 1, existing datasets for qa/qg and paraphrase generation do not include question variations with details. to fill this gap, we collected a new dataset by integrating the questions in the widely-used stanford question answering dataset (squad) with relevant facts obtained from google's entity search api as follows: collecting candidate question-entity pairs: we selected from squad, questions that refer to named entities in either the (1) question or (2) answer texts. in we obtained the subset of 25, 316 questions from the 100k questions in squad which reference 'tangible' named-entity types such as people, places, and organizations. entity types referring to concepts such as "quantity, percent etc" are not supported in currently-available knowledge resources. for example, for the ibm question in entity name: ibm type: 'corporation', 'thing', 'organization' description: computer hardware company detailed description: international business machines corporation is an american multinational technology company headquartered in armonk, new york, with operations in over 170 countries. ## generating fact-infused questions definition: given two input sequences of |n| base question words, q @xmath0, q 2 . . . q n , and |m| words of a fact related to an entity, @xmath1, f 2 . . . f m , the objective of fact-infused question generation is to generate an output sequence of |k| words, q @xmath2, p 2 . . . p k , such that q p is a fact-infused rewrite of q b . that is, q p includes specific details from f while maintaining the intent of q b and our goal is to find q p that maximizes the conditional likelihood: we follow standard question generation approaches and adopt an attention-based encoder-decoder architecture for estimating the probability function, p @xcite @xcite . the main components of our fact-infused question generator network (fiqg ) are depicted in the schematic diagram in figure 1 and summarized below: fact and question encoders: we use separate encoders for representing the question and fact sequences. the encoders are one-layer bidirectional lstms that extract contextual features from the input question (or alternatively, fact) and represent them as hidden states of the forward and backward lstms. let e i (q b ) and e i (f ) represent the feature-rich embeddings of our base question and fact, respectively @xcite . then, next, applying the attention mechanism (bahdanau et al., 2015) for the question encoders over its hidden states, the attention weighted sum of the contextualized question can be written as the context vector for the fact can be written as: the decoder takes the hidden states from the question and fact encoders to generate the paraphrased sequence of words. our decoder is a uni-directional lstm network whose state and context vectors are represented by s t and i t , respectively, such that: here e d refers to the embedding from the decoder for the paraphrase word, p t-1 . the current context and decoder state vectors are combined with the attention vector from the question encoder to obtain the readout state and subsequently the generative probability distribution over the vocabulary using a maxout layer @xcite : the matrices w y , w r , u r and v r are all learned during training. copy mechanism: recent works for qg handle rare words by employing a pointer network that enables both copying of the words from the input source (answer passage) and generation of words during the decoding process @xcite . for question rewriting using facts, we extend this copy mechanism to enable copying from both the input fact words as well as the question words. the copy switch in our case is a softmax function given by @xmath3, u @xmath4. during the decoding step, p is sampled to ( 1 ) copy the words from the question, based on y q t , the normalized, attention weights from the question encoder, or, (2) copy words from the fact based on y f t , the normalized, attention weights from the fact encoder, or (3) generate a new word, based on y v t , the generative distribution on the vocabulary estimated during learning. ## experiments and results implementation: we implemented fiqg in python. 15 the hidden state sizes for the two encoders and the decoder are set to 256, whereas the depth for the attention mechanism is set to 512. the readout size is 128 whereas vocabulary size is ∼ 20k words, and the target sequence length was set to 50. dropout rates are set to 0.5 for the dense layers and 0.3 for the attention layers, respectively. a learning rate of 0.001 was used. feature-rich embeddings @xcite were used for input representations using word, parts-of-speech tags and indicator embeddings. indicator features using the bio representation are incorporated in qg models to indicate the answer span in a passage to focus the question around the answer. for our case, this aspect corresponds to the named-entity whose fact we are integrating into the question. however, to differentiate the two cases, namely, when the entity is part of the answer versus when the entity name is part of the question, we use an extended set of tags: {ba, ia, bn, in, o}. fact extraction: the entity descriptions obtained from google are brief summaries comprising 1-3 long sentences. the crowdworkers however only use specific segments of these summaries or entity facts in their paraphrases. to model this aspect, we used minie 16 an unsupervised, domainindependent fact extraction tool on our entity descriptions and mapped a specific fact from the summary with each rewrite @xcite . we provide an example in the appendix (footnote 5). evaluation: following existing qg works, we use bleu @xcite , meteor @xcite , and rouge-l @xcite scores to characterize model performance. all three measures are based on calculating the n-gram overlap between human-generated "gold" references and machine-generated predictions. all baseline models were set up using the configuration settings shared by the authors. as in existing qg studies, we uniformly use pretrained embeddings from glove foot_15 @xcite for word representations and tune all models using the bleu-4 score on the dev portion of the dataset. all experiments were performed on a single gpu on an nvidia cluster and fiqg took approximately 2 hours to train. ## conclusions and future work we presented firs, to the best of our knowledge, a first-of-its-kind dataset containing fact-infused vari-ations of a subset of questions from squad. we proposed a double encoder-decoder model fiqg, for learning to generate question variations through fact infusion. fiqg is able to significantly outperform extensions of standard qg models on firs. in future, we would like to investigate the use of question variations on downstream tasks such as qa, reading comprehension, and interactive dialog @xcite @xcite . additionally, question variations available in firs can be used for learning diverse question generation, adversarial models for qa, and qg on multiple passages @xcite @xcite . we would like to explore these aspects as well as study novel learning methods such as variational auto-encoders and reinforcement learning for improving performance on the fact-infused question generation task @xcite .