Question Generation Using Sequence-to-Sequence Model with Semantic Role Labels (2023)

Abstract: Automatic generation of questions from text has gained increasing attention due to its useful applications. We propose a novel question generation method that combines the benefits of rule-based and neural sequence-to-sequence (Seq2Seq) models. The proposed method can automatically generate multiple questions from an input sentence covering different views of the sentence as in rule-based methods, while more complicated “rules” can be learned via the Seq2Seq model. The method utilizes semantic role labeling to convert training examples into their semantic representations, and then trains a Seq2Seq model over the semantic representations. Our extensive experiments on three real-world data sets show that the proposed method significantly improves the state-of-the-art neural question generation approaches.

## introduction question generation (qg) from text has gained increasing interest due to its usefulness in various applications such as educational reading comprehension assessment @xcite , data augmentation for training questionanswering systems @xcite , and response generation in conversational systems @xcite . we have been working on qg for the purpose of automatically creating the knowledge base (kb) for a conversational qa system of an industry partner 1 . the knowledge base consists of qa pairs extracted from a domain-specific document (such as car manuals). previously, the creation of their kb was done manually, which is very labor-intensive. to automate this kb generation process, we have tried both rule-based and neural sequence-to-sequence (seq2seq) qg methods. the rule-based methods create rules based on linguistic features that capture the relationships 1 inago @xcite among components of a sentence and can generate multiple questions from an input sentence to cover different aspects of the sentence. however, designing such rules is a very labour-intensive task as well. also, these rules may not capture the complexity of ways a human asks questions @xcite . as we will show in section 4, the rule-based method does not lead to good results compared to neural seq2seq models. while the seq2seq methods achieve better results, such methods are highly data-driven. for domains with limited training data (such as car manuals), relations that map the input text to questions cannot be well captured. in addition, seq2seq models often generate a single question from an input text. however, multiple questions can be asked about a piece of text from different aspects. one way to generate multiple questions with seq2seq models from an input text is to mark the input text with different answer spans or keywords to show the focus for qg so that multiple questions may be generated from the same text, one for each answer span/keyword. however, in our application, such answer spans or keywords are not available as marking answer spans or keywords when creating training data requires intensive labor work. our partner prefers an answer-unaware qg system that can automatically generate multiple factual questions without indicating answer spans or keywords in either training or inference time. another technique for generating multiple questions is to use diverse beam search @xcite however, the beam search methods require the user to specify the number of questions returned, which is hard to specify as the ideal number of generated questions varies among different input texts. to address these issues, we propose a novel approach to question generation, which uses semantic role labeling (commonly used in rule-based systems) that can label an input sentence in different ways corresponding to multiple semantic views of an input sentence, and then trains a seq2seq model with srl-labeled sequences for question generation. our method does not need keyword/answer span labels in the training data nor specifications of the number of multiple answers to be generated. the use of srl also increases the number of training examples, which may help alleviate the problem of the limited labeled data problem. we evaluate the proposed method on three realworld data sets and compared the proposed method with several state-of-the-arts qg methods including the ones that generate multiple questions. the extensive experiments on these data sets show that proposed framework is significantly better than the state-of-the-art seq2seq models and rule-based methods, especially in terms of coverage and overall scores considering both precision and recall. ## related work the question generation approaches broadly fall into two categories: rule-based approaches and neural seq2seq learning approaches. rule-based approaches mainly relay on hand-crafted templates/rules built upon linguistic features @xcite @xcite . these methods use rigid heuristic rules to transform a source sentence into one or more questions. however, rules have limited power in expressing the complicated mapping function that the human uses for question generation. designing a comprehensive set of rules is a very labour-intensive task. recently, neural seq2seq models have been successfully applied to question generation due to its capability to extract effective features and model complicated functions. early seq2seqbased qg models are based on rnn structures. examples include an lstm-based seq2seq model with the global attention mechanism @xcite and lstm-based model with the maxout pointer and gated self-attention network (mp-gsn) @xcite . more recent seq2seq models are based on transformer which relies entirely on self-attention to compute representations of its input and output without using sequence-aligned rnns. state-of-the-art transformer-based models that have been pre-trained for qg include t5 @xcite , bart @xcite , prophetnet @xcite , unilm @xcite , unilmv2 @xcite , and ernie-gen @xcite , to list some examples. while the seq2seq methods achieve better results than rule-based methods @xcite , they are highly data-driven. in addition, seq2seq models often generate a single question given an input text, which does not cover multiple views of a sentence. to solve this single-question-generation problem, different strategies have been proposed. one strategy is to use diverse beam search @xcite or sampling techniques (such as top-p nucleus sampling used in @xcite ). while these methods showed promising results, the user has to specify the bin/sample size and the number of questions to be generated (whose ideal number may depend on the input sequence). another strategy for generating diverse questions is to mark or extract keywords in the input text and generate questions by conditioning on keywords or keyword positions (e.g., @xcite @xcite @xcite ). however, extracting keywords (either automatically or manually) to build keyword-labeled training data often needs domain knowledge, a pre-defined keyword list, or documents beyond the training data. the qg method we propose solves these problems by learning seq2seq models with semantic role labeled qas. it can generate multiple and diverse questions without specifying the number of questions to be generated or requiring keyword-labeled training data. a recent method that also uses srl and seq2seq models is a 2-step method in @xcite . it tackles role question generation that, given a predicate mention and a passage, generates a set of questions asking about all possible semantic roles of the predicate. it first generates prototype questions for all the roles based on the ontology in prop-bank @xcite . it then trains a bart model to generate all questions (including ones that cannot be answered by the input text) given these prototype questions contextualized over the input text. both the problem definition and the methodology are very different from ours. our method does not need to generate prototype questions and we generate only the information-seeking questions that can be answered by the input text. ## proposed methodology given a set of answer (i.e., sentence) and question pairs, our goal is to train a model to generate from an unseen sentence one or more questions that can be answered by the sentence. foot_0 ## empirical evaluation we investigate (1) whether using srl with seq2seq models improves the qg performance of seq2seq without srl, and (2) how our method compares with the state-of-the-art qg methods. ## performance with different data sizes among the 3 datasets, car manuals is the smallest, and the improvements of our method on this dataset over the baselines are the largest. to further investigate whether our method makes better improvement when the data set is small, we conducted an experiment with one quarter of the squad dataset. figure 3 in appendix f illustrates the relative improvement of our method over the t5 baseline in f-measure, precision and recall. the relative improvement is computed as: scoreour-score t5 score t5 ## conclusions we proposed a novel qg method that learns a seq2seq model to convert an srl representation of an input sentence into an srl representation of a question, which is then converted to a natural language question. similar to rule-based methods, our srl-based seq2seq methods can generate multiple questions from an input sentence, significantly improving the recall and overall performance of seq2seq qg. it is also much better than rule-based methods because better and more complicated "rules" can be learned via the seq2seq model. our evaluation on three real-world datasets shows that the proposed method significantly outperforms both rule-based, original seq2seq methods and several other sota models, especially in recall and overall performance. as future work, we will extend this method to paragraph-based ques-tion generation.