Enhancing Question Generation with Commonsense Knowledge (2021)

Abstract: Question generation (QG) is to generate natural and grammatical questions that can be answeredby a specific answer for a given context. Previous sequence-to-sequence models suffer from aproblem that asking high-quality questions requires commonsense knowledge as backgrounds which in most cases can not be learned directly from training data resulting in unsatisfactory questions deprived of knowledge. In this paper we propose a multi-task learning framework tointroduce commonsense knowledge into question generation process. We first retrieve relevant commonsense knowledge triples from mature databases and select triples with the conversion information from source context to question. Based on these informative knowledge triples wedesign two auxiliary tasks to incorporate commonsense knowledge into the main QG modelwhere one task is Concept Relation Classification and the other is Tail Concept Generation. Ex-perimental results on SQuAD show that our proposed methods are able to noticeably improvethe QG performance on both automatic and human evaluation metrics demonstrating that incor-porating external commonsense knowledge with multi-task learning can help the model generatehuman-like and high-quality questions.

## introduction question generation (qg) has become an essential task for nlp, which aims to generate grammatical and fluent questions for a given context and answer. qg can create question-answer pairs as data augmentation for question answering (qa) @xcite @xcite . moreover, it is also useful in education @xcite and business applications @xcite , such as creating materials for language beginners, helping build chatbots, etc. existing question generation methods can be roughly grouped into two categories. first, rule-based methods utilize handcrafted paradigms to perform declarative-to-interrogative sentence transformations @xcite , but these methods often consume lots of efforts from domain experts and usually cover limited areas. second, neural network-based methods typically model the question generation task in a fully data-driven manner @xcite @xcite , which has made much progress in recent years. one key issue, however, is still up in the air: human beings often ask questions with commonsense knowledge that may exist in their brain but not appear in the given context. take the instance in actually, previous nlp works have investigated the structured commonsense knowledge to help text generation, such as story generation @xcite and response generation @xcite . they model commonsense knowledge from external databases as additional context through computational linguistics ## related work traditionally, qg is tackled by rule-based methods @xcite @xcite ) that rely heavily on extensive hand-crafted rules. different from these, neural network-based methods are completely data-driven and trainable in an end-to-end fashion @xcite @xcite @xcite @xcite @xcite . for better representing the input context, the answer position and token lexical features (e.g. ner, pos and word case) are treated as supplements for the neural encoder @xcite . pointer or copy mechanisms @xcite @xcite are also utilized to overcome the oov problem in question generation process. in order to optimize the parameters of qg model, recent works adopt the multi-task learning framework with different auxiliary tasks. zhou (2019a) use language modeling as a low-level task to provide coherent representations of the input context for the high-level qg task. to improve the accuracy of the start-up word generation, @xcite treat question type prediction as an auxiliary task and use the pre-c c l 2 0 2 1 computational linguistics squad knowledge-equipped pure conceptnet -12432 -wordnet -41049 -whole 75722 46455 (61.3%) 29267 (38.7%) in addition to unstructured knowledge as jia (2020b) used, many works in text generation utilize structured knowledge from mature database like conceptnet @xcite . @xcite employ external commonsense knowledge through dynamic memory mechanism to generate more diverse essays. @xcite apply structured commonsense knowledge through multi-source attention to facilitate story comprehension and generate coherent endings. @xcite incorporate commonsense knowledge graphs through graph attention to create more appropriate and informative responses. instead of combining knowledge triples into encoding process, our model creatively incorporates commonsense knowledge into qg procedure with two new auxiliary tasks. in this way, the commonsense knowledge can be effectively absorbed by question generation. ## knowledge extraction to incorporate structured commonsense knowledge into question generation, the most basic step is to extract proper knowledge triples for each training sample. we will describe the details of the knowledge extraction in this section. in order to obtain more commonsense knowledge, we extract structured knowledge triples from two commonly used databases: conceptnet and wordnet. conceptnet is a semantic network composed of triples (h, r, t) denoting that the head concept h has a relation r with tail concept t. wordnet is a lexical database organized in accordance with psycholinguistic theories, where lexicalized concepts are organized by semantic relations (synonymy, hyponymy, etc.). for each sample in the training set of squad, we use each non-stop word in the context passage as a query to retrieve corresponding triples from both conceptnet and wordnet. in the process of question generation, only those triples that provide essential knowledge for source contexts and target questions are useful rather than all retrieved triples. therefore, we design a rule to filter triples: keep only those triples where the head concept h appears in the context passage and the tail concept t appears in the question (in the case of reversed, we swap the head and tail concepts), since these triples can directly provide conversion information between the input context and the output question. for example, in from conceptnet, there are 12,432 training samples that have extracted knowledge triples, and from wordnet there are 41,049 training samples successfully extract knowledge triples. for each training sample, we merge these triples from conceptnet and wordnet and then remove repeated ones. finally we obtain 46,455 knowledge-equipped samples and each sample has 1.7 corresponding triples on average, as clearly shown in accordingly, we divide the origin squad training set into two parts: commonsense knowledgeequipped samples: (context passage, answer, question, knowledge triples) and pure samples: (context passage, answer, question). we will explain how to use these two parts of data in the following sections. as shown in ## model description in this section, we will describe our proposed question generation model, as is illustrated in figure 1 . based on the extracted knowledge triples which can provide commonsense transition information, we incorporate this knowledge into question generation via multi-task learning by employing two triplebased auxiliary tasks. ## conclusion in this paper, we propose a new multi-task learning framework to introduce commonsense knowledge into qg. we first extract relevant structured knowledge triples from external databases, conceptnet and wordnet. based on these knowledge triples, we design two auxiliary tasks to help the main qg model deeply absorb the commonsense knowledge. both the automatic and human evaluation results verify the effectiveness of our proposed methods. in the future, we will explore new ways to use multiple knowledge triples simultaneously in the multi-task learning framework. besides, we may also apply our framework in other text generation tasks, such as conversation generation and story generation.