Generating Deep Questions with Commonsense Reasoning Ability from the Text by Disentangled Adversarial Inference (2023)

Abstract: This paper proposes a new task of commonsense question generation, which aims to yield deep-level and to-the-point questions from the text. Their answers need to reason over disjoint relevant contexts and external commonsense knowledge, such as encyclopedic facts and causality. The knowledge may not be explicitly mentioned in the text but is used by most humans for problem-shooting. Such complex reasoning with hidden contexts involves deep semantic understanding. Thus, this task has great application value, such as making high-quality quizzes in advanced exams. Due to the lack of modeling complexity, existing methods may produce shallow questions that can be answered by simple word matching. To address these challenges, we propose a new QG model by simultaneously considering asking contents, expressive ways, and answering complexity. We first retrieve text-related commonsense context. Then we disentangle the key factors that control questions in terms of reasoning content and verbalized way. Independence priors and constraints are imposed to facilitate disentanglement. We further develop a discriminator to promote the deep results by considering their answering complexity. Through adversarial inference, we learn the latent factors from data. By sampling the expressive factor from the data distributions, diverse questions can be yielded. Evaluations of two typical data sets show the effectiveness of our approach.

## introduction text-oriented question generation (qg) aims to endow machines with the ability to ask relevant and thought-provoking questions about the given text. this task can support a wide range of real-world applications, such as yielding quizzes from course materials for education @xcite , and generating questions as synthetic data to train a qa system @xcite . according to bloom's taxonomy @xcite , questions can be classified into different levels of cognitive ability. the simple ones involve only the shallow meaning of the text. for example, the question "what is the longest river in the world?" about the given text "the nile is the longest river in the world" can be answered directly by matching. however, matching is far from a real understanding of the semantics @xcite . for example, in the field of education, simple questions are hard to fully evaluate students' learning effects, especially in advanced exams. thus, the deep questions that require semantic understanding and reasoning have attracted extensive attention. as shown in fig. (1) , the question asks about some kind of clothing. the answer needs to be deduced from multiple relevant but disjoint clues in the contexts, i.e., "traditional local costumes," "mountain," "mount fuji," as well as implicit commonsense knowledge, such as mount fuji is a famous mountain in japan, kimono is the traditional local costume of japanese, and japanese kimono is a kind of robe clothing. here, commonsense refers to the self-evident and unwritten knowledge shared by most humans, such as encyclopedism and causality. although it does not appear in the text, it is hard to find the correct answer without it due to the incomplete context. asking this kind of question requires a full understanding of commonsense and the ability to make inferences. that is a key ingredient for general intelligence. some works have studied how to answer such questions, represented by commonsense qa and multi-hop qa @xcite , but less effort explores how to generate them. we thus propose a new qg task to fill this research gap. raising deep questions involves three fundamental processes: what to ask, how to ask, and how to answer. what to ask is to identify the answer and its relevant reasoning contents. learning how to ask focuses on the language qualities, such as grammatical correctness and expressive diversity, since the question could be asked in various ways and each way needs to be fluent. respectively, how to answer reflects the question's complexity, shallow questions only need to match the text while deep reasoning ones require understanding the semantics in contexts with long-range dependencies and hidden commonsense knowledge. for these processes, traditional qg models have considerable defects. the rule-based method relies on handcrafted rules or transformation templates with a limited scale. that would restrict the coverage of results. due to the neglect of indispensable answering feedback, the results are not guaranteed to be inferable and deep. on the other hand, the neural model mainly follows the sequence-to-sequence framework which is data-driven and labor-saving, but this monotonous mapping is hard to learn the one-to-many diversified generation. besides, this method cannot cover the nuances of data by using a single vector to encode complex input features, especially when the training data is insufficient or has a long tail distribution. spurious correlations and unexpected variances would easily mislead the single-factor model and deteriorate its robustness. motivated by the above observations, we propose a practical model for the new commonsense reasoning qg task. concretely, we first leverage a knowledge-enhanced model to represent the text contexts, as well as relevant commonsense concepts and relations. we then learn the key factors related to the necessary ask contents and expressive ways. the first factor refers to the reasoning clues involved in asking deep questions, including entities and relations in the commonsense deductive context. another encompasses other variations not covered by the content factor, like the verbalized styles and patterns. these factors can be sampled from the data manifold and used as conditions to generate results. this sampling-then-generate way alleviates the difficulty of collecting real data at the lower ends of a distribution tail in order to learn diversified generation. all these unknown factors may be mutually interrelated. simply assuming that they are independent would oversimplify the latent manifold, leading to unsatisfied results due to the incorrect preservation of the redundant noises. we thus propose to disentangle such factors explicitly to ensure their independence and prevent information leakage between them. to achieve this goal, we introduce two kinds of latent variables to characterize the factors and impose constraints to learn their disentangled representations. these variables are forced to obey two prior non-overlapping distributions, including an isotropic gaussian for the expressive way and another conditional gaussian mixture for the reasoning content. each component can be viewed as a cluster of neural templates or prototypes, which can be used as a guide to control the detailed nuances of a generation process. to encourage the deep and inferable questions, we impose regularization on the distributions by considering the answering complexity, including whether the answer matches the question and involves multihop reasoning with implicit commonsense knowledge. moreover, we design an adversarial inference mechanism to derive optimal distributions for the disentangled factors. to facilitate deployment, we further employ the prefix-tuning technique @xcite that can support inference with limited labeled data. our model enables one-to-many generation by randomly sampling the expressive factor from the distributions to yield new reasoning questions. experimental results on two popular data sets show the effectiveness of our approach. ## approach as shown in fig. (2) , we propose a new framework for this task. we first encode the text context and relevant commonsense knowledge. we then disentangle the key ask-related factors on reasoning content and expressive way. the answering feedback is also considered. by sampling the learned data manifold, we can derive factors as conditions to yield new questions. next, we define some notations and then show the details of each component. ## evaluations we extensively evaluated the effectiveness of our method with quantitative and qualitative analysis. ## related works question generation (qg) is a hot research topic that can support many valuable applications, including synthesizing training data for the questionanswering (qa) task @xcite , producing exercises on the textbook @xcite , and clarifying users' needs for a dialog agent @xcite . previous studies mainly focus on shallow questions @xcite . they can be tackled by matching the text without demanding a real understanding of semantics @xcite . the researchers gradually pay attention to deep questions @xcite , such as multi-hop qg @xcite . however, these questions only involve the context that appears in the text without the need of understanding the commonsense knowledge. asking questions with this background knowledge is indispensable for machine intelligence, but has been less explored. thus, we propose a new qg task to fill this research gap. most of the earlier methods in the qg task were rule-based (dhole and manning, 2020). the handcrafted rules were labor-intensive with poor scalability @xcite . to reduce labor costs, recent attempts turned to a data-driven neural model with better language flexibility @xcite . they learned direct mappings from input texts to questions by an encoder-decoder framework @xcite . considering the question would be asked in diverse ways @xcite , it was hard to support one-to-many generation based on a fixed encoded vector @xcite . some studies proposed to enhance the generalization ability @xcite by variational autoencoder (vae) @xcite . it can learn an ask-related vector @xcite which can be resampled to produce multiple questions @xcite based on data distribution. however, one single vector was not sufficient to capture the complex and entangled asking features @xcite . in contrast, we consider multiple factors and disentangle them to control the generation finely. deep questions require reasoning the knowledge both inside and outside the text @xcite , including hidden commonsense context @xcite . to capture this context, we can resort to the knowledge graphs (kg) @xcite or pre-training models @xcite , such as bert @xcite , gpt-2 @xcite , and @xcite . the kg knowledge can be collected by matching @xcite , and the pre-training one is often obtained by prompt learning @xcite . in addition, the depth of questions is mainly reflected in how to answer @xcite . there are often two ways to incorporate the answering feedback @xcite . one is reinforcement learning which views the answer as a reward @xcite . since there is no prior guidance, the robustness of this method is weak @xcite . another way is to use generative adversarial learning (gan) to jointly train the qa and qg tasks @xcite . this method only judges the final answer but neglects to grasp the answering process, leading to the results' lack of commonsense reasoning ability @xcite . also, this discrete judge is non-differentiable @xcite , causing unstable training @xcite . in contrast, our discriminator simultaneously consider the matched answer and its reasoning complexity, which can facilitate the training of deep question generator. ## conclusions we have proposed a new commonsense reasoning qg task which aimed to generate valid and inferable questions about the given text. unlike traditional qg tasks, our questions needed to deduce multiple clues in disjoint contexts, where not all clues were provided in the given text, and some required to resort to commonsense knowledge outside the text. since understanding semantics is the prerequisite to asking high-quality questions, our complex qg task requires a higher level of machine intelligence. due to the lack of modeling complexity, traditional methods often yield shallow results. to address the problem, we proposed a practical framework that can flexibly incorporate the asking contents, expressive ways, and answering complexity to yield deep results by disentangling adversarial inference. we first retrieved the commonsense knowledge related to the given text. we then disentangled the key question-controlled factors in terms of reasoning content and verbalized way based on the independency priors and constraints. to promote deep results, we further designed a discriminator to regularize the generator by providing the answering feedback. by adversarial inference, we can derive the factors and use them as conditions to decode questions. by sampling the expressive factor from the data distribution, diverse results can be produced. experimental results on two typical data sets showed the effectiveness of our approach.