Simple or Complex? Complexity-controllable Question Generation with Soft Templates and Deep Mixture of Experts Model (2021)

Abstract: The ability to generate natural-language questions with controlled complexity levels is highly desirable as it further expands the applicability of question generation. In this paper, we propose an end-to-end neural complexity-controllable question generation model, which incorporates a mixture of experts (MoE) as the selector of soft templates to improve the accuracy of complexity control and the quality of generated questions. The soft templates capture question similarity while avoiding the expensive construction of actual templates. Our method introduces a novel, cross-domain complexity estimator to assess the complexity of a question, taking into account the passage, the question, the answer and their interactions. The experimental results on two benchmark QA datasets demonstrate that our QG model is superior to state-of-the-art methods in both automatic and manual evaluation. Moreover, our complexity estimator is significantly more accurate than the baselines in both in-domain and out-domain settings.

## introduction the task of question generation (qg) aims at generating natural-language questions from different data sources, including passages of text, knowledge bases, images and videos. for a variety of applications, it is highly desirable to be able to control the complexity of generated questions. for instance, in the field of education, a well-balanced test needs questions of varying complexity levels in suitable proportions for students of different levels @xcite . that is to say, the teacher can tailor the questions to the competence of the learner. in addition, it has recently been shown @xcite that question answering (qa) models can benefit from training datasets enriched by applying qg models. however, despite the growing interests of answering complex questions @xcite as well as questions with varying complex-ity levels @xcite , most existing work focus on generating simple questions @xcite . although @xcite explored the generation of complex questions, they do not consider controlling the complexity of generated questions. complexity-controllable question generation (ccqg) faces a number of challenges. high diversity. compared to simple questions, complex questions contain significantly more information and exhibit more complex syntactic structures. the complexity of questions is caused by compositional complexity because complex questions can be decomposed to a sequence of simple questions @xcite . generation of both simple and complex questions imposes even higher challenges because simple and complex questions demonstrate different semantic and syntactic patterns. to this end, the resulted distributions are expected to be multimodal, i.e., with different modes for different patterns of questions. existing works @xcite fail to capture the diverse nature of ccqg. they model complexity as discrete labels, such as easy and hard, and introduce a learnable embedding as the representation of the complexity labels in the initial hidden state at the decoding stage. however, the information contained in such an embedding plays a limited role in modelling multiple modes of the underlying distribution. similarly, it is observed that latent variables are ignored such that the posterior is always equal to the prior in variational autoencoders @xcite . limited training data. the training of ccqg models requires questions annotated with complexity levels. however, although there are a large number of qa datasets in various domains, few of them is annotated with complexity levels. therefore, in-domain training of high quality ccqg models becomes infeasible in most domains. in this paper, we propose a novel question generation model, ccqg, capable of controlling ques-tion complexity. we incorporate soft templates and deep mixture of experts (moe) @xcite to address the high diversity problem. inspired by a recent work @xcite , we posit that similar questions have similar templates, and that different modes of the underlying distributions should capture different question templates. instead of manually constructing templates, which is labor-intensive and time-consuming, we employ soft templates, each of which is a sequence of latent embeddings. inspired by @xcite , we apply moe to select templates, whereby we introduce a discrete latent variable to indicate the choice of an expert. taking as input a complexity level, a passage and an answer, our model selects an expert, which chooses a template of that complexity level to guide the question generation process. to address the challenge of limited training data, we design a simple and effective cross-domain complexity estimator based on five domain-independent features to classify questions w.r.t. their complexity levels. the predicted labels are incorporated into the training of ccqg. the main contributions of this work are three-folds: • an end-to-end neural complexity-controllable qg model, which incorporates mixture of experts (moe) and soft templates to model highly diverse questions of different complexity levels. our work is mainly relevant to question complexity estimation and question generation. ## methodology given a passage, an expected answer, and a complexity level, the task of ccqg is to generate questions with the specified complexity. according to @xcite , the complexity of a question depends on two factors: i) individual capability of answering a question, and ii) the common process required to answer a question (e.g. understanding content of a question and background knowledge, steps of reasoning to infer an answer). the former varies between individual learners so that it is infeasible to find a generally applicable criterion. despite that, we can determine the shared factors involved in the answering process and use them to quantify complexity of a question. the resulted score is then used to categorize complexity of a question. more details can be found in sec. 4. formally, given a passage denoted as a word sequence @xmath0, • • • , x n x ) with x i in a vocabulary v, a complexity level @xmath1, complex}, an answer @xmath2, • • • , x m ), our goal is to generate the most probable question ŷ = (y 1 , • • • , y n y ) with y @xmath3, which has a as its answer and the complexity level d. the estimation of complexity level d will be described in sec. 4. given the same passage, there are different ways of asking questions, which can be summarized into different question templates that model their semantic similarities @xcite and complexity similarities. templates provide a reference point as guidance for more nuanced question generation. @xcite also suggested that questions generated from templates tend to be fluent and natural. therefore, we argue that question generation would be more effective if the model chooses the appropriate templates at each decoding step. despite the usefulness of templates for question generation, template construction is labor-intensive and requires substantial domain knowledge. therefore, template-based qg approaches typically suffer from low coverage. to alleviate this problem, we employ soft templates and avoid explicitly designing string-based templates. a soft template is modeled as a sequence of elements, each of which provides a reference point at a decoding step t. this modeling allows sharing of elements across templates at the same complexity level. the selection of soft templates is conducted through a mixture of experts. each expert distinguishes from each other in terms of its preference of templates. for a given input, an expert from them is chosen to determine a probable template. both soft templates and experts are latent. as a template is a sequence of template elements, we introduce a latent variable π d @xmath4, • • • , n π } for the selection of template elements at the complexity level d at time t. the value of π d t indicates the choice of a particular template element. in the same manner, we introduce another latent variable @xmath5, • • • , n z } to represent the choice of an expert for a given input. each expert has its own dense vector representation e z . at each decoding step t, we obtain the probability of estimating y t by marginalizing over all template el-ements. we also marginalize over all latent experts for the same input. ## cross-domain complexity estimator it is desirable to build a cross-domain estimator to predict the complexity levels of questions because few domains have questions annotated with complexity levels for training ccqg models. as measuring complexity should be independent of domain-specific content, we use a simple classification rule without any training, which relies on the following five domain-independent features d f i . number of clauses in a question (d f 1 ): the number of events/facts is a strong indicator of question complexity. we observe that the number of clauses are often proportional to the number of events/facts mentioned in a question. we use nltk foot_0 to seek the question's syntactic tree to count the number of clauses. number of certain dependency relations in a question (d f 2 ): certain dependency relations across words influences the understanding of the content of a question @xcite . the more of them, the more difficult it is to understand. thus, we count the number of advmod, amod, nounmod, npmod, and possessive modifiers after running the spacy dependency parser foot_1 on questions. topic coherence of sentences in a passage (d f 3 ): @xcite observed that a passage is easy to understand if the topic coherence of its sentences is high. in light of this, we measure the topic coherence between sentences by calculating the jensen-shannon divergence j s @xcite between their topic distributions. frequency of question entities in a passage (d f 4 ): we observe that a question asking about an entity frequently appearing in a passage is often easier to answer than the one about an infrequent entity. thus, we recognize entities in questions and passages, compute the average frequency of entities mentioned both in a question and a passage by avg(q) = 1 distance between entities in a question and an answer span in a passage (d f 5 ): the answer to a question is often easy to find, if an entity mentioned in the question is located close to the answer in the same passage. therefore, d f 5 is such a distance by taking the average number of tokens between the entities in a question and an answer span in a passage. ## experiments in this section, we evaluate the effectiveness of the ccqg model and the complexity estimator. ## discussion on moe-based architecture we provide justifications of the moe-based architecture from the perspective of high-level cognition. humans can easily ask questions that are simple and complex questions @xcite , mainly because we can identify patterns through a certain mechanism and then combine these patterns for generalizing to various scenarios. that is, humans possess the capability for compositional generalization, which is critical for learning in real-world situations @xcite . some studies have shown the importance of modularity for this capability @xcite . they suggest that modularity is conducive to the specialization of different modules, which are responsible for different functions. in other words, specialization improves generalization. similarly, a modular neural network enables compositional generalization like human intelligence. moe-based architecture can be regarded as an implementation of this concept. moe is a tightly coupled modular structure designed so that similar inputs are mapped to similar expert modules, effectively making each module specialize in a different selection. ## conclusion and future work we propose a novel encoder-decoder model incorporating soft templates and moe to address the problem of complexity-controllable question generation. as most domains do not have training data for ccqg models, we propose a simple and effective cross-domain estimator to predict the missing complexity levels of questions. in the extensive experiments of both ccqg and complexity assessment tasks, our models achieve superior performance over the competitive baselines across all experimental settings. in the future, we will consider anaphora resolution and numerical reasoning in complexity estimator, and explore the performance of our model in different applications, such as examination and assisting qa systems.