Guiding the Growth: Difficulty-Controllable Question Generation through Step-by-Step Rewriting (2021)

Abstract: This paper explores the task of Difficulty-Controllable Question Generation (DCQG), which aims at generating questions with required difficulty levels. Previous research on this task mainly defines the difficulty of a question as whether it can be correctly answered by a Question Answering (QA) system, lacking interpretability and controllability. In our work, we redefine question difficulty as the number of inference steps required to answer it and argue that Question Generation (QG) systems should have stronger control over the logic of generated questions. To this end, we propose a novel framework that progressively increases question difficulty through step-by-step rewriting under the guidance of an extracted reasoning chain. A dataset is automatically constructed to facilitate the research, on which extensive experiments are conducted to test the performance of our method.

## introduction the task of difficulty-controllable question generation (dcqg) aims at generating questions with required difficulty levels and has recently attracted researchers' attention due to its wide application, such as facilitating certain curriculum-learningbased methods for qa systems @xcite and designing exams of various difficulty levels for educational purpose @xcite . compared to previous qg works which control the interrogative word @xcite or the context of a question @xcite , few works have been conducted on difficulty control, as it is hard to formally define the difficulty of a question. to the best of our knowledge, @xcite is the only previous work of dcqg for free text, and defines question difficulty as whether a qa model can correctly answer it. this definition gives only two difficulty levels and is mainly empirically driven, lacking interpretability for what difficulty is and how difficulty varies. in this work, we redefine the difficulty level of a question as the number of inference steps required to answer it, which reflects the requirements on reasoning and cognitive abilities @xcite . existing qa systems perform substantially worse in answering multi-hop questions than single-hop ones @xcite , also supporting the soundness of using reasoning hops to define difficulty. to achieve dcqg with the above definition, a qg model should have strong control over the logic and reasoning complexity of generated questions. graph-based methods are well suited for such logic modelling @xcite . in previous qg researches, @xcite and @xcite implemented graph-to-sequence frameworks to distill the inner structure of the context, but they mainly used graphs to enhance document representations, rather than to control the reasoning complexity of questions. in this paper, we propose a highly-controllable qg framework that progressively increases difficulties of the generated questions through step-bystep rewriting. specifically, we first transform a given raw text into a context graph, from which we sample the answer and the reasoning chain for the generated question. then, we design a question generator and a question rewriter to generate an initial simple question and step-by-step rewrite it into more complex ones. as shown in fig. 1 , "tom cruise" is the selected answer, and q 1 is the initial question, which is then adapted into q 2 by adding one more inference step (i.e. n 1 ←n 2 ) in the reasoning chain. that is, it requires to infer "top gun" is "the film directed by tony scott" before answering q 1 . similarly, we can further increase its difficulty level and step-by-step extend it into more difficult questions (i.e., q 3 , q 4 and q 5 ). to train our dcqg framework, we design effective strategies to automatically construct the training data from existing qa datasets instead of building one from scratch with intensive human efforts. specifically, we utilize hotpotqa @xcite , a qa dataset where most questions require two inference steps to answer and can be decomposed into two 1-hop questions. thus, we get the dataset that contains 2-hop questions and their corresponding 1-hop reasoning steps. having learned how to rewrite 1-hop questions into 2-hop ones with this dataset, our framework can easily extend to the generation of (n+1)-hop questions from n-hop ones only with a small amount of corresponding data, because the rewriting operation follows rather certain patterns regardless of the exact value of n, as shown in fig. 1 . extensive evaluations show that our method can controllably generate questions with required difficulty, and keep competitive question quality at the same time, compared with a set of strong baselines. in summary, our contributions are as follows: • to the best of our knowledge, this is the first work of difficulty-controllable question generation, with question difficulty defined as the inference steps to answer it; • we propose a novel framework that achieves dcqg through step-by-step rewriting under the guidance of an extracted reasoning chain; ## related work deep question generation most of the previous qg researches @xcite @xcite ) mainly focused on generating single-hop questions like the ones in squad @xcite . in the hope that ai systems could provoke more in-depth interaction with humans, deep question generation aims at generating questions that require deep reasoning. many recent works attempted to conquer this task with graph-based neural architectures. talmor and berant (2018) and kumar et al. (2019) generated complex questions based on knowledge graphs, but their methods could not be directly applied to qg for free text, which lacks clear logical structures. in sequential question generation, chai and wan (2020) used a dual-graph interaction to better capture context dependency. however, they considered all the tokens as nodes, which led to a very complex graph. yu et al. (2020) tried to generate multi-hop questions from free text with the help of entity graphs constructed by external tools. our work shares a similar setting with yu et al. (2020), and we further explore the problem of how to generate deep questions in a more controllable paradigm. difficulty-controllable question generation dcqg is a relatively new task. gao et al. (2019) classified questions as easy or hard according to whether they could be correctly answered by a bert-based qa model, and controlled the question difficulty by modifying the hidden states before decoding. another research on qg for knowledge graphs (kumar et al., 2019) estimated the question difficulty based on popularity of the named entity. they manipulated the generation process by incorporating the difficulty level into the input embedding of the transformer-based decoder. in our work, we control the question difficulty based on the number of its reasoning hops, which is more explainable. question rewriting it is another emerging trend in the recent researches, demonstrating benefits to both qg and qa tasks. with rewriting, qg models produced more complex questions by incorporating more context information into simple questions ( @xcite , and qa pipelines could also decompose the original complex question into multiple shorter questions to improve model performance @xcite . ## method given input context text c and a specific difficulty level d, our objective is to generate a (question, answer) pair (q, a), where a is a sub-span of c and q requires d-hop reasoning to answer. fig. 2 and algorithm 1 give an overview of our proposed framework. first, we construct a context graph g cg corresponding to the given context, from which a subgraph g t is selected to serve as the reasoning chain of the generated question. next, with the reasoning chain and other contextual information as input, a question generator (qg initial ) produces an initial simple question q 1 . then, q 1 is fed to a question rewriting module (qg rewrite ), which iteratively rewrites it into a more complex question q i (@xmath0, 3, . . . , d). in what follows, we will introduce the whole generation process in more details. ## automatic dataset construction manually constructing a new dataset for our task is difficult and costly. instead, we propose to automatically build a dataset from existing qa datasets without extra human annotation. in our work, the training data is constructed from hotpotqa @xcite , in which every context c consists of two paragraphs {p 1 , p 2 }, and most of the questions require two hops of reasoning, each concerning one paragraph. hotpotqa also annotates supporting facts f, which are the part of the context most relevant to the question. in addition to the information already available in hotpotqa, we also need the following information to train qg initial and qg rewrite : i) (q 1 , a 1 ), the simple initial question and its answer, which are used to train qg initial ; ii) r 2 , the type of rewriting from q 1 to q 2 ; iii) {n 0 , n 1 , n 2 }, the reasoning chain of q 2 ; and iv) s i (@xmath1, 2), the context sentences where we extract n 0 , n 1 and n 2 . algorithm 2 describes our procedure to obtain the above information. the construction process is facilitated with the help of a reasoning type classifier (typeclassify) and a question decomposer (decompq), referring to @xcite . for each question in hotpotqa (i.e. q 2 ), we first distinguish its reasoning type, and filter out those that are not bridge and intersection. the reasoning type here corresponds to the rewriting type r i . then, decompq decomposes q 2 into two subquestions, subq 1 and subq 2 , based on span prediction and linguistic rules. for example, the q 2 in fig. 2 will be decomposed into subq 1 ="to which film a perfect murder was a modern remake?", and subq 2 ="who directed dial m for murder?". after that, an off-the-shelf single-hop qa model @xcite is utilized to acquire the answer of the two sub-questions, which should be "dial m for murder" and "alfred hitchcock" in the example. as for q 1 , it is one of the sub-questions. when q 2 is of the intersection type, q 1 can be either subq 1 or subq 2 . for the bridge type, it is the subquestion that shares the same answer as a 2 . for the example above, q 1 is subq 2 because suba 2 = a 2 . the context sentence s i is supposed to provide supporting facts contained in the paragraph f that concerns q i (@xmath2, 2). for the reasoning chain, it is selected from the local context graph by first locating n 2 and then finding n 0 , n 1 through text matching with the two sub-questions. ## experiments in the following experiments, we mainly evaluate the generation results of our proposed method when required to produce 1-hop and 2-hop questions, denoted as ours 1-hop and ours 2-hop . in sec. 5.2, we compare our method with a set of strong baselines using both automatic and human evaluations on question quality. in sec. 5.3, we provide controllability analysis by manually evaluating their difficulty levels and testing the performance of qa systems in answering questions generated by different methods. in sec. 5.4, we test the effect of our generated qa pairs on the performance of a multi-hop qa model in a data augmentation setting. in sec. 5.5, we further analyze the extensibility of our method, i.e., its potential in generating questions that require reasoning of more than two hops. our code and constructed dataset have been made publicly available to facilitate future research. foot_0 ## conclusion we explored the task of difficulty-controllable question generation, with question difficulty redefined as the inference steps required to answer it. a step-by-step generation framework was proposed to accomplish this objective, with an input sampler to extract the reasoning chain, a question generator to produce a simple question, and a question rewriter to further adapt it into a more complex one. a dataset was automatically constructed based on hotpotqa to facilitate the research. extensive evaluations demonstrated that our method can effectively control difficulty of the generated questions, and keep high question quality at the same time. for transfer learning in question generation. in proceedings of the 12th international conference of knowledge science, engineering and management, volume 11776 of lecture notes in computer science, pages 339-352. springer.