KHANQ : A Dataset for Generating Deep Questions in Education (2022)

Abstract: Designing in-depth educational questions is a time-consuming and cognitively demanding task. Therefore, it is intriguing to study how to build Question Generation (QG) models to automate the question creation process. However, existing QG datasets are not suitable for educational question generation because the questions are not real questions asked by humans during learning and can be solved by simply searching for information. To bridge this gap, we present KHANQ, a challenging dataset for educational question generation, containing 1,034 high-quality learner-generated questions seeking an in-depth understanding of the taught online courses in Khan Academy. Each data sample is carefully paraphrased and annotated as a triple of 1) Context: an independent paragraph on which the question is based; 2) Prompt: a text prompt for the question (e.g., the learner’s background knowledge); 3) Question: a deep question based on Context and coherent with Prompt. By conducting a human evaluation on the aspects of appropriateness, coverage, coherence, and complexity, we show that state-of-the-art QG models which perform well on shallow question generation datasets have difficulty in generating useful educational questions. This makes KHANQ a challenging testbed for educational question generation.

## introduction question asking has long been considered a fundamental cognitive process in learning. an ideal learner should be an active, self-motivated, creative, inquisitive person who asks deep questions and searches for answers to thought-provoking questions, usually in the form of why, why-not, how, what-if, etc @xcite . for example, figure 1 shows a question raised by a learner after learning "allosteric regulation and feedback the learner 4 years ago allosteric regulation confuses me a lot. i don't really get it even after i read the article on khan academy . can anyone explain it to me briefly? ## related work question generation (qg) aims to automatically generate questions from textual input. earlier work relied on syntactic transformations to convert declarative sentences into questions @xcite . recent neural models @xcite @xcite rely on sequenceto-sequence models to generate questions from a given sentence or paragraph by considering the focus, type, and general specific relations of the question. however, these approaches only involves generating shallow factoid questions, which are typically trained and evaluated on the squad @xcite dataset. squad is insufficient to evaluate deep qg because more than 80% of its questions are only relevant to information confined to a single sentence @xcite . with the advent of pretraining language models, the challenge of generating single-hop questions similar to squad have largely been addressed. qg research has started to generate more complex questions that require multi-hop reasoning @xcite @xcite , benchmarking on the hotpotqa @xcite dataset. however, for questions in hot-potqa, the reasoning is often evident from the surface form of the question, simplifying the qg task. for example, @xcite show that hotpotqa-style multi-hop questions can be generated by composing single-hop questions through templates. different from squad and hotpotqa, questions in our khanq dataset are asked by real course learners, thus having a wider variety in both question forms and reasoning types. recently, a few works started to work on generating real human-raised questions. for example, @xcite collect real questions from online forums (reddit and yahoo). @xcite collect questions posted by customers on amazon product pages. however, questions collected in above works are noisy, with few in-depth questions, since they fail to carefully filter and validate the questions. compared to them, questions in khanq are carefully filtered and annotated, providing a more clean testbed for deep question generation. among these works, learningq @xcite is closest to us, which also collects questions from khan academy. however, our work has three key differences. first, learningq is more focused on the educational nature of the questions. they filter questions based on whether they are useful for learning, whereas we focus more on the depth of the question, keeping only the questions that involve deep reasoning. second, we use prompts to give the models additional guidance on what information to focus on when generating. third, the contexts used by learningq are entire articles or videos, causing most of the sentences in the contexts being irrelevant to the target question. in contrast, the contexts we use are answers that contain comprehensive explanations of the knowledge points relevant to the question. ## experiments we evaluate the performance of state-of-the-art question generation models on khanq, focusing on the following: • exploring whether existing models can generate reasonable educational questions by conducting both automatic evaluation (section 5.3) and human evaluation (section 5.4). ## conclusions and future works in this paper, we propose khanq, a dataset for generating in-depth educational questions. each sample in khanq is carefully annotated as context, prompt, and question to form a clean dataset. we evaluate the performance of state-of-the-art question generation models on khanq. we find that although it is feasible for the model to generate fluent and complex questions, the ability to understand and reason over the context and the prompt is still far from reaching the human level. there are several future directions that are worth investigating: (1) what different results the models will produce in terms of different types of questions; (2) how to enable models to obtain information from the context and the prompt separately and then make the inference, to enhance their ability to seek information under the given conditions.