Towards Process-Oriented, Modular, and Versatile Question Generation that Meets Educational Needs (2022)

Abstract: NLP-powered automatic question generation (QG) techniques carry great pedagogical potential of saving educators’ time and benefiting student learning. Yet, QG systems have not been widely adopted in classrooms to date. In this work, we aim to pinpoint key impediments and investigate how to improve the usability of automatic QG techniques for educational purposes by understanding how instructors construct questions and identifying touch points to enhance the underlying NLP models. We perform an in-depth need finding study with 11 instructors across 7 different universities, and summarize their thought processes and needs when creating questions. While instructors show great interests in using NLP systems to support question design, none of them has used such tools in practice. They resort to multiple sources of information, ranging from domain knowledge to students’ misconceptions, all of which missing from today’s QG systems. We argue that building effective human-NLP collaborative QG systems that emphasize instructor control and explainability is imperative for real-world adoption. We call for QG systems to provide process-oriented support, use modular design, and handle diverse sources of input.

## introduction decades of educational research has shown the benefit of active learning in improving students' learning outcomes where students actively engage with the materials, e.g., through question answering and problem solving @xcite @xcite , compared with passively receiving lectures or reading texts @xcite . however, most instructors still use traditional teaching methods, e.g., lecturing, in large-enrolment college courses @xcite @xcite , due to their limited time and resources in creating opportunities for active learning @xcite @xcite . meanwhile, there is a growing interest in leveraging nlp methods to serve educational needs. specifically, automatic question generation (qg) is promising in enhancing students' active learning experience by creating problem-solving activities at scale @xcite . still, the adoption of automatic qg systems in classrooms is low @xcite , mainly because those models are only suitable for specific domains, e.g., language learning, and the generated questions are often of low quality and limited in types and difficulty levels @xcite . for qg systems to provide more meaningful support to instructors' work, we need to address at least two fundamental challenges. first, designing questions to assist learning is a highly complex, creative, and knowledge intensive process. typical college instructors have received years of training on the subject domain, and their question construction process is hard to be fully automated by one single model. second, classroom is a high-stake environment, and instructors often have pre-defined goals. as a result, they may not prefer imperfect and fallible ai models @xcite . to fill the gap between the design of nlppowered qg systems and the reality in classrooms, we conducted a novel need finding study, with two major goals: (i) to understand how instructors create questions of high educational value, including decision making processes and information sources used throughout the procedure. (ii) to identify key touch points to improve, design, and re-create nlp systems that instructors would find useful in their practice. concretely, we interviewed and observed 11 instructors from 7 different universities as they created questions based on readings to support student learning of the content. we piloted the interview study protocol for several rounds, before finalizing a two-phase protocol. in the first phase, we conducted an in-depth case study of one instructor's quiz design experience for a semester-long course. we extracted the instructor's output and manually labeled the context of each action. we then propose a novel method: text replay enactment, where we replayed the instructor's process of generating the output and annotated what nlp tasks could be used to accomplish the instructor's text transformation operations. this method successfully addresses the challenge that users are unable to directly articulate what they want from ai @xcite . next, we summarized a list of nlp tasks that this instructor found useful, and used them as design probes in a subsequent interview study with 10 instructors. we then analyzed our data using affinity diagrams. on a subset of these tasks, we reported a qualitative analysis of current nlp models' performance and pointed out major issues that should be addressed to allow classroom adoption. our major findings include: 1) none of the instructors interviewed are currently using any type of automatic question generation tools, though they expressed a strong desire to find better ways to design quality questions to support active learning and effective reading. 2) instructors' natural behaviors and thought processes reveal multiple reasons why existing automatic qg techniques fall short: they leverage multiple sources of information that cover domain knowledge, educational goals, and student misconceptions; they view question creation as an iterative process; they often apply a suite of techniques and strategies. all of these traits are largely ignored by existing end-to-end qg models. 3) finally, instructors also reported challenges when constructing questions, and showed strong interests in receiving suggestions and support from nlp systems if the outputs are of high quality and interpretable, and that they have sufficient control when interacting with the system. based on these findings, we argue that developing effective human-nlp collaborative qg systems is a promising direction, and propose three design implications. first, it is critical to provide process-oriented support to instructors, rather than automating the process end to end. second, we advocate for modular system design with different nlp components that give instructors strong control over which nlp components to use to meet their goals. third, we argue that future qg systems should accept diverse data input, including expert domain knowledge, students' background, educational goals, and pedagogical context. finally, we stress the value of understanding stakeholders' needs to better design and deploy nlp systems. as anticipated, users have concerns about how well nlp can do, and are hesitant to use ai-based tools in high-stake environments. therefore, an important first step in developing human-nlp collaborative systems is to understand user needs and increase user buy-in. as we learned from the interviews, using simple models with reliable performance in the beginning while requesting more user input may be a good approach to bootstrap. we also encourage our community to collaborate with hci researchers to develop userfriendly interfaces and investigate users' adoption and preferences of nlp tools in context of use. foot_0 2 related work ## need finding study methodology we wanted to understand how instructors construct questions that align with their educational needs. we also wanted to probe on when instructors think an intelligent system might offer support or their work. to address these needs, we chose to conduct a qualitative study consisting of observations and semi-structured interviews. it has become a standard hci approach when designing new software systems that address human needs. in this work, we conducted an irb-approved two-phase need finding study. we went through rounds of piloting before reaching at the final protocol, which addressed two challenges that emerged during the pilots. 1) an interview-alone approach is insufficient since it is hard for instructors to recollect all details of question design. in addition, it requires concrete textual input for us to understand how humans use texts when designing questions. 2) instructors are not able to directly articulate their nlp needs @xcite . to address the first challenge, we propose a specific scenario where instructors design multiple-choice style quiz questions to support active reading. on one hand, it targets an authentic problem since text reading is a passive learning experience that occurs everyday in college classrooms. active reading opportunities are much needed to support students' comprehension and learning. on the other hand, it is a question design scenario where the text input is ex-plicit. to address the second challenge, we propose a text replay enactment method where we, as nlp researchers, review users' operation on the text and annotate possible nlp tasks that can be applied to make the text transformation. ## findings through affinity diagramming, we observe emerging themes from the interview data. we present our findings and use participants' quotes to illustrate these themes. we refer to participants as p1-p10. ## evaluation of existing nlp tools given the observations from §4, here we select popular nlp models for the most frequent text transformation operations used by the instructors and investigate existing issues. we show both instructorconstructed transformations and system outputs in here we use the context consisting of one or multiple sentences annotated on the quiz dataset as described in §3. we first conduct sentence selection and abstractive summarization based on bert-sumext and bertsumextabs models @xcite , which are extractive and abstractive summarization models fine-tuned on news articles. to build an abstractive summarization model that is suitable for scientific domains, we further fine-tune the sequence-to-sequence bart model @xcite on our newly collected hci article summarization dataset that consists of sectionsummary sentence pairs. given the same context, we then build models for text simplification using access @xcite and muss @xcite , which are built on top of bert and bart, respectively; a paraphrasing model that fine-tunes bart using parasci @xcite which contains paraphrase pairs from scientific papers; a negation generation model based on crossaug @xcite which fine-tunes bart to produce text that contradicts the given context. three major findings are made. for sentence se-lection and summarization models, it is often hard to interpret why the output is produced as is. for example, it is unclear why a sentence is treated as more important and thus selected from a given context. second, for the generation-driven models, such as abstractive summarization, simplification, and paraphrasing, their output is sometimes only tangentially relevant to the content. the output frequently contains content that cannot be inferred from the input as well as errors, which can hurt instructor experience and even raise ethical concerns. moreover, the diversity of the generations is rather limited. for example, simplification and negation models tend to focus on changing specific words. overall, all these nlp models are trained without user needs being specified, thus do not offer control over which topic to summarize, what phrase or content to simplify or paraphrase, and which knowledge to be used as a pivot to create distractors. these issues point to the future directions for building summarization models with explainability, guaranteed faithfulness and correctness, as well as that allow users to exert control over where the transformation should be applied. ## implications for research our study reveals instructors' natural processes of constructing questions, the challenges they have, and when they may benefit from nlp support. specifically, we see a strong desire for user control, where humans provide input to nlp systems and can decide when to use nlp outcomes. ## conclusion qg has been an area of interest in the nlp community. however, the adoption of qg systems in classrooms is low. the goal of this work is to investigate this gap and explore directions for future qg systems that can meet stakeholders' needs. our work reveals that existing qg systems do not take information that instructors deem critical when designing questions to support learning, including educational goals, and student misconceptions. we surface instructors' desires for receiving support during question design. we make recommendations on how to develop process-oriented, modular, human-nlp collaborative qg systems.