Quiz Design Task: Helping Teachers Create Quizzes with Automated Question Generation (2022)

Abstract: Question generation (QGen) models are often evaluated with standardized NLG metrics that are based on n-gram overlap. In this paper, we measure whether these metric improvements translate to gains in a practical setting, focusing on the use case of helping teachers automate the generation of reading comprehension quizzes. In our study, teachers building a quiz receive question suggestions, which they can either accept or refuse with a reason. Even though we find that recent progress in QGen leads to a significant increase in question acceptance rates, there is still large room for improvement, with the best model having only 68.4% of its questions accepted by the ten teachers who participated in our study. We then leverage the annotations we collected to analyze standard NLG metrics and find that model performance has reached projected upper-bounds, suggesting new automatic metrics are needed to guide QGen research forward.

## introduction question generation is a text generation task with practical applications in several settings such as asking clarification questions in dialogue systems @xcite , recommending questions during a reading session @xcite , or other educational scenarios such as creating quizzes to emphasize core concepts and engage learners through interaction @xcite . the most common automatic evaluation of qgen borrows from other nlg tasks, using metrics such as bleu @xcite to compare system-generated questions with heldout human-written references in terms of n-gram overlap @xcite . although they are straightforward to compute, these metrics have been shown to correlate weakly with human opinion in nlg @xcite , do not provide a ceiling performance, or insights into the which questions would you include in a quiz about the statue of liberty? reading material: the copper statue, [...], was designed by french sculptor frédéric auguste bartholdi and its metal framework was built by gustave eiffel. teacher selects quiz concept: gustave eiffel teacher picks questions added to quiz and selects error category otherwise: who design the copper satus? ## analysis with the annotations collected, we calculate interannotator agreement and use the data to benchmark commonly-used nlg metrics. ## limitations we now discuss the limitations of the work we've presented. first, even though we attempted to create a realistic scenario in which to evaluate qgen models, some components of the protocol are simplified for practical purposes. for example, the created quiz were not assigned to students, and we rely solely on the teacher's opinion of the questions as a signal of question quality. pushing the study further by assigning the quizzes to students and tying question quality to student performance on the quiz would add complexity, but render the protocol more realistic and provide practical learning signals from students. second, although we treat teacher annotations as the ground truth, there is some level of disagreement amongst the teachers we recruited, and we measured a moderate level of agreement in section 4.1. this emphasizes the necessity of thorough and precise guidelines requirements for evaluation protocols, as our lack of rules around the treatment of capitalization of questions led to low agreement on questions generated by an uncased model. third, although we gathered a large number of annotations overall, with 3,164 questions annotated in total, this remains small due to the fact that there are many variables on which to break down performance on (e.g., source document, model of origin, annotator). we plan to release the annotation interface as well as the content and models we used to allow future work to expand and reproduce the results. ## conclusion we introduce the quiz design task, a human evaluation protocol used to evaluate question generation models in an applied scenario. in the qd task, teachers creating a quiz for their students are recommended generated questions, which they can accept in their quiz or reject with a reason from a newly proposed error categorization. we run a qd task with 10 teachers, annotating 3,164 questions originating from seven models, and find that acceptance rates vary widely with the latest qgen models obtaining the highest acceptance rate of 68.4%. finally, analysis of automatic metrics on our task's data reveals that even though metrics correlate well with system-level ranks, models have reached potential metric upper-bounds, and improved metrics are required to guide nlg forward.