Evaluating the Knowledge Dependency of Questions (2022)

Abstract: The automatic generation of Multiple Choice Questions (MCQ) has the potential to reduce the time educators spend on student assessment significantly. However, existing evaluation metrics for MCQ generation, such as BLEU, ROUGE, and METEOR, focus on the n-gram based similarity of the generated MCQ to the gold sample in the dataset and disregard their educational value.They fail to evaluate the MCQ’s ability to assess the student’s knowledge of the corresponding target fact. To tackle this issue, we propose a novel automatic evaluation metric, coined Knowledge Dependent Answerability (KDA), which measures the MCQ’s answerability given knowledge of the target fact. Specifically, we first show how to measure KDA based on student responses from a human survey.Then, we propose two automatic evaluation metrics, KDA_disc and KDA_cont, that approximate KDA by leveraging pre-trained language models to imitate students’ problem-solving behavior.Through our human studies, we show that KDA_disc and KDA_soft have strong correlations with both (1) KDA and (2) usability in an actual classroom setting, labeled by experts. Furthermore, when combined with n-gram based similarity metrics, KDA_disc and KDA_cont are shown to have a strong predictive power for various expert-labeled MCQ quality measures.

## introduction multiple-choice question (mcq), comprised of a question stem, an answer, and a set of distractors, is one of the most widely used student assessment tools. since manually generating exam-style questions is a complex, labor-intensive process 1 our code is released here: https://github.com/ riiid/question-score (a) bad question (too easy) (b) good question figure 1 : problem formulation. a multiple-choice question designed to test the knowledge of a target fact must satisfy the following criterion: a question has to be answered by only a student who knows the fact, not a student who doesn't. despite the importance of aqg in educational purposes, the task did not receive much attention and was not actively introduced in the education field due to the limitation of evaluation methods. previous aqg works mostly evaluate their methods based on how similar the generated questions/distractors are to the gold questions/distractors, using n-gram based similarity metrics such as bleu @xcite , rouge @xcite , meteor @xcite , or bertscore @xcite . however, all metrics mentioned above share the following limitations: (1) reliability of the evaluation depends on the quality of the reference dataset against which similarity is measured, and (2) ngram-based similarity of a question does not directly measure the usability of the question as an assessment tool. though a prior work @xcite proposed an alternative metric of answerability, the suggested metric does not account for whether the question's answerability crucially depends on knowledge of the target fact and also requires reference made by humans. aqg will be applied in the real-world much more easily if we address these limitations. to this end, this paper suggests a new evaluation criterion for mcq generation, coined knowledge dependent answerability (kda). kda measures whether a real student who got the problem wrong can choose the correct answer after gaining the knowledge the problem wants to test. kda is based on the rationale that a well-designed mcq should be answerable if the student knows the target fact being tested, which is briefly described in figure 1 . however, as such human-dependent measurement is hard to be applied in real world scenario, we propose two automatic variants, kda cont and kda disc . to automate the measurement of kda, we regard the pre-trained language models (plms) as question solvers that imitate students, based on the fact that large plm's have strong question-answering abilities @xcite . to show the validity of the metrics, we ask two research questions: rq1) when replacing students with plms, do the automated metrics (kda cont and kda disc ) behave similarly to kda ? and rq2) can we use mcqs with high kda cont and kda disc in education field meaningfully? to answer these research questions, we asked 116 students to solve 480 mcqs, which include both automatically generated questions by various methodologies and gold questions from three mcq datasets. among them, 96 questions were randomly selected, and a qualitative survey was conducted with experts to see if they were suitable for educational use. through our experiments, we demonstrate that kda disc and kda cont both have a strong correlation not only with kda as measured based on student responses but also with expert likert score that measures the mcq's overall usability in the classroom setting. our main contributions are as follows: ## knowledge dependent answerability to properly evaluate the question's answerability, we propose to measure knowledge dependent answerability (kda). in particular, for each mcq, we measure the proportion of students who answer the question correctly when they know the target fact being tested. our rationale is that a good mcq's answerability must crucially depend on the knowledge of the target fact. then, in order to automatically measure kda without human trials, we replace the students with pre-trained language models based on their question-answering abilities @xcite . ## experiments in this section, we first discuss the settings of our experiments to verify the efficacy of kda, and then describe student and expert studies we conducted in detail. lastly, we show our results for respective experiments. ## conclusion in this paper, we proposed knowledge dependent answerability to measure the assessment value of the generated multiple choice question. we formulated kda as the probability that a student will solve the question correctly if the student know the target fact being tested. then, we proposed kda disc and kda cont to approximate kda, treating plms as individual human solvers. since both metrics are reference-free evaluation metrics, they can be applied without restrictions unlike previous metrics. they can be also applied as filters for question generation for educational use cases. on 3 real-world mcq datasets, kda cont and kda disc demonstrated a high correlation with kda and the expert likert score that measures the usability of the question as an assessment tool in a classroom setting. notably, using kda cont and kda disc along with current n-gram based metrics drastically increased the overall correlation with these expert labels, as well as the prediction power of specific rejection reasons such as irrelevancy and wrong answer. we released a code and model weights to easily measure kda cont and kda disc for a given question and a fact pair. future research may address expanding the metric's applicability to other types of assessment questions, such as short answer questions or multi-hop questions.