D r. A cademy: A Benchmark for Evaluating Questioning Capability in Education for Large Language Models (2024)

Abstract: Teachers are important to imparting knowledge and guiding learners, and the role of large language models (LLMs) as potential educators is emerging as an important area of study. Recognizing LLMs’ capability to generate educational content can lead to advances in automated and personalized learning. While LLMs have been tested for their comprehension and problem-solving skills, their capability in teaching remains largely unexplored.In teaching, questioning is a key skill that guides students to analyze, evaluate, and synthesize core concepts and principles.Therefore, our research introduces a benchmark to evaluate the questioning capability in education as a teacher of LLMs through evaluating their generated educational questions, utilizing Anderson and Krathwohl’s taxonomy across general, monodisciplinary, and interdisciplinary domains. We shift the focus from LLMs as learners to LLMs as educators, assessing their teaching capability through guiding them to generate questions. We apply four metrics, including relevance, coverage, representativeness, and consistency, to evaluate the educational quality of LLMs’ outputs. Our results indicate that GPT-4 demonstrates significant potential in teaching general, humanities, and science courses; Claude2 appears more apt as an interdisciplinary teacher. Furthermore, the automatic scores align with human perspectives.

## introduction large language models (llms) have demonstrated great performance in various natural language processing (nlp) tasks, including question answering @xcite , information retrieval @xcite , reasoning @xcite , and generation @xcite , etc. beyond these general nlp applications, llms are architecturally, the school has a catholic character. atop the main building's gold dome is a golden statue of the virgin mary. immediately in front of the main building and facing it, is a copper statue of christ with arms upraised with the legend "venite ad me omnes".it is a replica of the grotto at lourdes, france where the virgin mary reputedly appeared to saint bernadette soubirous in 1858. question: to whom did the virgin mary allegedly appear in 1858 in lourdes france? question: what is in front of the notre dame main building? question: what sits on top of the main building at notre dame? however, the capability of questioning is a crucial aspect in the educational field. as llms take on the role of teachers, can they pose high-quality questions like human educators? therefore, evaluating what constitutes a high-quality question in education becomes necessary. according to anderson and krathwohl's educational taxonomy @xcite , we consider that high-quality questioning in the educational field must meet the following characteristics: i) achieve a higher level across the six domains including memory, understanding, application, analysis, evaluation, and creation; ii) be relevant to the given context; iii) comprehensively cover the content of the context, and iv) also reflect the important knowledge of this context. we consider that questions meeting these characteristics can effectively assess students' knowledge levels, and llms capable of posing such questions can assume the role of competent human educators. the first characteristic is the most basic requirement for llms to act as human teachers, while the following three characteristics measure the excellence of llms in their role as a teacher. evaluating and enhancing the capability of llms to generate questions of high quality standards in the educational domain requires a benchmark. however, previous studies have mainly viewed llms from a student's perspective, focusing on tasks like reading comprehension @xcite @xcite @xcite @xcite and exam evaluations @xcite @xcite @xcite @xcite . however, these tasks focus on adopting contexts to passively answer questions or make reasoning, and these tests treat llms as students, assessing their abilities by how they answer questions, while the llm's questioning capability through generating educational questions is under-studied. current education-related research is far from adequate to determine llms' question raising capability as a teacher, and there isn't a benchmark that studies the overall teaching abilities of llms, seeing them as teachers. although some role-playing tasks @xcite mimic professional dialogues but don't truly assess the llms' teaching capabilities. therefore, if we want llms to assist in teaching effectively, we need to evaluate and enhance their teaching abilities, as possessing knowledge and guiding others to learn are distinct skills. therefore, in this paper, we have developed a benchmark for assessing whether llms generate high-quality questions in the field of education, guided by professional educational theories. unlike general questioning, as shown in fig. 1 (a) , our benchmark requires that the generated questions not only be fluent and readable but also meet the fundamental characteristics proposed earlier (i.e. the first characteristic), as shown in fig. 1(b) . specifically, we draw on anderson and krathwohl's educational taxonomy @xcite to prompt llms to generate questions at six levels for each context. we select tasks from three domains, including general, single-discipline, and interdisciplinary domains, to more comprehensively assess the strengths of llms as teachers in various fields. based on the four characteristics proposed earlier, we have also designed four evaluation metrics: consistency, relevance, coverage, and representativeness, to assess the value of questions posed by llms in the educational domain, thereby comprehensively evaluating the questioning capability of llms as teachers in education through evaluating their generated educational questions. our experiments reveal that llms like gpt-4, claude2, and gpt-3.5 demonstrate good questioning capability across domains as teachers in education through evaluating their generated educational questions. in summary, our contributions are threefolds: ## datasets and task setups we propose a benchmark named dr.academy, which has tasks from three domains. the first two request llms to generate questions in the general and monodisciplinary domain, respectively, based on the six levels of anderson and krathwohl's educational taxonomy (anderson and krathwohl, 2001), including memory, understanding, application, analysis, evaluation and creation. the third one requests llms to generate questions that intersect multiple subjects. the overview of dr.academy is shown in fig. 2 . ## experiments in this section, we conduct extensive experiments to evaluate different llms' questioning capability through evaluating their generated educational questions in the proposed dr.academy. ## conclusions and future work in conclusion, our study presents a pioneering investigation into the questioning capability in education as a teacher of large language models (llms) through evaluating their generated educational questions, shifting the traditional role of llms from learners to educators. we have developed a comprehensive benchmark, named dr.academy, based on educational taxonomies that assesses llms' abilities to generate questions across various domains with four evaluation metrics. our findings indicate that models like gpt4, claude2, and gpt3.5 demonstrate promising teaching potential. looking ahead, the future directions of this research include refining the evaluation metrics for even more nuanced assessments of teaching effectiveness and expanding the range of subjects and domains covered.