Unifying AI Tutor Evaluation: An Evaluation Taxonomy for Pedagogical Ability Assessment of LLM -Powered AI Tutors (2025)

Abstract: In this paper, we investigate whether current state-of-the-art large language models (LLMs) are effective as AI tutors and whether they demonstrate pedagogical abilities necessary for good AI tutoring in educational dialogues. Previous efforts towards evaluation have beenlimited to subjective protocols and benchmarks. To bridge this gap, we propose a unified evaluation taxonomy with eight pedagogical dimensions based on key learning sciences principles, which is designed to assess the pedagogical value of LLM-powered AI tutor responses grounded in student mistakes or confusions in the mathematical domain. We release MRBench – a new evaluation benchmark containing 192 conversations and 1,596 responses from seven state-of-the-art LLM-based and human tutors, providing gold annotations for eight pedagogical dimensions. We assess reliability of the popular Prometheus2 and Llama-3.1-8B LLMs as evaluators and analyze each tutor’s pedagogical abilities, highlighting which LLMs are good tutors and which ones are more suitable as question-answering systems. We believe that the presented taxonomy, benchmark, and human-annotated labels will streamline the evaluation process and help track the progress in AI tutors’ development.

## introduction human tutoring is a cornerstone of educational development, playing a crucial role in fostering societal growth by empowering learners. while oneon-one tutoring is highly effective @xcite , its ubiquitous implementation is hindered by the limited availability of qualified tutors. 1 the remarkable success of llms as conversational systems offers promising opportunities in education @xcite . @xcite , driving the development of llm-powered intelligent tutoring systems (its) @xcite and the deployment of llms as tutors using advanced prompting techniques @xcite . such ai tutors serve various educational objectives @xcite , among which the task of students' mistake and confusion remediation is one of the most popular, leading to active ai tutor development @xcite . while the development of ai tutoring systems presents significant challenges, evaluating their pedagogical abilities is even more challenging and crucial for tracking the efficacy and quality of ai tutoring. general domain-agnostic natural language generation (nlg) metrics @xcite @xcite @xcite are not well-suited for this context, as most of them fail to account for pedagogical values and require gold references, which are often not available, especially in online interactions. specifically, for the student mistake remediation task, we need to assess complex pedagogical aspects and abilities of such systems, ensuring that they provide students with sufficient, helpful, and factually correct guidance and do not simply reveal answers when a student makes a mistake. for instance, @xcite found that chatgpt as a tutor reveals the solution 66% of the time and provides incorrect feedback 59% of the time. despite recent efforts to incorporate pedagogical dimensions in the evaluation of ai tutoring systems, there is a notable lack of a unified evaluation taxonomy. for example, @xcite and @xcite evaluated models' responses from the perspective of whether they speak like a teacher, understand a student, and help a student. @xcite assessed the responses of models acting as tutors focusing on coherence, correctness, and equitable tutoring. finally, @xcite evaluated usefulness, care, and humanness, while @xcite focused on targetedness, correctness, and actionability to assess the quality of tutor responses. to address these issues, we propose the first unified evaluation taxonomy based on learning sciences principles to assess the pedagogical abilities of ai tutors. this taxonomy is centered around eight evaluation dimensions related to student mistake remediation including: (1) mistake identification, (2) mistake location, (3) revealing of the answer, (4) providing guidance, (5) actionability, (6) coherence, (7) tutor tone, and (8) human-likeness. as we elaborate in section 4, our taxonomy is strongly aligned with key pedagogical values and unifies the taxonomies used in previous research. in addition to the taxonomy, we compile and release mrbench -a new evaluation benchmark derived from two public datasets, mathdial @xcite and bridge @xcite . each instance in the benchmark includes a partial conversation between a tutor and a student, concluding when the student either makes a mistake or exhibits confusion. the instance is also associated with the following human tutor's response aimed at remediating the student's mistake. using these partial conversation histories exhibiting students' mistakes, we generated responses from seven state-of-the-art llms acting as tutors and conducted human and llm-based evaluations to assess the pedagogical abilities of these models. our findings indicate that while state-of-the-art llms like gpt-4 are effective question-answering systems, they are often not as competent as tutors. in summary, our key contributions are as follows: ## related work in this section, we first briefly overview and discuss the limitations of the existing general-purpose nlg metrics and then turn to pedagogically-oriented approaches to evaluation. ## evaluation taxonomy in this section, we first present our approach, narrowing the evaluation taxonomy down to eight measurable dimensions aligned with key pedagogical strategies @xcite . these dimensions are most suitable for the student mistake remediation task and are based on the learning sciences principles. we then dive into the details of each dimension and its relationship to previous research. an overview of the taxonomy is presented in grounding the taxonomy in the learning sciences principles considering tutors as expert advisors, we prioritize the following high-level pedagogical principles: ## pedagogical ability assessment settings in this section, we provide details on the benchmark data preparation and statistics, llms deployed as ai tutors, the human annotation process, and the llm-based evaluation. ## key findings this section summarizes the key findings of our study on the pedagogical abilities of llms as ai tutors, based on human and llm-based evaluation of mrbench, and the correlation between them. we consider human-based evaluations as gold standard. performance of the powerful gpt-4 and llama-3.1-405b models: both these llms perform well in identifying students' mistakes and their exact location, with llama-3.1-405b having a slight edge as gpt-4 reveals the answer approximately 47% of the time, making its responses less actionable and impacting student's learning experience. this shows that gpt-4 is a good questionanswering system but a relatively poor tutor. at the same time, gpt-4's responses tend to be more encouraging. the guidance score is also high because gpt-4's answer-revealing responses often offer useful explanations, providing the student with learning opportunities. llama-3.1-405b performs more robustly along these dimensions, though it is less encouraging. both models exhibit a high level of coherence, and their responses are human-like as indicated by high damr scores. performance of gemini, sonnet, and mistral: among these three llms, gemini performs the worst as its responses are often incoherent, while also achieving low scores for mistake identification and exact location. furthermore, even in coherent responses, the model frequently reveals the answer and receives low scores for actionability and guidance as its explanations for both correct and incorrect revealed answers are often factually inaccurate, harming students' learning. sonnet and mistral perform slightly better than gemini, with sonnet focusing primarily on encouraging tone and human-likeness while avoiding revealing the answer, though it is less effective along key pedagogical dimensions like mistake identification, location, guidance, and actionability. on the other hand, mistral shows a slight edge along each of the dimensions. in conclusion, among these three models, gemini performs the worst, and mistral performs slightly better than sonnet. ## conclusion this paper presents the first effort to unify ai tutor evaluation for the student mistake remediation task in the mathematics domain. specifically, we propose an evaluation taxonomy with eight pedagogical dimensions based on the key learning sciences principles. we also release the mrbench benchmark with seven state-of-the-art llm-as-tutors responses, along with gold human annotations. we discuss the limitations of each llm by pinpointing the lack of specific pedagogical abilities demonstrated in their responses based on human evaluation. we also assess the feasibility of llms as evaluators in this context by correlating their judgements with human annotations, indicating that they are often unreliable. the study demonstrates that the current state-of-the-art llms are not yet sufficiently good as ai tutors and there is a huge scope for improvement, also identifying the most relevant directions for such improvement. we hope that the resources released with this study will streamline the evaluation process and help track the progress in the development of ai tutors. furthermore, our study opens possibilities for creation and annotation of datasets that can be used for rlhf and fine-tuning, helping future ai tutors align with human and pedagogical values.