Bringing Pedagogy into Focus: Evaluating Virtual Teaching Assistants’ Question-Answering in Asynchronous Learning Environments (2025)

Abstract: Virtual Teaching Assistants (VTAs) can reduce the workload of teaching teams in Asynchronous Learning Environments (ALEs) where timely, personalized support is often limited. As VTA systems grow more capable, rigorous and pedagogically sound evaluation becomes essential. Existing assessments often rely on surface-level metrics and lack sufficient grounding in educational theory, making it difficult to meaningfully compare the pedagogical effectiveness of VTA systems. To bridge this gap, we propose a pedagogically-oriented evaluation framework that is rooted in learning sciences and tailored to asynchronous forum discussions, a common VTA deployment context in ALE. We construct classifiers using expert annotations of VTA responses on a diverse set of forum posts. We evaluate the effectiveness of our classifiers, identifying approaches that improve accuracy as well as challenges that hinder generalization. Our work establishes a foundation for theory-driven evaluation of VTA systems, paving the way for more pedagogically effective AI in education.

## introduction asynchronous learning environments (ales) have become a critical complement and extension to traditional synchronous learning (e.g., live lectures and real-time tutoring). by removing time and location constraints, ales enhance flexibility and inclusivity, supporting more equitable access to educational resources and opportunities @xcite . however, providing timely and effective instructional support remains one of the biggest challenges for ale adoption. their self-paced, decentralized nature makes it difficult for instructors to track progress and manage interactions, often leading to * co-first authors † co-senior authors delayed feedback that can reduce student motivation, engagement, and learning effectiveness @xcite . virtual teaching assistants (vtas) offer a promising approach to address this challenge by providing scalable, timely, and personalized support, helping reduce instructor workload in ales @xcite . they are often powered by retrieval-augmented generation (rag) approaches to respond quickly to student questions @xcite . while earlier systems focused on routine and syllabus-based questions @xcite @xcite , recent vtas powered by large language models (llms) can tackle more complex academic queries @xcite @xcite . although these advancements highlight the growing capabilities of vtas, the pedagogical effectiveness of their generated content still lacks a systematic and scalable evaluation framework, which is a critical issue. early approaches often rely on manual qualitative analyses @xcite or general-purpose language metrics @xcite to assess similarity to human responses. nevertheless, these metrics are not designed to assess teaching quality and often lack generalizability across different educational contexts. to bridge this gap, we propose a pedagogical teaching assistant evaluation framework grounded in learning science, focusing on vtas in asynchronous online forums, one of the most common deployment contexts in ales. the framework has two components: a five-level pedagogical skill model with progressively increasing "instructional difficulty": (1) clarify misunderstandings and confusions, (2) deepen disciplinary understanding, (3) foster higher-order thinking, (4) enhance metacognitive awareness, and (5) foster collaborative knowledge construction developed by linking core cognitive mechanisms to pedagogical goals, and an evaluation rubric that operationalizes this model by specifying observable instructional behaviors at each level. when establishing the usability and reliability of our rubric, we utilize a set of simulated vta responses on student discussion forum posts collected through an irb-approved procedure. note that level 5 of our framework is currently unattainable for most vtas, as their responses are typically limited to a single forum post and any retrieved content, excluding broader context from other posts in the discussion. however, asynchronous online forums are inherently community-driven spaces, and achieving collaborative knowledge construction requires access to broader, forum-level context. to simulate vtas with broader forum-level post access for level 5 evaluations, we use rag to create responses informed by similar peer posts. expert annotators apply our rubrics to these simulated responses. we then leverage these annotations to build automatic classifiers using llms. maurya et al. (2024) reports negative correlations between llm and expert annotations in tutor pedagogical assessment, indicating that llms may be unreliable evaluators. their study uses a single, unoptimized prompt without fine-tuning, which the authors acknowledge as a limitation. we extend this line of inquiry to the vta pedagogical evaluation context by systematically exploring whether prompt optimization and fine-tuning can improve alignment between llm-based evaluators and expert pedagogical judgments. our expert annotation results indicate that including peer forum posts reduces pedagogical capability in levels 1-4 within our specific simulated vta environment. furthermore, while our automated classifiers perform reliably on lower-level, in-distribution examples, they struggle at higher pedagogical levels and exhibit limited generalization, even for level 1. overall, our contributions are as follows: ## our evaluation framework evaluating pedagogical skills is challenging because teaching support is a dynamic process that varies by context, subject, and learner preferences. therefore, effective evaluation requires selecting the appropriate level of observation granularity and ensuring alignment with the specific learning context. in our framework, we follow three steps to develop the pedagogical skill model and the evaluation rubric:(1) identifying cognitive mechanisms in asynchronous online forums, (2) distilling contextspecific pedagogical goals, and (3) translating these goals into observable pedagogical behaviors. ## pedagogical evaluation of simulated vta responses we apply our evaluation framework to two sets of simulated vta responses, one set with similar peer posts and the other without. this is to examine if incorporating peer responses as forum-level context affects the expressed pedagogical capabilities of our llm-powered vta. ## automated pedagogical evaluations of vtas we explore automatic approaches for pedagogical assessment using llms to address practical challenges in using expert annotators and to investigate the claim from maurya et al. ( 2024 ) about the limitations of llm evaluators in education. the labeling process in section 4.3 yielded 600 unique annotated post-response pairs. we randomly sample 150 context-free and 150 contextual pairs as our training set, consisting of student forum post information, the corresponding vta responses generated by llama-3, and ratings on all five levels. note that the annotators do not label the context-free instances for level 5; as a result, all levels but level 5 have 300 training pairs. the rest of the data constitutes a held-out test set. we refer to the training set as llama3-train and the test set as llama3-test for the rest of the paper. ## discussion by analyzing vta responses using our pedagogical framework and automating assessment with llms, we highlight key insights into the development and evaluation of vtas in ales. we inspect the consequences of incorporating peer forum posts to aid collaborative knowledge since level 5 appears at the end of our prompt, models may over-prioritize it. this underscores the value of our pedagogical framework, which makes such degradations in performance detectable. this result pertains only to llama-3-70b-instruct, although it is plausible that other llms would exhibit similar patterns. prior work @xcite argues that llms may be unreliable annotators for pedagogical assessment. we explore this claim in the context of assessing vtas in ales, and our findings largely support this concern. while prompt optimization and supervised fine-tuning on synthetic data successfully enhance accuracy, the models we examine still struggle with higher-level pedagogical evaluations. this could reflect a broader lack of pedagogical awareness during the training of stateof-the-art models @xcite , which impacts both prompt-based classification and data synthesis. another limitation is that all our training data (and in-context examples for data generation) come from a single llm, which might restrict generalization. still, the fine-tuned classifiers match and outperform frontier llm performance on llama3-mooc, suggesting strong potential. we expect that introducing more diverse vta response styles to our training data will improve classifier robustness. we leave expanding expert annotations on a wider range of vta outputs as future work. one interesting finding is that markdown formatting from vta responses affects classifier performance. we hypothesize that removing markdown formatting brings the vta responses closer to the output distribution of llama-3-70b-instruct for the open-weight classifiers, while mitigating selfselection biases in the prompt-based classifiers @xcite . this is notable because it emphasizes a critical difference between human and model evaluators; humans are capable of disentangling stylistic features and pedagogical content, whereas current llms experience challenges. ## conclusion we introduce the first evaluation framework for the pedagogical capabilities of teaching assistants in asynchronous online forums. upon expert labeling of forum vta responses using our rubric, we utilize these annotations to build and test automatic classifiers using language models. our results indicate that while certain approaches can benefit classifier performance, language models continue to face challenges with higher-level, nuanced classifications and generalization across response types. our novel framework paves the way for rigorous comparisons between different vta systems, and we caution researchers when using llms for complex pedagogical evaluations.