Is C hat GPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom Instruction (2023)

Abstract: Coaching, which involves classroom observation and expert feedback, is a widespread and fundamental part of teacher training. However, the majority of teachers do not have access to consistent, high quality coaching due to limited resources and access to expertise. We explore whether generative AI could become a cost-effective complement to expert feedback by serving as an automated teacher coach. In doing so, we propose three teacher coaching tasks for generative AI: (A) scoring transcript segments based on classroom observation instruments, (B)identifying highlights and missed opportunities for good instructional strategies, and (C) providing actionable suggestions for eliciting more student reasoning. We recruit expert math teachers to evaluate the zero-shot performance of ChatGPT on each of these tasks for elementary math classroom transcripts. Our results reveal that ChatGPT generates responses that are relevant to improving instruction, but they are often not novel or insightful. For example, 82% of the model’s suggestions point to places in the transcript where the teacher is already implementing that suggestion. Our work highlights the challenges of producing insightful, novel and truthful feedback for teachers while paving the way for future research to address these obstacles and improve the capacity of generative AI to coach teachers.

## introduction classroom observation, coupled with coaching, is the cornerstone of teacher education and professional development internationally @xcite @xcite . in the united states, teachers typically receive feedback from school administrators or instructional coaches, who assess teachers based on predetermined criteria and rubrics. these structured evaluations often involve pre-and post-observation conferences, where the observer and teacher discuss teaching strategies and reflect on the observed instruction. despite its widespread adoption, classroom observation lacks consistency across schools and different learning contexts due to time and resource constraints, human subjectivity, and varying levels of expertise among observers @xcite . frequency and quality of feedback can vary significantly from one school or learning context to another, resulting in disparities in teacher development opportunities and, consequently, student outcomes. prior work has sought to complement the limitations of manual classroom observation by leveraging natural language processing (nlp) to provide teachers with scalable, automated feedback on instructional practice @xcite . these approaches offer low-level statistics of instruction, such as the frequency of teaching strategies employed in the classroomdifferent from the high-level, actionable feedback provided during coaching practice. receiving highlevel, actionable feedback automatically could be easier for teachers to interpret than low level statistics, and such feedback also aligns more closely with existing forms of coaching. recent advances in nlp have resulted in models like chatgpt that have remarkable few-shot and zero-shot abilities. chatgpt has been applied to various nlp tasks relevant to education, such as essay writing @xcite or assisting on mathematics problems @xcite , and providing essay feedback to students @xcite . a survey conducted by the walton family foundation shows that 40% of teachers use chatgpt on a weekly basis for tasks such as lesson planning and building background knowledge for lessons @xcite . given chatgpt's potential and teachers' growing task b is to identify highlights and missed opportunities for good instructional strategies. task c is to provide actionable suggestions for eliciting more student reasoning. to answer this question, we propose the following teacher coaching tasks for generative ai. task a. score a transcript segment for items derived from classroom observation instruments task b. identify highlights and missed opportunities for good instructional strategies task c. provide actionable suggestions for eliciting more student reasoning we evaluate the performance of chatgpt with zero-shot prompting on each of these tasks via the process in figure 1 . we use the ncte dataset @xcite , a large dataset of elementary math classroom transcripts. the data is annotated by experts with two observation protocols: the classroom assessment scoring system (class) @xcite and mathematical quality instruction (mqi) @xcite instruments. we prompt chatgpt to score segments from these transcripts (task a) and to identify highlights and missed opportunities (task b) with respect to items derived from class and mqi. finally, we prompt the model to generate suggestions to the teacher for eliciting more student mathematical reasoning in the classroom (task c). we evaluate chatgpt by comparing the model's numerical predictions to raters' scores in the ncte data (task a). we also recruit math teachers to rate the chat-gpt's responses along multiple helpfulness criteria @xcite . we find that chatgpt has significant room for improvement in all three tasks, but still holds promise for providing scalable high-quality feedback. on predicting scores, chatgpt has low correlation with human ratings across all observation items even with added rubric information and reasoning. on identifying highlights and missed opportunities, chatgpt generates responses that are often not insightful (50-70%) or relevant (35-50%) to what is being asked for by both instruments. finally, the majority of suggestions generated by @xcite describe what the teacher already does in the transcript. nonetheless, the model does generate a majority of suggestions that are actionable and faithfully interpret the teaching context. we believe that with further development, chat-gpt can become a valuable tool for instructional coaches and teachers. our work highlights an exciting area for future research to improve on the current limitations of automated feedback systems. in sum, we make the following contributions: we (1) propose three teacher coaching tasks for generative ai, (2) recruit expert teachers to evaluate chatgpt's zero-shot performance on these tasks given elementary math classroom transcripts, (3) demonstrate that chatgpt is useful in some aspects but still has a lot of room for improvement, and finally (4) highlight directions for future directions towards providing useful feedback to teachers. testing zero-shot capabilities of chatgpt. recent works have measured the capabilities of chat-gpt for annotation on established datasets and benchmarks @xcite @xcite . for example, in a non-education setting, @xcite evaluates the zero-shot ability of chatgpt to classify tweets. @xcite is a recent education work that investigates chatgpt's zero-shot ability to provide feedback to students on business project proposals. however, their study only utilizes a single broad prompt to solicit feedback and they do not evaluate for common model issues like hallucination @xcite . our work proposes three concrete tasks to generate different forms of feedback for teachers, and our evaluation targets common qualitative issues in model generations. for other recent applications of chatgpt, we refer the reader to @xcite . ## data we use the national center for teacher effectiveness (ncte) transcript dataset @xcite in this work, which is the largest publicly available dataset of u.s. classroom transcripts linked with classroom observation scores. the dataset consists of 1,660 45-60 minute long 4th and 5th grade elementary mathematics observations collected by the ncte between 2010-2013. the transcripts are anonymized and represent data from 317 teachers across 4 school districts that serve largely historically marginalized students. transcripts are derived from video recordings, which were scored by expert raters using two instruments at the time of the ncte data collection: the classroom assessment scoring system (class) @xcite and mathematical quality instruction (mqi) @xcite instruments. we evaluate chatgpt's ability to predict scores for both instruments, as described below. the class instrument. class is an observational instrument that assesses classroom quality in pk-12 classrooms along three main dimensions: emotional support, classroom organization and instructional support. each of these dimensions is measured by multiple observation items; we choose one item from each dimension to provide a proofof-concept. for emotional support, we focus on the positive climate (clpc) item, which measures the enjoyment and emotional connection that teachers have with students and that students have with their peers. for classroom organization, we focus on the behavior management (clbm) item which measures how well the teachers encourage positive behaviors and monitor, prevent and redirect misbehavior. finally, for instructional support, we focus on the instructional dialogue (clinstd) dimension which measures how the teacher uses structured, cumulative questioning and discussion to guide and prompt students' understanding of content. each item is scored on a scale of 1-7 where 1 is low and 7 is high. all items are scored on a 15-minute transcript segment, which is typically about a third or fourth of the full classroom duration. the mqi instrument. the mqi observation instrument assesses the mathematical quality of instruction, characterizing the rigor and richness of the mathematics in the lesson, along four dimensions: richness of the mathematics, working with students and mathematics, errors and imprecision, and student participation in meaning-making and reasoning. similar to class, each of these dimensions is measured by several observation items and we select one from each. for richness of the mathematics, we focus on the explanations (expl) dimension which evaluates the quality of the teacher's mathematical explanations. for working with students and mathematics, we focus on the remediation of student errors and difficulties (remed) which measures how well the teacher remediates student errors and difficulties. for errors and imprecision, we focus on the imprecision in language or nota-tion (langimp) dimension which measures the teacher's lack of precision in mathematical language or notation. finally, for student participation in meaning-making and reasoning, we focus on the student mathematical questioning and reasoning (smqr) dimension which measures how well students engage in mathematical thinking. these items are scored on scale of 1-3 where 1 is low and 3 is high. they are scored on a 7.5 minute transcript segment, which is typically a seventh or eighth of the full classroom duration. ## methods we use the gpt-3.5-turbo model through the openai api, the model that powers chatgpt. we decode with temperature 0. we employ zero-shot prompting in our study for three reasons. first, transcript segments are long, and the length of annotated example segments would exceed the maximum input size. second, zero-shot prompting mimics most closely the current ways in which teachers interact with chatgpt. third, we are interested in evaluating chatgpt's capabilities off-the-shelf, without additional tuning. ## results & discussion as for mqi, for all dimensions but remed the model tends to predict the middle score (2 out of 3); this observation is consistent across all prompting methods. another interpretation of this finding, consistent with the class results (which is on a 7 point scale), is that the model tends to predict the second to highest rating. we do not have sufficient data to disentangle these two interpretations. for remed, the model generally predicts the highest rating (figure 4 ). similar to the observations made in class, adding more information or reasoning does not help the model. the model seems to pick up on smqr better than the other items, but its correlation decreases with both added information and reasoning. altogether, the models' tendency to predict the same scores for the same mqi or class item suggest that the predicted scores are a function of the dimension description and not of the transcript evidence or the prompting methodology. task b: identify highlights and missed opportunities. figure 5a summarizes the ratings on model responses for the class instrument, and figure 5b for the mqi instrument. teachers generally did not find the model responses insightful or relevant to what was being asked for both instruments. hallucination, as rated by faithfulness, is not the most problematic dimension out of the three. nonetheless, it appears in a nontrivial amount of the model responses-around 20-30% of the model responses are marked with being unfaithful in interpreting the classroom transcript. interestingly, the mqi results are worse than the class results across all evaluation dimensions. concretely, the "no" proportions increase on every dimension from @xmath0: low scores on faithful increase 22 → 29% (+7), relevant 35 → 55% (+20), and insightful 51 → 71% (+20). this suggests that the model performs relatively worse on interpreting and evaluating technical aspects of math instruction quality. appendix c contains additional plots, including the cohen's kappa between raters. task c: provide actionable suggestions for eliciting student reasoning. figure 5c summarizes the ratings on the model suggestions. the most noticeable observation is that the model tends to produce redundant suggestions (opposite of novelty), repeating what the teacher already does in the transcript 82% of the time. nonetheless, most model responses were rated to be faithful to the transcript context, relevant to eliciting more student reasoning, and actionable for the teacher to implement. ## limitations this section discusses the limitations related to the evaluation process and potential ethical considerations associated with the use of chatgpt or similar language models in educational settings. human evaluation our evaluation is conducted with a limited sample size of two teachers. future work should aim to include a larger and diverse sample of teachers to capture a wider range of perspectives. this would help tease apart the potential teacher biases from generalizable claims about the feedback quality. ethical considerations the use of language models like chatgpt in educational contexts war-rants careful examination. for example, because the model relies on transcribed speech and is trained on primarily english, it might misinterpret the transcriptions of teachers or students who do not speak english fluently. additionally, deploying language models in education settings raises concerns regarding privacy and data security. for example, the raw classroom transcripts should not be directly fed into the model to provide feedback as it may contain personally identifiable information about students. guardrails should be set to prevent classroom data from being sent directly to external companies. ## avenues for future work as evidenced from our work, generating good feedback for teaching is challenging and chatgpt has significant room for improvement in this area. this section discusses potential future directions to overcome these obstacles. reducing hallucination. our results show that chatgpt does generate a non-trivial amount of misleading responses as measured by our faithfulness dimension (15-30% of the time). this observation is documented in the llm literature as model hallucination @xcite . in domains that leverage references or citations such as in factchecking, remedies include retrieving sources and checking the claims made by the model @xcite . in the domain of teacher feedback, however, it is not obvious what the "true" interpretation is, as even human observers may disagree slightly with respect to the teachers' intentions or actions. future work could decrease hallucination in these higher inference domains, e.g. by forcing the model to be conservative with respect to making inferences. involving coaches and educators in model tuning. our results show that chatgpt struggles to generate insightful and novel feedback for teachers; understandably, since such feedback is not present in its training data. involving coaches and educators in the reinforcement learning stage of model fine-tuning (christiano et al., 2017) could be an effective way to improve the models' performance for teacher coaching. one less costly alternative is to engineer the model's prompt collaboratively with teachers and coaches. however, we are sceptical about the effectiveness of prompt engineering for teacher feedback, as it does not address model's lack of exposure to teacher coaching examples during training. tailoring feedback to a teacher's needs and expanding to other subjects. what counts as helpful feedback may be different for each teacher, and look different in other subjects, eg. history and english. even for the same teacher, what they self-report to be helpful may be different from what what has a positive impact on their practice. an effective coach takes this into account, and is able to dynamically adapt the feedback based on the teacher's needs and based on what they observe to be effective for that teacher (thomas et al., 2015; kraft and blazar, 2018). improving chat-gpt's ability to differentiate feedback based on the teacher's needs, and update the feedback strategy based on teacher's subsequently observed practice would be a valuable direction for future work. to adapt our approach beyond mathematics, such as in subjects like history or english, re-searchers and instructors should collaborate and account for the subject's instructional practices and learning objectives. this would help identify the relevant dimensions of effective teaching and inform the design of feedback prompts. for example, they can build on the subject-specific observation instruments as done in our work. integrating automated feedback into human coaching practice. we envision automated coaching to complement, rather than replace coaching by experts for three reasons. first, as this paper shows, the capabilities of current technology is very far from that of an expert instructional coach. second, even with improved technology, having an expert in the loop mitigates the risks of misleading or biased model outputs. finally, even though automated feedback offers several benefits, including flexibility, scalability, privacy, lack of judgment, human interaction is still an important component of coaching and is perceived by teachers as such @xcite . automated coaching could complement human coaching in a teacher-facing way, e.g. by directly providing the teacher with feedback on-demand. such an automated tool can also be coach-facing, e.g. by generating diverse range of suggestions that the coach can then choose from based on what they think is most helpful for the teacher they are supporting. ## conclusion our work presents a step towards leveraging generative ai to complement the limitations of manual classroom observation and provide scalable, automated feedback on instructional practice. while our results reveal that chatgpt has room for improvement in generating insightful and novel feedback for teaching, our proposed tasks and evaluation process provide a foundation for future research to address the challenges of teacher coaching using nlp. our work underscores the challenge and importance of generating helpful feedback for teacher coaching. moving forward, we propose several directions for further research, such as improved prompting methods and reinforcement learning with feedback from coaches. ultimately, we envision a future where generative ai can play a crucial role in supporting effective teacher education and professional development, leading to improved outcomes for students. prompt for direct score prediction (da) on clpc consider the following classroom transcript.