Automated Evaluation of Teacher Encouragement of Student-to-Student Interactions in a Simulated Classroom Discussion (2024)

Abstract: Leading students to engage in argumentation-focused discussions is a challenge for elementary school teachers, as doing so requires facilitating group discussions with student-to-student interaction. The Mystery Powder (MP) Task was designed to be used in online simulated classrooms to develop teachers’ skill in facilitating small group science discussions. In order to provide timely and scaleable feedback to teachers facilitating a discussion in the simulated classroom, we employ a hybrid modeling approach that successfully combines fine-tuned large language models with features capturing important elements of the discourse dynamic to evaluate MP discussion transcripts. To our knowledge, this is the first application of a hybrid model to automate evaluation of teacher discourse.

## introduction scientific argumentation is an essential skill, and in elementary school classrooms, group science discussions are a natural modality for providing students with opportunities to engage in scientific argumentation @xcite . accordingly, it is essential that teachers are well equipped to facilitate such discussions. but facilitating them is not straightforward. many teachers are used to a lecture style of interaction where they deliver the facts and the students respond only to the teacher @xcite @xcite . in contrast, in an ideal group science discussion, students directly interact with their peers (rather than just the teacher) and engage with each other's ideas, rather than only their own and the teacher's @xcite . digitally simulated classroom experiences have become increasingly used to prepare teachers for the work of teaching @xcite . in a simulated classroom, the teacherin-training, also called a pre-service teacher (henceforth, pst), enacts a classroom scenario, interacting in real time with student avatars puppeteered by a trained human actor equipped with voice modulating software. in contrast to practicum experiences, simulated classrooms afford development of targeted skills in an environment that is both standardized and low-stakes @xcite @xcite . automating as much as possible of the simulation would help make the learning experience more affordable and thus accessible to a wider range of teachers; it would also allow teachers to engage in multiple rounds of practice to hone their teaching skills. of the two bottlenecks-the puppeteer enacting the student avatars and the human expert evaluating the performance-we here address the second, leaving the first to future work. the present paper is a case study of developing automated evaluation, with supervised learning, of a pst's performance in a simulated classroom. we focus on the mystery powder (henceforth, mp) task @xcite , a particular lesson that the pst is to teach in a simulated classroom (figure 1 ) designed to develop psts' competency in facilitating small group argumentation-focused science discussions at the elementary level. successful facilitation of a discussion is complex; in this work, we address one of its dimensions, namely, the extent to which the teacher encourages student-tostudent interactions where students engage directly with each other's ideas @xcite . in line with the manual evaluation process that produced the training data @xcite , our approach to automated evaluation had models on two levels: classifiers identifying pst utterances as positive examples of the desired teaching practices; and regressors scoring the transcript as a whole on the same practices @xcite . furthermore, we kept in mind two considerations: classifier training must deal with the fact that rater labels were non-exhaustive (only some utterances are labeled); and regressors must aggregate utterance-level information in an intuitive way. in terms of what features were used, we built three types of models: (a) models based on the analysis of the content of what the pst said, implemented using fine-tuned large language models (henceforth, llms); (b) models based on the structure of the interaction, who speaks when and in relation to whose utterance; and (c) combined models using both content and structure. to our knowledge, this is the first demonstration, in the context of automated analysis of teacher discourse, of a successful combination of fined-tuned llms and shallow features into a hybrid model that outperforms both components in isolation across the board, for multiple levels of analysis (utterancelevel and transcript-level) and multiple indicators of performance. ## results ## conclusion the goal of the current study was automated evaluation of teacher discourse when facilitating a discussion in a simulated elementary science classroom. we showed that models focused on the content of the teacher's utterances using fined-tuned large language models and models focused on the structure of the discourse modeled using handcrafted features captured complementary aspects of the target construct and could be fruitfully combined into hybrid models that outperformed both content and structure models. our results also demonstrated strong generalization from a small number of "score justifications" provided by expert human raters, suggesting a potentially more efficient data generation paradigm than an exhaustive annotation of discourse moves.