The Promises and Pitfalls of Using Language Models to Measure Instruction Quality in Education (2024)

Abstract: Assessing instruction quality is a fundamental component of any improvement efforts in the education system. However, traditional manual assessments are expensive, subjective, and heavily dependent on observers’ expertise and idiosyncratic factors, preventing teachers from getting timely and frequent feedback. Different from prior research that mostly focuses on low-inference instructional practices on a singular basis, this paper presents the first study that leverages Natural Language Processing (NLP) techniques to assess multiple high-inference instructional practices in two distinct educational settings: in-person K-12 classrooms and simulated performance tasks for pre-service teachers. This is also the first study that applies NLP to measure a teaching practice that is widely acknowledged to be particularly effective for students with special needs. We confront two challenges inherent in NLP-based instructional analysis, including noisy and long input data and highly skewed distributions of human ratings. Our results suggest that pretrained Language Models (PLMs) demonstrate performances comparable to the agreement level of human raters for variables that are more discrete and require lower inference, but their efficacy diminishes with more complex teaching practices. Interestingly, using only teachers’ utterances as input yields strong results for student-centered variables, alleviating common concerns over the difficulty of collecting and transcribing high-quality student speech data in in-person teaching settings. Our findings highlight both the potential and the limitations of current NLP techniques in the education domain, opening avenues for further exploration.

## introduction evaluating instruction quality is a crucial component for most efforts aimed at improving an education system. measuring teaching practices with validity and reliability can help advance the understanding of effective teaching strategies, enhance teacher professional learning, and thus improve students' academic performance @xcite @xcite . traditional methods to assess instruction quality require trained raters to observe classrooms based on established criteria. despite its wide use in early childhood, k-12, and higher education, a long line of research has identified a range of issues with this human-based classroom observation approach, such as its lacking consistency across schools and different learning contexts due to time and resource constraints, human subjectivity, and varying levels of expertise among raters, among others @xcite @xcite . in light of the limitations of human-based classroom observations, an emerging literature has started to use natural language processing (nlp) techniques to automatically evaluate teaching practices by analyzing transcripts of classroom recordings @xcite @xcite @xcite . however, other than a few exceptions, these methods and related applications mostly focus on relatively low-inference instructional practices that rely on specific and concrete teaching moves @xcite , such as the frequency of focusing (or open-ended) questions @xcite . in contrast, research is still scarce in exploring the feasibility of building automated measures on high-inference teaching practices, which require observers to draw conclusions and make interpretations based on many elements of teaching that are more aligned with existing classroom observation tools @xcite . one exception is @xcite , which uses zeroshot prompting to explore chatgpt's ability in classifying classroom transcripts for a widely-used observation tool, mathematical quality instruction (mqi) @xcite . however, the authors find the performance of chatgpt to be quite poor. in this paper, we set out to evaluate to what extent state-of-the-art nlp methods can measure high-inference instructional practices. we conduct our study using data collected from both traditional in-person math classrooms in k-12 schools and simulated performance tasks designed for preservice teachers. our study is particularly novel in its inclusion of teaching practices that are widely acknowledged to be effective for students with special needs in general math classrooms @xcite . while research has been progressing quickly in the field of using nlp to measure teaching practices, to our knowledge, this is the first study that incorporates measures that target students with special needs @xcite . for general math education, we evaluate 11 observation variables across four dimensions of math instruction quality in in-person classrooms from mqi. comparing the performance of pretrained language models (plms) for multiple highinference teaching practices allows us to evaluate the relative performance of language models based on the nature of measures. our analysis shows that the performances of finetuned plms on measuring high-inference teaching practices vary by the levels of required pedagogical expertise. plms work decently well for variables largely depending on lexical usage such as mathematical language richness, achieving agreement levels on par with human experts. in contrast, plms demonstrate worse performances on variables that require further inferences that are also challenging for human experts, such as the precision of mathematical language and clarity of mathematical content. we identify and address several challenges that negatively impact the performance of plms. the first challenge is the highly skewed distribution of labels due to the lack of high-rating teaching samples in practice. across both datasets, high-rating samples consist of less than 10% of the dataset for the majority of variables. we address this issue by adopting class-weighted loss to help predict some variables, but the performance gain is limited. another challenge comes from the length of transcripts used to infer teaching practices, which are typically from 5-to 15-minute classroom recordings. such noisy and long inputs impede plms from focusing on the relevant information. we adopt a two-stage strategy by first predicting relevant sentences and then making the model focus on predicted relevant sentences. we find this strategy benefits variables whose assessment relies on multiple sentences. thus, even if the relevance classifier falsely rules out some relevant sentences, there are still cues for plms to make predictions. to this end, we also use chatgpt for zero-shot aspect summarization as the relevance model in the first stage but the extracted sentences are mostly irrelevant according to the rubrics, which is consistent with @xcite . although it is preferable to use all dialogue in a classroom for measuring teaching practices, in traditional classroom settings, the collection of student speech data is often challenging due to microphone setups and consent requirements. surprisingly, using teachers' utterances alone achieves reasonable performances for student-oriented observation variables. this finding underscores the potential of focusing on teacher-led discourse for comprehensive classroom assessment. we summarize our contributions as follows. ## data to comprehensively evaluate plms' ability to evaluate instruction quality, we select two datasets-the simulation for special education (simse) dataset and the national center for teacher effectiveness (ncte) transcript dataset @xcite , which capture both simulated teaching sessions for pre-service teachers and in-person k-12 classrooms. we introduce the details of these two datasets and their corresponding instruction quality measurement. ## methods measuring instructional practices is naturally formulated as text classification tasks. to comprehensively understand language models' ability in these tasks, we experiment with a wide range of plms. specifically, we fine-tuned the following models by adding a classification head on top of the hidden layers, using the implementation from transformer @xcite , i.e., bert @xcite , distilbert @xcite , xl-net @xcite , and roberta @xcite . we also fine-tuned llama2-7b @xcite with qlora @xcite in the instruction-following style by converting the dataset to a next-word prediction setup. more discussions on model choices are in appendix c. we identify two primary challenges in automatically measuring teaching quality: (1) the text input is long and noisy, making it hard for plms to focus on the relevant sentences for each variable; and ## experiments to evaluate the models and techniques introduced above, we randomly split the simulation sessions and classroom segments from the simse and ncte datasets into train/dev/test sets with a ratio the test set results for selected models are shown in ## results and discussion in general, models perform much better on metacognitive modeling components than mqi variables, partly due to the shorter input length and the teacher-only setup. however, we observe considerable variations across the variables. we discuss the insights in detail as follows. llms work better for variables that require less pedagogical expertise. in using only teacher utterances as input captures most observations, even for student-oriented variables. for ncte, we experimented with using only teacher utterances and including student utterances ( the results above are from models selected from various model types and combined with different techniques described in section 3. we next present the results and discuss their implications. class-weighted loss marginally improves the performances. variables that have more balanced class distributions tend to have better performance, but the intrinsic difficulty of the variables seems to be the driving factor. for example, self-regulation has a more skewed distribution but worse performance compared to self-regulation. similar observations can be made when comparing unpacking with objective or ending. moreover, the adoption of class-weighted loss achieved best performance on 10 out of the 16 variables in the three-way setting (tables 7 and 12 ) and 9 out of 16 in the binary setting (tables 11 and 13 ), although the differences are small. therefore, classweighted loss helps but with a marginal effect. ## related work prior works on automatic teaching evaluations primarily focused on low-level instruction statistics such as the frequency of certain utterancelevel teaching strategies employed in the classroom. considering teachers' questions in the classroom play a crucial role in students' engagement and academic achievement @xcite , many researchers studied question-related behaviors such as detecting utterances that contain questions @xcite , identifying authentic questions without prescripted answers @xcite @xcite , identifying focusing and funneling questions @xcite , and so on @xcite . other discourse features of teachers have also been shown to influence learning achievement. for instance, @xcite detects socio-emotional supports. @xcite measures teachers' uptake to provide feedback for students' contribution. @xcite ; @xcite recognizes different types of teaching activities such as presenting, guiding, and management @xcite . despite previous efforts on low-level evaluation, wang and demszky (2023) provided some preliminary results on evaluating teaching quality and providing high-level feedback, which aligns more with the current forms of observation feedback in education. however, their study focused on testing chatgpt's zero-shot ability and only evaluated a small set of mqi variables in the ncte dataset. ## future work model choices. although our experiments cover a wide range of plms including the recently released llama2-7b model, many advanced large language models (llms) were beyond our scope due to resource demands. with the negative results in the zero-shot setting @xcite , future research could focus on adapting these models for similar tasks, i.e., make llms better follow the instructions in the rubrics and align them with experts' preferences. ## conclusion in this study, we present the most comprehensive study to date on assessing high-level instruction quality, spanning both in-person and simulated educational settings, with a wide coverage of observation variables for math education, including metacognitive components tailored for students with special needs. we find that the performance of plms largely depends on the level of pedagogical expertise required for assessing the observation variables, despite employing various techniques aimed at addressing the issues of lengthy inputs and the scarcity of high-rating samples. our study enhances the understanding of the extent to which current nlp techniques work in education and brings insights into automatically evaluating highinference instruction quality, which could reduce disparities in teachers' professional learning and, consequently, students' academic outcomes.