Building Robust Content Scoring Models for Student Explanations of Social Justice Science Issues (2024)

Abstract: With increased attention to connecting science topics to real-world contexts, like issues of social justice, teachers need support to assess student progress in explaining such issues. In this work, we explore the robustness of NLP-based automatic content scoring models that provide insight into student ability to integrate their science and social justice ideas in two different environmental science contexts. We leverage encoder-only transformer models to capture the degree to which students explain a science phenomenon, understand the intersecting justice issues, and integrate their understanding of science and social justice. We developed models training on data from each of the contexts as well as from a combined dataset. We found that the models developed in one context generate educationally useful scores in the other context. The model trained on the combined dataset performed as well as or better than the models trained on separate datasets in most cases. Quadratic weighted kappas demonstrate that these models are above threshold for use in classrooms.

## introduction this study investigates the robustness of natural language processing (nlp)-based automatic content scoring models that assess secondary school students' ability to integrate their science and social justice ideas to explain social justice science issues (sjsi; @xcite in two different contexts. in particular, we investigate the robustness of content scoring models in terms of their ability to score out-of-distribution responses as we attempt to generalize the models from one sjsi context to another. the contexts are (a) a unit about combustion reactions and asthma caused by exposure to particulate matter pollution and (b) a unit about global climate change and exposure to extreme heat occurring in urban heat islands. in both units, which are also aligned to state science standards, the students explore the racially disparate impacts of the environmental hazard (particulate matter pollution, extreme heat in urban spaces). in the units, students are supported to explore typical disciplinary content and make connections to local justice issues. they answer the impacts item, explaining whether all people are impacted by the environmental hazard in the same way. we explore the possibility of building a robust domain general model that can be used across multiple sjsi contexts. the curriculum, assessments, and scoring rubrics were developed by a research practice partnership (rpp) including classroom teachers, computer scientists, and learning scientists guided by the knowledge integration pedagogy (ki; @xcite . the automatic content scoring models were created to assess the degree to which students connect their understanding of the environmental concepts with understanding of the social justice issues when explaining whether everyone is impacted in the same way. as teachers reformulate their instruction to include social justice perspectives, automatic content scoring models can help teachers by capturing student progress. they are especially valuable for social justice ideas that might be new to science teachers. we investigate the accuracy and robustness of automatic content scoring models that can quickly assess student explanations, particularly when those explanations contain social justice ideas. in this study we ask: this study builds on prior research integrating social justice into science curriculum and leveraging ai techniques to score student essays. ## data and experimental design we developed automatic content scoring models to automatically score the impacts item which is found in several units: global climate change and ki score criteria asthma example uhi example 1 irrelevant idk asifhsdif 2 vague yes, climate change will effect everyone in the whole world. i think in some ways yes and in some ways no. 3 partial link: one target idea yes, because if you have more freeways or factories where you live you could have more of the effects of incomplete combustion. no, because there is less greenery, and plants and trees help to keep things cool in urban heat islands. 4 full link: links two target ideas people who are lower income are impacted by climate change more than people who aren't because they sometimes have to live closer to factories and other places where there could be harm. no, some people who for example live in poorer or redlined areas will be more impacted. as those areas don't have as much greenery or architecture that can help with the heat. 5 full links: links three or more target ideas no! racially oppresed groups are affect more by climate change. these groups are in redlined communities which put near industrial areas which produce green house gases. these greenhouse gas emmisions give you a higher chance to have asthma. black and hispanic people who live in poorer residences have less trees and grass nearby, as an effect of redlining, which makes poorer neighborhoods hotter. the rich white neighborhoods are invested in by banks, and have much more trees and grass, making their neighborhood 5-20 degrees cooler. urban heat islands (uhi; 9th grade) and chemical reactions and asthma (asthma; 7th grade). in this section, we describe the item, scoring rubrics, training data, and experimental design. the section that follows details our model development approach. ## conclusions and next steps this study investigates the robustness of pedagogically aligned automatic content scoring models trained for one sjsi context when used for a different sjsi context and of the model trained on multiple sjsi. we found that the models are robust across these contexts. models developed in one context generate educationally useful scores in the other context. the model trained on the combined dataset is as good or better than the models trained on separate datasets in most cases. these findings underscore the value of using classroom data to fine-tune encoder-only transformer models using a pedagogically-grounded scoring rubric. in particular, the models were robust for scoring student responses for knowledge integration. they also demonstrate the potential for using "instancebased" models across contexts when it is unlikely that words or phrases associated with a high score from one context would be associated with a low score in another context. results demonstrate that these models are above threshold for use in classrooms to give students adaptive, personalized guidance based on their essay scores. they can also be used to synthesize classroom data for teachers in real time. thus, the automatic content scoring generates ki scores, disciplinary scores, and justice scores that can be displayed in class-level histograms along with illustrative student responses to help teachers monitor class progress. these results suggest promise for generalizing models across similar contexts, increasing the efficiency of design of automatic content scoring models for adaptive instructional materials. next steps include validating the educational value of the models in classroom settings. we plan to engage the rpp in designing and testing adaptive guidance informed by ki pedagogy for each of the automatically generated scores. in addition classroom observations and interviews with teachers are needed to understand how the scores generated by the models align with teachers' assessment of student explanations of sjsis and how access to student scores from the models shapes their instruction.