An empirical investigation of neural methods for content scoring of science explanations (2020)

Abstract: With the widespread adoption of the Next Generation Science Standards (NGSS), science teachers and online learning environments face the challenge of evaluating students’ integration of different dimensions of science learning. Recent advances in representation learning in natural language processing have proven effective across many natural language processing tasks, but a rigorous evaluation of the relative merits of these methods for scoring complex constructed response formative assessments has not previously been carried out. We present a detailed empirical investigation of feature-based, recurrent neural network, and pre-trained transformer models on scoring content in real-world formative assessment data. We demonstrate that recent neural methods can rival or exceed the performance of feature-based methods. We also provide evidence that different classes of neural models take advantage of different learning cues, and pre-trained transformer models may be more robust to spurious, dataset-specific learning cues, better reflecting scoring rubrics.

## introduction the next generation science standards (ngss) call for the integration of three dimensions of science learning: disciplinary core ideas (dcis), cross-cutting concepts (cccs), and science and engineering practices (seps) @xcite . science teachers can promote knowledge integration of these dimensions using constructed response (cr) formative assessments to help their students build on productive ideas, fill in knowledge gaps, and reconcile conflicting ideas. however, the time burden associated with reading and scoring student responses to cr assessment items often leads to delays in evaluating student ideas. such delays potentially make subsequent instructional interventions less impactful on student learn-ing. effective automated methods to score student responses to ngss-aligned cr assessment items hold the potential to allow teachers to provide instruction that addresses students' developing understandings in a more efficient and timely manner and can increase the amount of time teachers have to focus on classroom instruction and provide targeted student support. in this study, we describe a set of cr formative assessment items that call for students to express and integrate ideas across multiple dimensions of the ngss. we collected student responses to each item in multiple middle school science classrooms and trained models to automatically score the content of responses with respect to a set of rubrics. this study explores the effectiveness of three classes of models for content scoring of science explanations with complex rubrics: feature-based models, recurrent neural networks, and pre-trained transformer networks. specifically, we investigate the following questions: ## scoring item and rubric design in this section we describe the design of the cr items that comprise the datasets for the content scoring models. the cr items formatively assess student understanding of multiple ngss dimensions, namely, using seps while demonstrating integrated understanding of dcis and cccs. we designed formative assessment items and associated rubrics for four units currently used in the online classroom system: musical instruments (mi), photosynthesis and cellular respiration (ps), solar ovens (so), and thermodynamics challenge (tc). ## error analysis in this section, we explore the differences in the two neural models (rnn and pt) in more detail by looking at patterns of errors. we focus on instancelevel saliency maps -gradient-based methods that identify the importance of tokens to the model by examining the gradient of the loss. for each dataset, we sample 100 responses and generate saliency maps for each. we use the simple gradient method @xcite via allennlp @xcite . the item developers manually analyzed the generated saliency maps for each response and model. we analyzed two sets of cases: in the following, due to space constraints, we focus on error analysis for the scoring model for the musical instruments knowledge integration dataset. one correct, one incorrect. cases where one model accurately predicted the human score while the other did not illuminated several differences in the two neural models. the rnn model tended to ignore or deemphasize some keywords, while overemphasizing high frequency and function words. for example, figure 2a shows a simple example where the rnn fails to emphasize the keyword pitch. the bert model accurately registers this word as salient, and predicts the correct score. similarly, in figure 2b , the rnn misses the keyphrase full glass while the bert model catches it. in figure 2c , the rnn spuriously treats the function words when and you as salient and over-predicts the score. for its part, the bert model may de-emphasize many high frequency words but at the same time may regard discourse markers as salient. an example is in figure 3a , where the bert model emphasizes because since, and this may in part help the model reach the correct prediction. if the bert model is able to better learn important keywords (while ignoring more function words), it may sometimes "overlearn" the importance of those tokens, leading to over-prediction of scores. there are several examples where the model uses the word piece ##brate to overpredict a score (figure 3b ). both incorrect with the same prediction. in many cases, the models made the same incorrect predictions for different reasons. an example is figure 3c , where the rnn emphasizes deeper and dense while the bert model focuses on because and cup. overall, the same differences in the models identified above held for these cases of making the same incorrect prediction. in general, although there was some variability across models, both models correctly identified the keywords necessary for scoring responses correctly, leading to good human-machine agreement. the rnn model may be more sensitive to tokens that are good indicators of the score in the training data (either high or low) but not in language in general, such as high frequency and function words, while 148053 @xmath0 ## @xmath1 the tap of a full glass is more low pitched and an empty glass is more high pitched because there is no bellow 207529 @xmath2 ## @xmath3 the pitch of the tapped full glass is lower than the pitch of the tapped empty glass because since there is water inside you are not going to be able to hear it as much . ## @xmath4 the one taht is full will vibrate less so it will be higher than the one that is empty . ## related work the task of automated content scoring has recently gained more attention @xcite @xcite . our work is similar to @xcite , who developed a multi-task neural model for assigning an overall holistic score as well as content-based analytic subscores. we leave a multi-task formulation of our application setting for future work. @xcite demonstrated state-of-the-art performance for similarity-based content scoring on the semeval benchmark dataset @xcite . in this work, we use pre-trained transformer models for instance-based content scoring (cf. @xcite ). that is, we use whole responses as training data and fine-tune pretrained representations for response tokens on the content score prediction task. recently, methods have been introduced to incorporate "saliency" directly into the model training process @xcite . the current work focuses on interpreting the predictions of models trained without additional annotations (for an overview of interpretability in nlp, see @xcite . exploring the contribution of augmented datasets and training algorithms is future work. to our knowledge, our work is the first to to explore the relevance of the saliency in the predictions of neural methods for the content scoring task. ## conclusion we described a set of constructed response items for middle-school science curricula that simultaneously assess students on expression of ngss disciplineary core ideas (dcis), cross-cutting concepts (cccs), and science and engineering practices (seps), and the integrative linkages between each, as part of engaging in scientific explanations and argumentation. we demonstrated that human and automated scoring of such crs for the ngss dimensions (via independent subscores) and the integration of knowledge (via knowledge integration scores) is feasible. we demonstrated that automated scoring can be developed with promising accuracy. comparing feature-based, rnn, and pre-trained transformer models on these datasets, we observed that the pre-trained transformer models obtained higher rates of human-machine agreement on most holistic ki score and ngss subscore datasets. while the rnn models were often competitive with the pre-trained transformer models, an analysis of the different kinds of errors made by each model type indicated that the pre-trained transformer models may be more robust to strong dataset-specific, but spurious, cues to score prediction. results showed that, in the formative setting targeted by the online science learning environment used in this study, students often scored at the lowest levels of all three rubrics, which increased skewness in the datasets and likely contributed to reduced model accuracy. future research will explore more robust methods for learning scoring models from less data in formative settings, especially from highly skewed score distributions, while continuing to provide accurate scoring. our findings demonstrate the ability to both develop and automatically score ngss-aligned cr assessment items. with further refinement, we can provide teachers with both the instructional and technological assistance they need to effectively and efficiently support their students to demonstrate the multidimensional science learning called for by the ngss.