Context-based Automated Scoring of Complex Mathematical Responses (2020)

Abstract: The tasks of automatically scoring either textual or algebraic responses to mathematical questions have both been well-studied, albeit separately. In this paper we propose a method for automatically scoring responses that contain both text and algebraic expressions. Our method not only achieves high agreement with human raters, but also links explicitly to the scoring rubric – essentially providing explainable models and a way to potentially provide feedback to students in the future.

## introduction in this paper we present work on automatically scoring student responses to constructed-response mathematics items where the response should contain both text and mathematical equations or expressions. existing work on automated scoring of mathematics items has largely focused on items where either only text is required (c.f. related work on automated short-answer-scoring @xcite ) or only an expression or equation is required @xcite @xcite . this is the first work, to our knowledge, that attempts to automatically score responses that contain both. items that elicit such responses could be algebra, trigonometry, or calculus items that ask the student to solve a problem and/or provide an argument. items at levels much below algebra most likely would not require the student to include an equation -at least one that requires an equation editor for proper entry -in the text, and items at a higher level might require the student to include abstract mathematical expressions that would themselves present automated scoring difficulties. these kinds of items are quite common on paper-and-pencil algebra exams. however, they are less common on computer-delivered exams, primarily because the technology of calling up an equation editor to insert equations in text is new and not generally used. the challenge with automatically scoring these kinds of responses, in a construct-valid way, is that the system needs to be able to interpret the correctness of the equations and expressions in the context of the surrounding text. our goal is not just to achieve accurate scoring but to also have explainable models. explainable models have a number of advantages including (i) giving users evidence that the models are fair and unbiased; (ii) the ability to leverage the models for feedback; and (iii) compliance with new laws, e.g. the general data protection regulation (eu) 2016/679 (gdpr) which requires transparency and accountability of any form of automated processing of personal data. in this paper we present an approach that not only achieves high agreement with human raters, but also links explicitly to the scoring rubric -essentially providing explainable models and a way to potentially provide feedback to students in the future. ## data in this paper we use data from 3 pilot-study items that elicited responses containing both textual explanations as well as equations and expressions. an example item is given in figure 1 , and a sample response (awarded 2 points on a 0-3 point scale) is given in figure 2 . 1 the pilot was administered as part of a larger project in four high schools located in various regions of the united states. the items assumed one year of algebra and involved writing solutions to algebra problems, similar to what a student would be expected to write on a paper-based classroom test. responses were collected digitally; explain, using words and equations, how you would use the quadratic formula to find two values of x for which 195 = -2x foot_2 + 40x. you may also use the on-screen calculator. figure 1 : sample item that elicits textual explanations as well as equations and mathematics. to solve this you must first put your equation in standard form, which gives you @xmath0. you then plug your a, b, and c values into the quadratic formula. to start finding your x value, you must first multiply all your values in parentheses. you must then simplify the square root you get from multiplying. with your new equation, you make two more equations, one adding your simplified square root and one subtracting it. the two answers you get from those equations are your two values of x. . the student has put the equation into standard form with a slight error. of all equations entered by the students, only 34% were entered via the equation editor since most of the students preferred to write simple equations as regular text. 2 there were over 1,000 responses collected for each item, however some responses were blank and therefore not included in this study. ## results coming back to our original research questions: character level features certainly do capture a large portion of mathematical expressions. we see that in the exp 0 results, where there is no interpretation of the mathematical expressions, that systems perform almost as well as the systems that do explicit interpretation. yes, we can include model interpretability without compromising scoring accuracy. the differences between the best models from exp 0 and exp3 ranged from -0.004 to +0.041). by explicitly linking aspects of the rubric to each response, we yield interpretable models that perform comparably to systems without this interpretative layer. although the overall results are lower, they are not statistically significantly lower. ## conclusion to summarize, this work presented a hybrid scoring model using a deterministic system for evaluating the correctness (or partial correctness) of mathematical equations, in combination with text-based automated scoring systems for evaluating the appropriateness of the textual explanation of a response. we contribute the following: chris sangwin. 2004. assessing mathematics automatically using computer algebra and the internet. teaching mathematics and its applications: an international journal of the ima, 23(1):1-14. dinghan shen, guoyin wang, wenlin wang, martin renqiang min, qinliang su, yizhe zhang, chunyuan li, ricardo henao, and lawrence carin. 2018. baseline needs more love: on simple word-embedding-based models and associated pooling mechanisms. in proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers), pages 440-450, melbourne, australia. association for computational linguistics. sudhir varma and richard simon. 2006. bias in error estimation when using cross-validation for model selection. bmc bioinformatics, 7(1):91.