Automated Evaluation of Writing – 50 Years and Counting (2020)

Abstract: In this theme paper, we focus on Automated Writing Evaluation (AWE), using Ellis Page’s seminal 1966 paper to frame the presentation. We discuss some of the current frontiers in the field and offer some thoughts on the emergent uses of this technology.

## a minimal case for awe in a seminal paper on the imminence of automated grading of essays, @xcite showed that a high correlation between holistic machine and human scores is possible. he demonstrated automated scoring of 276 essays written by high school students by a system with 32 features, resulting in a multiple @xmath0.65 between machine and average human score, after adjustment. he also provided a thoughtful discussion of his ambitions for automated scoring and of the possible objections. page made the case that automated evaluation of student writing is needed to take some of the evaluation load off the teachers and to provide students evaluations of their (potentially multiple) drafts with a fast turnaround. he then appealed to the then-burgeoning interest and fascination with machine learning to argue for the feasibility of such an enterprise, namely, that machines can learn how to give the right grades to essays, if trained on an expert-scored sample. as part of the feasibility argument, page emphasized the need to carefully define the goal so that success can be judged appropriately. the goal is not a "real" master analysis of the essay the way a human reader would do but merely an imitation that would produce a correlated result (using what page called proxes -approximations). page considered this goal to be both useful and achievable. ## going off the page so far, page's outline of the promises and challenges of awe have provided a good framework for surveying the field. there are also a number of developments that were not mapped on page's chart; we turn to reviewing those next. ## discussion once an invention is out in the open, it is difficult to predict what specific uses people would put it to. how do we go about evaluating the tool if we don't know what the user's goal is? while it isn't possible to anticipate all specific uses, it is possible, we believe, to consider the types of uses that suggest different evaluation strategies. from the current vantage point, we see three types of uses. ## summary & conclusion in his visionary paper from 1966, ellis page provided a proof-of-concept demonstration of the possibility of automated grading of essays, as well 19 1989 curriculum and evaluation standards for school mathematics from the national council of teachers of mathematics recommend in the summary of changes to content and emphasis in k-4 mathematics (p.21) decreasing the attention devoted to long division specifically and to "complex paper-and-pencil computations" in general; the recommendation for grades 5-8 is likewise to decrease emphasis on "tedious paper-and-pencil computations" (p.71). https: //archive.org/details/curriculumevalua00nati . the document has sparked substantial controversy, including with regards to long division @xcite . as outlined some potential challenges to its adoption. subsequent research and practice have delivered on page's minimum desiderata for an awe system; current research is working to address the outstanding challenges dealing with a variety of languages, content domains, and writing tasks. the field of awe has thus progressed according to the trajectory charted by page to a large extent, though not completely. in particular, while page imagined the main use case of awe to be in the service of a harried english teacher and his feedback-thirsty students, in reality, the most visible use case has arguably evolved to be automated scoring of essays for standardized testing, which, in turn, has led to new challenges, such as ensuring the validity and fairness of scores. the other development that page could not anticipate is the sheer pervasiveness of technology in people's daily lives; awe software can be made available not only in classrooms to be used under the watchful eye of the english teacher, but (almost) anywhere and at any time, including on mobile devices. while it is difficult to predict specific uses people would find for such software, we outlined a number of types of use, depending on the goal: (a) consequential decision making about the user; (b) delivery of the best possible written product in partnership with the user; and (c) assisting the user in improving her writing skills. we believe that we, as researchers, can help users find value in our technology by considering the goals, engaging partners from other relevant disciplines, and designing the tools as well as their evaluations to focus on specific types of use.