Essays are considered by many researchers as the most useful tool to assess learning outcomes, implying the ability to recall, organize and integrate ideas, the ability to express oneself in writing and the ability to supply merely than identify interpretation and application of data. It is in the measurement of such outcomes, corresponding to the evaluation and synthesis levels of the Bloom's (1956) taxonomy that the essay questions serve their most useful purpose. One of the difficulties of grading essays is represented by the perceived subjectivity of the grading process. Many researchers claim that the subjective nature of essay assessment leads to variation in grades awarded by different human assessors, which is perceived by students as a great source of unfairness. This issue may be faced through the adoption of automated assessment tools for essays. A system for automated assessment would at least be consistent in the way it scores essays, and enormous cost and time savings could be achieved if the system can be shown to grade essays within the range of those awarded by human assessors. This paper presents an overview of current approaches to the automated assessment of free text answers. Ten systems, currently available either as commercial systems or as the result of research in this field, are discussed: Project Essay Grade (PEG), Intelligent Essay Assessor (IEA), Educational Testing service I, Electronic Essay Rater (E-Rater), C-Rater, BETSY, Intelligent Essay Marking System, SEAR, Paperless School free text Marking Engine and Automark. For each system, the general structure and the performance claimed by the authors are described. In the last section of the paper an attempt is made to compare the performances of the described systems. The most common problems encountered in the research on automated essay grading is the absence both of a good standard to calibrate human marks and of a clear set of rules for selecting master texts. A first conclusion obtained is that in order to really compare the performance of the systems some sort of unified measure should be defined. Furthermore, the lack of standard data collection is identified. Both these problems represent interesting issues for further research in this field. Keywords: Automated Essay Grading, NLP, Computer-based Assessment Systems Assessment is considered to play a central role in the educational process. The interest in the development and in use of Computer-based Assessment Systems (CbAS) has grown exponentially in the last few years, due both to the increase of the number of students attending universities and to the possibilities provided by e-learning approaches to asynchronous and ubiquitous education. According to our findings (Valenti, CucMaterial published as part of this journal, either on-line or in print, is copyrig hted by the publisher of the Journal of Information Technology Education. Permission to make digital or paper copy of part or all of these works for personal or classroom use is granted without fee provided that the copies are not made or distributed for profit or commercial advantage AND that copies 1) bear this notice in full and 2) give the full citation on the first page. It is permissible to abstract these works so long as credit is given. To copy in all other cases or to republish or to post on a server or to redistribute to lists requires specific permission and payment of a fee. Contact Editor@JITE.org to request redistribution permission. Overview of Current Research on Automated Essay Grading 320 chiarelli, & Panti., 2002) more than forty commercial CbAS are currently available on the ma rket. Most of those tools are based on the use of the so-called objective-type questions: i.e. multiple choice, multiple answer, short answer, selection/association, hot spot and visual identification (Valenti et al., 2000) . Most researchers in this field agree on the thesis that some aspects of complex achievement are difficult to measure using objective-type questions. Learning outcomes implying the ability to recall, organize and integrate ideas, the ability to express oneself in writing and the ability to supply merely than identify interpretation and application of data, require less structuring of response than that imposed by objective test items (Gronlund, 1985). It is in the measurement of such outcomes, corresponding to the higher levels of the Bloom's (1956) taxonomy (namely evaluation and synthesis) that the essay question serves its most useful purpose. One of the difficulties of grading essays is the subjectivity, or at least the perceived subjectivity, of the grading process. Many researchers claim that the subjective nature of essay assessment leads to variation in grades awarded by different human assessors, which is perceived by students as a great source of unfairness. Furthermore essay grading is a time consuming activity. According to Mason (2002) , about 30% of teachers' time in Great Britain is devoted to marking. "So, if we want to free up that 30% (worth 3 billion UK Pounds/year to the taxpayer by the way) then we must find an effective way, that teacher will trust, to mark essays and short text responses." This issue may be faced through the adoption of automated assessment tools for essays. A system for automated assessment would at least be consistent in the way it scores essays, and enormous cost and time savings could be achieved if the system can be shown to grade essays within the range of those awarded by human assessor. Furthermore, according to Hearst (2000) using computers to increase our understanding of the textual features and cognitive skills involved in the creation and in the comprehension of written texts, will provide a number of benefits to the educational community. In fact "it will help us develop more effective instructional materials for improving reading, writing and other communication abilities. It will also help us develop more effective technologies such as search engines and question answering systems for providing universal access to electronic information." Purpose of this paper is to present a survey of current approaches to the automated assessment of free text answers. Thus, in the next section, the following systems will be discussed: Project Essay Grade (PEG), Intelligent Essay Assessor (IEA), Educational Testing service I, Electronic Essay Rater (ERater), C-Rater, BETSY, Intelligent Essay Marking System, SEAR, Paperless School free text Marking Engine and Automark. All these systems are currently available either as commercial systems or as the result of research in this field. For each system, the general structure and the performance claimed by the authors are presented. In the last section, we will try to compare these systems and to identify issues that may foster the research in the field. Current Tools for Automated Essay Grading PEG is one of the earliest and longest-lived implementations of automated essay grading. It was developed by Page and others (Hearst, 2000; Page, 1994 Page, , 1996 and primarily relies on style analysis of surface linguistic features of a block of text. Thus, an essay is predominantly graded on the basis of writing quality, taking no account of content. The design approach for PEG is based on the concept of "proxes", i.e. computer approximations or measures of trins, intrinsic variables of interest within the essay (what a human grader would look for but the computer can't directly measure) to simulate human rater grading. Proxes include: essay length (as the amount of words) to represent the trin of fluency; counts of prepositions, relative pronouns and other parts of speech, as an indicator of complexity of sentence structure; variation in word length to indicate diction (because less common words are often longer). Proxes are calculated from a set of training essays and are then transformed and used in a standard multiple regression along with the given human grades for the training essay to calculate the regression coefficients. These regression coefficients represent the best approximation to human grades when obtained according to proxes. Then, they are used with the proxes obtained from unmarked essays to produce expected grades. PEG purely relies on a statistical approach based on the assumption that the quality of essays is reflected by the measurable proxes. No Natural Language Processing (NLP) technique is used and lexical content is not taken in account. PEG also requires training, in the form of assessing a number of previously manually marked essays for proxes, in order to evaluate the regression coefficients, which in turn enables the marking of new essays. PERFORMANCE: Page's latest experiments achieved results reaching a multiple regression correlation as high as 0.87 with human graders. IEA was developed in the late nineties (Hearst, 2000; Jerrams-Smith, Soh, & Callear, 2001) and is based on the Latent Semantic Analysis (LSA) technique that was originally designed for indexing documents and text retrieval (Deerwester, Dumais, Landauer, Furnas, & Harshman, 1990) . LSA represents documents and their word content in a large two-dimensional matrix semantic space (Whittington & Hunt, 1999; Williams, 2001) . Using a matrix algebra technique known as Singular Value Decomposition (SVD), new relationships between words and documents are uncovered, and existing relationship are modified to more accurately represent their true significance. A matrix represents the words and their contexts. Each word being analysed represents a row of in the matrix, while each column represents the sentences, paragraphs, and other subdivisions of the context in which the word occurs. The cells of the matrix contain the frequencies of the words in each context. This initial matrix is then transformed according to an inverse document frequency weighting approach, a very well known technique of the indexing and information retrieval domain. The SVD is then applied to the matrix to subdivide it into three component matrices that reproduce the original one if multiplied together. Using a reduced dimension of these three matrices in which the word-context associations can be represented, new relationships between words and context are induced when reconstructing close approximations to the original matrix from the reduced dime nsion component SVD matrices. These new relationships are made manifest, whereas prior to the SVD they were hidden or latent. To grade an essay, a matrix for the essay document is built, and then transformed by the SVD technique to approximately reproduce the matrix using the reduced dime nsional matrices built for the essay topic domain semantic space. (The semantic space typically consists of human graded essays). Cosine correlation is used to measure the similarity of the reduced dime nsional space constructed from a "model answer", such as an instructional text taken from a course text or an essay prepared by the class tutor, against a student essay. LSA makes no use of word order since the authors claim that this is not the most important factor for grabbing the sense of a passage. It also requires large amounts of data to construct a suitable matrix representation of word use/occurrence, and due to the size of the matrices involved computations are very cumbersome. Key features of IEA include relatively low unit cost, quick customized feedback, and plagiarism detection. Furthermore, the authors claim that the system is very well suited to analyse and score expository essays on topics such as science, social studies, history, medicine or business, but not suitable to assess factual knowledge. IEA automatically assesses and critiques electronically submitted text essay, and represents a useful domain-independent tool. It supplies instantaneous feedback on the content and the quality of the student's writing. PERFORMANCE: A test conducted on GMAT essays using the IEA system resulted in percentages for adjacent agreement with human graders between 85%-91%. This system, developed by Burstein and Kaplan of the Educational Testing Service in early nineties, works only on a sentence fragment of between 15 and 20 words (Whittington & Hunt, 1999) . The technique uses lexical-semantic techniques to build a scoring system, based on small data sets. It uses a domain-specific, concept-based lexicon and a concept grammar, both built from training data. The training data essays are parsed by Microsoft Natural Language Processing (MsNLP) tool, any suffixes are removed by hand, and a list of stop words is also removed. This produces a lexicon. The list of words and terms in the lexicon remain constant whilst the features associated with each entry are modular, so can be replaced as necessary. Some manual classification is necessary, such as specification of some words as metonyms of each other and so on. Grammar rules are then constructed, again manually, for each category of answer (each category should contain all the paraphrases for that possible answer) using syntactic parses of sentences from the training data along with the lexicon. New essays are then parsed by the phrasal node extraction program that outputs the sentences' noun, verb, adjective, adverb and preposition phrases etc. The system does not make use of specific parts of speech at this stage so they are collapsed into a generic XP phrase type and the sentence, or what is left of it now, is checked for matches against the grammar and the lexicon. The XP phrase type is taken from X-bar syntax, which attempts to model common properties between the different syntactic components of noun, verb, adjective and preposition phrase etc. Instead of building a different grammar rule for each different type of phrase, X-bar syntax generalizes out to a single rule that applies universally to all. The system involves lots of pre-processing and much of it is manual, although the authors argue that the cost, in time, is still worth the saving. PERFORMANCE: The authors of the ETS I system claim an accuracy of 80% when marking the essay test set and 90% when marking both the training and test set, i.e. using the training the system on a set of essays and then including it in the test set for the marking part as well. The most common source for errors is due to gaps in the lexicon, i.e. to words not manually identified as metonyms. In a second reported experiment, the authors adopted an augmented lexicon. Constructing this new lexicon involved examining the test set as well as training set to manually place metonyms. This achieved accuracy of 93% when marking on the test set and of 96% when marking both the training and test sets. E-Rater was developed by Burstein and others (Burstein, Kukich, Wolff, Chi, & Chodorow, 1998; Burstein, Leacock, & Swartz, 2001 ). E-Rater uses the MsNLP tool for parsing all sentences in the essay. E-Rater uses a combination of statistical and NLP techniques to extract linguistic features from the essays to be graded. Essays are evaluated against a benchmark set of human graded essays. With E-Rater, an essay that stays on the topic of the question, has a strong, coherent and well-organized argument structure, and displays a variety of word use and syntactic structure will receive a score at the higher end of a six-point scale. E-Rater features include the analysis of the discourse structure, of the syntactic structure and of the vocabulary usage (domain analysis). E-Rater adopts a corpus-based approach to model building by using actual essay data to analyse the features of a sample of essay responses. The application is designed to identify features in the text that reflect writing qualities specified in human reader scoring criteria and is currently composed by five main independent modules. Three of the modules identify features that may be used as scoring guide criteria for the syntactic variety, the organization of ideas and the vocabulary usage of an essay. A fourth independent module is used to select and weigh predictive features for essay scoring. Finally, the last module is used to compute the final score. E-Rater is currently embedded in Criterion a web-based real-time version of the system developed by ETS Technologies. A further feedback component with advisory features has been added to the system. The advisories are based on statistical measures and are completely independent from the score generated by E-Rater, thus providing additional feedback about qualities of writing related to topic and fluency only. E-Rater is trained on a collection of 270 essays that have been manually scored by trained human raters. E-Rater is far more complex and requires more training than many other available systems. Furthermore, no on-line demonstration and no downloadable trial version of E-Rater have been made available to the scientific community. PERFORMANCE: Over 750000 GMAT essays have been scored, with agreement rates between human expert and system consistently above 97%. By comparing human and E-Rater grades across 15 test questions, the empirical results range from 87% to 94%. C-rater is a NLP based prototype aimed at the assessment of short answers related to content-based questions such as those that may appear in a textbook's chapter review section (Burstein et al., 2001) . Crater adopts many of the some natural language processing tools and techniques developed for E-Rater, even if the two systems differ in many important ways. E-Rater assigns a score for writing skills rather than for specific-content while C-rater is aimed to score a response as being either correct or incorrect. This goal is achieved by evaluating whether a response contains information related to specific domain concepts; if the response expresses these concepts it is rated as correct, otherwise it is rated as incorrect without any regard to writing skills. Moreover, E-Rater provides grades that are partly based on the rhetorical structure of an essay while Crater needs to identify specific content. It generates a fine-grained analysis of the logical relations between the syntactic components of each sentence appearing in the response. C-rater does not require a large collection of graded answers for training. Instead, it uses the single correct answer that is found in an instructor's guide or answer key, because it is believed unrealistic to require extensive data collection for the purpose of grading relatively low stakes quizzes, especially given that a set of short questions is often provided at the end of chapters in a textbook. PERFORMANCE: C-Rater achieved over 80% agreement with the score assigned by an instructor. BETSY is a program that classifies text based on trained material and is being developed by Lawrence M. Rudner at the College Park of the University of Maryland with funds from the U.S. Department of Education (http://ericae.net/betsy). According to Rudner and Liang (2002) the goal of the system is to determine the most likely classification of an essay into a four point nominal scale (e.g. extensive, essential, partial, unsatisfactory) using a large set of features including both content and style specific issues. The underlying models for text classification adopted are the Multivariate Bernoulli Model (MBM) and the Bernoulli Model (BM). With the MBM each essay is viewed as a special case of all the calibrated features, and the probability of each score for a given essay is computed as the product of the probabilities of the features contained in the essay. With the BM the conditional probability of presence of each feature is estimated by the proportion of essays within each category that contain the feature. This model can require a long time to compute since every term in the vocabulary needs to be examined. Both the Bernoulli Model and the Multivariate Bernoulli Model are considered naive Bayes models because they assume conditional independence. According to its authors BETSY relies on an approach that may incorporate the best features of PEG, LSA (Landauer, Foltz, & Laham, 1998 ) and E-rater "plus it has several crucial advantages of its own. It can be employed on short essays, is simple to implement, can be applied to a wide range of content areas, can be used to yield diagnostic results, can be adapted to yield classification on multiple skills, and is easy to explain to non statisticians" (Rudner & Liang, 2002) . BETSY is a Windows-based program written in Power Basic, and is computationally intensive. Moreover, BETSY is the only software in the field that ma y be freely downloadable and useable. PERFORMANCE: Rudner and Liang (2002) report about two text classification models that were calibrated using 462 essays with two score points. The calibrated systems were then applied to 80 new prescored essays, with 40 essays in each score group. An accuracy of over 80% was achieved with the described dataset. IEMS is based on the Pattern Indexing Neural Network (the Indextron) developed at NGEE ANN Polytechnic (Ming, Mikhailov, & Kuan, 2000) . The system can be used both as an assessment tools and for diagnostic and tutoring purposes in many content-based subjects. Students can be given immediate feedback and can learn where and why they had done well or not made the grade. Thus it can be embedded in an intelligent tutoring system that will help students to write better by grading papers fast and providing the feedback quickly. The essay grading is based on qualitative type of questions rather than numerical type. Indextron is defined as a specific clusterisation algorithm. In itself, such an algorithm is not a neural network. However, the clusterisation algorithm can be implemented as a neural network. The Indextron-based neural network attempts to overcome a slow, nonincremental training, which is typical of traditional Artificial Neural Networks. PERFORMANCE: According to Ming et al. (2000) an experiment involving the evaluation of the essays produced by 85 students doing a module on Project Report Writing and coming from six classes of third-year Mechanical Engineering, obtained a correlation of 0.8. Automark is a software system developed in pursuit of robust computerized marking of free-text answer to open-ended questions (Mitchell, Russel, Broomhead, & Aldridge 2002) . Automark employs NLP techniques to mark open-ended responses. This software has been under development for almost three years, and has been employed in a commercial e-Learning product for the last months (ExamOnline as cited in Mitchell et al., 2002) . Computer Assisted Assessment procedures based on Automark are currently being developed at a number of higher education establishments, including Brunel University where an online Java test for first year engineering student is under developed. The system incorporates a number of processing modules specifically aimed at providing robust marking in the face of errors in spelling, typing, syntax and semantics. Automark looks for specific content within free-text answers, the content being specified in the form of a number of marking scheme templates. Each template represents one form of a valid or a specifically invalid answer. Development of the templates in the computerized marking scheme is an offline process, achieved through a custom-written system configuration interface. The representation of the templates is robust enough to cope with multiple variations of the input text. The marking process progresses through a number of stages. First, the incoming text is pre-processed to standardise the input in terms of punctuation and spelling. Then, a sentence analyser identifies the main syntactic constituents of the text and how they are related. The pattern-matching module searches for matches between the marking scheme templates and the syntactic constituents of the student text. Finally, the feedback module processes the result of the pattern match. Feedback is typically provided as a mark, but more specific feedback is claimed to be possible. PERFORMANCE: Automark has been tested on National Curriculum Assessment of Science for eleven years old pupils. The form of response was: single word generation, single value generation, generation of a short explanatory sentence, description of a pattern in data. The correlation achieved ranged between 93% and 96%. SEAR is a software system developed by Christie (1999) as a result of his PhD research work. According to Christie, the automated grading of essays requires the assessment both of style and content (where appropriate). Thus, the system provides an expandable, flexible method for the automated marking of the essay content and provides a method for the automated marking of the essay style too. The methodology adopted to assess style is based on a set of common metrics, and requires some initial calibration. In essence the computer-based marking of the style is based on pre-determining what would be candidate metrics, use a subset of essays as training set and mark them manually. Then a process of calibration is started by adjusting the weight of each metric until an acceptable agreement between human and computer marking is achieved, processing the whole essay set. For content assessment only those essay that are technical are candidates for this type of marking (they lead to a bounded spectrum of content) are taken into consideration. For SEAR the content schema is prepared once and is revised fairly quickly and easily. Further, the SEAR content schema requires neither 'training' nor 'calibration', although the usual practice of taking a sample to verify the method would be recommended. The schema is held as a simple data structure. Two measures have been devised to aid the automated marking process: usage and coverage. The former aimed to measure how much of each essay is used, while the latter measures of how much of the essay schema is used by the essay under examination. Both measures were devised to envision the relationship between each essay and the schema. PERFORMANCE: SEAR is still a work in progress. At this point, no performance indicators are available yet. PS-ME is designed as an integrated component of a Web-based Learning Management System (Mason & Grove-Stephenson, 2002) and is still under development. Due to its processing requirements, the PS-ME does not grade essays in real-time. This system applies NLP techniques to assess student essays in order to reveal their level of compete ncies as for knowledge, understanding and evaluation. The student essay is submitted to the server, together with information about the task in order to identify the correct master texts for comparison. Each task is defined via a number of master texts that are relevant to the question to be answered. An interesting issue is raised by the existence of 'negative' master texts containing a set of false statement composed using typical student mistakes and misconception. The essay to be rated is compared against each relevant master text to derive a number of parameters reflecting the knowledge and understanding exhibited by the student. The ability to evaluate parameter is calculated through a linguistic analysis as described above. When multiple master texts are involved in the comparison, each result from an individual comparison gets a weighting, which could be negative in the case of a master text containing common misconceptions. The weights are derived during the initial training phase. The individual parameters computed during the analysis phase are then combined in a numerical expression to yield the assignment's grade (typically a National Curriculum grade or a GCSE level). The parameters are also used to select specific comments from a comment bank relevant to the task. With a fine-grained set-up it is possible to provide the student with formative feedback regarding his/hers perOverview of Current Research on Automated Essay Grading 326 formance in different areas within a given subject. The output from the marking process is then passed back to the client for presentation to the teacher. This includes details on sections of essay that are particularly good or bad in relation to the knowledge, understanding and evaluation factors. The process of setting up the auto marker for a particular task is very straightforward: select master text, from a number of sources such as textbooks, encyclopedias or relevant websites (the system is highly tolerant of duplication of content between master texts, but can lose accuracy if the master texts use extremely complex grammar); have a sample hand-marked (can be as few as 30, this needs to be done once only per task, in order to derive the right weightings for the parameter values computed by the marking system); run the same sample through the marker and perform regression analysis, which tries to get a best fit between the grades given by the marker and those resulting from the combination of the parameters; upload the resulting data to the server. PERFORMANCE: The PS-ME is still a work in progress. Thus, no performance indicators are available yet.
Applications include essay grading for standardized tests such as for the GMAT (Graduate Management Admission Test) or TOEFL (Test of English as Foreign Language) REF .
An Overview of Current Research on Automated Essay Grading
JITE
JITE
Computer Science