Automated A rabic Essay Evaluation (2020)

Abstract: Although the manual evaluation of essays is a time-consuming process, writing essays has a significant role in assessing learning outcomes. Therefore, automated essay evaluation represents a solution, especially for schools, universities, and testing companies. Moreover, the existence of such systems overcomes some factors that influence manual evaluation such as the evaluator’s mental state, the disparity between evaluators, and others. In this paper, we propose an Arabic essay evaluation system based on a support vector regression (SVR) model along with a wide range of features including morphological, syntactic, semantic, and discourse features. The system evaluates essays according to five criteria: spelling, essay structure, coherence level, style, and punctuation marks, without the need for domain-representative essays (a model essay). A specific model is developed for each criterion; thus, the overall evaluation of the essay is a combination of the previous criteria results. We develop our dataset based on essays written by university students and journalists whose native language is Arabic. The dataset is then evaluated by experts. The experimental results show that 96% of our dataset is correctly evaluated in the overall score and the correlation between the system and the experts’ evaluation is 0.87. Additionally, the system shows variant results in evaluating criteria separately.

## introduction automated essay scoring (aes) or so-called automated essay evaluation (aee) systems came to facilitate the evaluation task of students' writings. assigning questions that demonstrate writing skills, such as linguistic skills and creativity, is crucial. however, the evaluation process is laborious, particularly with a large number of essays, such as during examination boards. aes systems assist essay authors, editorial boards, publishers, and newspaper editors by overcoming some of the shortcomings of traditional evaluation. for instance, aes systems reduce variabilities between instructors' viewpoints or biases resulting from a good point in essay that causes an evaluator to ignore other mistakes @xcite . moreover, aes systems are tools that can assist both new instructors and students for training and improving writing skills. for the english language, there have been many studies drafted on aes in addition to the development of commercial applications used in english-learning institutes. in comparison, aes systems for arabic seem restricted to short-answer questions with predefined answer models from instructors. this limitation stems from complexities of the arabic language and a lack of arabic natural language processing (nlp) resources. in this paper, we attempt to fill this gap by providing an arabic essay evaluation system using a machine learning method that does not require a model essay. ## related work in english, there are several related efforts. for example, the project essay grader (peg) is one of the earliest scoring systems. the system was based on a statistical method to predict the score. peg succeeded in predicting the surface structure of an essay, but it did not meet the semantic criteria @xcite . in the intelligent essay assessor (iea), latent semantic analysis (lsa) is used to evaluate essay content. in addition, iea can be used to evaluate writing style and detect plagiarism (dessus and @xcite . the e-rater @xcite and intellimetric @xcite are evaluation systems that rely on linguistic features extracted by nlp techniques to evaluate common criteria. in contrast to previous systems, the bayesian essay test scoring system (betsy) is noncommercial and can be used for research. betsy uses bayesian models to evaluate essays for content and style @xcite . although commercial systems have restrictions to accessing system details, aes systems have attracted the attention of many researchers. most studies, such as @xcite , have focused on the automatic student assessment prize (asap) dataset. @xcite used machine learning with nine surfaces, deep features, and three algorithms including support vector machine (svm), knearest neighbors (knn), and linear regression. the obtained accuracy ranges from 73% to 93% according to the dataset class and the algorithms used. in addition, @xcite and @xcite have employed deep-learning algorithms and obtained encouraging results. in arabic, there are several studies for shortanswer questions scoring. @xcite based on light stemming and assigning weights over words in the model answer. for the system to consider semantics, the instructor must manually attach the synonyms. in comparison, gomaa and fahmy (2014) combined similarity measures, such as string, corpus, and knowledge-based similarity for 610 arabic short answers, which were translated to english due to the lack of arabic resources. however, this approach requires great effort for translation and then scoring. apart from semantics, al-shalabi (2016) proposed a scoring system for online exams using stemming and levenshtein string-based similarity measures. @xcite conducted a comparison between several stringbased (damerau-levenshtein and n-gram) and corpus-based similarity measures (lsa and disco) on 210 answers. they found that applying n-gram with removing stop words produced the best results. other than short-answer scoring, we only came across three studies for essay scoring @xcite @xcite . first, @xcite conducted a study based on a linear regression algorithm. they used lsa, the number of words, and spelling mistakes to predict an essay's score. the proposed system was applied to 579 essays collected from undergraduate university students and scored by two instructors. the results showed that 96.72% of essays were scored correctly, while the correlation between the system and manual scoring was 0.78, which is close to the value of 0.7 obtained by interhuman correlation. second, azmi, al-jouie, and hussain (2019) collected 360 essays from students in middle and high schools. the essays were evaluated by two school instructors according to criteria obtained from a questionnaire given to instructors. the criteria can be presented as semantic analysis, writing style, and spelling mistakes. however, there was a difference in assigning each criterion weight, but the majority was 5%, 40%, and 10% for semantic analysis, writing style and spelling mistakes, respectively. the proposed system was based on lsa to evaluate semantics while rhetorical structure theory (rst) and other features were used to evaluate the writing style. finally, the system employed aracomlex @xcite to detect spelling mistakes. the system achieved an accuracy of 90% while the correlation to the manual evaluation was 0.756, which outperformed the inter-human correlation of 0.709. both @xcite and azmi, al-jouie, and hussain (2019) relied on the lsa approach, which requires domain-representative essays. furthermore, outputs of @xcite are given as an overall score without details about the score for each criterion. in contrast, @xcite employ rules to evaluate writing style and spelling mistakes separately. however, half of the assigned score is based on lsa, which in turn requires pre-defined models. recently, @xcite proposed a rule-based system to evaluate arabic essays without the need to train on domain-representative essays. they adopted an evaluation criteria scheme based on arabic literary resources and university instructors' experiences as following: spelling, grammar, structure, cohesion, style and punctuation marks. then, scheme was given to two experts to evaluate 100 essays for university students. the system follows a set of rules to evaluate the previous criteria based on some facts and analyses to evaluate the overall score beside the specific criteria scores. the system accuracy was 73% in the overall score while there were variations in evaluating criteria separately. although this study does need a model essay, it is limited to set of rules that does not include semantic. therefore, in this paper, we propose an arabic essay evaluation system does not need to train on predefined essays represent domain by using a machine learning algorithm and a wide range of features to evaluate the specific criteria besides the overall score. ## dataset our dataset can be classified into three parts: essays written by undergraduate/graduate arabic native students with university-level; unedited essays written to be published in one of the saudi newspapers and essays have been published in different newspapers. any handwritten copies have been retyped by computer exactly as they were written. altogether, our dataset contains 200 (msa) essays with an approximately average length of 250 words (3kb) in several topics. this dataset has been given to two arabic experts hold master's degree in arabic language to evaluate essays following the criteria and the evaluation rubric shown precisely in @xcite which include: spelling, grammar, structure, coherence and cohesion level, style and punctuation marks. therefore, in spelling criterion, evaluators concern about the correct spell of words; therefore, they detect each spelling mistake and classify it to one of four types: mistakes on ‫,>ء/ھﻤﺰة/‪<hmzp‬‬ replacement letters, extra letters and neglecting letters. for structure criterion, they check the existence of four essential parts; title, introduction, body and conclusion. in coherence, they evaluate the coherence between the title and remaining parts and check the cohesion between essay parts. this criterion also concerns with the correct use of connectives. to evaluate style of essay, they consider, words repetition, length of sentences, word choice and avoiding lengthy speech. in addition, they evaluate the punctuation marks according to the arabic rules. ## automated essay evaluation we intend to model arabic essay evaluation based on the supervised linear model support vector regression (svr) @xcite along with different levels of features. we develop a specific model for each criterion. the overall evaluation of an essay will be a combination of the models' outputs. we follow this procedure to ensure that each criterion model takes the full advantage of the used features hence more accurate in the overall score. in addition, existence of separated model per criterion will assist in the future to provide a valuable feedback for the user. in the preprocessing step, we applied normalization, then stemming using buckwalter stemmer @xcite . our features extracted by considering the criteria followed by humans. according to the system output, which is numeric scores, we try a wide range of features represented by numbers at different levels including: ## experiments and results all experiments in this study were carried out using the weka tool @xcite , based on a tenfold cross-validation for the entire dataset. we built a specific model for each criterion: spelling, coherence, structure, punctuation marks, and style. then, we computed the models' results to predict the overall score. during each model development, we considered the size of our dataset, the appropriate features, and the number of features to achieve the most accurate model with the minimum number of features possible to avoid overfitting problems @xcite . it is worth noting that we only included the effective features in each model rather than including all features. to evaluate the system's performance, we used accuracy "acc," which refers to the number of essays correctly evaluated by the system, as well as pearson's correlation r to measure the relationship and association between manual scores and system scores @xcite . as our dataset contains fractions, and to align with scoring in similar studies @xcite @xcite , we considered a threshold value t in our results. therefore, an essay is considered as correctly evaluated if the difference between the manual score and the system score does not exceed t. @xcite set t to be approximately 17% of the overall score, whereas @xcite set t to be 25% of the essay score. in our case, although our dataset contains many fractional numbers, we will show the results when @xmath0. ## conclusion this paper introduced an arabic essay evaluation system based on the svr algorithm and features from different linguistic levels. separately, we conducted experiments to predict five criteria scores; spelling, structure, coherence, style, and punctuation marks. the essay holistic score was assigned by a combination of the previous criteria scores. the experiments conducted on our dataset consisted of 200 essays. in the overall evaluation, the proposed system achieved 96% accuracy and 0.87 in correlation with manual evaluation, while it achieved 77%, 91%, 87%, 78%, and 93% in accuracy for spelling, structure, coherence, style, and punctuation marks, respectively. in the future, we look forward to expanding our dataset as our system performance improved by increasing the dataset size from 100 essays to 200 essays. furthermore, we intend to involve some criteria that have been annotated by humans but not yet automated, such as grammar. similar studies in foreign languages have had promising results by applying deep learning algorithms while it is unexplored in aes for arabic writing. therefore, we believe it is worth to apply deep learning algorithms utilizing the features extracted in this work.