QAES : First Publicly-Available Trait-Specific Annotations for Automated Scoring of A rabic Essays (2024)

Abstract: Automated Essay Scoring (AES) has emerged as a significant research problem within natural language processing, providing valuable support for educators in assessing student writing skills. In this paper, we introduce QAES, the first publicly available trait-specific annotations for Arabic AES, built on the Qatari Corpus of Argumentative Writing (QCAW). QAES includes a diverse collection of essays in Arabic, each of them annotated with holistic and trait-specific scores, including relevance, organization, vocabulary, style, development, mechanics, and grammar. In total, it comprises 195 Arabic essays (with lengths ranging from 239 to 806 words) across two distinct argumentative writing tasks. We benchmark our dataset against the state-of-the-art English baselines and a feature-based approach. In addition, we discuss the adopted guidelines and the challenges encountered during the annotation process. Finally, we provide insights into potential areas for improvement and future directions in Arabic AES research.

## introduction automated essay scoring (aes) is used to automatically evaluate essays, eliminating the need for human intervention. aes has gained significant importance in educational assessment, offering an efficient way to evaluate written essays promptly. traditionally, aes focused on assigning a single holistic score to an essay, reflecting its overall quality @xcite . this approach simplifies the evaluation by summarizing the performance of the essay. however, relying solely on a uni-dimensional score is insufficient for guiding students on how to improve areas of weakness. consequently, recent research has shifted towards trait-specific scoring, which assigns scores to distinct traits of the essay @xcite . by assessing each trait separately, educators and students gain deeper insight into areas for improvement. although significant progress has been made in aes for languages such as english @xcite , the assessment of arabic essays remains understudied. this is primarily due to the lack of publicly available annotated datasets tailored for arabic essay scoring. to address this challenge, we introduce the qcaw annotations for automated essay scoring (qaes), foot_0 the first freely available annotated arabic essay corpus for trait-specific scoring. we build qaes by annotating the qatari corpus of argumentative writing (qcaw) @xcite across seven traits: relevance, organization, vocabulary, style, development, mechanics, and grammar. we release these annotations to the research community, aiming to foster advancements in arabic aes. we also evaluate the performance of featurebased and state-of-the-art english baselines on the dataset. our efforts help bridge the resource gap and empower educators and researchers with valuable insights into the nuances of arabic writing. the contributions of this paper are four-fold: (1) annotating the arabic essays of the qcaw dataset, providing both holistic and trait-specific scores, (2) releasing qaes, foot_1 the first publiclyavailable trait-specific annotations for arabic aes, (3) providing insights into the annotation challenges and areas for improvement, and (4) evaluating the performance of feature-based and stateof-the-art baselines on qaes. the rest of the paper is organized as follows: section 2 reviews related work on arabic essay datasets. section 3 presents the qcaw dataset considered for annotating. section 4 outlines the annotation process. section 5 provides a detailed analysis of the annotations. section 6 discusses the experimental setup and results. finally, section 7 concludes with future directions. ## related work in automated essay scoring (aes), a significant contrast exists between english and arabic datasets. while english aes benefits from well-established large-scale benchmark datasets, such as the automated student assessment prize (asap) foot_2 dataset, there is a lack of publicly available annotated datasets for arabic aes. this could be attributed to the limited research focus on the arabic language, coupled with its high ambiguity, rich morphology, and complex morpho-syntactic rules. despite these challenges, some initiatives have developed modest arabic essay corpora for preliminary studies. these datasets, summarized in ## the qcaw corpus the qatari corpus of argumentative writing (qcaw) @xcite is a publicly available corpus comprising 195 argumentative essays written in both arabic and english by first-year arabic native university students as part of a compulsory university-level course. it provides a valuable resource for aes as its lengthy essays offer insights into arabic linguistic styles and different writing proficiency levels. furthermore, the essays were collected from a diverse student sample, considering factors such as gender, year of study, and major. in our work, we only consider the arabic essays. these essays constitute responses to two distinct argumentative writing tasks, originally derived from toefl writing prompts: task 1: 'telephones and emails have made communication between people less personal"; and task 2: 'technology has enabled students nowadays to learn more information quickly". ## constructing qaes to create qaes, annotators were hired to annotate the arabic essays of the qcaw dataset across seven traits: relevance (rel), organization (org), vocabulary (voc), style (sty), development (dev), mechanics (mec), and grammar (gra) in addition to a holistic (hol) score of overall quality. the process entails the deployment of trusted assessment guidelines/rubrics, the selec-tion of specialist annotators, and the delivery of indepth training sessions to ensure consistency and understanding throughout the annotation process. ## annotation analysis in this section, we analyze the annotations via two essay examples explaining score assignments, examine the inter-annotator agreement, and study trait distributions across tasks. ## arabic aes: preliminary experiments to explore the potential of qaes to develop arabic aes models, we conducted several preliminary experiments. this section details the baseline models and discusses their performance. ## conclusion and future work in conclusion, qaes represents a pioneering step in the field of arabic aes by introducing the first publicly available trait-specific annotations for arabic essays. our work addresses a critical gap in arabic aes research, providing a valuable resource for exploring and developing advanced essay scoring systems based on multiple traits including relevance, organization, vocabulary, style, development, mechanics, and grammar. benchmarking against english state-of-the-art baselines and a feature-based approach has demonstrated qaes's potential to serve as a platform for future studies. despite the challenges encountered during the annotation process, the insights gained pave the way for enhancing the reliability of arabic aes systems. moving forward, it is imperative to continue refining the annotation guidelines, expanding the dataset, and exploring innovative methodologies to further improve the progress of arabic aes. we believe that the journey towards improving aes for arabic is just beginning, and qaes is a promising step forward in this ongoing quest for automated essay scoring proficiency.