Towards Explainable C hinese Native Learner Essay Fluency Assessment: Dataset, Tasks, and Method (2024)

Abstract: Grammatical Error Correction (GEC) is a crucial technique in Automated Essay Scoring (AES) for evaluating the fluency of essays. However, in Chinese, existing GEC datasets often fail to consider the importance of specific grammatical error types within compositional scenarios, lack research on data collected from native Chinese speakers, and largely overlook cross-sentence grammatical errors. Furthermore, the measurement of the overall fluency of an essay is often overlooked. To address these issues, we present CEFA (Chinese Essay Fluency Assessment), an extensive corpus that is derived from essays authored by native Chinese-speaking primary and secondary students and encapsulates essay fluency scores along with both coarse and fine-grained grammatical error types and corrections. Experiments employing various benchmark models on CEFA substantiate the challenge of our dataset. Our findings further highlight the significance of fine-grained annotations in fluency assessment and the mutually beneficial relationship between error types and corrections

## introduction essay fluency refers to the coherence of a sentence or a whole composition, as well as grammatical accuracy @xcite , serving as a foundational component in automated essay scoring (aes). the study of essay fluency has significant applications in fields such as education @xcite ), text generation @xcite and publishing @xcite . recent advancements in aes have integrated grammatical error correction (gec) to improve explainability @xcite , with gec focusing on automatic text error correction @xcite . in chinese aes, the prevalent chinese gec (cgec) categorizes errors into four modification types @xcite and make corrections. subsequently, an overall score of the essay is conducted based on the errors and other linguistic features. this method, while adding some explainability to the scoring process, offers limited insights for students seeking to understand complex grammatical rules. additionally, it lacks a distinct fluency score to assess the specific impact of grammatical errors on essay fluency and the overall level of fluency in the essay, which is a crucial component in essay grading. the existing cgec dataset is not directly applicable for assessing essay fluency. primarily, most cgec methods rely on corpora from chinese-asa-second-language (csl) learners, who are more prone to lexical confusion errors, such as confusing "关爱" (care and love) and "爱情" (romantic love), both translated as "love" in english @xcite , which is seldom seen among native speakers. additionally, existing corpora often derive from online texts, which typically do not adhere to language usage norms and grammars. moreover, the definition of error types is not sufficiently detailed. recent datasets either predominantly focus on orthographic errors like typos @xcite , or solely target syntactic errors like constituent omissions @xcite , which lacks comprehensiveness and diversity. lastly, existing datasets lack annotations for cross-sentence errors @xcite , which are common in documents, as illustrated in figure 1(c ) error 1. to tackle the issues, we propose an detailed assessment guideline for automatic essay assessment in fluency and developed the chinese essay fluency assessment (cefa) corpus. this dataset addresses limitations in prior work: firstly, it simultaneously annotates essay fluency grades, grammatical error types and the corrected sentences, which facilitates a comprehensive and detailed evaluation of the essay in fluency. secondly, it encompasses 5 coarse-grained and 18 fine-grained grammatical error types, providing a basis for scoring and correction, and offering teachers and students precise insights into writing issues and targeted feedback. finally, it originates from essays written by native primary and secondary school students, encompassing a diverse range of topics, genres, and score ranges, and annotates errors from documentlevel perspectives, which is especially beneficial for a more in-depth study of cgec. to assess the complexity of our cefa dataset, we explored several baseline models and large language models (llms) on our dataset. the results show that our dataset is challenging. furthermore, we investigated the impact of detailed annotation on fluency grading, as well as the mutually benefits between grammatical error types and corrections through experiments. the findings emphasize the importance of fine-grained annotations and the strong mutual benefit between error types and corrections. we summarize our contributions as follows: ## discussion we explored the importance of fine-grained annotations. specifically, we studied the significance of grammatical errors for fluency grading and the mutually beneficial relationship between grammatical error types and corrections. additionally, we also discussed the significance of studying crosssentence errors. ## conclusion we present cefa, a comprehensive dataset derived from native chinese student essays. it captures document-level errors, fluency grading, and fine-grained grammatical error details, advancing the field of automated essay fluency assessment. through experiments using popular existing models, we have demonstrated the challenging nature of our work. furthermore, we have validated the importance of fine-grained annotation for fluency rating of compositions and the mutually beneficial relationship between error types and corrections.