C hinese Essay Fluency Evaluation ( CEFE ) Task (2024)

Abstract: “This paper presents a detailed review of Task 7 in the CCL24-Eval: the second Chinese Essay Fluency Evaluation (CEFE). The task aims to identify fine-grained grammatical errors that impair readability and coherence in essays authored by Chinese primary and secondary school students, evaluate the essays’ fluency levels, and recommend corrections to improve their written fluency. The evaluation comprises three tracks: (1) Coarse-grained and fine-grained error identification; (2) Error sentence rewriting; and (3) Essay Fluency Level Recognition. We garnered 29 completed registrations, resulting in 180 submissions from 10 dedicated teams. The paper discusses the submissions and analyzes the results from all participating teams.”

## introduction education is an enduring and evolving journey, continuously reshaping itself, particularly in the wake of the internet's proliferation and the development of large language models (llms) @xcite , which has also significantly expanded the scope of chinese essay evaluation. the marked increase in the volume of essays requiring evaluation has highlighted concerns regarding the cost-effectiveness and efficiency of manual essay corrections, positioning these issues as salient factors in contemporary educational methodologies. in light of these challenges, a growing number of scholars and educational institutions have initiated investigations into the feasibility of utilizing computer technologies for automated essay correction (aec) @xcite . these relative methods fulfill a twofold purpose. firstly, it facilitates the provision of objective, precise, and timely feedback by analyzing various dimensions of an essay, such as language, content, and structure, and addressing inherent writing challenges. this, in turn, potentially enhances students' comprehension of their writing difficulties, thereby improving their overall writing competencies. secondly, it enables educators to more accurately assess students' writing proficiency and offer more focused instructional support, furthering the educational advancement of students. in practical educational settings, a critical component that teachers assess during essay evaluation is the fluency of expression. this aspect reflects the essay's coherence and grammatical accuracy, offering insights into the writer's proficiency and ability to convey ideas effectively. improving fluency is essential for enhancing the accuracy of essay evaluations and raising the authors' writing standards. nevertheless, the current evaluation of essay fluency at primary and secondary education levels faces significant challenges: 1) lack of detailed criteria most existing assessments focus broadly on overall essay quality without delving deeply into fluency. there is a noticeable absence of systematic criteria, which hampers a thorough understanding and development of students' writing skills. 2) limited interpretability previous studies often approach fluency as merely a scoring endeavor, providing only an aggregate score or rating. alternatively, they treat it as a basic grammatical error correction (gec) task @xcite . such approaches predominantly target simple grammatical mistakes, reviewing them through simple corrections like additions, deletions, and modifications, which limits comprehensive feedback on the text's flow and structure. however, these methodologies typically overlook the analysis of specific grammatical error types and fail to specify the exact nature of the errors encountered. providing students with detailed descriptions of error types and suggested corrections is beneficial as it enables them to recognize their mistakes, refine their essays, and prevent the recurrence of these errors in future writing. 3) scarcity of authentic data from primary and secondary educational contexts there is a notable shortage of public datasets for studying essay fluency among chinese primary and secondary school students. previous research relying on gec often utilizes rule-based or inter-language datasets from learners of chinese as a second language. however, the errors in compositions written by native chinese students are more diverse and require a deeper understanding of complex grammatical structures. figure 1 presents examples of sentences taken from essays written by primary and secondary school students. these excerpts showcase various errors along with recommended corrections. typically, each sentence contains multiple types of errors that extend beyond simple spelling mistakes, illustrating the complexity of issues found in student writing. english transla¡on chinese sentence figure 1 : an example of our task. in modern chinese, adverbials are typically positioned between the subject and the predicate rather than at the end of the sentence, thereby leading to an 'inappropriate word order' error. moreover, in the first two short sentences, there is a problem of 'subject redundancy' where the subject 'i' is repeated unnecessarily. these shortcomings in current methodologies highlight the need for a more detailed and nuanced approach that not only identifies fine-grained errors but also delivers specific, actionable feedback to students. it underscores the importance of utilizing composition data from the authentic writing contexts of primary and secondary school students to better address their specific learning needs. motivated by this, we present the ccl24-eval task 0 : chinese essay fluency evaluation (cefe), which aims to identify and correct errors that impede the fluency of writing in essays by primary and secondary school students and to assess the overall fluency level of an essay. compared to task 8 of the ccl23-eval @xcite , we have introduced a new track and removed an original one, taking into account their practical application and relevance. this task featured three tracks: (1) coarse-grained and finegrained error identification; (2) error sentence rewriting; (3) essay fluency level recognition, aiming at providing a higher-quality evaluation of fluency in primary and secondary school essays. this task attracted 29 teams to sign up for the competition, and in the end, we received 180 submissions from 10 teams. the task description is presented in section 2. we describe the data we used in this task in section 3. we explain baselines used for each track and list participants' information and results from their submissions and provide a more in-depth discussion in section 5. ## task description our evaluation is structured into three distinct tracks, each crafted to tackle specific aspects of identifying and correcting errors in essays written by primary and secondary school students. these tasks are designed to shed light on the common types of errors these students make, offering a basis for focused enhancements in their writing skills. 0 https://github.com/cubenlp/2024ccl_cefe c c l 2 0 2 4 ## datasets in an effort to enhance research on essay fluency among primary and secondary school students, we meticulously annotated a dataset with fine-grained grammatical error types and provided corresponding corrections that impact sentence fluency. we developed the second chinese essay fluency evaluation (cefe 2.0) dataset based on cefe 1.0 dataset @xcite , aiming to offer in-depth insights into the typical grammatical errors students make. this fine-grained dataset not only helps in identifying common mistakes but also supports the development of targeted teaching strategies to improve writing skills in young learners. ## evaluation metrics different evaluation metrics are utilized across the various tracks of the task; however, the calculations for precision and recall remain consistent throughout. precision is calculated as the ratio of correctly identified instances to the total number of instances identified by the model. conversely, recall is calculated as the ratio of correctly identified instances to the total number of instances labeled in the ground truth. the f1-score, frequently employed in binary or multi-class classification tasks, represents the harmonic mean of precision and recall, and is computed using the formula: f 1 = 2p r p +r . ## participant systems in this task, the competing teams employed a variety of approaches to detect and correct errors in student essays and to assign grades based on fluency. this section will provide an overview of the methods that were successful in each track. the unique approaches of each team demonstrate the diversity of methods that can be exploited in automated essay scoring and present various potential directions for future research.