Overview of CCL 23-Eval Task 8: C hinese Essay Fluency Evaluation ( CEFE ) Task (2023)

Abstract: “This paper provides a comprehensive review of the CCL23-Eval Task 8, i.e., Chinese EssayFluency Evaluation (CEFE). The primary aim of this task is to systematically identify the typesof grammatical fine-grained errors that affect the readability and coherence of essays writtenby Chinese primary and secondary school students, and then to suggest suitable corrections toenhance the fluidity of their written expression. This task consists of three distinct tracks: (1)Coarse-grained and fine-grained error identification; (2) Character-level error identification andcorrection; (3) Error sentence rewriting. In the end, we received 44 completed registration forms,leading to a total of 130 submissions from 11 dedicated participating teams. We present theresults of all participants and our analysis of these results. Both the dataset and evaluation toolused in this task are available1.”

## introduction as a life-long and continuous process, education continually evolves and adapts, especially with the widespread of the internet. consequently, the task of student essay assessment has considerably broadened in scale. this significant growth has brought the issues of cost-effectiveness and efficiency in manual essay correction to the forefront, marking them as noteworthy considerations in modern educational practices. in response to this, numerous researchers and institutions have begun exploring the potential of computer technology for automated essay correction @xcite . this initiative serves a dual purpose. first, by analyzing various aspects of an essay, including its language, content, structure, and the challenges that exist within, it allows for the provision of objective, precise, and timely feedback and may equip students with an enriched understanding of their writing challenges, enhancing their overall skills. second, this allows teachers to effectively gauge students' writing proficiency and provide more targeted guidance, thereby advancing the educational development of students. among the key aspects considered during essay correction by teachers is the fluency of expression. the fluency of an essay mirrors the coherence and grammatical correctness of the text, in addition to giving an insight into the author's writing proficiency and expressive capabilities. enhancing this aspect carries significant implications for improving the quality of essay corrections and elevating the writing standards of the authors themselves. however, existing evaluation of essay fluency at primary and secondary levels has the following issues: 1) lack of specifications: current work mainly evaluates essay quality overall, with little in-depth research in fluency and a lack of systematic evaluation specifications, which is not beneficial for comprehensive understanding and improvement of students' writing skills. 2) poor interpretability: prior research typically treats fluency as a scoring task @xcite , providing only an overall rating or score. alternatively, it is treated as a simple grammatical error correction (gec) task @xcite . these studies focus primarily on identifying and rectifying simple grammatical errors in sentences, examining them through the lens of revisions such as additions, deletions, and modifications. however, these approaches often neglect to study the specific types of grammatical errors and do not indicate the particular error type. however, both detailed error types and correction references are helpful to students, enabling them to understand their mistakes, revise their essays, and avoid the same errors in the future. 3) lack of data from authentic writing contexts of primary and secondary school students: public datasets for researching essay fluency among chinese primary and secondary school students are scarce, and previous gec-based research often relies on rule-based or inter-language datasets from chinese language learners. however, the types of errors found in chinese students' compositions are more diverse and involve more complex grammar knowledge. figure 1 provides exemplars of sentences extracted from compositions penned by primary and secondary school students, highlighting their respective errors alongside appropriate corrective suggestions. usually, an individual sentence encompasses multiple categories of errors that go beyond the confines of mere spelling errors. english transla¡on chinese sentence figure 1 : an example of our task. in modern chinese, adverbials are typically positioned between the subject and the predicate rather than at the end of the sentence, thereby leading to an 'inappropriate word order' error. moreover, in the first two short sentences, there is a problem of 'subject redundancy' where the subject 'i' is repeated unnecessarily. these gaps in existing methodologies underscore the necessity for a fine-grained, interpretable approach that not only identifies errors but also provides detailed, actionable feedback for students, and emphasize the importance of using composition data from authentic writing contexts of primary and secondary school students. motivated by this, we present the ccl23-eval task: chinese essay fluency evaluation (cefe), which aims to identify and correct errors that affect the fluency of writing for primary and secondary school students. the task featured three tracks: (1) coarse-grained and fine-grained error identification; (2) character-level error identification and correction; (3) error sentence rewriting, aiming at providing a higher-quality evaluation of fluency in primary and secondary school essays. this task attracted 44 teams to sign up for the competition, and in the end, we received 130 submissions from 11 teams. the task description is presented in section 2. we describe the data we used in this task in section 3. in section 4, we discuss the metrics used to rank participant submissions. we list participants' information and results from their submissions and provide a more in-depth discussion in section 6. in section 7, we introduce the methods of the excellent teams. we finally conclude the paper in section 8. ## task description the mission of our evaluation is categorized into three distinct tracks, each designed to address a specific aspect of identifying and rectifying errors within primary and secondary school compositions. the tasks aim to illuminate the types of errors that students commonly make, thereby providing a foundation for targeted improvements in writing skills. ## datasets in a bid to promote and advance research on essay fluency in primary and secondary school students, we annotated both fine-grained grammatical error types and corresponding correction references that affect sentence fluency. we constructed a fine-grained dataset for chinese essay fluency evaluation (cefe 1.0), which aims to provide meaningful insights into the nature and types of grammatical errors students typically make in their writing. ## evaluation metrics we use different evaluation metrics in different tracks of the task. our precision and recall calculations are the same in all tracks. precision is defined as the ratio of correctly identified instances to the total c c l 2 0 2 3 number of identified instances. recall is defined as the ratio of correctly identified instances to the total number of instances labeled in the ground truth. the f1-score, often used in tasks involving binary or multi-class classification, is the harmonic mean of precision and recall, calculated using the formula: f 1 = 2p r p +r . ## baselines we present the outcomes of our baseline models for reference. the training dataset and test a dataset are utilized for training, and we evaluate the performance of our model on test b dataset. for track 1, we fine-tune bert @xcite on our dataset over 100 epochs, employing batch sizes within the range of @xcite , a learning rate of 2e ## participant systems the participating teams in the task adopted diverse approaches for error detection and correction in primary and secondary school students' essays. this section gives an overview of the methods that have been successful in each of the tracks. each team's unique approach illustrates the diversity of methods that can be utilized in automated essay assessment and presents various potential directions for future research. ## conclusions and future work this paper presents an overview of the ccl23-eval task chinese essay fluency evaluation (cefe). we conduct this evaluation using our meticulously annotated cefe 1.0 dataset. the evaluation is divided into three distinct tracks: (1) coarse-grained and fine-grained error identification; (2) characterlevel error identification and correction; (3) error sentence rewriting. each one aims at addressing a specific facet of grammatical error identification and correction within primary and secondary school compositions. we received a total of 44 completed registration forms, culminating in 130 submissions from 11 participating teams. in addition, we provide a comprehensive analysis and summary of the methodologies employed by the participants, which will contribute to future research in this field of natural language processing. in the future, we will continue to explore methods to improve the identification of fine-grained error types and moderate correction, as well as further investigate the effectiveness of llms in our task.