Multi-Error Modeling and Fluency-Targeted Pre-training for C hinese Essay Evaluation (2024)

Abstract: “This system report presents our approaches and results for the Chinese Essay Fluency Evaluation (CEFE) task at CCL-2024. For Track 1, we optimized predictions for challenging fine-grained error types using binary classification models and trained coarse-grained models on the Chinese Learner 4W corpus. In Track 2, we enhanced performance by constructing a pseudo-dataset with multiple error types per sentence. For Track 3, where we achieved first place, we generated fluency-rated pseudo-data via back-translation for pretraining and used an NSP-based strategy with Symmetric Cross Entropy loss to capture context and mitigate long dependencies. Our methods effectively address key challenges in Chinese Essay Fluency Evaluation.”

## introduction with the growing integration of smart education and deep learning technologies, automated text evaluation systems have become increasingly critical. these systems aim to accurately and efficiently assess students' compositions, reduce teachers' workload, and provide instant feedback for error correction and writing improvement. the china national conference on computational linguistics (ccl-2024) has presented the chinese essay fluency evaluation (cefe) as a public assessment task. this task focuses on three primary text evaluation strategies: error sentence type recognition, error sentence correction, and essay fluency evaluation, offering an in-depth research direction for the field of automatic text evaluation. this system report provides an overview of our work on the cefe evaluation task, highlighting the different strategies employed for each track: the remainder of this report is structured as follows: section 2 presents related research in the field of chinese composition fluency evaluation. sections 3, 4, and 5 detail our methodologies, experiments, and results for tracks 1, 2, and 3, respectively. finally, section 6 concludes the report with an analysis of our findings, discusses the limitations of our work, and potential future research directions. our code and data are available at https://github.com/astro-jon/ccl2024-coherence . ## related research the field of chinese composition fluency evaluation has gained significant attention from researchers in recent years, with the three different track directions involved in this review being popular topics for related research. ## conclusions this report presented our approaches and results for the three tracks of the chinese essay fluency evaluation (cefe) task at ccl-2024. for track 1, we employed a hierarchical method combining token-level error identification, coarse-grained modeling on the chinese learner 4w corpus, fine-grained binary models, and prediction fusion to handle both broad and specific error types. in track 2, constructing pseudo-data with multiple error types per sentence improved performance in correcting real-world sentences compared to previous single-error methods. our track 3 approach, which achieved first place, utilized back-translated pseudo-data with triple fluency labels, an nsp-based strategy to incorporate context while mitigating long sequence issues, and symmetric cross entropy loss for increased robustness. by addressing challenges such as limited data, error diversity, long-range dependencies, and label noise, our methods contribute to advancing intelligent assessment of chinese essays. potential future directions include cross-lingual generalization, few-shot learning to reduce annotation requirements, and generating more detailed feedback to further enhance student learning. repetition strategy may lead to overfitting. therefore, we selected the 4w corpus with the uniform allocation strategy (i.e. 4w/repetitive) as the best solution for track1 coarse-grained model training. based on the f1 scores of the two strategies, we decided to train with the second strategy of variant error sentence, which involves using a corpus that consists of both specified error types and a variety of other error types.