C hinese Essay Rhetoric Recognition and Understanding ( CERRU ) (2024)

Abstract: “Rhetoric is fundamental to the reading comprehension and writing skills of primary and middle school students. However, current work independently recognize single coarse-grained categories or fine-grained categories. In this paper, we propose the CCL24-Eval Task6: Chinese Essay Rhetoric Recognition and Understanding (CERRU), consisting of 3 tracks: (1) Fine-grained Form-level Categories Recognition, (2) Fine-grained Content-level Categories Recognition and (3) Rhetorical Component Extraction. A total of 32 teams registered to participate in CERRU and 9 teams submitted evaluation results, with 7 of these teams achieving an overall score that surpassed the baseline.”

## introduction in the learning process of primary and middle school students, rhetoric is not only a core component of reading comprehension and writing skills, but also an indispensable element in shaping excellent literary works. recognizing and understanding the use of rhetoric in students' essays can help improve their expressive abilities in writing. however, this requires a significant amount of manual effort, posing challenges to teachers in term of essay assessment and instruction. with the development of education and the widespread availability of the internet, many researchers have begun to explore the use of computer technology for automatic grading of essays @xcite , where the use of rhetoric is a crucial part of teachers' essay grading. the use of rhetoric in essays reflects the level of literacy grace and language expression ability @xcite , which is significant for helping teachers assess the quality of essays and guide students in improving their expressive skills. in recent years, research on the recognition of rhetoric in essays often employs alignment strategies and other rules to perform coarse-grained recognition of rhetoric such as parallelism and metaphor from the perspectives of sentence structure and semantic information @xcite or designs model structures specifically to recognize simile @xcite . these efforts independently recognize different major rhetorical categories such as metaphor, personification, hyperbole and parallelism, lacking universality. on the other hand, they are coarse-grained and lack fine-grained definitions of rhetorical categories. furthermore, beyond recognizing rhetorical categories in sentences, some researches treat the understanding of rhetoric as a component extraction task, for example, extracting the tenor and the vehicle from metaphorical sentences @xcite . these researches lack definitions for the rhetorical subjects and contents of other rhetorical devices, and thus cannot provide systematic and comprehensive guidance and feedback on the essays of elementary and middle school students. therefore, to address the aforementioned challenges, we propose the ccl24-eval task6: chinese essay rhetoric recognition and understanding (cerru). the dataset for the evaluation originates from examination essays written by elementary and middle school students whose native language is chinese. the genres of these essays include narrative and argumentative writing, among others. our task settings systematically define the fine-grained rhetorical categories found in these essays, recognizing them from both form level and content level based on the linguistic definitions of rhetoric @xcite . furthermore, we define the subjects and contents of each rhetorical category, which aids teachers in understanding the use of rhetoric at the sentence level in student essays. it also supports elementary and middle school students in practicing appropriate rhetorical techniques in their writing. cerru categorizes rhetorical devices into metaphor, personification, hyperbole and parallelism, and further subdivides these four rhetorical categories into fine-grained categories. as shown in figure 1 track 1 rhetorical category: metaphor rhetorical category: simile (coarse-grained) (fine-grained, form-level) track 2 rhetorical category: metaphor rhetorical category: concrete (coarse-grained) (fine-grained, content-level) ## evaluation metrics in this section, we introduce the metrics used in cerru. f 1 refers to macro-f1 score in track1, track2 and track3. the overall score of cerru is the arithmetic mean of track1, track2 and track3. ## baselines in this section, we introduce the baseline approaches used in cerru and the scores on track1, track2 and track3. for track1 and track2, we take both the tasks as multi-label classification problems and fine-tune roberta 1 @xcite on the training set. a dropout @xcite layer and a linear layer are concatenated to roberta, and the output after applying sigmoid function is used to represent the probabilities of each category in the given sentence. for track3, we take the task as named entity recognition and fine-tune roberta on the training set. a dropout layer and a linear layer are concatenated to roberta. furthermore, we utilize the iob tagging format @xcite to tag the comparator, tenor, vehicle, personification object, personification content, hyperbole object, hyperbole content and parallelism marker. the output from roberta after applying argmax function is represented as an entity tag on each token. subsequently, the consecutive "b-" prefix tag and "i-" prefix tag are combined to represent the corresponding rhetorical components. ## results in this section, we first discuss the overall results, including the statistics of the participating teams and their scores on each track (see section 6.1). considering the correlation between different tracks, most of the teams choose to combine the dataset from different tracks for joint training. therefore, we then discuss the approaches they use respectively (see section 6.2 -section 6.6). finally, an overall analysis will be discussed in section 6.7. ## conclusion in this paper, we propose the ccl24-eval task6: chinese essay rhetoric recognition and understanding (cerru), consisting of 3 tracks: (1) fine-grained form-level categories recognition, (2) fine-grained content-level categories recognition and (3) rhetorical component extraction. a total of 32 teams registered to participate in cerru and 9 teams submitted evaluation results and obtained valid scores. furthermore, we discuss the approaches used by the top 5 teams based on their overall scores. the results demonstrate that the usage of llms and data augmentation methods help improve the overall scores.