Essay Rhetoric Recognition and Understanding Using Synthetic Data and Model Ensemble Enhanced Large Language Models (2024)

Abstract: “Natural language processing technology has been widely applied in the field of education. Essay writing serves as a crucial method for evaluating students’ language skills and logical thinking abilities. Rhetoric, an essential component of essay, is also a key reference for assessing writing quality. In the era of large language models (LLMs), applying LLMs to the tasks of automatic classification and extraction of rhetorical devices is of significant importance. In this paper, we fine-tune LLMs with specific instructions to adapt them for the tasks of recognizing and extracting rhetorical devices in essays. To further enhance the performance of LLMs, we experimented with multi-task fine-tuning and expanded the training dataset through synthetic data. Additionally, we explored a model ensemble approach based on label re-inference. Our method achieved a score of 66.29 in Task 6 of the CCL 2024 Eval, Chinese Essay Rhetoric Recognition and Understanding(CERRU), securing the first position.”

## introduction essay writing is a crucial means of assessing students' language proficiency and logical thinking skills. the development and application of natural language processing (nlp) technologies have significantly advanced the field of education. typically, grading essays demands substantial time and effort from teachers. by applying automation technologies to this process, we can alleviate the workload on teachers, allowing them to focus more on instruction and student guidance. rhetorical devices are an essential component of essays, making it necessary to automate the extraction and classification of these devices as a key dimension in the automated assessment of writing quality. previous work in the fields of machine learning and deep learning has extensively explored the identification and classification of rhetorical devices in writing. for instance, @xcite used convolutional neural networks and support vector machines to identify metaphors in both chinese and english datasets. @xcite employed sequential models to recognize metaphors in text, while @xcite adopted a multi-task learning approach to classify rhetorical sentences and extract their rhetorical components. additionally, @xcite utilized bert models to identify metaphors in compositions by overseas chinese students. the recognition and extraction of rhetorical devices can be fundamentally categorized as tasks of text classification and entity recognition, both of which have been extensively studied. the introduction of the gpt series @xcite @xcite has sparked significant interest in llms within the nlp community, marking the advent of the era of large language models @xcite . following the release of the llama series @xcite , the open-source community for llms has flourished. in this new era, some studies have utilized the in-context learning capabilities of large models for text classification @xcite . furthermore, @xcite demonstrated the effectiveness of supervised instruction fine-tuning of large language models for information extraction tasks. against this backdrop, we explored the use of generative large language models for rhetorical recognition and extraction in the ccl24-eval task 6 chinese essay rhetoric recognition and understanding(cerru). for this task, we processed the dataset into fine-tuning instructions and applied parameterefficient fine-tuning methods to the yi @xcite and qwen1.5 @xcite models with supervised instruction fine-tuning. to further enhance model performance, we incorporated multi-task learning methods and augmented the training set with synthetic data generated by llms. finally, we developed a model ensemble approach involving re-inference for hierarchical rhetorical classification tasks. our method achieved a final score of 66.29 in ccl 2024 eval task 6 cerru, ranking 1st. ## chinese essay rhetoric recognition and understanding ccl 2024 eval task 6: chinese essay rhetoric recognition and understanding includes three tracks: track 1: this track classifies rhetorical devices in each sentence at a coarse-grained level into five categories: metaphor, simile, hyperbole, parallelism, and no rhetoric. additionally, each rhetorical category is further classified into subcategories based on form, resulting in a total of 4 coarse-grained categories and 12 fine-grained subcategories. track 2: similar to track 1, this track classifies rhetorical devices in each sentence at a coarse-grained level into the same five categories. however, the fine-grained classification is based on content, resulting in 4 coarse-grained categories and 11 fine-grained subcategories. track 3: this track focuses on identifying rhetorical components within sentences, specifically conjunction, tenor, and vehicle. overall, tracks 1 and 2 fall under hierarchical text classification tasks, while track 3 is an entity extraction task. ## conclusion in this paper, we explored the use of large language models (llms) for the tasks of essay rhetoric recognition and understanding. by employing instruction fine-tuning, multi-task learning, synthetic data augmentation, and a model ensemble method based on label re-inference, we significantly improved the model's performance. ultimately, our approach achieved a score of 66.29 in task 6 of the ccl 2024 evaluation, cerru, securing the first place. this outcome demonstrates the tremendous potential of llms in the automated processing of complex natural language tasks.