TOREE : Evaluating Topic Relevance of Student Essays for C hinese Primary and Middle School Education (2024)

Abstract: Topic relevance of an essay demands that the composition adheres to a clear theme and aligns well with the essay prompt requirements, a critical aspect of essay quality evaluation. However, existing research of Automatic Essay Scoring (AES) for Chinese essays has overlooked topic relevance and lacks detailed feedback, while Automatic Essay Comment Generation (AECG) faces much complexity and difficulty. Additionally, current Large Language Models, including GPT-4, often make incorrect judgments and provide overly impractical feedback when evaluating topic relevance. This paper introduces TOREE ( To pic Re levance E valuation), a comprehensive dataset developed to assess topic relevance in Chinese primary and middle school students’ essays, which is beneficial for AES, AECG and other applications. Moreover, our proposed two-step method utilizes TOREE through a combination of Supervised Fine-tuning and Preference Learning. Experimental results demonstrate that TOREE is of high quality, and our method significantly enhances models’ performance on two designed tasks for topic relevance evaluation, improving both automatic and human evaluations across four diverse LLMs.

## introduction topic relevance of an essay requires that the composition not only revolves around a clear theme but also its content aligns with the requirements of essay prompt, which is a critical aspect in evaluating the quality of an essay, as shown in figure 1 . automating the assessment of topic relevance not only effectively reduces teachers' workload but also efficiently enhances students' writing skills, which holds significant educational implications. current research on automatic essay evaluation predominantly focuses on the development of automatic essay scoring (aes), which aims to determine holistic and trait-specific rankings, or discrete scores to essays by analyzing a range of textual features, such as coherence, grammatical correctness, and expressive clarity @xcite @xcite . however, existing research in this field, while considering topic as an aspect of evaluation, has neglected the importance of topic relevance @xcite (see appendix a for detailed explanations of these two aspects in chinese essay evaluation). furthermore, despite the efficiency in scoring offered by aes systems, it is the detailed comment and feedback that is particularly beneficial for students to directly refine their essays and strengthen their writing abilities. automatic essay comment generation (aecg) @xcite seeks to provide students with constructive feedback tailored to improve particular areas of their writing as delineated by the strengths and weaknesses highlighted in the comments, which is commonly adopted by teachers @xcite . nevertheless, the intersection of complexity and difficulty in aecg results in a the essay excels in focus and relevance. the author chose a girl who excels academically and is eager to help others as the subject for the essay, using descriptions of her qualities and actions to convey admiration and gratitude. the piece also highlights the significance of friendship and the spirit of mutual assistance. from the perspective of relevance, the essay fully meets the requirements, using a specific individual to delve into her moral fiber and the ethos she represents while expressing the author's emotional stance. hence, i believe this essay excels in both focus and relevance. the essay fails to meet the requirements in relevance and focus. the title is "such people touch me" which should concentrate on the deeds of the person who moved the author, thereby showcasing their personality and qualities. however, the essay starts with friendship, discussing how two people become friends and develop their relationship, deviating from the topic. the author is advised to carefully review the question before writing and organize facts around the central theme. paucity of research within chinese essay evaluation. @xcite represents a limited foray into this area, concentrating on comment generation for chinese narrative essays at the sentence and paragraph levels for student cohorts in the fifth grade and beyond. however, it neglects the diverse requirements posed by various writing tasks and is confined to a singular literary genre. recently, large language models (llms) have been exhibiting considerable promise across various applications @xcite . nonetheless, these models, including gpt-4 foot_0 and domain-specific iterations like educhat @xcite , often make incorrect judgements regarding the topic relevance and generate overly generic or even misleading feedback when tasked with assessing topic relevance of essays, which is related to hallucination of llms, as shown in figure 2 . to address these issues, we introduce a novel dataset toree (topic relevance evaluation) fo-cusing on evaluating topic relevance of essays written by chinese primary and middle school students. to the best of our knowledge, toree offers three primary advantages over existing works in this field. firstly, toree is the first attempt to evaluate chinese essays based on their adherence to topic relevance, which is more comprehensive, considering both the objectives of the writing assignment and the substance of the essay, rather than focusing solely on content. secondly, toree encompasses a broad range of participants, specifically targeting students in the compulsory education stage of china, which includes grades 3 through 9. this span captures 13 distinct literary genres and covers 74 diverse essay prompts, thereby providing insights into student writing abilities across different styles and levels. thirdly, toree is constructed as a comparison dataset that is apt for contrasting and evaluating models through both supervised fine-tuning (sft) and preference learning (pl) @xcite like reinforcement learning from human feedback (rlhf), making it highly versatile for aes, aecg, and other various applications. two tasks based on toree are designed for a comprehensive evaluation: topic relevance recognition and topic relevance comment generation. moreover, we propose a twostep method combining sft and pl to elicit abilities of open-source llms to identify topic relevance level and generate specific and constructive comments. experimental results suggest that our method effectively enhances models' abilities on two tasks, improving both automatic and human evaluations across 4 diverse llms. to sum up, our contributions can be summarized as follows. ## method we propose a two-step method for recognizing topic relevance levels and generating corresponding topic relevance comments in a joint format. ## case study as shown in ## conclusion in conclusion, this paper introduces toree, a novel comprehensive dataset for assessing the topic relevance of student essays, addressing a critical need in educational assessment. toree serves as a valuable resource for future research and development in the field of automatic essay evaluation, which can be further utilized through supervised fine-tuning or preference learning. moreover, our two-step approach has demonstrated success in enhancing the performance of open-source large language models in recognizing topic relevance and providing constructive feedback, presenting great value in real-world applications.