Prompting GPT -4 for C hinese Essay Fluency Evaluation (2024)

Abstract: “This report presents the methodology and results of utilizing GPT-4 for CCL24-Eval Task 7 of Chinese Essay Fluency Evaluation (CEFE). The task is divided into three tracks: Identification of Error Sentence Types, Rewriting Error Sentences, and Essay Fluency Rating. We employed a few-shot prompt engineering to guide GPT-4 in performing this task. Our approach integrated fine-grained error analysis with advanced NLP techniques to provide detailed, actionable feedback for students and teachers. Despite some successes, particularly in generating semantically similar and syntactically relevant corrections, our analysis revealed significant challenges, especially in multiple-label classification and the accurate identification of error types. The report discusses these findings and suggests areas for further improvement.”

## introduction the rapid development of education and the widespread use of the internet have significantly increased the volume of essay evaluations. this growth poses significant challenges in terms of the cost and efficiency of human evaluation. to overcome these challenges, researchers and institutions have increasingly integrated artificial intelligence (ai) and machine learning (ml) algorithms into educational settings. various nlp tasks have emerged, including automated item generation @xcite , grammar error detection and correction @xcite , reading and text complexity assessment, writing analysis and feedback @xcite , and automated writing evaluation. for essay evaluation systems, particularly those targeting primary and middle school students' writing, the goal is to provide objective, accurate, and timely evaluations by analyzing various aspects of essays, including language use, content, and structural coherence. one of the critical aspects of essay evaluation is the fluency of expression, which reflects not only the smoothness and normative use of language but also the overall writing proficiency and ability to convey ideas clearly. essay fluency is essential for assessing and improving writing quality. it encompasses several linguistic features, such as sentence length, lexical complexity, and sentence structure @xcite . current approaches to essay fluency evaluation typically involve scoring based on these linguistic features, treating it as a grammar correction task to identify and correct spelling and grammatical errors, or regarding it as a faulty sentence detection task to determine the presence of errors. however, these methods often consider essay fluency assessment as an isolated natural language processing (nlp) task, lacking systematic integration across multiple levels and perspectives. moreover, existing studies tend to define grammatical error types in broad categories like redundancy, omission, misuse, and disorder, which are insufficiently detailed for providing precise feedback. to advance the field of grammatical error recognition and correction in examination essays written by native chinese-speaking primary and secondary school students, the china national conference on computational linguistics (ccl-2024) has included chinese essay fluency evaluation (cefe) as one of its shared tasks. this task systematically classifies text errors at various granularities, provides humanannotated data, and proposes three tracks that cover error detection and correction, sentence rewriting, and essay fluency rating. to address this task, we employed few-shot prompting techniques and leveraged the advanced generation capabilities of @xcite to make predictions across the three tracks. our approach integrates fine-grained error analysis with advanced nlp techniques to deliver detailed, actionable feedback for students and teachers. specifically, we used prompt engineering @xcite to instruct gpt-4 to rewrite erroneous sentences with minimal modifications. additionally, we guided gpt-4 to rate the overall fluency of essays. ultimately, our goal is to improve the quality and fluency of student essays, providing them with insights necessary for continuous improvement in their writing skills. ## task description the cefe shared task systematically defines fine-grained error types that affect essay fluency and offers corrective suggestions. this fine-grained approach helps students understand their writing issues more clearly and assists teachers in quickly gauging students' writing levels, facilitating more effective writing instruction. the evaluation task involves a comprehensive analysis of essay fluency from multiple linguistic angles: lexical, syntactic, and semantic, and provides modification suggestions. the task is divided into three tracks: track 1: identification of error sentence types this track focuses on recognizing both coarsegrained and fine-grained errors commonly found in student essays. it approaches the issue from two perspectives: character-level and component-level errors. there are four types of coarse-grained grammatical errors @xcite : character-level error (cl), incomplete component error (ic), redundant component error (rc), and incorrect constituent combination error (icc). additionally, there are fourteen fine-grained error types, which provides a more detailed understanding of specific errors. additionally, there are fourteen fine-grained error types, providing a more detailed understanding of specific errors. the detailed error types are listed in the ## experiment results and analysis track 1 result in track 1, gpt-4 (openai, 2023) encountered significant challenges in error identification, leading to incomplete and empty predictions: the model failed to recognize certain error types, resulting in missing annotations. this indicates that gpt-4 struggled with multiple label classification tasks, particularly in accurately identifying all relevant error types within a sentence. as for the empty predictions, there were instances where the model outputted empty results for error types, even when multiple errors were present. this further highlights gpt-4's difficulty in handling tasks requiring multiple labels. the performance issues observed indicate that gpt-4 is not well-suited for multiple label classification tasks, especially in the context of identifying fine-grained grammatical errors. we summarize the key challenges and insights for track 1: complexity of multiple labels: these findings suggest that further refinement and alternative approaches are needed to improve gpt-4's performance in multiple label classification tasks. track 2 result our evaluation process for sentence rewriting involved comparing the corrected sentences generated by gpt-4 against reference annotations. to evaluate the performance of our model, we utilized several metrics: exact match (em), bert perplexity (bert ppl), levenshtein distance, bleu-4, bertscore @xcite , precision, recall, and f0.5. an aggregated score, avgscore, was calculated using a specific formula. exact match (em) measures the percentage of sentences that match the reference sentence exactly. bert perplexity (bert ppl) @xcite indicates the fluency of the generated text based on the perplexity score from a bert model. levenshtein distance measures the number of single-character c c l 2 0 2 4 edits required to change the generated sentence into the reference sentence. bleu-4 @xcite evaluates the n-gram precision up to 4-grams, indicating the similarity between the generated and reference sentences. bertscore @xcite uses bert embeddings to measure the semantic similarity between the generated and reference sentences. precision is the ratio of correctly predicted positive observations to the total predicted positives. recall is the ratio of correctly predicted positive observations to all observations in the actual class. f0.5 is a weighted harmonic mean of precision and recall that gives more weight to precision. the avgscore was calculated using the following formula: the results are shown in ## conclusion the results of our evaluation indicate that while gpt-4 shows potential in tasks such as rewriting sentences and rating essay fluency, it faces significant challenges in accurately identifying error types, particularly in multiple-label classification tasks. our approach demonstrated strengths in generating semantically similar corrections and achieving moderate agreement with human ratings. however, these findings highlight the need for further refinement and alternative strategies to improve the model's accuracy and overall performance in these tasks. 中小学作文病句改写任务的目标是对中小学生作文中出现的错误句子进行最小化修改， 同时确保语义保持不变。这是一个文本生成任务，输入为错误的句子，输出为修改后的句子。 操作指南： 请仔细分析输入的错误句子，并进行适当的修改以纠正错误，确保修改后的句子语义保持一 致。 输出应仅包含修改后的句子(revisedsent)，不包含其他信息。 示例： example 1 example 2 作文句子="{sentences}" 请确保所有修改尽可能地简洁，并严格遵循语法规则，以提供清晰、准确的改写。 figure 2: the final few-shot learning prompt used for track 2.