LEAF : Language Learners’ E nglish Essays and Feedback Corpus (2024)

Abstract: This paper addresses the issue of automated feedback generation for English language learners by presenting a corpus of English essays and their corresponding feedback, called LEAF, collected from the “essayforum” website. The corpus comprises approximately 6K essay-feedback pairs, offering a diverse and valuable resource for developing personalized feedback generation systems that address the critical deficiencies within essays, spanning from rectifying grammatical errors to offering insights on argumentative aspects and organizational coherence. Using this corpus, we present and compare multiple feedback generation baselines. Our findings shed light on the challenges of providing personalized feedback and highlight the potential of the LEAF corpus in advancing automated essay evaluation.

## introduction the educational technology landscape is undergoing a radical transformation, shaped by the profound influence of the internet and ai. assessment and evaluation are integral parts of education @xcite , and the cost-effectiveness and efficiency issues in providing corrections and feedback to students are noteworthy considerations in modern educational practices @xcite . recent work have mainly focused on sentencelevel grammatical feedback comment generation @xcite @xcite @xcite and there have been extensive efforts in the field of nlp focused on essay scoring @xcite , but the task of generating effective and personalized feedback remains insufficiently explored and studied. current automated essay scoring models offer useful overall scores; however, they lack the granularity desired by both learners and instructors seeking more detailed insights. delivering effective feedback on student essays is essential for enhancing learning outcomes, yet it poses significant challenges and could be very time-consuming, particularly when providing personalized and informative guidance @xcite . some studies looked at the use of nlp techniques to provide feedback comments to learners. criterion @xcite was one of the first tools that offered comments on different parts of the essay but was not open-sourced. argrewrite @xcite is a revision assistant for argumentative writings. @xcite present a feedback system that incorporates predefined questions for each predefined essay feature (e.g. grammar, sentence diversity, supporting ideas). these questions advise students to review their essays, but they are not very specific, for example: the communication of your ideas needs more strong cohesive cues and devices, such as transitions and connective phrases that link ideas. do you use them correctly? more recently, @xcite and @xcite proposed methods for chinese feedback generation based on sets of keywords and features, and sets of templates respectively. feedback prize competition @xcite introduced tasks involving segmenting essays into sections and assigning discourse labels like lead, position, claim, and evidence. participants also predicted the effectiveness rating of these labels. @xcite presented essaycot, a chain-of-thought prompting strategy that uses scores predicted from their essay scoring system to generate feedback. @xcite shows that while llms such as gpt-4 are capable of producing reasonable feedback, they still lack certain crucial elements associated with constructive feedback @xcite @xcite . most current feedback systems cannot account for the diversity in students' writing and often appear generic, failing to be customized according to the specific content of the essay. in fact, to the best of our knowledge, there are no publicly available datasets as a starting point for this direction of research. in this paper, (i) we present leaf 1 : a dataset of language learners' english essays and feedback, which is the first publicly available english essay feedback generation corpus including open-ended essay-level feedback on multiple aspects such as grammar, organization and arguments of the essay, (ii) we present a few strong baselines using llama2-7b and retrieval-augmented prompts, sharing results on how they perform on different aspects of constructive feedback, and (iii) our human evaluation shows that llama2-7b, although smaller, performs better than llama2-13b on this task. our proposed retrieval-augmented approach (primed on a few examples of leaf) improves these results further by better highlighting essay weaknesses. ## leaf corpus essay forum 2 is a platform that assists both native english speakers and those who learn english as a second language in enhancing their writing skills. users can upload their essays, and educators can provide them with feedback. our data was collected by crawling the writing feedback forum of the website. this forum is mainly used by english language learners preparing for english proficiency tests. data from essayforum has been used in previous work but for the argument generation task @xcite . we used similar approaches for preprocessing but added extra steps to account for feedback posts. these steps included filtering essays based on topic (if an essay needs a reference image/figure, it is removed), and feedback posts based on length (if feedback posts were too short, we removed them). to ensure high feedback quality, we only kept feedback posts from active users (who posted at least 5 feedback posts), and used like/post ratio as another filtering step. note that in many instances, students may find the feedback helpful, but the act of "liking" the feedback is not a widespread practice on this platform. thus, we established a like/post rate threshold of 0.3, considering the overall distribution of all ratios. comparison with gpt-4. several works have compared the feedback provided by instructors/peers with that generated automatically, discussing the advantages and disadvantages associated with each feedback source @xcite @xcite . in a previous work by the same authors @xcite human feedback from essayforum.com was studied and compared with feedback from @xcite . human evaluators were trained to evaluate feedback comments based on the following evaluation criteria: c1: which feedback is more accurate and relevant to the essay content? c2: does this feedback highlight the essay's weaknesses? c3: does this feedback highlight the strengths of the essay? c4: is the feedback specific and actionable? c5: which feedback is more helpful overall? these criteria are synthesized from the literature on constructive feedback @xcite @xcite , except for the first one which we specifically added to see if there are hallucinations in llm-generated feedback. multiple different prompting strategies were explored @xcite including detailed task description in the prompt, few-shot (using random examples in the prompt, @xcite , and rubric-based prompt. we developed the rubricbased prompt tailored explicitly for this task by reviewing various rubrics related to learner and argumentative writing @xcite @xcite , we summarized the essential points for the model's attention in the prompt. refer to @xcite for more details on the prompt design. the study showed that the use of a rubric-based prompt with gpt-4 yielded the highest quality feedback. additionally, we discovered that although gpt-4 feedback tends to be generally more encouraging and positive, humans excel in providing more specific and actionable comments that concentrate on the most critical issues in the essay. motivated by these findings, we opted to integrate the feedback of educators in leaf and gpt-4, creating a new set of feedback that encompasses high-quality elements from both sources. this approach aimed ## retrieval-augmented feedback generation to demonstrate the effectiveness of leaf in enhancing the qualitative aspects of (open-source) ai-generated feedback, we conducted experiments with two llama 2 models (7b and 13b) @xcite . two of the authors studied different aspects of 30 llama-generated feedback comments and findings were similar to @xcite : llms may overlook crucial aspects in their feedback such as relevance to the topic, and guidance on organizing the essay based on the writing task prompt. furthermore, ai-generated comments may lack specificity and actionability in certain instances. as part of this paper, we present a retrieval-augmented feedback generation baseline that would use a few relevant leaf examples to prime the llms to the task. we use the leaf training set as a reference corpus and retrieve relevant examples to use in a few-shot setting @xcite ; mi to find relevant examples, we look at similarity in writing task prompt since analyzing instances of our dataset revealed that a common issue among learners is that they are not always addressing the topic properly. furthermore, human feedback frequently provides insights into structuring the essay in accordance with the specific task prompt. for instance, when the prompt requires expressing agreement or disagreement, or necessitates discussing multiple perspectives, guidance is offered by educators on structuring individual paragraphs to ensure a coherent essay that effectively addresses the prompt within the allotted examination timeframe. such recommendations are typically absent in feedback generated by automated systems. we hypothesize by priming llms with a few essays on related task prompts, coupled with their corresponding feedback, the generated feedback is more likely to incorporate suggestions concerning essay organization, and relevance to the topic and given task prompt. to identify similar essays, we employed sentence-bert @xcite . we first conducted a semantic search with a biencoder (paraphrase-mpnet-base-v2) and then rerank using a cross-encoder (ms-marco-minilm-l-6-v2, trained on the ms marco passage ranking task @xcite ). then we picked the top 3 most similar instances and arranged them in the prompt, with the most relevant example positioned last. ## results and discussion automatic metric scores are presented in we evaluated 20 essays with their feedback comments from our 4 baselines (80 feedback in total). six human evaluators experienced in nlp/linguistics research (3 female, 3 male. native or nativelike english speakers) were given detailed guidelines (appendix b) on aspects discussed in §2 and asked to rate each feedback (from 1 to 3, ordinal scale). each sample was evaluated by 2 people and then their scores were averaged. human evaluation results are available in surprisingly, in the vanilla setting, llama 7b outperforms llama 13b, suggesting that smaller models may still yield reasonable results for this task. refer to despite the improvements observed, the evaluation scores suggest significant room for further improvement, particularly for future use by students in the real world. furthermore, another key concern is relevance and accuracy, as models often propose unnecessary additions and edits, especially in well-written essays. for instance, common suggestions, such as requesting additional evidence and specific data, may be deemed unnecessary for students striving to meet a word count. in particular, llama-13b performed better in scenarios involving well-written essays. lastly, evaluators observed that generated feedback tends to inaccurately praise aspects not represented in the essay. for example, statements such as "your essay presents a clear stance on the topic" may be given despite the arguments in the essay lacking clarity and coherence. ## conclusion in this paper, we introduced leaf: a dataset of language learners' english essays and feedback, representing the first publicly available english essay feedback generation corpus. each essay is paired with both human-written feedback and feedback collaboratively generated by a human and gpt-4. we contend that the latter demonstrates higher quality, considering various crucial aspects of constructive feedback. our experiments reveal that (i) llms face inherent challenges in generating personalized and constructive feedback effectively, and (ii) leaf can serve as an evaluation benchmark and a valuable resource for training and reference to improve the performance of aigenerated feedback.