LLM -as-a-tutor in EFL Writing Education: Focusing on Evaluation of Student- LLM Interaction (2024)

Abstract: In the context of English as a Foreign Language (EFL) writing education, LLM-as-a-tutor can assist students by providing real-time feedback on their essays. However, challenges arise in assessing LLM-as-a-tutor due to differing standards between educational and general use cases. To bridge this gap, we integrate pedagogical principles to assess student-LLM interaction. First, we explore how LLMs can function as English tutors, providing effective essay feedback tailored to students. Second, we propose three criteria to evaluate LLM-as-a-tutor specifically designed for EFL writing education, emphasizing pedagogical aspects. In this process, EFL experts evaluate the feedback from LLM-as-a-tutor regarding (1) quality and (2) characteristics. On the other hand, EFL learners assess their (3) learning outcomes from interaction with LLM-as-a-tutor. This approach lays the groundwork for developing LLMs-as-a-tutor tailored to the needs of EFL learners, advancing the effectiveness of writing education in this context.

## introduction personalized feedback is known to significantly enhance student achievement @xcite . however, providing real-time, individualized feedback at scale in traditional classroom settings is challenging due to limited resources. large language models (llms) can be particularly beneficial to address this challenge by enabling real-time feedback in educational settings @xcite @xcite . however, llms often struggle to generate constructive feedback within educational contexts. unlike human feedback, which consistently identifies areas for improvement, llm-generated feedback frequently fails to effectively highlight students' weaknesses @xcite . therefore, it is essential to identify the advantages and limitations of llms as english writing tutors and to develop methods for providing effective feedback for students. the evaluation of llms for educational purposes differs significantly from their generalpurpose evaluation. general-purpose llm evaluation primarily focuses on assessing the quality of responses @xcite @xcite . however, as @xcite emphasize, merely evaluating the final output quality is insufficient to capture the full dynamics of human-llm interactions. in particular, educational feedback needs a more nuanced consideration of factors beyond traditional metrics. it also requires the expertise of education professionals to evaluate the learning process and outcomes due to its inherent challenges. our work incorporates metrics specifically tailored to pedagogical considerations by involving real-world education stakeholders to better assess student-llm interactions. in summary, the main contributions of this work are as follows: in this section, we report preliminary findings that display both the advantages and limitations of llm-as-a-tutor. ## student-llm interaction evaluation in this section, we introduce evaluation methods for student-llm interaction. we provide feedback generated with score-based prompting to student. english experts then evaluate llm-generated feedback with our evaluation metrics on a 7 point likert scale. ## conclusion this paper advances efl writing education by generating and evaluating feedback tailored to students' needs, incorporating pedagogical principles, and involving real-world educational stakeholders. our focus on essay feedback through llm-as-a-tutor aims to more effectively support efl students in their writing process. in the future, we plan to customize the llm-as-a-tutor to provide individualized support. for instance, our evaluation metric and dataset can be utilized to personalize feedback, aligning with students' varying preferences. this customization would allow llm-as-a-tutor to adapt to the specific needs and desires of each student, thereby enhancing the learning experience. ultimately, we envision personalized llm agents in efl education, offering tailored support to each learner based on their unique needs.