Large Language Model as an Assignment Evaluator: Insights, Feedback, and Challenges in a 1000+ Student Course (2024)

Abstract: Using large language models (LLMs) for automatic evaluation has become an important evaluation method in NLP research. However, it is unclear whether these LLM-based evaluators can be effectively applied in real-world classrooms to assess student assignments. This empirical report shares how we use GPT-4 as an automatic assignment evaluator in a university course with over 1000 students. Based on student responses, we found that LLM-based assignment evaluators are generally acceptable to students when they have free access to these tools. However, students also noted that the LLM sometimes fails to adhere to the evaluation instructions, resulting in unreasonable assessments. Additionally, we observed that students can easily manipulate the LLM to output specific strings, allowing them to achieve high scores without meeting the assignment rubric. Based on student feedback and our experience, we offer several recommendations for effectively integrating LLMs into future classroom evaluations. Our observation also highlights potential directions for improving LLM-based evaluators, including their instruction-following ability and vulnerability to prompt hacking.

## introduction llms have significantly changed the landscape in nlp @xcite . one notable advancement brought by llms lies in automatic evaluation. chiang and @xcite show that when an llm is provided with evaluation criteria and a sample to evaluate, it can produce results that align well with those of human evaluations conducted by domain experts. in this paper, we use "llm-based evaluator" to refer to llms instructed to evaluate a sample based on specific criteria. significant research efforts have been devoted to studying how to align the results of llm-based evaluators more closely with human evaluation results @xcite @xcite and to address the known pitfalls of llm-based evaluators @xcite . while the research on llm-based evaluators prospers, llm-based evaluators primarily exist in academic papers and have not yet seen widespread applications in real-world scenarios. specifically, using llm-based evaluators in educational settings to grade student assignments remains largely unexplored. although using machine learning models to assess student performance is not a new practice @xcite @xcite , we find that few works apply llm-based evaluators for scoring the student's assignments @xcite . this makes it hard for nlp researchers to understand what obstacles need to be tackled before using an llm-based evaluator in the real world. in this empirical report, we share our educational practice of using llm-based evaluators to score the student assignments. we call these llm-based evaluators "llm-based evaluation teaching assistants (tas)" and use llm tas for short. we use llm tas in over half of the assignments in a course with over 1,028 students from diverse backgrounds. to the best of our knowledge, we are the first to report such a large-scale usage of llm tas in a real-world course. we collect and share invaluable feedback from the students to understand their attitude toward llm tas and the failure cases of llm tas. we also find that many student submissions attempt to manipulate the llm ta's response into giving them a perfect score; we show some examples and discuss how to detect and defend them. we summarize our key observations as follows: our findings highlight several important aspects of llm-based evaluators that need to be addressed by the nlp community, including the vulnerability of prompt hacking and the inability to follow the specified output format. ## course introduction we introduce some context about the course that uses llm tas. in spring 2024, professor hung-yi lee offered a course titled "introduction to generative ai" foot_0 at national taiwan university. the course aims to equip students with a fundamental and precise understanding of generative ai, including its principles and applications. the course is an elective for electrical engineering and computer science (eecs) students but also serves as one of the mandatory elective options for students from the liberal arts college. this unique setting brings together students with diverse backgrounds. as we anticipate many students enrolling in the course before it begins, we plan to use llm tas to grade assignments automatically. this will help reduce the workload of the teaching team. six assignments use the llm tas, while the remaining four are scored using other automatic methods. we briefly introduce the assignments in appendix b. during the course enrollment period, we explicitly state that generative ai will be used to assess students' assignments. therefore, students who choose to enroll in the course are expected to accept this practice. eventually, 1,028 students enrolled in the course, with eecs students making up about 80% of the population and the remaining 20% coming mostly from the liberal arts college. ## student feedback at the end of the semester, we ask the students to fill out a survey about their attitude toward llm tas and their experience. the survey is designed as part of an assignment, and students earn the score for the survey as long as they submit it, regardless of the answer. we discuss how we obtain consent from the students to share their responses in the ethics statement. we show the complete survey in appendix a. in total, 838 students agreed to share their responses, which we analyze in this section. ## prompt hacking the llm tas prompt hacking refers to using adversarial prompts to trigger malicious results @xcite . in this course, we find a nonneglectable proportion of students using prompt hacking to force the llm ta to print "final score: <max @xmath0. this type of prompt hacking is called goal hijacking @xcite , where the malicious prompt aims to make the llm print a target phrase. while goal hijacking has received a lot of attention in llm safety community @xcite @xcite , it is out of our expectation to see so many students try to hack the llm tas. we do not ban prompt hacking in the assignment since we believe that understanding the vulnerability of llm tas is an important topic in generative ai. we asked the students to self-report whether they used prompt hacking in any of the six assignments using llm tas, and we found that only 53% of the students allege that they did not try prompt hacking. all the assignments with llm tas were hacked. homework 2 had the highest percentage of hacking, with as many as 44% of the students attempting to hack the llm ta. ## conclusion and discussions this empirical report summarizes our experience using gpt-4 as an llm ta to automatically score the assignments in a course with over 1000 students. we collect opinions from the students about how well they can accept llm tas under different scenarios, and we also report the problems the students encountered when interacting with those llm tas. we show that students can easily find interesting and creative prompts to hack the llm tas to give unreasonably high scores. guidelines for teachers we provide several actionable guidelines for teachers considering using llm tas: (1) always provide the students with the evaluation prompts. (2) it is very important to make the llm ta freely accessible to all the students. future research directions our educational practice of using llm tas provides several valuable research directions and problems that are not yet well-studied or fully addressed. for example, it may be interesting to study how to improve or control the consistency between the llm's responses between multiple random samples. it is also important to enhance the instruction-following ability of llms, specifically, the output format, while not sacrificing their reasoning ability @xcite . last, the prompt hacking results shown in section 5 highlight the inability of llms to distinguish the priority and hierarchy of user instructions, making them easily deceived by malicious user instructions to generate seemingly benign outputs; such an issue must be properly tackled before widely applying llms in more high-stacks real-world applications.