Assessing Student Explanations with Large Language Models Using Fine-Tuning and Few-Shot Learning (2024)

Abstract: The practice of soliciting self-explanations from students is widely recognized for its pedagogical benefits. However, the labor-intensive effort required to manually assess students’ explanations makes it impractical for classroom settings. As a result, many current solutions to gauge students’ understanding during class are often limited to multiple choice or fill-in-the-blank questions, which are less effective at exposing misconceptions or helping students to understand and integrate new concepts. Recent advances in large language models (LLMs) present an opportunity to assess student explanations in real-time, making explanation-based classroom response systems feasible for implementation. In this work, we investigate LLM-based approaches for assessing the correctness of students’ explanations in response to undergraduate computer science questions. We investigate alternative prompting approaches for multiple LLMs (i.e., Llama 2, GPT-3.5, and GPT-4) and compare their performance to FLAN-T5 models trained in a fine-tuning manner. The results suggest that the highest accuracy and weighted F1 score were achieved by fine-tuning FLAN-T5, while an in-context learning approach with GPT-4 attains the highest macro F1 score.

## introduction interactivity is critical to learning @xcite . it has been widely demonstrated that by increasing interactivity in the classroom, we can significantly improve students' learning outcomes @xcite ). student-teacher interaction is one of the most influential factors in learning @xcite , and when classrooms are interactive, students become more engaged, more participative, and are more motivated to learn @xcite @xcite . in addition, interactivity can improve comprehension and lead to improved learning @xcite . despite these benefits, many stem classrooms use lectures as the primary method of instruction. the lack of interactivity poses serious issues in undergraduate education @xcite , and large class sizes can inhibit meaningful exchanges between instructors and students in traditional classrooms @xcite ). the passive nature of lectures is particularly problematic in stem courses, as research shows that undergraduate students in classes that use a traditional lecture format are much more likely to fail than students in classes that use a more active learning method @xcite . classroom response systems have been touted as a potential solution to this problem. these systems capture and grade student responses to multiple choice questions posed by instructors during lectures. each student submits a response using a handheld transmitter (a "clicker"), and software on the instructor's computer records, grades, and displays students' answers for the class to view. while research has shown that classroom response systems can promote student engagement and facilitate the learning of factual knowledge @xcite , studies have also shown that "clickers" are less effective for promoting deep and meaningful learning. in fact, traditional classroom response systems may actually obstruct students from developing a conceptual understanding of concepts and principles, particularly for novice students @xcite . because students simply select an answer from a list of choices, "clickers" do not enable students to construct or generate their own responses to questions, which is a key component of active and constructive learning @xcite . decades of research have shown that selfexplanation has a significant impact on student learning @xcite . by explaining concepts and examples to themselves as they learn, students trigger the selfexplanation effect, where they actively probe their own understanding and address gaps in their knowledge. enabling students to generate shortanswer textual explanations to prompts posed by instructors during lectures could open a rich communication channel between instructors and students. eliciting self-explanations from students has the potential to yield substantial learning benefits for students in undergraduate stem classrooms, and it has been widely demonstrated that self-explanation helps students learn much more effectively than students who do not engage in self-explanation @xcite @xcite . because self-explanation requires students to explain concepts to themselves in their own words, they learn much more deeply. however, despite the great potential offered by selfexplanation for promoting learning, students in undergraduate stem classrooms often have limited opportunities to engage in this type of active and constructive learning activity due to limited class time for discussion. similarly, instructors have limited time to assess students' self-explanation responses and provide formative and timely feedback during lectures. in this paper, we present a large language modelbased approach that automatically assesses students' written responses. we investigate the performance of four transformer-based large language models-llama 2 @xcite , gpt-3.5 (openai 2023), gpt-4 (openai 2023), and flan-t5 (chung et al. 2022)-in assessing the correctness (i.e., fully correct, partially correct, and incorrect) of student self-explanations to undergraduate computer science questions. these explanations were collected from undergraduate students, including those who participated in an undergraduate course using the explainit system we have developed. our findings suggest that flan-t5 demonstrates high performance in terms of accuracy and weighted f1, when fine-tuned using a prompt that includes information taken from a grading rubric in combination with an exemplar response provided by the instructor. however, we also find that the highest macro f1 score is achieved by gpt-4 in a few-shot learning setting, where examples of only ten students' explanation responses are provided without any additional information from a rubric or an exemplar response. we discuss the tradeoffs between these models and the implications of our research for practical applications of llm-based explanation assessment in classroom response systems. ## related work it has been found that students explaining concepts to themselves has a profound effect on learning. known as the self-explanation effect @xcite @xcite , the result of self-explanation goes beyond simply rehearsing information: it requires students to express concepts in their own words, relate concepts to prior knowledge, make inferences, integrate information with prior knowledge, and monitor and repair faulty knowledge. thus, selfexplanation is a deeply constructive activity @xcite . the significant learning gains associated with self-explanation have been demonstrated in a wide range of stem disciplines including computer science @xcite , engineering @xcite , chemistry (crippen and earl 2007), algebra @xcite @xcite , and physiology @xcite . our explainit classroom response system leverages the self-explanation effect to improve stem classroom learning. widely known as "clickers," classroom response systems have emerged as a tool to bridge the gap between students and instructors and to make lectures more interactive. used by millions of students, classroom response systems allow students to anonymously respond to multiple choice questions presented during lectures. research has shown that students appreciate the ability to compare their own answers to those of their peers, receive immediate feedback, and test their knowledge, and that "clickers" can increase student interactivity during lectures @xcite @xcite . however, studies have also shown that clickers fail to promote deep and meaningful learning, which can be particularly problematic for students in stem classes who are required to conceptually understand important concepts, relationships, and theories to effectively solve problems @xcite . closely related to our work, commercial classroom response systems have been explored in various classroom settings. these systems typically support students through classroom discussions, questions, and assignments, and they support instructors with features for course material creation and assessment, which are incorporated with learning management systems. while they provide a range of functionalities required for a classroom response system, such as the ability to pose various types of questions (e.g., multiple choice, fill-in-the-blank, short answer questions), their automated assessment is typically limited to multiple choice and fill-in-the-blank types of questions that accept a predetermined set of answers, while they require a manual assessment process for other types of questions. deep learning-based language models such as bert (e.g., @xcite , flan-t5 (e.g., chung et al. 2022), gpt (e.g., @xcite , and llama @xcite have been pivotal in the recent advancements in natural language processing (nlp; @xcite . in learning analytics, additional sources of training data, including data collected for free-response prompts @xcite ), text providing additional context for free-response prompts @xcite , response assessment rubrics @xcite , and synthetic data generated via data augmentation strategies @xcite , have effectively enhanced the training of nlp models, leading to improved predictive performance. nlp techniques have been used to accurately analyze student textual responses in the context of short-answer science assessment @xcite , student written reflections @xcite , student-tutor dialogue @xcite , and student selfexplanations @xcite . while previous work demonstrated considerable success with llms for short answer grading @xcite ) and short answer question generation @xcite , a research area that has seen limited exploration is assessing students' free-text explanations @xcite . building on recent advances in nlp and deep learning-based language modeling techniques, our work makes a novel contribution by investigating an approach to assess students' self-explanations, collected from an undergraduate artificial intelligence course, utilizing large language models with fine-tuning and few-shot learning. ## explainit classroom response system the explainit classroom response system leverages the self-explanation effect and active, constructive, and interactive learning, along with state-of-the-art natural language processing, to significantly improve stem undergraduate education. with a specific focus on computer science, biology, and physics, it aims to create highly engaging classroom learning experiences. explainit offers the opportunity to fundamentally improve classroom dynamics by supporting both students and instructors. the system is designed to support both students and instructors in undergraduate stem courses by analyzing and providing feedback on students' explanations through an integrated five-step explanation feedback loop (figure 1 ): (1) the instructor issues an explanation prompt, which appears in the explainit app on students' computing devices (e.g., laptops, tablets, phones); our initial prototype of the explainit classroom response system was implemented using a web-based application architecture to support enhanced scalability, where instructors and students can choose their platforms of choice such as laptops and handheld devices, while the software enables real-time interactions with the user interfaces. the explainit user interfaces consist of an instructor authoring tool, instructor dashboard, and student explanation app. the streams of communication data are uploaded into a cloud-based database by the server. for data synchronization and analysis purposes, all interaction data is timestamped. to support these functionalities, we implemented the software modules to include apis using the http protocol. we use microsoft's azure cloud computing service to host our cloud-based services. the instructor authoring tool enables instructors to create or edit questions and an exemplar correct response per question. all questions and responses are categorized by their subjects and topics in the tool. all authored content is stored and accessed from the cloud, allowing the original instructor to reference their own created questions for future courses. the instructor dashboard presents the pool of questions per subject and topic, and it allows instructors to select and send questions to the student explanation app, so that students can view and interact with the questions in real-time during lectures. the instructor dashboard is also designed to display student-written responses and nlp assessment results in visual analytics. the student explanation app enables students to receive questions posed by instructors and write self-explanation responses to instructor-posed questions. when students submit their responses, the student explanation app taps into the explanation analyzer, which performs nlpdriven assessment of student explanations, generates tailored feedback to students, and dispatches analytical summaries to instructors through instructor dashboard. the explanation analyzer is in the development phase, and our findings about the explanation analyzer's nlp performance are presented in this paper. ## study and data this work uses data collected during a classroom pilot study of the explainit system. the participants in the classroom study consisted of 36 consented undergraduate students enrolled in a computer science course focused on artificial intelligence. thirty-two students completed a demographic pre-survey, and among them 8 indicated that they identified as female, 23 as male, and 1 preferred not to indicate gender identification. participants ranged in age from 18 to 28 (@xmath0.1, @xmath1.64). of these participants, 40.6% were asian, 50.0% were white, and 9.4% preferred not to answer. prior to using explainit in the class, the instructor used the instructor authoring tool to prepare a set of questions, each accompanied by an exemplar correct answer. these answers were presented to students immediately after they submitted their responses to the respective questions. the classroom implementation unfolded over 6 weeks within a single semester. throughout this period, a total of 13 questions were sent to the class, eliciting 356 responses from 36 participants, which were utilized in our evaluation ( students' responses to the questions were labeled by two of the researchers, who are experts in computer science. first, a rubric item was constructed for each question that described the qualities of a correct, partially correct, or incorrect answer to the question. for example, the rubric for the question "in a neural network, what function is responsible for introducing non-linearity to the model?" indicated that a correct response should mention the term "activation function", a partially correct response might present an example of an activation function (e.g., "sigmoid") without explicitly mentioning the term "activation function", and that an incorrect response would not include any of this information. we also referenced instructor-provided exemplar answers to further refine the rubrics for each question. these were comprehensive and well-reasoned responses, serving as a representative correct answer to each question. then, based on the developed rubric, both researchers labeled twenty percent of the student responses. after one cycle of rubric refinement, a cohen's kappa of 0.702 was achieved, indicating substantial agreement @xcite . all labels that the annotators did not agree on were discussed and agreement on a single label was reached. across all questions, 73% of explanations were labeled as correct, 22% were labeled as partially correct, and 5% were labeled as incorrect. ## method we evaluated the performance of llama 2 @xcite , gpt-3.5 (openai 2023), gpt-4 (openai 2023), and flan-t5 (chung et al. 2022) on the self-explanation assessment task. large language models (llms) have been demonstrated to achieve state-of-the-art performance on many natural language processing tasks, with gpt-4 particularly excelling with fewshot prompting where training examples are integrated into the task description (openai 2023). this enables gpt-4 to readily adapt to new tasks without re-training, avoiding the prohibitive cost of updating its extensive parameters. however, gpt models' proprietary nature and associated costs pose barriers to its educational adoption, such as explainit. to address this challenge, we also evaluated the performance of open-source models, flan-t5 and llama 2. flan-t5 is an instruction-fine-tuned language model that has demonstrated competitive performance with other state-of-the-art models across a range of tasks when it was released @xcite . llama 2 is an open-source pre-trained large language model that has demonstrated leading performance compared to other opensource models and performs similarly to gpt-3.5 on several tasks @xcite . in this work, we investigate the performance of the base flan-t5 model (250m parameters) and llama 2-7b, the smallest version of the model. these versions of flan-t5 and llama 2 were selected due to their computational efficiency. for all models, default hyperparameters were used. we investigated several different zero-shot and few-shot prompting approaches to evaluate the performance of llama 2, gpt-3.5, and gpt-4 for automated assessment of students' selfexplanation. as a baseline, these models were provided with instructions that described the task (i.e., "please evaluate a student's explanation response to the following question.") in addition to the question and student response. then, we systematically evaluated the impact on model performance of including the following information in the prompt: (a) rubric items for the current question, (b) an exemplar correct response provided by the instructor, and (c) other students' labeled responses to the current question. prompts were constructed with all possible combinations of the different information elements, and model performance was evaluated for each combination. for the prompts incorporating student selfexplanation responses, 10-fold student-level crossvalidation was used to prevent bias from students' individual writing styles and to ensure generalizability, avoiding data leakage in model evaluation. additionally, this approach accurately represents the real-world scenario that will be faced when deploying explainit in future classroom implementations, as the students interacting with the system will be new but the models will have access to past student's responses to each question. due to llm token limits and the per-token cost of proprietary models like gpt-4, we sampled ten responses from the training set for each crossvalidation fold to include in the prompts rather than including the entire training set. in comparison to llama 2, gpt-3.5, and gpt-4, the flan-t5 base model's smaller parameter count facilitates easier and more cost-effective training. given its sufficient size for fine-tuning using our available resources, we chose this approach over few-shot prompting. we applied lora for efficient fine-tuning, changing only a subset of the model's parameters to conserve time and computational resources, while achieving similar performance to full fine-tuning @xcite . the evaluation of fine-tuned flan-t5 models is also based on 10-fold student-level crossvalidation using the same data split as was used for in-context learning with the other models. however, rather than including example explanations and their assigned labels in the prompt, they were used as training examples in a supervised learning approach. as with the incontext learning approach, we explored variants of prompts including the rubric item for each question and/or the exemplar correct response created by the instructor. a separate flan-t5 model was finetuned for each prompt variant. ## results results from all experiments are presented in across all experiments, flan-t5 models that were fine-tuned with rubric information and the instructor's exemplar response achieved the highest accuracy (acc.=0.824). this was a substantial improvement over the majority baseline, which always predicts the most common class (acc.=0.730), as well as the next-highest performing approach, which was gpt-4 with ten student examples included in the prompt (acc.=0.775). in terms of macro f1 score, gpt-4 with ten labeled student explanation responses included in the prompt achieved the highest performance (@xmath2.664). this was a significant improvement over the majority baseline (@xmath3.281) and the next-highest performing approach, which was gpt-4 with all three information elements included in the prompt (@xmath4.641). in terms of weighted f1 score, flan-t5 models that were fine-tuned with rubric information and the instructor's exemplar response achieved the highest performance (@xmath5.798). this was an improvement over the majority baseline (@xmath6.616) and a small improvement over the nexthighest performing approach, which was gpt-4 with ten labeled student explanation responses included in the prompt (@xmath7.792). in general, our results demonstrate that including rubric information in the prompt improved model performance. for flan-t5, llama 2, and gpt-4, both accuracy and f1 score were improved relative to the prompting approach that only provided high-level instructions for the explanation assessment task. we observed the largest improvement in model performance when the sole additional information was a set of ten labeled explanation responses from other students. with this prompt, llama 2 and gpt-4 demonstrated improved accuracy over the instruction-only approach, while llama 2, gpt-3.5, and gpt-4 exhibited improved macro f1 scores. however, we found that including the instructor's exemplar response into the prompt led to reduced model performance across all models except for llama 2, compared to the instructiononly approach. this reduction may stem from the exemplar responses often containing comprehensive details that exceed the question's scope, leading the models to apply a very strict standard in assessing student responses. consequently, responses were more frequently categorized as partially correct or incorrect, even though they should be labeled correct within the question's intended scope. next, we looked at the effects of including two information elements in the prompt. note that these results are omitted from a distinct trend emerged when all three information elements were included in the prompt. gpt-3.5 and gpt-4 models with access to all three information elements performed worse than models provided with only ten labeled student example responses, both in terms of accuracy and f1 scores. however, for llama 2 models, incorporating all three information elements in the prompt resulted in the highest accuracy and f1 scores compared to any other combinations of information. these results suggest that the best results are not necessarily guaranteed by providing the model with the maximum amount of task-related information. models consistently performed well when the prompt included labeled examples of other students' responses, but including the instructor-created exemplar response tended to reduce model performance as discussed. adjusting the exemplar response provided to the models, by adding clarification or simplifying its content, could potentially lead to improved performance when this information element is included. this underscores an important area for future research. overall, these results demonstrate that finetuning flan-t5 and utilizing few-shot learning with gpt-4 are both viable approaches to this explanation assessment task. although flan-t5 requires more training data than gpt-4 to reach high performance levels (our preliminary analysis indicated that the predictive accuracy of a flan-t5 model, fine-tuned with only the data from five focus group students, was 60%), this tradeoff may be acceptable considering that flan-t5 is opensource and gpt-4 is proprietary. this consideration becomes more critical as our classroom implementation scales, especially in large classroom settings with multiple sessions where deployment costs become a significant factor. conversely, if the explainit system is implemented in a course where flan-t5 models have not been trained with student data from that course, gpt-4 with one-shot learning (with rubric information) might significantly outperform flan-t5, making gpt-4 potentially more suitable for the classroom response system. it will be crucial to weigh practical benefits, scalability, and cost considerations when deploying a runtime version of the explanation assessment system during the classroom use of explainit. in practice, these results suggest that a hybrid system may be a viable approach. when a new question is deployed using the system, zero-shot learning with gpt-4 can be used based on a pre-defined rubric that was created for assessing responses to the question. since this information can be created at the same time as the question, it can be provided to the system when the new question is first deployed. then, as student responses to the question are collected, they can be used to fine-tune a flan-t5 model, which can then replace the gpt-4 model once it starts showing superior performance. ## conclusion prompting students to craft self-explanations has demonstrated to offer numerous educational advantages. however, it often requires substantial time and effort necessary for instructors to manually assess student responses and provide feedback for students, which renders them unsuitable in large classroom environments. to address this challenge, we present explainit, a self-explanation-based classroom response system specifically designed to encourage students in formulating written self-explanations during undergraduate stem lectures. our nlp framework builds on transformer-based large language models, such as flan-t5 and gpt-4, in assessing the correctness of student explanations, and it is evaluated using our dataset collected from classroom interactions with the explainit system. results demonstrate that fine-tuned flan-t5 models using prompts with rubric information and an exemplar response achieved the highest accuracy and weighted f1 score, while few-shot prompting that provided gpt-4 with ten labeled student response examples achieved the highest macro f1 score. these results indicate the potential to use large language models for automated explanation assessment, which can be leveraged to provide adaptive support for students' selfexplanations in classroom environments. moving forward, there are several promising directions for future work. first, it will be important to implement the full suite of explainit system functionalities, including nlp assessment models, in a classroom environment and investigate their impact on students' learning outcomes. it would also be interesting to incorporate ai capabilities to support question and rubric generation, thereby reducing the amount of work required by instructors to use explainit in their classes. additionally, the explanation assessment system could be expanded to support a finer-grained assessment of students' self-explanations. for example, concept-level assessment of students' self-explanations could provide more insightful feedback for both students and instructors. also, it will be important to investigate this explanation assessment approach in disciplines other than computer science to evaluate its performance in other domains. finally, it will be important to explore how different types of exemplar responses and rubric items impact model performance. if we are able to identify characteristics of exemplar responses and rubric items that most improve the predictive accuracy of our llm-based framework for self-explanation assessment, that will enable our classroom response system to more effectively support student learning in new settings where there is limited student data that can be used to inform the assessment models.