Using LLM s to simulate students’ responses to exam questions (2024)

Abstract: Previous research leveraged Large Language Models (LLMs) in numerous ways in the educational domain. Here, we show that they can be used to answer exam questions simulating students of different skill levels and share a prompt, engineered for GPT-3.5, that enables the simulation of varying student skill levels on questions from different educational domains. We evaluate the proposed prompt on three publicly available datasets (one from science exams and two from English reading comprehension exams) and three LLMs (two versions of GPT-3.5 and one of GPT-4), and show that it is robust to different educational domains and capable of generalising to data unseen during the prompt engineering phase. We also show that, being engineered for a specific version of GPT-3.5, the prompt does not generalise well to different LLMs, stressing the need for prompt engineering for each model in practical applications. Lastly, we find that there is not a direct correlation between the quality of the rationales obtained with chain-of-thought prompting and the accuracy in the student simulation task.

## introduction large language models (llms) currently represent the state of the art in text generation, with some capable of generating human-like texts, such as gpt-4 (openai, 2023), llama @xcite , and gemma (gemma @xcite . in this work we focus on the educational domain, which can massively benefit from llms @xcite @xcite . specifically, we study whether it is possible to leverage llms to simulate the response patterns of students of different skill levels to exam questions. previous research tried to simulate the responses of human participants to surveys with llms @xcite @xcite , but nothing similar has been done for simulating students answering exam questions. there have been some concerns about the fairness of using llms instead of (or in addition to) human survey participants @xcite , and we agree that this is an important aspect to consider in the educational domain, as well. however, we believe that it might be less of an issue with respect to general-domain surveys, due to the factual nature of learning content and exam questions, which are built to evaluate domain knowledge and to minimise the effects that the wording has on the students' outcomes @xcite . in this work, we aim at answering the following research questions. rq1: can llms be prompted to answer multiple choice questions (mcqs) while role-playing as (i.e., simulating) learners of different skill levels, and does this generalise to unseen data? foot_0 rq2: how do these findings compare across different models? we work primarily on gpt-3.5 foot_1 and three publicly available datasets of science mcqs (arc) and english reading comprehension mcqs (race and cup&a), and engineer a prompt (referred to as "reference prompt" or rp) that leads the llm to answer exam questions with different levels of accuracy, thus representing students of different skill levels. also, we observe a small but positive correlation between the difficulty obtained from virtual pretesting with llms and the difficulty from pretesting with human learners. although the prompt was engineered using only gpt-3.5 and one dataset, this behaviour proved generalisable to new questions (also from different educational domains), but not to other llms, thus stressing the need for prompt engineering for each model. lastly, we find that there is not a direct correlation between the quality of the rationales obtained with chain-of-thought prompting and the accuracy of the models in the student simulation task. the code, prompts, and llm outputs are publicly available at github.com/lucabenedet to/llm-student-simulations. ## methodology we proceed in three steps: we i) perform prompt engineering to search for the "best" prompt using one llm and one dataset, ii) evaluate its generalisation capabilities to unseen data and to other llms, and iii) perform some additional analyses of the models' responses. ## related work previous research discussed the possibility of using llms instead of (or in addition to) human participants in surveys @xcite @xcite , and studied whether llms can be prompted to show human-like behaviours in a series of task @xcite . however, it is not agreed whether this is actually a good practice. indeed, some researchers argue that llms cannot (and should not) replace human research participants @xcite . we mostly agree with the latter, but believe that exam simulations are a different application scenario, as knowledge-based exam questions are built to assess students knowledge in an objective (as much as possible) manner. still, possible biases of this approach will have to be studied before an application in the real world. an approach like the one proposed by @xcite (i.e., using llms as a preliminary step before the human annotations) might be adopted in education, for instance pretesting with human learners only a fraction of the original items. previous research also discussed profusely the potential of llms in education @xcite @xcite . closer to our work, previous research experimented on knowledge tracing with lms @xcite , but without using them for simulating students. also related to the current work is the previous research of question difficulty estimation with nlp @xcite @xcite , especially when performed in an unsupervised manner @xcite . indeed, the students simulation we propose in this paper could be used as an alternative to previous approaches for difficulty estimation. ## conclusions and future work in this paper, we have shown that it is possible to prompt gpt-3.5 to simulate students of different levels, and the reference prompt we have engineered proved capable of generalising across datasets. however, even though the prompt seems to generalise well to unseen data, it does not seem to generalise to different llms, thus stressing the need for prompt engineering for each model. although we found some strong indications that it might be possible to simulate students of different levels with llms, there are questions still to be addressed. for a better simulation, one could try to use retrieval augmented generation (rag) @xcite on topic specific documents to better define the level of the role-played student. for a better virtual pretesting, it will be needed to have a larger set of simulated students. also, it might be helpful to simulate whole exams, instead of one question at a time as we did here. future work could also iterate on the reference prompt, possibly using automatic prompt optimization @xcite , and experiment with open models, which is particularly relevant since specific versions of closed llms can become deprecated.