Large Language Models are Students at Various Levels: Zero-shot Question Difficulty Estimation (2024)

Abstract: Recent advancements in educational platforms have emphasized the importance of personalized education. Accurately estimating question difficulty based on the ability of the student group is essential for personalized question recommendations. Several studies have focused on predicting question difficulty using student question-solving records or textual information about the questions. However, these approaches require a large amount of student question-solving records and fail to account for the subjective difficulties perceived by different student groups. To address these limitations, we propose the LLaSA framework that utilizes large language models to represent students at various levels. Our proposed method, LLaSA and the zero-shot LLaSA, can estimate question difficulty both with and without students’ question-solving records. In evaluations on the DBE-KT22 and ASSISTMents 2005–2006 benchmarks, the zero-shot LLaSA demonstrated a performance comparable to those of strong baseline models even without any training. When evaluated using the classification method, LLaSA outperformed the baseline models, achieving state-of-the-art performance. In addition, the zero-shot LLaSA showed a high correlation with the regressed IRT curve when compared to question difficulty derived from students’ question-solving records, highlighting its potential for real-world applications.

## introduction the advancement of online learning platforms such as coursera foot_1 and udemy foot_2 has recently emphasized the importance of personalized education. these platforms utilize extensive educational question data to recommend questions with suitable difficulty levels to students. this enables students to effectively learn by solving questions that match their proficiency levels @xcite . to provide questions that match students' proficiency levels, it is important to accurately estimate the difficulty of the questions before presenting them @xcite . question difficulty estimation (qde) has traditionally been performed using manual estimation @xcite or the item response theory (irt) @xcite . manual estimation was performed by educational experts, such as teachers and course instructors, who assigned difficulty labels to each question @xcite . however, manual estimation has the drawback of varying results based on the subjective judgment of experts @xcite . by contrast, qde using the irt predicts question difficulty based on student question-solving records, thereby minimizing subjective bias. this method offers the advantages of explainability, and the ability to track changes in the abilities of students and difficulties of questions over time @xcite . however, a significant limitation lies in the need to collect vast amounts of student question-solving records. to overcome these limitations, recent studies have explored new approaches using natural language processing (nlp) techniques to perform qde based on textual information. for instance, the study @xcite employed a tacnn, a cnn-based sentence classifier, and attention layers to estimate question difficulty from a textbased perspective. leveraging the powerful language understanding capabilities of transformerbased pre-trained language models (plms), studies @xcite @xcite have utilized plms to comprehend the textual information of questions and answers in qde. nlp-based qde methodologies have various advantages; however, they solely focus on the information of the questions themselves, not on the students solving them. the same question may have different difficulty levels depending on the proficiency level of the student group. although it is possible to address this aspect by training on the difficulty of each question measured through the irt, there are still drawbacks. these include the requirement for separate question-solving records and the need to train models for each student group. to address these limitations, we focus on the general question-solving capabilities of large language models (llms). noting the achievement of human-level performance by llms across diverse domains @xcite , we hypothesize that llms can substitute for students at various levels. based on this hypothesis, we propose a novel framework, llms are students at various levels (llasa). in llasa, we target the abilities of student groups to form llm clusters with question-solving abilities similar to those of students. considering llms as representatives of students, llasa can effectively predict the question difficulty perceived by student groups using the question-solving records of llms. in contrast to traditional qde methods, our approach can easily adapt to changes in the perceived difficulty of questions among different student groups by modifying the composition of the llms. in particular, llasa utilizes individual student ability levels derived from the irt to form an llm cluster that represents the student group. typically, llasa requires student question-solving records to estimate these abilities. however, if alternative information is available (e.g., grades and levels), llasa can perform qde without any questionsolving records. to demonstrate this, we propose a zero-shot llasa that performs qde using alternative information about student abilities without any question-solving records. to validate the effectiveness of our approach, we evaluated llasa on two qde benchmarks: dbe-kt22 @xcite and @xcite @xcite . regarding the performance in regressing the question difficulty, llasa achieved a performance comparable to those of state-ofthe-art (sota) qde models, despite not being trained itself. remarkably, in the classification setting, llasa achieved sota performance on both benchmarks. compared to question difficulty de-rived from students' question-solving records, the zero-shot llasa achieved over 74% of the performance of the strongest baseline, even without using any of these records. this result strongly supports llasa's ability to substitute students using only approximate distributions, without any student question-solving records. in summary, our contributions are three-fold: ## method our framework, llasa, estimates question difficulty by performing the irt on llm-generated question-solving records. in section 2.1, we describe the methods used to answer the questions using llms within the llasa framework. in section 2.2, we describe llasa, which performs the irt on the question-solving results of students to estimate their abilities and select similar llm clusters. in section 2.3, we describe the zero-shot llasa, which assigns student groups into low/middle/high ability categories based on teacher intuition, and selects the appropriate llms. ## conclusion in this study, we proposed llasa framework by leveraging llms to estimate question difficulty in personalized education. llasa demonstrated a competitive performance with strong baseline models, even without extensive training data. the zero-shot llasa exhibited a high correlation with the student irt, indicating its potential for effective real world applications. this study highlights the potential of llms in qde, suggesting that they can substitute for human abilities in mathematics and computer science domains.