Towards Explainable Computerized Adaptive Testing with Large Language Model (2024)

Abstract: As intelligent education evolves, it will provide students with multiple personalized learning services based on their individual abilities. Computerized adaptive testing (CAT) is designed to accurately measure a student’s ability using the least questions, providing an efficient and personalized testing method. However, existing methods mainly focus on minimizing the number of questions required to assess ability, often lacking clear and reliable explanations for the question selection process. Educators and students can hardly trust and accept CAT systems without an understanding of the rationale behind the question selection process. To address this issue, we introduce L LM- A gent-Based CAT (LACAT), a novel agent powered by large language models to enhance CAT with human-like interpretability and explanation capabilities. LACAT consists of three key modules: the Summarizer, which generates interpretable student profiles; the Reasoner, which personalizes questions and provides human-readable explanations; and the Critic, which learns from past choices to optimize future question selection. We conducted extensive experiments on three real-world educational datasets. The results demonstrate that LACAT can perform comparably or superior to traditional CAT methods in accuracy and significantly improve the transparency and acceptability of the testing process. Human evaluations further confirm that LACAT can generate high-quality, understandable explanations, thereby enhancing student trust and satisfaction.

## introduction as intelligent education has evolved significantly, computerized adaptive testing (cat), which aims to accurately measure a student's ability with as few questions as possible, provides an efficient and personalized testing method for downstream educational services, such as learning resource recommendation, and learning status report. (van der @xcite . generally, cat selects appropriate questions that align with each student's individual performance, further delivering equivalent testing accuracy levels with fewer test items compared to conventional paper-and-pencil exams @xcite . figure 1 illustrates a toy example of a typical cat. in practice, cat involves two core components: the cognitive diagnosis model (cdm), which estimates students' abilities, and a selection strategy that identifies the most appropriate questions based on their past performance @xcite . the workflow of the cat system is as follows. given a student, in step t -1, the selection algorithm selects an exercise based on the cdm's estimate of the student's current abilities, denoted by θ t-1 . subsequently, the students accept and answer the exercise. after receiving the answer, the cdm updates the estimate to θ t based on the new answer. traditionally, heuristic rules such as maximizing fisher information @xcite , kli @xcite have been used for question selection. however, these methods struggle with generalization and modeling complex interactions between students and questions. recently, the focus has shifted to deep learning methods like ncat @xcite , bob-cat @xcite , and gmocat @xcite , which learn selection algorithms from large-scale student response data and have improved test accuracy. despite their success, these black-box systems lack transparency and fail to provide reasonable explanations @xcite , leading to potential distrust among teachers and students. hence, developing interpretable and transparent adaptive testing models is crucial for their widespread adoption and trust @xcite . nowadays, the emergence of large language models @xcite offers a new perspective on solving this problem. recent research has demonstrated their power in textual reasoning @xcite , achieving significant results in tasks like text summarization @xcite and sentiment analysis @xcite . additionally, these models possess powerful language generation capabilities, enabling them to produce human-readable explanatory text for their decision-making processes. however, although large language models excel in language processing tasks, the integration of llm within the framework of computerized adaptive testing remains highly challenging. three primary challenges can be delineated in tackling the explainable computerized adaptive testing task using large language models (llms). firstly, the cognitive diagnostic model (cdm) generally represents the estimation of a student's abilities using a single or multi-dimensional vector. however, relying solely on a vector is insufficient for guiding an llm in question selection. secondly, while the primary goal of computerized adaptive testing is to assess students' abilities accurately, the lack of a precise standard for question selection and determining the optimal question poses a significant challenge. this is especially true for llms, which must also explain their question selections. thirdly, the lack of an effective feedback mechanism to optimize question selection is a challenge. in computerized adaptive testing, decisionmaking is continuous, with past selections and experiences crucial for current decisions. ensuring past decisions improve future question selection is essential for maintaining accurate assessments. to address these challenges, we propose an llm-agent-based cat (lacat). lacat incorporates three essential components: (1) the summarizer, which accurately depicts user profiles from student response data and further enhances the understanding of students' abilities, addresses the challenge of insufficient selection guidance from student's abilities estimation vectors. ## experiments in this section, we conduct extensive experiments on three real-world datasets to evaluate the performance of lacat. our experiments focus on answering the following research questions: furthermore, we conduct several quantitative experiments to further demonstrate the effectiveness of lacat. more details can be found in appendix b. ## conclusion we introduce lacat, a framework integrating large language models with computerized adaptive testing. it comprises the summarizer for student profiling, the reasoner for question selection and explanation, and the critic for process optimization, enhancing adaptive testing in real educational settings. compared to traditional rule-based and datadriven methods, lacat can generate clearer, more transparent, and more reasonable questions. this integration of large language models with computerized adaptive testing demonstrates a promising avenue for enhancing educational assessment.