Exploring the Cognitive Knowledge Structure of Large Language Models: An Educational Diagnostic Assessment Approach (2023)

Abstract: Large Language Models (LLMs) have not only exhibited exceptional performance across various tasks, but also demonstrated sparks of intelligence. Recent studies have focused on assessing their capabilities on human exams and revealed their impressive competence in different domains. However, cognitive research on the overall knowledge structure of LLMs is still lacking. In this paper, based on educational diagnostic assessment method, we conduct an evaluation using MoocRadar, a meticulously annotated human test dataset based on Bloom Taxonomy. We aim to reveal the knowledge structures of LLMs and gain insights of their cognitive capabilities. This research emphasizes the significance of investigating LLMs’ knowledge and understanding the disparate cognitive patterns of LLMs. By shedding light on models’ knowledge, researchers can advance development and utilization of LLMs in a more informed and effective manner.

## introduction large language models (llms), such as gpt series @xcite , flan @xcite , and palm @xcite , have gained significant attention worldwide due to their remarkable ability. given their unprecedented human-like performances, researchers have started to explore alternative evaluation metrics beyond traditional benchmarks like mmlu @xcite and big-bench @xcite . existing works on llms evaluation with exams. researchers have long sought models capable of passing human exams @xcite . recently, a new approach simulates professional exams designed for humans to evaluate llms. for example, openai (2023) reports the performance of gpt series on a variety of exams, including ap exams, sat, leetcode, and so on. there are also emerging benchmarks that comprise common standardized exams, such as agieval @xcite , c-eval @xcite , m3exam @xcite , and cmexam @xcite . however, although standardized exams contain diverse information, these works condense them into a single overall score, lacking structured understanding of llms' knowledge and cognitive patterns. for example, while llms demonstrate exceptional performance on tasks challenging for humans, they might still struggle with basic knowledge, as illustrated in figure 1 , which may lead to over-estimation of the validity of model generated contents. therefore, there is a pressing need for further research of models' knowledge and cognitive distribution in comparison to humans. ## experiment in this section, we conduct experiments and analyze the results from three perspectives in the following subsections. we assign a score of 1 to each question type. following standardized exams, for multiple questions, models receive a score of 0.5 if they fail to select all correct options. we then calculate the average score across questions. ## conclusion in this work, we introduce a new research question on llms analyzing, which calls for a deeper understanding of the knowledge structure of these models. we use educational diagnostic assessment as a tool to test the performance of llms on various dimensions, and develop a metric to measure the similarity of their knowledge structure with humans. we provide findings and discussion for insight into research on the cognition of llms.