From Remembering to Metacognition: Do Existing Benchmarks Accurately Evaluate LLM s? (2025)

Abstract: Despite the rapid development of large language models (LLMs), existing benchmark datasets often focus on low-level cognitive tasks, such as factual recall and basic comprehension, while providing limited coverage of higher-level reasoning skills, including analysis, evaluation, and creation. In this work, we systematically assess the cognitive depth of popular LLM benchmarks using Bloom’s Taxonomy to evaluate both the cognitive and knowledge dimensions.Our analysis reveals a pronounced imbalance: most datasets concentrate on “Remembering” and “Understanding”, with metacognitive and creative reasoning largely underrepresented. We also find that incorporating higher-level cognitive instructions into the current instruction fine-tuning process improves model performance. These findings highlight the importance of future benchmarks incorporating metacognitive evaluations to more accurately assess and enhance model performance.

## introduction in recent years, large language models (llms) based on the transformer architecture have made remarkable progress @xcite . gpt-4 introduced multimodal capabilities and achieved human-level cognition on professional benchmarks. the success of chatgpt has spurred major tech companies to invest heavily in foundational llm research and to release their own models, such as llama @xcite , palm @xcite , gemini @xcite , among others. the latest generation of llms exhibits exceptionally remarkable capabilities in text generation, reasoning, and knowledge-based qa, unlocking a wide range of applications from chatbots @xcite to programming copilots @xcite . the release of open-source llms represents a significant milestone in foundational model * the corresponding author. designing benchmarks for llms has become a significant task. substantial work has been devoted to evaluating models' capabilities, including domain-specific knowledge @xcite @xcite , commonsense @xcite @xcite , mathematical @xcite @xcite and code @xcite @xcite reasoning. the current evaluation datasets exhibit two clear trends: first, extensive research has carefully designed more refined benchmarks to assess the capabilities of large models; second, the construction of benchmarks is becoming increasingly complicated.this trend stems from continuous advancements in model performance, necessitating benchmarks with enhanced distinguishable capabilities to effectively evaluate these models. humanity's last exam @xcite is a prime example. this reveals that current most benchmarks do not truly measure a model's true capabilities; they merely construct comparatively harder tasks rather than following any theoretical foundation to guide the design of instructions. we contend that simply augmenting the number or complexity of benchmarks does not inherently enhance the accuracy of model performance assessment. the bias stemming from task-driven design means that when datasets are structured solely around specific tasks, they fail to capture a broader, cognitive evaluation of a model's abilities, which can consequently distort the results @xcite . therefore, we argue that it is essential to provide a guiding framework for bench-mark design, grounded in llms' capabilities, to enable precise measurement of their abilities. given that llms demonstrate their capabilities through cognitive abilities, we propose a comprehensive evaluation framework anchored in cognitive abilities, encompassing instructions partition as well as assessment methodologies. to address this gap, we turn to human cognition theories to support our evaluation of benchmark datasets for large models. we use an assessment framework based on the notion of ability, aligning it with human cognitive capacities rather than merely instruction difficulty. there have been substantial advances in the study of human cognitive theories within the field of education, and bloom's taxonomy framework stands out as the significant theory @xcite . specifically, we draw upon bloom's taxonomy, a framework integrating cognitive and knowledge dimensions to characterize data to systematically organize benchmarks. this approach enables us to address the following four research questions: rq1: do the current benchmarks align with human cognitive dimensions? a1: no, benchmarks are misaligned with human cognition; uneven distribution. rq2: do the current benchmarks align with human knowledge dimensions? a2: no, benchmarks are misaligned with human knowledge; uneven distribution. rq3: do existing evaluations accurately reflect the model's performance? a3: no, benchmarks are inaccurate due to misalignment and lack of advanced metacognitive tests. rq4: how can we organize instructions to improve model performance? a4: the model can be enhanced with analytical and creative-like instructions for fine-tuning. ## conclusion the paper evaluates the reasonableness of existing llm benchmarks through the lens of bloom's taxonomy, analyzing their coverage across cognition and knowledge dimensions. we find that most current benchmarks are heavily biased toward lowlevel cognitive skills such as "remembering" and "understanding" , while higher-level abilities like "analyzing" , "evaluating" , and "creating" are un-derrepresented. similarly, "factual" and "conceptual" knowledge dominate, with limited emphasis on "procedural" and "metacognitive" knowledge. our results suggest that commonly used benchmarks may not fully reflect the comprehensive cognitive capabilities of llms. we propose that future benchmark designs integrate a broader range of high-level, cognitively diverse tasks and include a structured framework to guide their evaluation.