PATCH ! P sychometrics- A ssis T ed B en CH marking of Large Language Models against Human Populations: A Case Study of Proficiency in 8th Grade Mathematics (2025)

Abstract: Many existing benchmarks of large (multimodal) language models (LLMs) focus on measuring LLMs’ academic proficiency, often with also an interest in comparing model performance with human test takers’. While such benchmarks have proven key to the development of LLMs, they suffer from several limitations, including questionable measurement quality (e.g., Do they measure what they are supposed to in a reliable way?), lack of quality assessment on the item level (e.g., Are some items more important or difficult than others?) and unclear human population reference (e.g., To whom can the model be compared?). In response to these challenges, we propose leveraging knowledge from psychometrics—a field dedicated to the measurement of latent variables like academic proficiency—into LLM benchmarking. We make four primary contributions. First, we reflect on current LLM benchmark developments and contrast them with psychometrics-based test development. Second, we introduce PATCH: a novel framework for Psychometrics-AssisTed benCHmarking of LLMs. PATCH addresses the aforementioned limitations. In particular, PATCH enables valid comparison between LLMs and human populations.Third, we demonstrate PATCH by measuring several LLMs’ proficiency in 8th grade mathematics against 56 human populations. We show that adopting a psychometrics-based approach yields evaluation outcomes that diverge from those based on current benchmarking practices. Fourth, we release 4 high-quality datasets to support measuring and comparing LLM proficiency in grade school mathematics and science with human populations.

## introduction large language models (llms), including their multimodal variants like vision language models, have witnessed significant advancements in recent years. these models are typically evaluated on established benchmarks that assess their performance across a diverse set of tasks such as commonsense reasoning @xcite @xcite , coding @xcite and academic proficiency. academic proficiency, in particular, has become a crucial part of llm evaluation, as evidenced by the large number of related benchmarks like mmlu, arc, gsm8k, drop and math @xcite @xcite @xcite , as well as recent model technical reports' increasing focus on them @xcite . in these benchmarks and reports, the contrast between llm performance and human performance is often highlighted, sparking media coverage and discussions. despite their success in advancing llm research and shedding light on the artificial versus human intelligence debate, existing benchmarks have notable limitations. the first concern is measurement quality: do these benchmarks measure what they are supposed to in a reliable way? many benchmarks are created via crowd-sourced knowledge, by asking a convenience group of individuals (e.g., crowd workers, paper authors) to create new test items (e.g., gsm8k, drop) or collecting them from (often undocumented) sources (e.g., websites, textbooks, school exams) (e.g., math, mmlu, arc). without domain expert input and rigorous testing of item quality, undesirable outcomes can occur, including a mismatch between a benchmark and its claimed measurement goal, missing information in a question, wrong answer keys, and low data annotation agreement (e.g., @xcite @xcite . second, current benchmarks do not account for differences across test items, such as item discrimination and difficulty (see section 3.1). for example, consider three items a (easy), b (hard) and c (hard). while answering correctly to a and b would result in the same accuracy score as answering correctly to b and c, the latter (i.e., answering correctly to more difficult items) would imply higher proficiency. furthermore, items that are too easy or too difficult (i.e., low discrimination) will fail to differentiate models (and humans) of different proficiency levels. thus, without accounting for item differences, benchmarking results, especially model (versus human) rankings, can be misleading. third, while many benchmarks compare llms against humans, the human populations under comparison remain unclear @xcite . for instance, human performance in math is based on the benchmark's authors; in mmlu, crowd workers; in math, 6 university students. using such convenience samples (with little information about sample characteristics), the resulting human performance cannot be generalised to other human samples or populations. to address these challenges, we propose leveraging insights from psychometrics-a field dedicated to the measurement of latent variables like academic proficiency-into llm benchmarking practices. in particular, we draw on two research areas in psychometrics: item response theory (irt) (section 3.1) and test development (section 3.2). the former enables more accurate estimation of academic proficiency on a standardised scale by taking into account both the characteristics of the test items as well as the abilities of the llms and individuals being assessed, compared to common practices in llm benchmarks (e.g., using mean scores, percentages of correct responses). it can also provide diagnostic information about the quality of each test item. the latter, test development knowledge, can help to build high quality llm benchmarks where valid comparison to specific human populations can be made (section 3.3). our paper makes four primary contributions. first, we reflect on current llm benchmark development processes and contrast them with psychometrics-based test development, thereby revealing the limitations of current llm benchmarks and the potential benefits that patch/psychometrics can bring to llm benchmarking. second, we present patch: a novel framework for psychometrics-assisted benchmarking of llms (figure 1 ). patch is built upon irt and test development insights from psychometrics and addresses the aforementioned limitations of existing benchmarks. third, we demonstrate the irt part of patch by testing several llms' proficiency in 8th grade mathematics using the released test items and data from trends in international mathematics and science study foot_0 (timss) 2011. we show empirically that an irt-based approach can lead to evaluation outcomes that diverge from those obtained through conventional benchmarking practices and that are more informative, underscoring the potential of patch/psychometrics to reshape the llm benchmarking landscape. fourth, we make our evaluation code based on the patch framework available foot_1 , along with three other mathematics and science datasets based on timss 2011 and 2008 foot_2 . ## related work we are not the first to propose leveraging psychometrics for research on llms and other areas in nlp. for instance, psychometric scales have been used to examine the psychological profiles of llms such as personality traits and motivations @xcite @xcite . the text in these scales can also be used to improve encoding and prediction of personality traits @xcite @xcite . psychometrics-based reliability and validity tests have also been proposed or/and used to assess the quality of nlp bias measures @xcite ), text embeddings @xcite , political stance detection @xcite , annotations @xcite , user representations @xcite , and general social science constructs @xcite . the most closely related work to our paper is the use of item response theory (irt) models in nlp for constructing more informative test datasets @xcite , comparison of existing evaluation datasets and instances (e.g., difficulty, discrimination) @xcite @xcite @xcite , as well as identification of difficult instances from training dynamics @xcite . our work distinguishes itself from these papers in two aspects. first, we do not apply irt to existing llm datasets/benchmarks. instead, we introduce a framework for benchmarking llms by leveraging both irt and test development knowledge from psychometrics. the goal of this framework is to generate new, high-quality benchmarks for llms that warrant valid comparison with human populations. second, we demonstrate our framework with a mathematics proficiency test validated on 56 human populations, and compare llm performance with human performance. to the best our knowledge, we are the first to apply psychometrically validated (mathematics) proficiency tests to llms and make valid model versus human comparison. ## preliminaries in this section, we provide background knowledge on item response theory and test development in psychometrics. ## patch: psychometrics-assisted benchmarking of llms next, researchers use the items from the validated psychometric test to construct prompts for the llm under evaluation and sample responses from the llm. similarly, researchers administer the psychometric test to a representative sample of the reference human population, and collect their responses. these human responses are then used to train and validate the irt model(s) that match the type(s) of items in the psychometric test. the resulting irt model(s) will be fitted to the responses from the llm and the humans, and will subsequently estimate each test taker's latent proficiency score-whether human or llm-along with uncertainty estimates. these final proficiency scores 8 enable valid comparison between the llm and the reference human population. at the heart of patch lies the psychometric test, which not only provides the basis for accurate measurement of the construct (i.e., capability of interest) but also enables valid comparison between llms and human test takers. unfortunately, developing such a test can be a long and expensive process; utilising existing tests can be a shortcut, which should satisfy three conditions: 1) clear human population reference; 2) test items available in the public domain; 3) human responses and/or item parameter estimates available. the second and third are in practice difficult to meet, as many test institutes do not make their test items public due to commercial interests (e.g., sat) or the need to measure trends over time (e.g., pisa). collaboration with test institutes would alleviate this problem and additionally mitigate the likely data contamination issue with many public benchmarks. to the best of our knowledge, among academic proficiency tests, only timss and pirls tests from certain years can be readily used for patchbased llm benchmarking without formal collaboration with test institutes. timss measures proficiency in grade school mathematics and science (4th grade, 8th grade, and final year of secondary well distributed across different difficulty levels, and showing high reliability (e.g., high internal consistency) and validity (e.g., the sizes and directions of the empirical correlations among test items match theoretical expectations). 8 these latent proficiency scores are typically standardised z-scores (i.e., mean of 0 and standard deviation of 1), which sometimes go through further normalisation (e.g., re-scaling to a mean of 500 and a standard deviation of 100). ## demonstration: leveraging irt in patch to measure llm proficiency in 8th grade mathematics in this section, we conduct an experiment to demonstrate the benefits and differences irt modelling can make under our patch framework, compared to the traditional benchmark scoring approach. note that we focus on the irt part of patch instead of also on test development, because we lack the resources to develop our own valid test, and that there is no existing llm benchmark that has the same measurement target as timss 2011 (i.e., 8th grade mathematics proficiency) to enable comparison in terms of test development. nevertheless, we hope our detailed comparison between llm benchmark devleopment and psychomtric test development in section 3.3 suffices to fill in this gap. ## conclusion in this paper, we propose patch, a psychometricsinspired framework to address current limitations of llm benchmarks, including questionable measurement quality, lack of quality assessment on the item level and unwarranted comparison between humans and llms. we demonstrate the irt part of patch with an 8th grade mathematics proficiency test and show evaluation outcomes that diverge from those based on existing benchmarking practices, especially when comparison with human test takers is made. this underscores the potential of patch to reshape the llm benchmarking landscape. furthermore, we release 4 datasets that meet the requirements of patch, supporting the measurement of llm proficiency in grade school math and science and its comparison with human performance. we hope to bring the llm research community a step forward towards more scientific benchmarking and inspire more research in this direction. we also encourage researchers to collaborate with test institutes when developing new benchmarks, especially those that aim to measure cognitive capabilities.