Triangulating LLM Progress through Benchmarks, Games, and Cognitive Tests (2025)

Abstract: We examine three evaluation paradigms: standard benchmarks (e.g., MMLU and BBH), interactive games (e.g., Signalling Games or Taboo), and cognitive tests (e.g., for working memory or theory of mind). First, we investigate which of the former two—benchmarks or games—is most effective at discriminating LLMs of varying quality. Then, inspired by human cognitive assessments, we compile a suite of targeted tests that measure cognitive abilities deemed essential for effective language use, and we investigate their correlation with model performance in benchmarks and games. Our analyses reveal that interactive games are superior to standard benchmarks in discriminating models. Causal and logical reasoning correlate with both static and interactive tests, while differences emerge regarding core executive functions and social/emotional skills, which correlate more with games. We advocate for the development of new interactive benchmarks and targeted cognitive tasks inspired by assessing human abilities but designed specifically for LLMs.

## introduction evaluating llms is critical to track progress, identify blind spots, and ultimately advance towards the kind of language-based ai systems we want as a society @xcite . currently, the most widespread way to evaluate llms is by means of large benchmarks made up of miscellaneous tasks. pre-llm benchmarks such as glue and superglue @xcite have been replaced by even larger evaluation suites such as mmlu (measuring massive multitask language understanding; @xcite , mmlu-pro @xcite , or bbh (big-bench hard; @xcite . models with high performance on these benchmarks are taken to possess extensive world knowledge along with complex problem-solving abilities. this trend has promoted standardisation in llm evaluation protocols, with online leaderboards constantly updated as new models are released. despite this undeniable benefit, general-purpose benchmarks, like those mentioned above, are not without problems. evaluation results may be inflated by data contamination (see, e.g., gema et al. 2025 for mmlu) and distorted by model sensitivity to prompt format @xcite . moreover, by design, such benchmarks overlook actual language use in favour of knowledge-intensive tasks where success is measured against gold reference answers provided in a single conversational turn. this contrasts with the view, put forward by philosophers and psycholinguists alike @xcite @xcite , that the quintessence of language resides in situated language use, i.e., using language for a purpose in social and task-based multi-turn interactions @xcite . this situated and interactive view underpins a parallel trend where llms are evaluated as goal-directed language users by means of interactive games @xcite @xcite . 1 this interactive evaluation paradigm goes beyond single-turn text generation, which is critical for deploying llms as agents. additionally, it is less susceptible to data contamination because the vast space of possible multi-turn interactions is unlikely to be fully represented in the training data. yet, despite these advantages, it is not easy to pinpoint which specific abilities underpin models' performance on interactive language games-a difficulty that to some extent also applies to static question-answering benchmarks such as mmlu. in this paper, we study these two evaluation paradigms-general-purpose benchmarks and interactive games-and show that they can provide complementary perspectives. in particular, we investigate whether standard benchmarks or games are more effective in gauging qualitative differences between models, evaluate a selection of current llms from five model families, and find that games highlight more nuanced differences between llms compared to standard benchmarks. while scaling model size leads to systematic improvements on benchmarks, game-based evaluation reveals differences between models of the same size. to shed light on the abilities underlying models' performance on these two evaluation frameworks, we resort to targeted cognitive tests. in the absence of specifically designed tests for llms, we propose a taxonomy of cognitive skills motivated by human neurocognitive science and compile a list of existing evaluation datasets designed to assess each skill in isolation. specifically, we investigate to what extent increased performance on specific abilities correlates with performance gain in standard benchmarks and interactive games, respectively. in sum, our work brings in a unique perspective by studying the interrelations between a diverse set of evaluation paradigms which, to our knowledge, have not been jointly investigated before. ## models we apply our evaluation framework to the following open-weight llms ranging from 3.8b to 72b models: phi-4-mini with 3.8b parameters @xcite , olmo-2-1124 with 7b and 13b parameters @xcite , falcon3 with 10b parameters (falcon team, 2024), qwen2.5 with 7b, 32b, and 72b parameters @xcite , and llama-3 with 8b and 70b parameters @xcite . all these models score above 70% on ifeval (zhou et al., 2023), which we take as indication that they possess the instruction-following capabilities neces-sary for playing games and understanding task descriptions in the cognitive tests. see appendix a for further model details. ## how to identify blind spots in llms evaluation paradigms have most practical use when they allow us to track progress by identifying blind spots in llms. here we compare the two paradigms under study on the extent to which they highlight differences between current models, helping us form hypotheses about possible performance gaps and successful mitigation strategies. ## cognitive abilities we now turn to targeted cognitive tests, which assess specific cognitive abilities deemed essential for effective language use in real-world situations. we explore the use of such tests as a complementary evaluation method to reference-based benchmarks and interactive games. ideally, such an investigation should be carried out using tests tailored to machine intelligence, which unfortunately have not yet been systematically developed @xcite . to make progress in this direction, @xcite advocate for the development of an "artificial cognition evaluation toolkit" through a cyclic process that takes cognitive tests designed for humans as the starting point. following this literature, we present a taxonomy of abilities and a suite of existing evaluation datasets inspired by human cognition, with the aim to make a first step towards more adequate testing of artificial cognition. ## related work a few recent papers have evaluated the abilities of llms beyond standard benchmarks. for example, @xcite propose a framework to evaluate models' linguistic competence (in chomsky 1965's sense) using probing classifiers to disentangle knowledge of, e.g., syntax and semantics from other abilities such as instruction following. @xcite posit that some perceived functional linguistic abilities (e.g., related to emotional intelligence) may be the result of in-context learning, model memory, and formal linguistic knowledge, whereas @xcite propose an evaluation protocol for assessing machine theory of mind based on a taxonomy of existing benchmarks driven by psychology research. in contrast, the present work includes a uniquely comprehensive selection of evaluation tasks, encompassing standard benchmarks, interactive games, and cognitive ability assessments. to our knowledge, this diverse range of evaluations has not been jointly tested in prior work. moreover, we systematically compare their discriminative power and analyse their interrelations-an approach that, to our knowledge, is unprecedented. ## conclusion in this paper, we have examined three llm evaluation paradigms. our analyses revealed that interactive games have superior model discrimination power over standard reference-based benchmarks: models with comparable parametric capacity perform similarly on standard benchmarks, but very differently on games. this suggests that scaling is not sufficient to learn strategies for effective language use in interaction. we have further argued for the use of targeted cognitive tests as a complementary method to triangulate llm progress. while each cognitive ability test alone does not get us very far in the quest for robust llm evaluation, we contend that this type of evaluation paradigm has the potential to enhance our understanding of what fundamental abilities llms must develop to be able to function effectively as language agents, where multiple skills may be required and possibly interact. in our study, we have drawn inspiration from human cognitive tests as a first step in this direction. we hope to encourage the future development of targeted tests tailored to artificial cognition and supplemented with mechanistic analyses.