A Comprehensive Evaluation of Cognitive Biases in LLM s (2025)

Abstract: We present a large-scale evaluation of 30 cognitive biases in 20 state-of-the-art large language models (LLMs) under various decision-making scenarios. Our contributions include a novel general-purpose test framework for reliable and large-scale generation of tests for LLMs, a benchmark dataset with 30,000 tests for detecting cognitive biases in LLMs, and a comprehensive assessment of the biases found in the 20 evaluated LLMs. Our work confirms and broadens previous findings suggesting the presence of cognitive biases in LLMs by reporting evidence of all 30 tested biases in at least some of the 20 LLMs. We publish our framework code and dataset to encourage future research on cognitive biases in LLMs: https://github.com/simonmalberg/cognitive-biases-in-llms.

## introduction transformer-based llms @xcite and other foundation models (e.g., @xcite have gained significant attention in recent years. at an accelerating pace, models are becoming larger and more capable, conquering additional modalities such as vision and speech @xcite . this makes llms increasingly attractive for complex reasoning @xcite and decision-making tasks @xcite . however, using llms for high-stakes decision-making, e.g., for managerial or public policy decisions, comes with severe risks, as they may produce flawed yet convincingly articulated outputs, such as hallucinations @xcite . humans are at most boundedly rational @xcite and biased @xcite . llms are trained on human-created data and typically fine-tuned on human-defined instructions @xcite and through reinforcement learning from human feedback (rlhf) @xcite figure 1 : an llm changes its answer as the framing of the decision changes, indicating the susceptibility of the llm to the framing effect. building on previous work that found some cognitive biases in llms, we share three main contributions for a much broader understanding of cognitive biases in llms: cognitive biases in llms recently, llms' presence in high-stakes decision-making has rapidly become ubiquitous @xcite . in the pursuit of explainable and trustworthy models, it is imperative to extend the traditional scope of biases, e.g., gender and ethical ones @xcite , to account for biases and heuristics of cognition that directly impact the rationality of llms' judgments @xcite . earlier works in this direction (talboy and fuller, 2023; @xcite focused on detecting effects on the level of individual prompts. separate research directions investigated challenges of cognitive bias detection and mitigation for lists of less than six cognitive biases @xcite , particular llm roles @xcite @xcite , or specific domains @xcite . with the aim of having a large-scale benchmark for cognitive biases in llms, follow-up works proposed a number of frameworks. notably, a framework proposed by @xcite encapsulates quantitative evaluation and automatic mitigation of cognitive biases; however, its variability is constrained to only five biases and a single scenario of student admissions -two limitations we directly address in this paper. the recent contribution of @xcite explores a similar direction through multi-agent systems. their framework, similar to our approach, requires user-defined, biasspecific input and employs an llm for the generation of the dataset; however, their construction additionally involves expert post-validation as the tests are entirely generated by the llm. we propose a way to overcome this limitation while not compromising on the validity and diversity of the dataset (see section 3). the development of a scaleable, systematic, and expandable benchmark would allow for further progress in the task of comprehensive mitigation of cognitive biases in llms (e.g., @xcite and thus comprises the main motivation for this paper. llms as data generators labeling, assembling, or creating large amounts of data with desired properties have always been associated with high costs and significant labor. moreover, this process is inherently intricate due to the annotator's and the instructions' biases @xcite . recent impressive performance by the state-of-theart llms (e.g., @xcite has shifted the perspective on these tasks, calling llms to the rescue. the surveys by @xcite , @xcite summarize the progress in this direction. notably, @xcite showed the costeffectiveness of llm data creation and competitive performance of models trained on this data. diversity of prompts is shown to directly impact the diversity of generated data @xcite , with works proposing self-generated instructions @xcite and multi-step @xcite approaches to achieve the respective enhancement. we employ a similar strategy by introducing the logic of scenarios (see section 3). earlier findings @xcite ) elicited flaws in llms' instruction-following, and more recent work @xcite still indicates the struggle with complex instructions. moreover, synthetic datasets are found to exhibit biases @xcite . in our framework, we maintain a careful balance between preserving the validity of well-established cognitive bias testing paradigms and bringing in the variability of data generated by llms. ## test framework we introduce a novel framework for reliably generating diverse and large-scale sets of tests for evaluating llms. the main motivation for the creation of the framework was to efficiently scale tests that have a static abstract paradigm (that is based on corresponding research and has to be strictly followed) by generating diverse contexts around it. the framework comprises four entities and three functions. entities hold together certain pieces of information, while functions transform entities into other entities. all entities and functions are explained in the following. we use lower case letters t, s, c, r, b, ... to denote entities and their contents. functions are denoted by upper case letters test case: anchoring bias control template treatment template situation: suppose you are [[a/an]] [[type]] manager at [[organization]]. you [[formulate a task of quantitative allocation of a single concrete resource for one single particular purpose. do not include any numbers.]]. prompt: which allocation level do you choose for this purpose? answer options: option 1: 0% option 2: 10% ... option 11: 100% situation: suppose you are [[a/an]] [[type]] manager at [[organization]]. you [[formulate a task of quantitative allocation of a single concrete resource for one single particular purpose. do not include any numbers.]]. prompt: do you intend to allocate more than {{anchor}}% for this purpose? which allocation level do you choose for this purpose? answer options: option 1: 0% option 2: 10% ... option 11: 100% scenario a marketing manager at a company from the telecommunication services industry deciding the best strategy to launch a new service package on social media platforms. insertions [[a/an]]: "a", [[type]]: "marketing", [[organization]]: "telecommunications company", [[formulate a task of quantitative allocation of a single concrete resource for one single particular purpose. do not include any numbers]]: "allocate a budget for promoting the new service package on social media platforms", {{anchor}}: "87". ## results & discussion a perspective on the absolute biasedness foot_3 of the models in relation to other model characteristics such as size and general capability is provided in figure 5 . as a proxy for a model's general capability, we show each model's chatbot arena foot_4 score on the horizontal axis. while there seems to be no when no such score is available, we take the mean of the other models' scores and mark the model with a '*'. gpt-4o gpt-4o mini gpt-3.5 turbo llama 3.1 405b llama 3.1 70b llama 3.1 8b llama 3.2 3b llama 3.2 1b claude 3 haiku gemini 1.5 pro gemini 1.5 flash gemma 2 27b gemma 2 9b mistral small mistral large wizardlm-2 8x22b wizardlm-2 7b phi-3.5 qwen2.5 72b yi-large random average information bias in-group bias survivorship bias framing effect anchoring halo effect loss aversion hindsight bias bandwagon effect hyperbolic discounting self-serving bias confirmation bias illusion of control mental accounting negativity bias availability heuristic fundamental attribution error stereotyping not invented here escalation of commitment risk compensation social desirability bias optimism bias conservatism reactance planning fallacy endowment effect anthropomorphism status-quo bias disposition effect average average absolute 3.2 1b and 3b) show similar bias characteristics. further, three of the largest models tested can be found in the bottom three branches of the dendrogram, apparently showing similar behaviors. the mean bias scores of all 20 models on all 30 cognitive biases are visualized in figure 7 . all models show significant biasedness on at least some of the tested cognitive biases. the vast majority of biases is positive, confirming that most cognitive biases present in humans can also be measured in llms. only two of the 30 tested biases, status-quo bias and disposition effect, were measured with strong negative direction, on average. on both biases, negative scores express a model's preference for change. the random model shows no biasedness on average, highlighting our met-ric's strength as an unbiased estimator. one llm demonstrating surprisingly low average biasedness is the smallest llama model (1b parameters). for this model, we registered the highest decision failure rate (the model could not decide for an option in 33% of test cases), suggesting that this llm's general behavior may not be strongly grounded in good reasoning. ## conclusion we have presented a comprehensive evaluation of 30 cognitive biases in 20 state-of-the-art llms. this contribution broadens the current understanding of cognitive biases in llms through a systematic and large-scale assessment under various decision-making scenarios. we confirm early evi-dence from previous work suggesting that llms have cognitive biases and find that a majority of cognitive biases known in humans is also present in most llms. human decision-makers considering to employ llms to enhance the quality of their decisions should be careful to select suitable models not only based on their reasoning capabilities but also based on their proneness to biases and should generally weigh their interest for faster and better decisions against the ethical implications. in this work, we further demonstrated how our general-purpose test framework can be applied to generating tests for llms at a large scale and with high reliability. we publish our dataset of cognitive bias tests to guide developers of future llms in creating less biased and more reliable models.