Cognitive Bias in Decision-Making with LLM s (2024)

Abstract: Large language models (LLMs) offer significant potential as tools to support an expanding range of decision-making tasks. Given their training on human (created) data, LLMs have been shown to inherit societal biases against protected groups, as well as be subject to bias functionally resembling cognitive bias. Human-like bias can impede fair and explainable decisions made with LLM assistance. Our work introduces BiasBuster, a framework designed to uncover, evaluate, and mitigate cognitive bias in LLMs, particularly in high-stakes decision-making tasks. Inspired by prior research in psychology and cognitive science, we develop a dataset containing 13,465 prompts to evaluate LLM decisions on different cognitive biases (e.g., prompt-induced, sequential, inherent). We test various bias mitigation strategies, while proposing a novel method utilizing LLMs to debias their own human-like cognitive bias within prompts. Our analysis provides a comprehensive picture of the presence and effects of cognitive bias across commercial and open-source models. We demonstrate that our selfhelp debiasing effectively mitigates model answers that display patterns akin to human cognitive bias without having to manually craft examples for each bias.

## introduction llms exhibit strong performance across multiple tasks @xcite , such as summarizing documents @xcite , answering math questions @xcite or chat-support @xcite . these capabilities lead humans to increasingly use llms for support or advice in their day-to-day decisions @xcite . however, models suffer from various algorithmic biases, requiring procedures to evaluate and mitigate bias @xcite @xcite . in addition to societal bias, llms can show answer patterns similar to human-like cognitive bias, which can implicitly mislead a user's decision-making @xcite . cognitive bias refers to a systematic pattern of deviation from norms of rationality in judgment, where individuals create their own "subjective reality" from their perception of the input @xcite , and leads to inconsistent decision-making. cognitive bias arises in human decision-making as well as human-ml interaction @xcite . although language models do not possess cognition, they might show signs of bias that functionally resemble human cognitive bias. hence, when llms aid humans in decision-making, such as evaluating individuals, these models must be properly audited @xcite . cognitive and social biases are highly connected. cognitive biases are systematic tendencies leading to error -such as the tendency to interpret information in a way that confirms and reinforces preexisting beliefs and opinions. connected to these are social biases, formed automatically by impressions of people, based on the social group that they are a member of @xcite . different from societal bias where behavior is influenced by social and cultural background, cognitive bias arises from the information processing mechanisms in human decision-making procedures, often influenced by the setup of the task @xcite . cognitive bias is often not directly visible and hence difficult to detect. our work introduces a novel approach to quantifying and mitigating patterns akin to human cognitive bias in llms using cognitive bias-aware prompting techniques. our work proposes biasbuster (figure 1 ), a systematic framework that encapsulates quantitative evaluation and automatic mitigation procedures for human-like cognitive bias. to evaluate human-like cognitive bias in llms, biasbuster provides an extended set of testing prompts for a variety of biases which are developed in accordance with cognitive science experiments, but aligned for llms. we create metrics to assess how large language models respond to prompts categorized as either "biased" or "neutral" in relation to human-like cognitive biases. biasbuster compares different debiasing strategies (some shown to also be effective on humans) in zero-shot and few-shot prompting. to minimize manual effort in prompt creation, we propose a novel prompting strategy where a language model debiases its prompts and helps itself to be less subject to bias (we call it selfhelp). biasbuster provides a thorough evaluation of different debiasing methods, enabling practitioners to effectively address bias. to avoid cross-contamination with existing data that a model might have been trained on, bias-buster provides novel prompts for a high-stakes decision-making scenario -student admissions for a college program, where we generate and provide sets of cognitive bias testing prompts and debiased prompts. these testing prompts quantitatively evaluate various patterns for cognitive biases, focusing on llm self-consistency and decision confidence. the debiased prompts assess the utility of various mitigation techniques, specifically focusing on the ability of llms to debias their prompts. ## testing for patterns of cognitive bias in llms we classify human-like cognitive bias into three novel categories concerning their manifestation in llms. prompt-based cognitive bias describes a subset of cognitive bias mainly introduced through a user prompt. inherent bias is instilled in the model through the training data. inherent bias is not necessarily obvious in the prompt but within the general task itself. sequential bias is induced by previous model answers. we design experiments to test decision patterns similar to human cognitive bias, based on definitions from human cognitive bias literature. our work aims to align all bias groups (prompt-based, sequential, inherent) as much as possible with the same evaluation metrics. however, the detection of individual biases in each group has to be tackled with separate metrics to be able to account for the nuances of the bias group. for all biases, we strive to find a metric of "consistency". compared to evaluating decision patterns on human participants, llms have the distinct advantage of being testable under various study conditions through repeated prompting to evaluate consistency. in the following, we describe the creation of the prompt dataset. ## mitigating cognitive bias in llms there are different approaches to mitigating decision patterns similar to human cognitive bias in llms. we group these approaches into zero-shot approaches, which can give additional information about the existence of cognitive bias without giving any examples, few shot approaches which can give examples of specific desired or undesired behavior, and self-mitigation approaches, which use the model to debias themselves (figure 2 ). ## results we evaluate four language models with different capabilities. we evaluate state-of-the-art commercial language models gpt-3.5-turbo and gpt-4 foot_1 , as well as open-source large language models llama 2 in sizes 7b and 13b. ## conclusion a model showing patterns similar to human cognitive bias can make inconsistent decisions, which can lead to unfair treatment in high-stakes decisionmaking. our work provides a dataset of 13,465 prompts to test for inherent, prompt-based, and sequential patterns of cognitive bias in llms. we propose metrics to evaluate patterns of different kinds of biases and different mitigation procedures. our mitigation procedures include a novel selfdebiasing technique for patterns of cognitive bias that enables models to autonomously rewrite their own prompts, successfully removing bias-inducing parts of the prompt and enabling more consistent decisions in llms. we observe our self-debiasing technique to be specifically successful in highcapacity models. this method has the advantage of not requiring manually developed examples as debiasing information to give to the model and applies to a variety of biases.