Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Bias (2024)

Abstract: Recent studies show that instruction tuning (IT) and reinforcement learning from human feedback (RLHF) improve the abilities of large language models (LMs) dramatically. While these tuning methods can help align models with human objectives and generate high-quality text, not much is known about their potential adverse effects. In this work, we investigate the effect of IT and RLHF on decision making and reasoning in LMs, focusing on three cognitive biases—the decoy effect, the certainty effect, and the belief bias—all of which are known to influence human decision-making and reasoning. Our findings highlight the presence of these biases in various models from the GPT-3, Mistral, and T5 families. Notably, we find a stronger presence of biases in models that have undergone instruction tuning, such as Flan-T5, Mistral-Instruct, GPT3.5, and GPT4. Our work constitutes a step toward comprehending cognitive biases in instruction-tuned LMs, which is crucial for the development of more reliable and unbiased language models.1

## introduction advanced fine-tuning methods, like instruction tuning (it) and reinforcement learning from human feedback (rlhf), have recently emerged as essential paradigms for improving the alignment of language models (lms) with human objectives @xcite . although widely adopted @xcite , the specific cases in which it and rlhf enhance model behavior to resemble human behavior, and the mechanisms involved in this process, remain unclear. in this work, we delve into the impact of it and rlhf on decision-making and reasoning in lms. recent studies highlighted to some extent cognitive-like biases in pretrained lms @xcite and instruction-tuned models @xcite . we take a step further, exploring the consequences of it and rlhf interventions on lms' cognitivelike behavior. we inspect three well-researched and fundamental biases: the decoy effect @xcite , the certainty effect @xcite , and belief bias @xcite . these biases reflect basic inconsistencies in human decisionmaking (decoy and certainty effects) and fallacies in logical reasoning (belief bias) that are prevalent, persistent, and consequential @xcite . the conventional approach to studying cognitive biases in humans is to design simple experiments that elicit from human subjects either judgments or decisions that are likely to reflect a target bias. many of these experiments involve question answering; within this setup, we empirically evaluate the degree of bias exhibited by several pretrained lms, and compare them to their corresponding fine-tuned variants. our findings indicate that applying it or rlhf tuning either introduces cognitive-like biases into text generation, or amplifies these biases if they already exist. given that fine-tuned models are typically considered to be superior, our results point to an important limitation of tuning based on instructions or human feedback. fine-tuned models are also often regarded as potentially less biased, such as in domains like gender or race, since they can be explicitly trained to avoid these biases or having personal preferences (openai, 2023). our results suggest that, similarly to debiasing attempts @xcite , improving align-ment with respect to one human objective may result in behavior that is unintended with respect to others. ## cognitive biases: background and experimental setup rational choice theory depicts humans as making choices in a manner that maximizes value on the basis of fixed preferences. a large body of literature is devoted to describing how actual human behavior deviates from this ideal. cognitive biases aim to explain regular inconsistencies in choice behavior by revealing our susceptibility to 'supposedly irrelevant' factors, such as the context of the decision task, or its framing. cognitive biases are therefore defined and measured by how judgments and decisions deviate from the rational or logical ideal in response to contextual changes. our research targets three biases that are both prevalent and well-established. the first two are the decoy effect and the certainty effectdecision-making biases that relate to special cases of the more general prospect theory @xcite , with each capturing one of its distinct aspects: the perception of value, and the perception of uncertainty. the third bias is the belief bias, a logical fallacy in judgment, which was previously observed in a closed-off pretrained model @xcite . in this section, we provide for each bias some general background and a description of its classic experimental setup, which we later build on. ## data and evaluation we next describe our data generation process and evaluation scheme. section 3.1 outlines our semi-automatic approach for generating specific datasets, each designed to probe a certain cognitive bias and to evaluate the existence of biased 'behavior' in models. section 3.2 provides further details on these generated datasets. section 3.3 formally introduces our proposed bias score, intended to quantify the degree of bias exhibited by a model based on its predictions on the generated data. ## results we measured the significance of the differences between models using the difference-indifferences method @xcite . all differences were significant except for davinci and davinci-002 in the belief valid and in decoy expensive, t5 and flan-t5 in the certainty effect, and mistral and mistral-instruct in the decoy cheaper. lms exhibit biases that align with human bias theory. intriguingly, our investigation reveals a convergence between the decisionmaking biases observed in the models and the well-established theory on irrational biases inherent in human decision-making processes. recall from section 3.3 that positive values indicate alignment between bias scores and human biases. indeed, tuning using instructions or human preferences generally makes bias scores increasingly high. the negative bias score exhibited by davinci in the decoy biases can be explained by choice criteria which, unlike humans, are not value-depended. in this exceptional case, the model chose the last option offered almost all the time, regardless of the options' content, making its choice more focused on positional preferences. this finding emphasizes the role of fine-tuning on bias amplification on previously undiscovered biases. in addition, the similarity between the theory on human biases and model biases highlights the potential connection of inherent biases ingrained in human decision-making processes to tuning methods that induce the models to replicate human behaviors. ## analysis we delve into the effects of few-shot in section 6.1 and explore different attributes of the decoy effect and belief bias in sections 6.2 and 6.3. ## discussion and conclusions our study examines the influence of it and rlhf on lms' decision-making through cognitive bias analysis. we reveal the presence of these biases across models, notably in models amplified with it and rlhf. these insights enhance our understanding of biases in fine-tuned models, widely considered superior to the pretrained models. in section 7.1, we explore the potential consequences of identifying these biases and the challenges in addressing them. section 7.2 delves into research paths investigating the source of these biases in the training of lms.