Mind the Biases: Quantifying Cognitive Biases in Language Model Prompting (2023)

Abstract: We advocate the importance of exposing uncertainty on results of language model prompting which display bias modes resembling cognitive biases, and propose to help users grasp the level of uncertainty via simple quantifying metrics. Cognitive biases in the human decision making process can lead to flawed responses when we are under uncertainty. Not surprisingly, we have seen biases in language models resembling cognitive biases as a result of training on biased textual data, raising dangers in downstream tasks that are centered around people’s lives if users trust their results too much. In this work, we reveal two bias modes leveraging cognitive biases when we prompt BERT, accompanied by two bias metrics. On a drug-drug interaction extraction task, our bias measurements reveal an error pattern similar to the availability bias when the labels for training prompts are imbalanced, and show that a toning-down transformation of the drug-drug description in a prompt can elicit a bias similar to the framing effect, warning users to distrust when prompting language models for answers.

## introduction cognitive biases describe the flawed human response patterns for decision making under uncertainty @xcite @xcite @xcite . for example, when people are biased by the availability heuristic, they make probability judgments based on the ease with which information comes to mind @xcite . knowing cognitive biases can help predict what types of error will be made, which is also helpful for interpreting behaviors of generative systems such as language models, as they may err in a similar pattern as humans do, especially when the data used to build the systems carry man-made biases @xcite . we are inspired by leveraging cognitive biases -systematic error patterns which deviate from rational decisions -to study error patterns of language models. we highlight the importance of exposing uncertainty to users of language models @xcite , and leverage cognitive biases to quantify the level of imprecision in results when performing language model prompting via simple, perceptual metrics. some would argue that the biases in machines are a result of unmatched data distributions in training and test sets. however, merely matching training and test distributions does not solve the problem of biased predictions for long-tailed input distributions. for example, on the drug-drug interaction (ddi) dataset @xcite , the training and test distributions are identically skewed, and there are 100 times more negative (non-interacting) drug pairs than the interacting drug pairs in both sets. though performances on the development set and test set are not too bad for positive class inputs with a prompt-based bert model @xcite , the model still most frequently mistakes positive pairs for negative pairs, as shown by the confusion matrix in the left part of figure 1 . this label bias towards negative mimics the availability bias. the availability bias is one of the most common cognitive biases in real life, especially in doctors' diagnoses which increase with years of training @xcite . moreover, an equal number of samples in each class during training does not guarantee that a "majority" class does not exist, especially when the input distribution of the negative class has a higher variance (i.e., highly diversified samples in the negative class) and the samples within the positive class share more common characteristics. considering the input variance, even though sample sizes are the same, the negative class still can be viewed as the majority class. more on this can figure 1 : left: an illustration of the negative class vs. the four positive classes of the ddi data. the confusion matrix of the development set predictions shows that the most frequent wrong prediction for positive input pairs is the negative class (the majority label), calling for the need to quantify the availability bias. right: an observation on the framing effect paraphrase (in purple) and the original sentence (in blue): the paraphrase describes the two non-interacting drugs in a slightly more toned-down way compared to the original sentence, and it leads to the correct prediction whereas the original sentence does not, calling for the need to quantify the framing effect. in addition to the availability bias, the framing effect is another common cognitive bias. the framing effect is prevalent in medical diagnoses @xcite , where doctors intentionally frame diagnoses positively ("90% chance to survive") or negatively ("10% chance to die") to make patients perceive the results differently. it was recently found that a failure mode of a code generation language model codex @xcite resembles the framing effect -how an input prompt is framed changes the model output predictably @xcite . in our study, prompting bert with paraphrases generated by toning-down the original inputs improves prediction results, suggesting a bias brought by the tone of the input sentences. it is important to see that the biases found above are not the expected behavior of bert as a promptbased classification model, and our goal of this paper is to analyze these failure modes from the lens of cognitive biases and quantify them via simple metrics. we are devoted to warning practitioners about the risks of biased language model predictions, especially on biomedical tasks. on a case study of the ddi extraction task, we measure output label distribution with content-free prompts and how model output changes when applying a toning-down transformation to prompt texts. our key findings are: ## proposed metrics we propose two metrics for quantifying the bias modes by the availability bias and the framing effect respectively in prompt-based bert, helping users perceive how much bias comes with prompt-ing results. ## discussion few-shot training vs. the availability bias. we have seen from ## conclusion in this work, we identify and quantify two bias modes in bert's prompt-based predictions, leveraging the availability bias and the framing effect on biomedical drug-drug interaction extraction. the error mode of the availability bias suggests that the label for a prompt still matters for prompt-based learning, as shown by a large availability bias score towards the majority class, which is 52% on a scale of 0 to 80%. we also find that a toning-down transformation of the drug-drug description in a prompt can elicit a bias similar to the framing effect, since when we tone down the input description, 84.4% of drug pairs that are wrongly classified with the original text are now correctly predicted with their toned-down paraphrases. for highly toned-down paraphrases (as measured by f fep above 0.99), this conditional probability reaches 98.7%. the magnitude of these biases suggests that language model users need to be aware of the imprecision of their prompting results.