Mitigating Label Biases for In-context Learning (2023)

Abstract: Various design settings for in-context learning (ICL), such as the choice and order of the in-context examples, can bias the model’s predictions. While many studies discuss these design choices, there have been few systematic investigations into categorizing them and mitigating their impact. In this work, we define a typology for three types of label biases in ICL for text classification: vanilla-label bias, context-label bias, and domain-label bias (which we conceptualize and detect for the first time). Our analysis demonstrates that prior label bias calibration methods fall short of addressing all three types of biases. Specifically, domain-label bias restricts LLMs to random-level performance on many tasks regardless of the choice of in-context examples. To mitigate the effect of these biases, we propose a simple bias calibration method that estimates a language model’s label bias using random in-domain words from the task corpus. After controlling for this estimated bias when making predictions, our novel domain-context calibration significantly improves the ICL performance of GPT-J and GPT-3 on a wide range of tasks. The gain is substantial on tasks with large domain-label bias (up to 37% in Macro-F1). Furthermore, our results generalize to models with different scales, pretraining methods, and manually-designed task instructions, showing the prevalence of label biases in ICL.

## introduction large language models (llms) can perform unseen tasks by conditioning on a context prompt that consists of a few training example-label pairs @xcite . however, such in-context learning ability is highly sensitive to various design settings, such as the choice @xcite and order @xcite of the in-context samples. recently, @xcite showed that the instability of icl largely arises from the fact that these design settings bias the model toward predicting certain answers (e.g., llms often predict the label of the last in-context example). as a result, the sensitivity of the results in icl studies calls for a systematic discussion of biases in icl and new methods to properly categorize, detect, and comprehensively mitigate various types of biases. in this work, we conduct a thorough investigation of biases in icl for text classification. we start by defining a typology of three types of label biases (the model's undesirable preference toward certain label names): vanilla label bias, context-label bias, and domain-label bias. what we term vanilla label bias captures the model's non-contextualized preference for the label names (e.g., the common token bias mentioned by @xcite caused by different frequencies of label names in the pretraining corpus). context-label bias summarizes the effects of the context prompt (e.g., llms tend to prefer the majority and last label of the in-context examples). finally, domain-label bias captures the effects of the task corpus on the model's predictions. we show that domain-label biases significantly affect a model's prediction in icl. for example, on a hate detection task with two nearly balanced classes, simply seeing random words sampled from the dataset severely biases the model towards predicting the label hate (fig. 1(a) ), while seeing random english words does not show such an effect. more importantly, on many tasks with large domain-label bias, llms achieve no better than random performance, regardless of the choice of in-context examples (fig. 2 ). moreover, we find that existing bias mitigation methods, such as contextual calibration (cc; @xcite do not combat this effect. to this end, we propose domain-context calibration (dc) to mitigate label biases in icl. dc first estimates the effects of different label biases holistically using random words sampled from the task corpus. specifically, we compute the probabilities assigned by the model to each label using random in-domain words as the task input (with optional real in-context learning examples prepended). using random words limits the semantic meaning of the input, allowing us to estimate the vanilla-label and context-label biases while using in-domain words accounts for the effect of the task corpus. then, at inference time, we use this label bias estimate to calibrate the model's output probabilities. we evaluate the impact of dc on 24 classification datasets, showing that dc improves the average icl performance of gpt-j @xcite and gpt-3 by 20% and 18%. we observe substantial gains on tasks with large domainlabel bias (up to 37% in macro-f1). dc also benefits models with different scales, instruction-tuning (e.g., instruct-gpt, @xcite , and provided with task instructions. finally, we show that dc improves the zero-shot prompting performance of smaller models like roberta @xcite , demonstrating that label bias can be mitigated in prompt-based frameworks beyond icl. overall, our work proposes a new typology of label biases in prompt-based methods, and a simple method for mitigating them. when studying icl on a diverse collection of datasets, the results on datasets with severe label bias can obfuscate the actual behaviors of the model. thus, rigorous design for dataset selection (that accounts for confounders) and fine-grained analysis of individual datasets are essential for effectively studying icl. ## categorizing label biases in icl in this paper, we focus on in-context learning (icl; fig. 3 ) for classification tasks. formally, we consider a dataset of examples {x i , y i } where x i are text inputs and each y i can be mapped to a verbalization in a label name set l. we assume each class has one label name. for example, in a sentiment task, l could be composed of positive and negative as label names. given a context prompt c consisting of a few labeled examples and an input text x i , the model m determines the label of x i by computing: arg max @xmath0, c). using this notation, we define our typology of label biases based on the mathematical formulation of icl. ## domain label bias to illustrate how the domain of a task can induce label bias, consider a case where an llm predicts whether a patient is sick or healthy based on some medical descriptions. because medical descriptions are associated more often with people having health problems in the natural corpus, frequently used words in such documents are likely to have a stronger correlation with sick than healthy, leading to a systematic bias in the model's predictions. supporting this intuition, we find that for many datasets, conditioning on random words from the dataset examples biases the model toward predicting certain label names. for example, in the hate speech detection task depicted in fig. 1 , we compute the model's preference (prior) on both label names given random words as the input. a model such as gpt-j has no preference for either of the classes (neutral v.s. hate) given random english words, but given random in-domain words sampled from the dataset, the label priors shift dramatically, becoming 0.95 (hate) v.s. 0.05 (neutral). motivated by this experiment, we quantify the domain-label bias of a model for a particular task using the distance of the model's priors estimated using random english words and in-domain words. to make the measure more comparable on tasks with different numbers of classes, we define the following metric: ## domain-context calibration in this section, we propose domain-context calibration (dc), which mitigates the effects of the multiple label biases of our typology ( § 2.1). following contextual calibration (cc; @xcite , we estimate the overall label bias of a model with respect to a task by estimating the label probabilities on a content-free example text. however, unlike cc, which uses a single, seemingly contentfree token (e.g., "n/a") to approximate the label bias, we use random words sampled from the unlabeled evaluation dataset as the content-free text. then, for all examples we classify for the task, we re-calibrate the model's prediction probability using this estimated bias. more formally, given a dataset, we first construct a bag-of-words b using the unlabeled texts {x i }. assuming x i 's have average length l, we sample l words randomly from b to form a content-free random text, which captures the word distribution of the dataset domain. however, the random text still remains nearly content-free as it is not grammatically meaningful and potentially contains words from all classes, making it suitable for calibration. we repeat this process m times and average the estimated priors: the model then makes predictions according to the following estimate: ## experimental setup we conduct comprehensive experiments to analyze the effectiveness of our domain-context calibration in mitigating label biases in icl. ## experimental results we report the average macro-f1 scores of gpt-j (6b) and gpt-3 (175b) across the entire evaluation suite in figure 6 . furthermore, we stratify our results into three equal-sized subsets according to their levels of domain-label bias. our main finding is that domain-context calibration generally improves in-context learning, especially on tasks with large domain-label bias. ## discussion label name selection as label bias mitigation our results outline how llms can be biased to certain label names for different tasks. intuitively, because the task is formulated as generating the label name given an example, the mechanism elicits the model's prior knowledge about the task. to better understand whether domain-label bias could be mitigated through more careful label name selection, we test gpt-j with three different pairs of label names on tweet hate: 1) neutral v.s. hate, which is the most task-relevant set of label names but introduces severe domain-label bias; 2) favor v.s. against, a pair of less task-relevant antonyms used by @xcite ; 3) x v.s. y, which are meaningless placeholders. as shown in fig. 12 , calibrating using random english words or in-domain words makes little difference when choosing (x, y) or (favor, against) as the label names, showing that they do not introduce domain-label bias. however, although gpt-j is able to achieve better original and cc performance on these label names, (neutral, hate) yields the best performance after removing domain-label bias using dc. thus, with proper calibration, using the most task-indicative words as label names is likely to be the best option. surprisingly, the manually picked label names (favor, against) under-perform the meaningless ones (x, y) after applying dc, hinting that human-plausible label names are not necessarily good label names for llms. ## related work in-context learning (icl) is the standard paradigm for adapting llms @xcite @xcite . many recent works focus on understanding its mechanism to improve adaptation performance. for example, @xcite showed that icl is sensitive to the order of in-context examples. @xcite demonstrated that the icl performance on numerical reasoning tasks is strongly correlated with the pretraining term frequencies. @xcite found that using examples semantically close to the input texts is beneficial. @xcite showed that for classification tasks, the input-label pairing format plays the most crucial role in icl. @xcite found that structure of the prompt also significantly affected icl performance, and that better prompts could be selected based on mutual information between the prompts and the model's output. complementary to these works, we comprehensively study the label bias problem in icl. the existence of domain-label bias indicates that icl is largely affected by the word-level associations llms learn during pre-training. other recent works discuss the bias problem in icl. @xcite proposed contextual calibration to mitigate three types of biases in icl: the majority bias, recency bias, and common token bias. @xcite focused on the zeroshot setting and found that different surface forms of the answer can compete for probability mass given a question, leading to a bias when predicting with a single label name for each class. in contrast to these works, which consider a specific type of bias, we propose a typology of label biases and propose domain-context calibration that handles all biases in our typology. the ability of the largest models, like palm @xcite , to perform in-context learning under the flipped label or semanticallyunrelated label settings @xcite is very relevant to this work. as the largest models tend to have emergent abilities, it would be interesting to test how vulnerable these models are to label biases (especially domain-label bias) and how domaincontext calibration would help. unfortunately, we currently do not have access to them (e.g., @xcite ). nevertheless, the similar behavior of palm and instruct-gpt (as shown in @xcite ) and the fact that instruct-gpt also suffers from domain-label bias ( ## conclusion in this work, we define a typology of label biases that affect in-context learning (icl). we categorize existing findings of label biases into two types: vanilla-label bias and context-label bias, and identify a new type of bias, domain-label bias, that significantly influences icl performance. to mitigate these label biases, we propose domain-context calibration, which significantly improves icl performance on a wide range of tasks, particularly on datasets with large domain-label bias. the various levels of domain-label bias in different datasets also suggest that when analyzing icl, we need to select datasets with diverse types of label biases and report stratified that acknowledge this diversity beyond single aggregated scores.