In-Context Learning (and Unlearning) of Length Biases (2025)

Abstract: Large language models have demonstrated strong capabilities to learn in-context, where exemplar input-output pairings are appended to the prompt for demonstration. However, existing work has demonstrated the ability of models to learn lexical and label biases in-context, which negatively impacts both performance and robustness of models. The impact of other statistical data biases remains under-explored, which this work aims to address. We specifically investigate the impact of length biases on in-context learning. We demonstrate that models do learn length biases in the context window for their predictions, and further empirically analyze the factors that modulate the level of bias exhibited by the model. In addition, we show that learning length information in-context can be used to counter the length bias that has been encoded in models (e.g., via fine-tuning). This reveals the power of in-context learning in debiasing model prediction behaviors without the need for costly parameter updates.

## introduction in-context learning (icl) has emerged as a new ability in large language models (llms), representative of a novel learning paradigm @xcite . with in-context learning, an llm learns to perform an unseen task by seeing a number of demonstrations in the context window @xcite . whereas previous methods such as fine-tuning update the model parameters to teach the model a desired task, icl provides the model with input-output pairs as task exemplars directly at inference, with no parameter updates. while the goal of increased task accuracy is the same, the underlying mechanisms contributing to in-context learning are still being understood. this has motivated a body of work aiming to understand how in-context learning works. some works have demonstrated simlarities between finetuning and in-context learning. for example, instability due to the choice of examples occurs both in few-shot finetuning (schick and schütze, 2021; @xcite and in-context learning @xcite @xcite . however, other work has shown counterintuitive results when comparing the apparent learning mechanisms of in-context learning and finetuning @xcite . a key area that is underexplored is whether incontext learning exhibits similar biases to finetuning with regard to statistical data biases. statistical data biases can be defined as correlations between features and class labels. under traditional learning paradigms such as fine-tuning, language models can learn exploitable statistical biases in the data. such biases, or shallow features, can be exploited by a model as discriminatory features when they exhibit biased distributions across classes or are correlated with a specific class. this can lead to overestimates of a model's performance on the underlying task @xcite . prior work has identified length as an exploitable statistical bias in natural inference datasets. for example, in the multinli and snli datasets, length has been shown to be a discriminatory feature @xcite , and on the roc story cloze llm the cat is friendly. \n positive this coffee tastes bitter. \n negative … what a fun puzzle! \n positive the bird's chirping is pretty. \n _______ x 1 x 2 x k y 1 y 2 y k x test {prediction} y test y test context window task choosing the longer ending performs above random baseline @xcite . however, length biases have been largely ignored in prior work on icl, and some existing studies on which factors impact icl have treated length as a static variable, selecting examples with similar lengths to test inputs @xcite . it is therefore unclear whether models can exploit length biases in the data under an in-context learning setting (similar to finetuning) or whether length is another factor with counterintuitive tendencies. in this work, we perform a series of empirical studies to investigate the ability of llms to learn statistical data biases in the context window during icl (figure 1 ). this has been studied in the finetuning literature, yet is underexplored in the icl literature. we demonstrate empirically the ability of llms to learn length biases in-context. in the sections to follow, we analyze which factors influence these results, and we show the efficacy of icl in debiasing finetuned models. our results show that icl can introduce biases to llms that negatively influence task performance. specifically, our contributions are as follows: ## background in-context learning in-context learning is an emergent ability of llms that enables pre-trained models to learn an unseen task using a set of exemplars concatenated in the context window (see figure 2 ). formally, given a test example x, in-context learning concatenates k demonstration examples to the task instruction i, where @xmath0, y i } k @xmath1 in-context learning & bias while in-context learning has shown significant potential as a way to extract relevant information from an llm and align the model with user expectations, it has also exhibited brittleness to an assortment of factors. these include selected examples @xcite @xcite and their orderings @xcite , which have recently been categorized under the umbrella of demonstration biases @xcite . beyond demonstration bias, instability of icl has been attributed to biases in the model toward predicting certain answers due to majority label bias, recency bias, and common token bias @xcite . correspondingly, several works have looked at identifying and mitigating label bias @xcite @xcite with respect to lexical information, and ali et al. (2024) have looked at mitigating "copy bias", where llms copy lexical information from demonstrations rather than learning underlying task information. however, statistical data biases such as length information have been largely ignored in the incontext learning literature, yet received wide attention in the natural language inference literature with respect to traditional finetuned models @xcite @xcite . our work bridges this gap by looking at in-context learning with relation to a specific statistical bias: length bias. ## experiment setup category dataset task #train #val inference qnli (wang et al., 2018) natural language inference 105k 5.46k rte 1 natural language inference 2.49k 277 wnli (levesque et al., 2011) natural language inference 635 71 hans (mccoy et al., 2019a) natural language inference 30k 30k single sentence sst-2 (socher et al., 2013) sentiment analysis 67.3k 872 paraphrase mrpc (dolan et al., 2004) paraphrase detection 3.67k 408 detection paws-x en @xcite paraphrase detection 49.4k 2k ## length biases in finetuning and icl in this section, we investigate the question do models learn length biases in-context? we demonstrate empirically the ability of llms to learn length biases in-context. ## analysis of influencing factors in this section, we investigate a further question of what factors influence how llms learn length biases in-context? we find that increased numbers of examples can exacerbate learned biases, and models across a range of sizes can learn length biases. further, we find that length bias can be learned from as little as a few tokens of difference in average length between classes. ## icl for debiasing finetuned models in-context learning eliminates the need for expensive model parameter updates incurred when fine-tuning. however, it is often the case that a model may have encoded biases picked up from the pretraining and/or finetuning. as our previous experiments show that in-context learning can learn length information, a natural extension is to question whether icl can be used to "unlearn" or mitigate previously learned length biases. in this section, we answer the question can icl be used as an intervention to mitigate biases in finetuned models? we use previously finetuned models from section 4 and modify the length distribution to try to counteract the biases. specifically we experiment with two conditions: 1) using in-context demonstrations drawn from the opposite tail of the length distribution from what was seen during finetuning, and 2) using randomly sampled in-context demonstrations. we again use llama3 8b and gpt-neo 2.7b for these experiments using the datasets in results results using llama3 (8b) on hans, paws-x en , and wnli are reported in figure 10 , figure 11 and figure 12 , respectively. we find that random sampling was able to counteract the bias, essentially "unlearning" the finetuned bias. this suggests that balanced data sampling is critical to minimize the likelihood of learning biases in-context. further, if a dataset is balanced, random sampling may be sufficient. however, if a dataset contains shortcut features, more sophisticated sampling methods to mitigate the bias may be necessary. moreover, our results suggest balanced sampling over showing the models demonstrations of opposite lengths with respect to the finetuned bias. specifically, the models learned the bias in the length information in the context window, regardless of how it contradicted what was seen during finetuning. one possible explanation is that the task may be implicitly encoded during pretraining and icl extracts the knowledge @xcite , however, further study is warranted on whether knowledge-extraction from icl overrides knowledge-gain during finetuning. ## discussion in this work, we investigate the impact of demonstration length bias on model performance when learning in-context. under this setting, length is a statistical data bias, where the shallow feature (length) is correlated with class labels. it is important to make the distinction, however, between length as a linguistic feature containing information relevant to the underlying task, and length as an artifact of the data collection protocol. for example, length is an informative syntactic feature in classifying truthfulness vs. deceptiveness @xcite and authorship detection @xcite , however, length biases can also arise from artifacts reflective of heuristics used by human data annotators @xcite . our work pertains to the latter settings where length is an artifact rather than a task-informative natural language feature. which variables have the greatest impact on models learning length biases? in subsection a.3, we observe that when varying the number of in-context examples, the distance between classes is greater with fewer in-context examples. however, the amount of bias increases with increased numbers of examples. further, while we observe that bias increases with increased length difference, we still observe learned bias when class length difference is reduced to as few as 3 tokens on the hans dataset. this suggests that a key factor in learning bias is the number of examples the model sees. additionally, our results suggest that any model can learn bias, and model parameter size is not necessarily correlated with increased ability to learn biases in-context. ## conclusion in this work, we empirically investigated the ability of llms to learn length biases under an in-context learning paradigm. our results show that llms can learn statistical biases in the data. we further show the impact of model parameter sizes, number of examples, and class length difference on length biases. finally, we demonstrate the potential for icl to be used as a tool to debias fine-tuned models with previously learned length biases.