Mitigate Position Bias in LLM s via Scaling a Single Hidden States Channel (2025)

Abstract: Long-context language models (LCLMs) can process long context, but still exhibit position bias, also known as “lost in the middle”, which indicates placing key information in the middle of the context will significantly affect performance. To mitigating this, we first explore the micro-level manifestations of position bias, concluding that attention weights are a micro-level expression of position bias. Then we identify that, in addition to position embeddings, positional information in hidden states also contributes to position bias, and it manifests itself in specific channels of hidden states, called positional hidden states. Based on these, we propose a method to mitigate position bias by scaling positional hidden states. Experiments on NaturalQuestions Multi-document QA, KV retrieval and LongBench, using various models including RoPE models, context window-extended models, and Alibi models, demonstrate the effectiveness and generalizability of our approach. Our method can improve performance by up to 15.2% in “lost in the middle” benchmark by modifying just one channel of hidden states. Our code is available at https://aka.ms/PositionalHidden.

## introduction long-context language models (lclms) @xcite @xcite @xcite have recently garnered significant attention within the community, enabling llms to handle longer and more complex tasks such as long-context questionanswering @xcite . however, recent research @xcite @xcite @xcite @xcite shows that even long-context llms often fail to utilize all context information effectively, exhibiting a "lost in the middle" bias where middle context information † work during internship at microsoft. is ignored. this issue affects all types of llms, regardless of their architecture or size, and worsens with longer contexts. previous works have analyzed this issue from the perspectives of data distribution @xcite @xcite and position embeddings @xcite . for example, film @xcite addresses position bias by constructing data with key information distributed in various positions for supervised finetuning (sft). @xcite mitigates position bias by interpolating @xcite using head-wise scaling factors. on the other handsome works @xcite @xcite have proven that, besides position embeddings, the hidden states of llms can also convey positional information, which is generated by the causal attention mask. although various works (see related works in appendix a) have attempted to mitigate position bias, few of them have related position bias to positional information in hidden states, which may be another non-negligible source of position bias. to verify the relation between position bias and positional information in hidden states, we conduct a series of experiments. first, as many previous works @xcite @xcite have observed, we find u-shaped attention weights consistent with position bias in specific layers. second, we use perturbation experiments to prove position bias is indeed affected by positional information in hidden states. third, we identify some channels called "positional channel" (the hidden states corresponding to them are called "positional hidden states"), whose values can manifest absolute positional information, through a process of hypothesis and subsequent verification. finally, we confirm these channels can indeed affect position bias. naturally, to mitigate position bias, a direct way is to eliminate (or minimize) its potential cause: positional hidden states. so, we propose a position bias mitigation method named "scale positional hidden states". specifically, we first design a heuristic searching algorithm that quickly identifies which channels (along the hidden size axis) of the hidden states are potential positional channel, using monotonicity and smoothness as indicators. then we select the optimal channel among them based on the loss on a calibration dataset. next, based on extensive engineering experience, we design a well-designed attention modification algorithm that let the scaled hidden states only influence the attention queried by the last token, to eliminate positional bias in attention, while avoiding interfering too much with the original attention leads to unstable model performance. extensive experiments on various models, including llama-2 @xcite , vicuna @xcite , mistral @xcite , gemma @xcite , qwen @xcite , and mpt (team, 2023), and across different tasks, including multi-document qa, kv retrieval, longbench @xcite benchmark, demonstrate that our method effectively mitigates position bias by modifying only one channel of the hidden states of the model. in addition, we test on timeline reorder task @xcite and mmlu @xcite to show our methods has minimal side effects on the original capabilities of the model. our main contributions are as follows: ## positional information in hidden states affects position bias in this section, we first identify patterns in attention weights that closely correspond to position bias. then, through modifying the causal mask in former layers and observing the change of attention in latter layers, we demonstrate position-related attention bias can also be affected by the positional information in hidden states. third, we extract specific hidden states channels which has monotonicity with the token position, named "positional hidden states", and find that they bears responsibility for the emergence of position bias. to sum up, we show our findings about position bias in this section in figure 1 . ## mitigating position bias previous work focusing on pe often attempted to refine the application of rope, essentially minimizing the gap in position information (introduced by rope) between tokens in different positions. in section 2, we have identified positional information in hidden states as another factor causing position bias, and found positional channels strongly related to positional information. thus, similarly, to minimize the gap in position information (introduced by positional information in hidden states) between tokens, the most direct way is to scale the activation values of those channels. therefore, we propose a method to mitigate position bias by scaling the positional hidden states, as shown in figure 5 . specifically, it consists of two steps: identifying the positional hidden states h t and scaling (multiplying) them by a factor s. ## conclusion we propose a method which scales positional hidden states to mitigate the position bias issue of llms. specifically, our study first explore the relationships between position bias (in the model's performance), attention weights, causal mask, positional information in hidden states and positional channels, to confirm the positional channels can affect position bias. based on these findings, we design a positional channel search algorithm to identify positional hidden states, and mitigate the model's position bias by scaling the positional hidden states. these findings provide a new perspective for research related to position information and position bias of llms.