Characterizing Positional Bias in Large Language Models: A Multi-Model Evaluation of Prompt Order Effects (2025)

Abstract: Large Language Models (LLMs) are widely used for a variety of tasks such as text generation, ranking, and decision-making. However, their outputs can be influenced by various forms of biases. One such bias is positional bias, where models prioritize items based on their position within a given prompt rather than their content or quality, impacting on how LLMs interpret and weigh information, potentially compromising fairness, reliability, and robustness. To assess positional bias, we prompt a range of LLMs to generate descriptions for a list of topics, systematically permuting their order and analyzing variations in the responses. Our analysis shows that ranking position affects structural features and coherence, with some LLMs also reordering or omitting topics. Nonetheless, the impact of positional bias varies across different LLMs and topics, indicating an interplay with other related biases.

## introduction large language models (llms) are widely used for text generation, ranking, and evaluation tasks. however, they exhibit distinct biases that can impact their reliability, fairness, and robustness @xcite . one such bias, positional bias, occurs when the placement of information within a prompt influences the model's output @xcite . this bias can lead to systematic prioritization of inputs based on order rather than content. specifically, an item's position within a list given as part of a prompt may influence how llms describe, emphasize, or interpret its importance, leading to inconsistent judgments. for example, in content generation tasks like summarization, question answering, and text completion, llms may disproportionately prioritize information appearing earlier in a prompt, affecting coherence and informativeness @xcite . in ranking and scoring applications, such as machine translation assessment or essay grading, bias toward the first or second option can result in unfair rankings @xcite . similarly, in structured tasks (like chain-of-thought reasoning), positional bias may distort logical consistency by prioritizing certain steps @xcite . positional bias not only undermines the fairness of llm-generated assessments but also raises concerns about their robustness across contexts and domains. moreover, understanding biases in llms is critical given their increasing role in generating educational @xcite , informative @xcite , and decision-support content. when models systematically favor items based on position rather than content, they may subtly shape human preferences, judgments, and beliefs in ways that go unnoticed. for instance, biased ordering in generated summaries, comparisons, or lists could mislead users about what is most relevant or important (e.g., as shown for decision making tasks @xcite ). while some studies have revealed ranking biases in llms, the study of positional effects in list-based prompts has received less attention. to address this gap, we conduct a systematic evaluation of positional bias in llm-generated lists, aiming to answer the following research questions: our study contributes to addressing this gap by providing a systematic, cross-model analysis of how topic order affects generation outcomes across different families of llms. to this end, we prompt llms to generate descriptions for a list of topics, systematically varying their order to analyze how the generated descriptions change as a function of position, aiming at uncovering patterns of topic pri-figure 1 : schematic steps of the applied method oritization, structure and coherence shifts linked to position, or the specific characteristics of the selected topics themselves. we not only quantify structural and content-level variations but also surface qualitative differences in how models treat position-sensitive inputs. our findings reveal that positional bias affects not only the structure and coherence of outputs but also the relative attention and completeness of item descriptions, with variations across models and topics. ## related work while llms have gained popularity, research on positional bias is still limited. prior studies have demonstrated that llms exhibit positional bias in decision-making tasks, often favoring options based on their placement rather than intrinsic quality @xcite @xcite @xcite , for example, by prioritizing early and, to a lesser extent, late information, while overlooking the middle @xcite . this bias has been shown to be not random, and to vary across tasks, llms, and candidate similarity, with inconsistencies in how llms assess options @xcite . also, while prompt length has been shown to have small effects on bias, bias direction has been inconsistent across tasks @xcite . beyond positional bias, research has also identified verbosity bias (favoring longer responses) and self-preference bias @xcite , while @xcite introduced a benchmark identifying six types of biases, including selection preferences based on response length, author and order. our study extends prior work on positional bias, which has largely focused on selection tasks and pairwise comparisons with direct/reverse orderings @xcite . rather than selecting items from a list, in our study, llms generate responses for each item in a list, enabling analysis of both selection tendencies and quality-related metrics. we systematically evaluate all possible permutations of items, offering a more comprehensive view of positional effects. additionally, our analysis spans a broad set of llm families, allowing us to assess whether these effects generalize across architectures. ## task and methodology figure 1 provides a schematic overview of the applied method for evaluating positional biases in llms' responses, illustrating topic selection, description generation (along with the prompt used for each llm) and description evaluation foot_0 . topic selection. we selected six diverse topics, albert einstein, world war ii, quantum computing, climate change, blockchain, and the james webb telescope, covering historical, scientific, and technological domains or categories. these topics, commonly found in trivia games, represent general knowledge and public interest while varying in complexity and controversy. selection criteria included relevance in scientific discourse, presence in widely available training data, and potential to reveal variations in response quality. this approach follows prior studies on llm biases @xcite , which highlighted the role of topic diversity in assessing llm behavior, as they may exhibit varying biases across subject areas, which might reflect underlying cultural values. ## experimental analysis rq1: positional effects in descriptions. characterization of descriptions per ranking position. figure 2 shows score distribution for a selection of metrics and llms across ranking positions and topics foot_5 . fre scores varied significantly across ranking positions, particularly for llama, gpt, and gemini 1.5 flash-8b, with higher scores at the top positions. despite these variations, topics maintained a consistent relative readability order across positions, indicating that while position influenced readability, relative differences across topics remained stable, suggesting that llms' responses were primarily affected by position. llama and gemma showed the most significant differences across all positions, whereas gpt models exhibited differences mainly at the top and last positions. topics followed a consistent ranking pattern for kc, even more pronounced for fre. named entity density was less affected by ranking position, gpt-4o mini was the only model displaying significant differences across all positions. in contrast, string length followed a u-shaped pattern, with longer responses at the first and last positions and shorter ones in between. solar produced notably shorter responses than the other llms while maintaining named entity density, suggesting a distinct summarization approach foot_6 . unlike other metrics, for string length, the lack of clear topicbased separation hints that ranking position, rather than topic, plays a dominant role in determining response length. topic ordering. most models preserved input order, with kendall's tau values close to 1, except for llama 3.2 1b, gemma, and mistral. gemma 2 2b displayed the most reordering, often inverting ranks and clustering related topics (e.g., science). mistral and gemma 2 27b made occasional swaps, while llama 3.2 1b showed moderate reordering, particularly in middle to last positions. missing topics per position. figure 3 shows the missing topics per position foot_7 . llms primarily omitted topics at the last position, with gemini 1.5 flash-8b and gemma 2 2b missing topics across all positions, and llama 3.2 1b also missing topics at the top. gemini 1.5 flash-8b exhibited a uniform omission pattern (i.e., it consistently missed topics at all positions, even missing all topics altogether), while llama 3.2 3b showed varied last-position omissions, with certain topics omitted more frequently than others. ## conclusions this study shows that positional bias in llmgenerated responses is real and multifaceted. across metrics and models, we found that ranking position systematically influenced structural and lexical features of outputs, including readability, complexity, and length. while some models also reordered or omitted topics, others produced consistent outputs, highlighting that positional effects are not universal but instead vary across architectures and families. moreover, our study shows that positional bias does not operate in isolation. it interacts with model-specific training patterns and topic-specific priors, meaning that variation arises not only from list position but also from how models encode and prioritize different kinds of content. some models exhibited clear position-based scoring trends, while others showed topic-dependent shifts in framing or emphasis. this variability suggests that mitigation strategies may need to be tailored to models rather than assuming a single correction method. overall, understanding and mitigating positional bias is critical for fairness, transparency, and reliability in llm applications. left unaddressed, these biases could distort information presentation, subtly shaping user perceptions and decisions. by surfacing both systematic patterns and model-or topic-specific nuances, our work contributes empirical evidence to guide the evaluation and mitigation of positional bias in large language models.