Unveiling Selection Biases: Exploring Order and Token Sensitivity in Large Language Models (2024)

Abstract: In this paper, we investigate the phenomena of “selection biases” in Large Language Models (LLMs), focusing on problems where models are tasked with choosing the optimal option from an ordered sequence. We delve into biases related to option order and token usage, which significantly impact LLMs’ decision-making processes. We also quantify the impact of these biases through an extensive empirical analysis across multiple models and tasks. Furthermore, we propose mitigation strategies to enhance model performance. Our key contributions are threefold: 1) Precisely quantifying the influence of option order and token on LLMs, 2) Developing strategies to mitigate the impact of token and order sensitivity to enhance robustness, and 3) Offering a detailed analysis of sensitivity across models and tasks, which informs the creation of more stable and reliable LLM applications for selection problems.

## introduction large language models (llms) have demonstrated remarkable abilities across various tasks @xcite @xcite , leading to their widespread adoption in downstream applications. in particular, the utilization of zero-shot or few-shot prompting techniques emerged as a highly convenient approach in harnessing the potential of llms, since these techniques empower end-users to solve a wide range of tasks without the need for extensive fine-tuning. despite llms' impressive performance and convenience, empirical investigations have found that their output is highly sensitive to the choice of prompts, and even subtle modifications of instructions or demonstrations have considerable influence on their performance. to address this issue, several works have been dedicated to the identification and mitigation the inherent biases in llms, aiming to enhance their robustness and reliability @xcite @xcite . in this study, our focus is directed towards the domain of "selection problem", where llms are instructed to select the optimal choice from an ordered sequence of choices. this problem encompasses a variety of downstream applications, including but not limited to classification, multiplechoice questions @xcite , and model evaluation scenarios @xcite @xcite . in our analysis, we identifies specific biases within the context of selection problems, which we call "selection biases" to encapsulate these discernible tendencies. these biases manifest as systematic deviations in llms' preferences, and a thorough understanding of these biases is pivotal for enhancing the robustness of llms across the spectrum of applications under the scope of selection problems. our subsequent exploration delves into the characterization, quantification, and mitigation strategies to address these biases. it is crucial to highlight that our analysis centers on the zero-shot setting. this choice distinguishes our work from previous endeavors, which predominantly concentrate on few-shot settings, making it difficult to disentangle biases stemming from in-context demonstrations. our contributions can be summarized as follows: 1) we quantify the influence of option order and token on the decision-making processes of various llms when tackling selection problems, providing clear insights into how these factors affect model performance; 2) we introduce strategies to mitigate the effects of token and order sensitivity, leading to performance improvements across a broad spectrum of tasks; 3) we offer a thorough understanding of the sensitivity landscape through an empirical study encompassing different models, tasks, and sensitivity settings. the analysis enables us to identify the most effective strategies for addressing sensitivity issues in diverse task scenarios. ## investigation on llm sensitivity while prior research has touched upon biases in llms concerning mcqs, with notable findings on position bias and token bias, our work stands out by delving deeper into unexplored territories of the combined impact of option order and token usage within mcqs. we uncover novel insights into the decision-making processes of llms that have yet to be extensively explored in the existing literature. ## methodology to mitigate the impact of sensitivity to tokens and/or order and improve model stability, we propose three methods tailored to different contexts of llms. we categorize these contexts into two scenarios: gray-box and black-box. in a black-box scenario, the llm provides only the generated text upon request, without additional information. conversely, a gray-box scenario allows access to more detailed output, such as token probability information. in our experiments, gpt-3.5 falls into the gray-box category, as the openai api enables retrieval of the top 5 token log probabilities, whereas other models are black-box ones. furthermore, all experiments adhere to the sensitivity settings mentioned in section 4. ## conclusion this study investigate into the effects of token and order sensitivity on llms when addressing selection problems, incorporating an empirical analysis of both powerful commercial llms such as gemini pro and gpt-3.5 and open-sources models like llama 2. by concentrating on zero-shot settings, we aim to isolate and better understand biases that previous works identified in in-context demonstrations, thereby offering a clearer perspective on how these sensitivities influence llm decision-making processes. our findings underscore the significance of task difficulty as a crucial determinant of sensitivity impact on llm performance. moreover, we introduce cost-effective mitigation strategies, including gray-box and black-box approaches, tailored for practical application scenarios. the results demonstrate that our gray-box methods, namely probability weighting and probability calibration, outperform baselines with minimal additional expenditure, contrasting with more complex methods like majority voting. additionally, our two-hop strategy for black-box scenarios proves to be effective in a significant portion of tasks. we anticipate that our contributions will aid future research in enhancing the robustness of llms across various types of selection problem applications.