Order Matters: Investigate the Position Bias in Multi-constraint Instruction Following (2025)

Abstract: Real-world instructions with multiple constraints pose a significant challenge to existing large language models (LLMs). An observation is that the LLMs exhibit dramatic performance fluctuation when disturbing the order of the incorporated constraints. Yet, none of the existing works has systematically investigated this position bias problem in the field of multi-constraint instruction following. To bridge this gap, we design a probing task where we quantitatively measure the difficulty distribution of the constraints by a novel Difficulty Distribution Index (CDDI). Through the experimental results, we find that LLMs are more performant when presented with the constraints in a “hard-to-easy” order. This preference can be generalized to LLMs with different architecture or different sizes of parameters. Additionally, we conduct an explanation study, providing an intuitive insight into the correlation between the LLM’s attention and constraint orders. Our code and dataset are publicly available at https://github.com/meowpass/PBIF.

## introduction large language models (llms) have made impressive progress in massive natural language tasks @xcite and have been applied to various real-world scenarios @xcite . to achieve satisfactory performance, it is crucial for llms to understand the user's instructions and convey desired outputs, which is known as the instruction following capacity of llm @xcite . in practice, instructions are usually incorporated with multiple constraints of different types, e.g., format constraint which limits the model's output to a specific format. nevertheless, existing llms often struggle to follow multi-constraint instructions, making multi-constraint instruction following an obstacle to hinder llms' real-world application @xcite . recently, a lot of works have demonstrated that llms are sensitive to the position of the referred context in many tasks, such as multi-document question answering, text evaluation, and list-wise ranking @xcite @xcite . since there are usually multiple constraints coexisting in the complex instruction, the position bias problem is also significant in multiconstraint instructions. as shown in fig. 1 , in the single-round scenario, the llm's performance varies significantly when presented with instructions that have different constraint orders, even though the two instructions are semantically iden- seed inst … seed inst multi-round inference 4 answer in lowercase letters only… there should be two paragraphs… do not say the word "popular"… end with the phrase "really love… 1 2 3 4 seed inst 1 2 3 4 seed inst 1 2 3 4 seed inst 1 2 3 4 ＞ ＞ seed inst 1 2 3 4 seed inst tical. when it comes to the multi-round scenario, different constraint orders impose different impacts on the intermediate responses, thus inevitably leading to a discrepancy in the quality of the final responses. nevertheless, the position bias of constraint orders in the multi-constraint instruction following remains an under-explored problem. existing work manually assigns difficulty to different constraints based on a predefined rule and orders the constraints according to their difficulty. they empirically demonstrate the existence of llms' performance fluctuation brought by different constraint order @xcite . however, on the one hand, handcraft difficulty categorization fails to reflect the real difficulty disparity of different constraints @xcite . on the other hand, they merely analyze the constraint order in a qualitative way, lacking a quantitative metric to measure the disparity of constraint order. additionally, none of the existing works has provided an intuitive explanation for the position bias in multi-constraint instructions. it remains unclear how the llms handle instructions with different constraint orders. to address all the problems above, we systematically investigate the position bias problem in the multi-constraint instructions. first, we propose a novel metric called the constraint difficulty distribution index (cddi) to quantitatively describe the disparity of constraint order from the perspective of constraint difficulty. we leverage the accuracy of the llm to quantify the difficulty of different constraints, thus precisely reflecting their disparity. then, for a thorough study of the position bias problem, we design a probing task. as shown in fig. 2 , we construct a large number of multi-constraint instances with different constraint orders and explore two practical scenarios: single-round inference and multi-round inference. our experiments find existing llms commonly perform better with the "hard-to-easy" constraint orders, i.e., possibly placing harder constraints in former positions. finally, to make an intuitive explanation of our findings, we resort to a gradient-based method @xcite . we visualize the importance of different constraints located in different positions. we observe that the constraint order will affect how the llm handle the constraints and is highly correlated to the llm's performance on a specific constraint. in summary, our main contributions are as follows: (1) we are the first to systematically investigate the position bias problem in multi-constraint instruction following. (2) we propose a novel cddi metric to quantify the disparity of different constraint orders in the multi-constraint instructions. (3) through extensive experiments, we find that existing llms can achieve a better performance when presented with constraints in "hardto-easy" orders. this finding can be generalized in both single-round and multi-round scenarios, regardless of the architecture of llm, the size of llm's parameters and the number of constraints. (4) our explanation study explores how the llms assign attention when provided with instructions in different constraint orders and demonstrates the significant correlation between the attention patterns and the llms' performance on specific constraints. ## conclusion in this paper, we systematically investigate the position bias problem in the multi-constraint instruction following. to quantitatively measure the disparity of constraint order, we propose a novel difficulty distribution index (cddi). based on the cddi, we design a probing task. first, we construct a large number of instructions consisting of different constraint orders. then, we conduct experiments in two distinct scenarios. extensive results reveal a clear preference of llms for "hard-toeasy" constraint orders. to further explore this, we conduct an explanation study. we visualize the importance of different constraints located in different positions and demonstrate the strong correlation between the model's attention distribution and its performance.