When Instructions Multiply: Measuring and Estimating LLM Capabilities of Multiple Instructions Following (2025)

Abstract: As large language models (LLMs) are increasingly applied to real-world scenarios, it becomes crucial to understand their ability to follow multiple instructions simultaneously. To systematically evaluate these capabilities, we introduce two specialized benchmarks for fundamental domains where multiple instructions following is important: Many Instruction-Following Eval (ManyIFEval) for text generation with up to ten instructions, and Style-aware Mostly Basic Programming Problems (StyleMBPP) for code generation with up to six instructions. Our experiments with the created benchmarks across ten LLMs reveal that performance consistently degrades as the number of instructions increases. Furthermore, given the fact that evaluating all the possible combinations of multiple instructions is computationally impractical in actual use cases, we developed three types of regression models that can estimate performance on both unseen instruction combinations and different numbers of instructions which are not used during training. We demonstrate that a logistic regression model using instruction count as an explanatory variable can predict performance of following multiple instructions with approximately 10% error, even for unseen instruction combinations. We show that relatively modest sample sizes (500 for ManyIFEval and 300 for StyleMBPP) are sufficient for performance estimation, enabling efficient evaluation of LLMs under various instruction combinations.

## introduction large language models (llms) have demonstrated remarkable capabilities across various tasks @xcite @xcite @xcite * equal contribution 1 manyifeval/stylembpp data and evaluation codes are available at https://github.com/kenoharada/ multiple-instructions-following @xcite . however, their practical utility depends on their ability to simultaneously follow multiple instructions. simple task descriptions such as "write a piece of code" or "summarize this text" typically produce outputs with limited practical value. adding specific instructions transforms these basic prompts into practically useful tools -for instance, code generation with style guide compliance enables effective team collaboration, while text summarization with formatting requirements (e.g., bullet points, length limits) ensures human-readable outputs. these kinds of instructions are ubiquitous in real-world applications, and models must be able to follow multiple instructions to be useful in practice. while some existing benchmarks have made valuable contributions to evaluate multipleinstructions-following capabilities for real-world applications, there are several limitations towards accurate and fair evaluation. for instance, ifeval @xcite and complexbench @xcite assign a different task description across different number of instructions, which prevents from isolating the effect of instruction counts as a controlled variable. followbench @xcite utilized llm-as-a-judge for measuring multiple-instructions-following performance, which prevents from reliable and stable verification compared to rule-based and programmable one. without controlled experimental design, it becomes difficult to systematically and precisely analyze how the number of instructions impacts model performance. our work focuses on measuring the simultaneous multiple-instructions-following performance, which we consider crucial in real-world applications. specifically, we introduce two specialized benchmarks: manyifeval for text generation (up to 10 instructions) extended from ifeval @xcite and stylembpp for code generation (up to 6 instructions) extended from mbpp @xcite . our benchmarks maintain consistent task descriptions while varying the number of instructions, allowing us to isolate and analyze the effect of instruction counts on model performance (figure 1 and based on the created benchmarks, we conducted a broad range of experiments to evaluate the multiple-instructions-following performance with a wide variety of llms. specifically, our experiments used ten state-of-the-art llms including api based models (gpt-4o, claude3.5 sonnet, gemini 1.5 pro, o3-mini) and open models (llama3.1-8b, gemma2-9b, gemma2-2b, qwen-2.5 72b, deepseek-v3, deepseek-r1), revealing that performance consistently and drastically degrades as the number of instructions increases. furthermore, given the fact that evaluating all the possible combinations of multiple instructions is computationally impractical in actual use cases, we developed regression models that can estimate performance on both unseen instruction combinations and different numbers of instructions which are not used during training. specifically, we explore three modeling approaches: naive estimators, beta-binomial, and logistic regression. through this estimation, we demonstrate that a logistic regression model using instruction count as an explanatory variable can predict performance with approximately 10% error, even for unseen instruction combinations. the approach generalizes effectively to unseen numbers of instructions, achieving a mean absolute error of 0.03 ± 0.04 when predicting performance on 10 instructions using training data from up to 9 instructions. our experiments reveal that relatively modest sample sizes are sufficient for accurate performance estimation -500 samples for manyifeval and 300 for stylembpp. this suggests that our benchmarks can provide reliable performance estimates for llms under various instruction combinations, enabling efficient evaluation and guiding future model development. ## benchmark creation to systematically investigate multiple instructionsfollowing capabilities of llms, we introduce two benchmarks: manyifeval for general writing task with up to ten instructions and stylem-bpp for code generations with up to six instructions. both benchmarks are constructed with programmatically verifiable instructions to ensure objective and reliable evaluation. our benchmarks are specifically created using sets of instructions that are non-conflicting and simultaneously satisfiable. the primary goal is to investigate how performance degrades as the number of instructions increases, keeping the instructions themselves compatible. we ensured this compatibility through careful curation. manyifeval extends ifeval @xcite , while stylembpp builds upon mbpp @xcite . manyifeval and stylembpp are under the cc by 4.0 international license same as original mbpp and ifeval. ## performance prediction as parameters of models grow, their evaluation becomes increasingly computationally expensive. in multiple-instruction settings, evaluating all of them would require substantial computational resources because the number of possible instruction combinations are massive. to address this challenge, we develop models capable of estimating performance on both unseen instruction combinations and different numbers of instructions. we explore three modeling approaches: naive estimators, beta-binomial, and logistic regression, each offering different perspectives on modeling multiple-instructions-following behavior. ## conclusion we introduced two benchmarks, manyifeval and stylembpp, to evaluate models' ability to follow multiple instructions simultaneously. our experiments have demonstrated that performance degrades as the number of instructions increases, with the degradation pattern varying across models and instruction types. we also showed that this degradation can be modeled using simple approaches like logistic regression, enabling accurate prediction of model performance on unseen instruction combinations with relatively small sample sizes. these findings provide insights for understanding and improving models' ability to handle multiple instructions simultaneously, an essential capability for real-world applications.