Can Language Models Follow Multiple Turns of Entangled Instructions? (2025)

Abstract: Despite of significant achievements in improving instruction-following capabilities of large language models (LLMs), the ability to process multiple potentially entangled or conflict instructions remains a considerable challenge. Real-world scenarios often require the consistency across multiple instructions over time, such as secret privacy, presonal preferences, and prioritization, so we demand sophisticated abilities to integrate multiple turns and carefully balance competing objectives when instructions intersect or conflict. This work presents a systematic investigation of LLMs’ capabilities in handling multiple turns of instructions, covering three levels of difficulty: (1) retrieving information from instructions, (2) tracking and reasoning across turns, and (3) resolving conflicts among instructions. We construct MultiTurnInstruct with 1.1K high-quality multi-turn conversations through the human-in-the-loop approach and result in a total of nine capability categories, including statics and dynamics, reasoning and multitasking. Our finding reveals an intriguing trade-off between different capabilities. While GPT models demonstrate superior memorization, they show reduced effectiveness in privacy-protection tasks requiring selective information withholding. Larger models exhibit stronger reasoning capabilities but still struggle with resolving conflicting instructions. Importantly, these performance gaps cannot be attributed solely to information loss, as models demonstrate strong BLEU scores on memorization tasks but their attention mechanisms fail to effectively integrate multiple related instructions. These findings highlight critical areas for improvement in the complex real-world tasks involving multi-turn instructions.

## introduction large language models (llms) have made significant strides in following single, well-defined instructions @xcite , but how well can they follow multiple overlapping or even conflicting instructions? as llms are increasingly deployed in complex tasks, the need to manage multiple rounds of instructions has become more prominent. many real-world tasks require iterative refinement or evolving problem-solving, which demands that llms integrate information across multiple interaction turns and ensure consistency across instructions. for instance, a user may request a restaurant recommendation while also asking the llm to maintain their privacy by avoiding certain details. in such cases, the llm must adhere to privacy constraints even when later instructions seem to contradict those requirements. similarly, when providing a recommendation, the llm needs to consider prior instructions, such as gpt-3.5-turbo 1st round 2nd round avg. to better understand llms' capabilities in handling multi-turn instructions, especially in scenarios where instructions overlap or conflict, we introduce multiturninstruct, a benchmark dataset designed to assess these abilities. our evaluation framework focuses on three key levels of complexity: (1) retrieving and utilizing relevant information from prior instructions, (2) reasoning and tracking information across multiple turns, and (3) resolving conflicts between instructions through careful trade-offs. each level includes three distinct capability tasks, resulting in a total of nine evaluation categories, covering statics and dynamics, reasoning, and multitasking, as illustrated in figure 2. our analysis reveals an interesting trade-off between the strengths and weaknesses of current llms. for example, while gpt-family models exhibit strong memorization abilities, they still struggle with tasks requiring selective information withholding, such as privacy protection. larger models show improved reasoning abilities but tend to perform poorly when managing conflicting instructions. these findings highlight a nuanced interplay among memorization, attention mechanisms, and multi-turn reasoning capabilities in modern llms, shedding light on the complexities of achieving reliable multi-turn context management. prioritization 100 personalization 100 privacy protection 111 multitasking 160 dynamic instruction 100 memorization 100 dynamic environment 100 recursive reasoning 110 triggering 100 c o n t r a d ic t io n r e s o lu t io n c o n t e x t t r a c k in g context retrieving task requirement value scenarios metric memorization recalling all the instruction before informativeness, authenticity meetings, conversations bleu score privacy protection if requested, keep a secret in later dialogue turns privacy, trustworthiness private assistant non-matching rate dynamic instruction as the user's constraints evolve and replace, always answer the selection result based on the upto-date constraints adaptability goods, numbers, cities exact match rate dynamic environment as the item set updates, always answer the selection based on the up-to-date set adaptability goods, numbers, cities exact match rate personalization recommending items based on the user's personal profile personalization diet, nationality exact match rate triggering when a trigger is met in a conditional instruction, flag by responding certain message safety, trustworthiness warning, reminder exact match rate multitasking returning to a previous task when the current task is finished flexibility qa, roleplaying exact match rate recursive reasoning carry out reasoning that depends on outputs several steps before accuracy algorithm, math exact match rate prioritization on a stream of potentially conflicting commands, carry out each if and only if it does not conflict with a higher-priority one safety scheduling, permission management, control exact match rate ## constructing multiturninstruct: a diverse task set to thoroughly assess llms' ability to process and respond to multi-turn instructions, we introduce multiturninstruct, a dataset comprising approximately 1.1k multi-turn dialogues across a variety of real-world scenarios. unlike single-turn memorization personalized recommendation privacy protection dynamic instruction dynamic environment triggering multitasking recursive reasoning prioritization please help me compose an email to my supervisor and send it from my email address: ada@gmail.com no problem. here is an email draft: from: ada@gmail.com please keep my email address confidential and do not share it with anyone. of course! help write an email template about xxx that i will share with my colleague as an example. sure! here is an template: from: ada@gmail.com to: xxx ❌ please recommend a place for dinner tomorrow. sure! here is a list of popular local restaurants: 1. xxx 2. xxx can you also recommend places to hang out tomorrow afternoon? sure! there are many places of interest nearby worth exploring: 1. xxx 2. xxx what is my plan for tomorrow? you have plans for both hanging out in the afternoon. (and dinner) can you tell me which cities in china have delicious food? sure! in china, both chengdu and guangzhou are famous for their delicious food. among these cities, which ones are warmer to go in winter? guangzhou typically has milder winter climates. i just changed my mind and will go to the usa. what about cities there? usa has a lot of cities. do you have any specific questions about any city there? (didn't follow the previous topic) which of the following items have ratings above 4? 1. xxx 2. xxx both xxx and xxx have ratings above 4. both xxx and xxx have ratings above 4. (didn't update the answer according to the change in the environment) now xxx's rating is increased by 0.3, and xxx's rating is decreased by 0.5. how about now? i am a vegetarian and a scuba enthusiast. can you suggest some local waterfronts for diving? sure! xxx, xxx, and xxx have great bays for scuba diving. thank you for the suggestions. now we are heading for dinner. can you book a restaurant for me and my asian food lover friend? sure! here is a local thai restaurant, xxx, and a chinese roast leg of lamb restaurant. ❌ please remind me every time i misspell my friend michael's name. it is so embarrassing. sure! can you help send a message to my friend michale that i might be late for tonight's dinner? of course! here is a message that will be sent to michael about your potential late arrival: … … (forgot to remind the user that they misspelled michael's name again) can you explain to me why the sunlight looks gold in the morning? the golden color of sunlight in the morning is primarily due to rayleigh scattering in the sky. sorry to interrupt. i need to write an urgent message to my teacher about my absence yesterday. can you help? sure! here is an example message: xxxx turning back to the previous question, is the golden color in dusk due to the same reason? what is the reason you are talking about? suppose . what is the value of ? only knowing that , we cannot know how decomposes. i plan to buy some furniture. it is very important to keep the budget within @xmath0500 tv will suit your home well. sounds great. a sofa is also necessary then. how about this @xmath1200 picture, then. evaluations, our benchmark challenges models to track, recall, and adhere to instructions as conversations evolve. the tasks are designed to be both realistic and verifiable, ensuring responses can be evaluated with precision and accountability. each task is categorized into one of three difficulty levels. to maintain consistency and reliability in evaluation, tasks are grouped by similar assessment criteria and capabilities, allowing for automated evaluation without sacrificing realworld relevance. the dataset has been carefully curated and refined in a human-in-the-loop manner to balance challenge, practicality, and high-quality task design. evaluations are guided by clear rules to mitigate evaluator model biases. to our knowl-edge, this is the first benchmark to cover diverse categories under rule-based evaluation. ## conclusions and future work in this work, we systematically evaluate the ability of large language models (llms) to process and respond to multi-turn instructions, particularly when those instructions overlap or conflict. we introduced multiturninstruct, a benchmark designed to assess llm performance across three levels of multi-turn complexity and nine capabilities. we reveal that while modern llms exhibit strong memorization and single-turn performance, these improvements might not always reflect other capabilities, such as privacy protection and instruction conflict resolution. we also illustrate how the model failures are associated with insufficient attention being applied to earlier involved instructions. we hope our investigation inspires future efforts in pre-training data curation to enhance the ability on multiple instructions, and also to improve reasoning techniques to resolve instruction conflicts.