Divide-Verify-Refine: Can LLM s Self-align with Complex Instructions? (2025)

Abstract: Recent studies show LLMs struggle with complex instructions involving multiple constraints (e.g., length, format, sentiment). Existing research enhances open-source LLMs using closed-source guidance (e.g., GPT-4), but this heavily relies on generated data quality. An alternative is leveraging LLMs’ self-correction to refine responses for better constraint adherence. However, this is limited by the feedback quality, as we found LLMs cannot generate reliable feedback or detect errors. Moreover, the self-correction effectiveness relies on few-shot examples illustrating response modifications. As constraints in complex instructions are diverse, manually crafting such examples for each constraint type can be labor-intensive and sub-optimal. To address these two challenges, we propose the Divide-Verify-Refine (DVR) framework with three steps: (1) Divide complex instructions into single constraints and prepare appropriate tools; (2) Verify responses using tools that provide rigorous check and textual guidance (e.g., Python scripts for format checks or pre-trained classifiers for content analysis); (3) Refine: To maximize refinement effectiveness, we propose dynamic few-shot prompting, where a refinement repository collects successful refinements, and these examples are selectively retrieved for future refinements. Recognizing the lack of complexity in existing datasets, we create a new dataset of complex instructions. DVR doubles Llama3.1-8B’s constraint adherence and triples Mistral-7B’s performance.

## introduction large language models (llms), like chatgpt, have shown significant improvements across various language tasks @xcite @xcite . the success of llms relies on the ability to execute complex instructions. failures to follow instructions can result in unintended outputs, which may have severe consequences @xcite @xcite . this issue becomes critical when llms are deployed in high-stakes environments, such as legal documentation or technical writing. for example, when drafting legal contracts, llms must strictly adhere to constraints related to format, specific terminology, and precise language usage to avoid misinterpretations or legal liabilities. similarly, in technical writing, adhering to strict format guidelines, word limits, and inclusion of essential technical terms is critical to ensure clarity and compliance with industry standards. recent studies show that llms, especially opensource ones, struggle to follow complex instructions with multiple constraints like response length or formatting @xcite @xcite . while this issue is well recognized, research on enhancing llms' constraint adherence ability is still limited, with most efforts focused on evaluating the ability @xcite . few studies improve llms' constraint-following via fine-tuning @xcite @xcite . for example, @xcite adopt a teacher model (e.g., to generate data and fine-tune a student model with the generated data to improve its multi-constraint adherence ability. while effective, it requires a large amount of computation resources and heavily depends on the generated data quality. in contrast, the concept of "self-correction" offers an alternative approach, where llms autonomously correct their responses @xcite . self-correction has been applied on other tasks such as question answering @xcite or mathematics @xcite , where an llm will evaluate its responses, give feedback, and further refine responses. however, whether llms can effectively self-align with diverse and complex constraints remains an open question. this is particularly important for agent llms to be deployed in high-stakes environments. for constraint-following, this self-correction process can be divided into two phases: verification and self-refinement (fig. 1 ). during the verification phase, llms assess whether their responses align with the specified constraints. if the responses do not align with the constraints, the llms will give feedback that pinpoints errors and suggests adjustments. following this, the self-refinement phase takes place where llms use the feedback to refine and improve their responses accordingly. however, there are several challenges. the first one is feedback reliability. studies show that the performance gains by llm self-correction is unstable, occasionally even degrading performance in question answering @xcite and code generation @xcite . the bottleneck in self-correction lies in the feedback quality @xcite @xcite . llms, including advanced models like gpt-4 and claude 3, tend to have low recall in detecting llms errors, underperforming significantly compared to humans @xcite . on the other hand, research reveals that the self-correction performance on reasoning tasks is boosted if the error location is given, indicating llms have the selfcorrection ability given reliable feedback @xcite . from the constraint-following perspective, llms are also not good at checking simple and easy-to-verify constraints. as shown in fig. 1 , given a response, llms struggle to accurately count the number of words. the second challenge is constraint diversity which lies in the self-refinement process. given the response and the feedback, llms should refine the response according to the feedback. however, to perform this task effectively, a set of representative few-shot examples is needed to demonstrate how to appro-priately modify the response @xcite . these constraints can vary widely, from adhering to a length limit to including specific keywords. each type of constraint needs distinct modifications. for example, meeting a length limit might require removing content, whereas incorporating specific keywords requires adding text. manually crafting representative few-shot examples for each constraint type is labor-intensive. to address these challenges, we propose a novel framework named divide-verify-refine (dvr) as shown in fig. 2 . to enhance feedback reliability, we observe that constraints that llms struggle to verify can be readily assessed using external tools. these tools include python toolkit for quantitative measures, such as regular expressions (re) @xcite and the natural language toolkit (nltk) @xcite for counting the number of words, sentences, paragraphs, or bullet points, as well as pre-trained classifiers for content analysis, such as topic and sentiment analysis, which are easily accessible and widely available on hugging face @xcite . we enhance llms by enabling interaction with external tools. first, we instruct llms to break down complex instructions into individual constraints and assign an appropriate tool to each. these tools then rigorously verify the llm's response and provide textual guidance for refinement if any constraints are violated. to address the problem of constraint diversity, we propose the dynamic few-shot prompting. since the verification reliability is ensured by external tools, we incorporate a novel refinement repository, which serves as a memory module to collect and store successful refinements for future use. when a new refinement task arises, we select few-shot examples with the same constraint type to maximize refinement effectiveness. our main contributions are: (i) our framework enhances feedback reliability by integrating easily accessible tools that provide strict verification and textual guidance for llms. (ii) to maximize the refinement effectiveness, we propose dynamic fewshot prompting with a refinement repository that stores successful refinements. this enables llms to learn from past experiences and retrieve more similar few-shot examples for future refinements, improving refinement effectiveness. (iii) most benchmarks only contain 1-2 constraints @xcite . we construct a new complex instruction dataset with instructions containing 1-6 constraints. self-correction of llms. self-correction is a framework where llms refine their responses during inference by reflecting on their initial responses @xcite . this process has two phases. initially, llms are prompted to analyze and provide feedback on their responses. subsequently, based on the feedback llms refine the responses to correct their mistakes. however, recent studies report negative results indicating that llms cannot self-correct their own mistakes @xcite @xcite . a study @xcite reveals that top llms like gpt-4 and claude 3 have low recall in detecting llm errors, with llms significantly underperforming compared to humans. additionally, feedbacks provided by llm self-correction tend to hallucinate and lack reliability. this unreliability suggests that even when errors are detected, the guidance offered for corrections may be incorrect or misleading. @xcite find that llms struggle to accurately identify logical fallacies, casting doubt on their inherent ability to detect errors and conduct self-verification reasoning effectively. however, the self-correction performance on reasoning tasks is boosted if the error location is given @xcite . all these observations indicate that llms are not reliable in analyzing their responses and a more reliable feedback mechanism is needed to pinpoint the mistakes. more introduction to related works is in appendix a.8. ## the proposed framework: dvr as shown in fig. 2 , we propose the divide-verify-refine (dvr) framework, which consists of three modules: (a) divide instructions and prepare tools accordingly, (b) verify responses and provide feedback, and (c) refine and store responses in a repository. first, the tool preparation module aims to identify constraints, select appropriate tools, and fill out parameters. in this module, llms first decompose the complex instructions into single constraints. for each single constraint, the llms will prepare appropriate tools for verification. second, the prepared tools will verify the response and give detailed textual guidance if the response does not adhere to the constraint. third, in the self-refinement module, given the textual guidance, llms will refine the response to adhere to the target constraint. since similar few-shot examples usually yield better results, dvr retrieves past refinement experience with the same constraint type few-shot examples. the successfully refined response will be stored for future use. next, we introduce each module in detail. ## empirical validation in this section, we conduct experiments to answer the following research questions: (rq1) can our dvr improve the ability of llms to follow complex constraints? (rq2) how does the performance of llms differ across various types of constraints, and which constraints pose the greatest challenges? (rq3) how does each module of dvr (the tool-assisted verification and the fewshot self-refinement library) individually contribute to improving llms' ability to follow constraints? ## conclusion we propose the divide-verify-refine (dvr) framework to enhance llms' ability to follow multiconstraint instructions. dvr has three steps: (1) divide complex instructions into single constraints and assign appropriate tools for each constraint.