LLM Self-Correction with D e CRIM : Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024)

Abstract: Instruction following is a key capability for LLMs. However, recent studies have shown that LLMs often struggle with instructions containing multiple constraints (e.g. a request to create a social media post “in a funny tone” with “no hashtag”). Despite this, most evaluations focus solely on synthetic data. To address this, we introduce RealInstruct, the first benchmark designed to evaluate LLMs’ ability to follow real-world multi-constrained instructions by leveraging queries real users asked AI assistants. We also investigate model-based evaluation as a cost-effective alternative to human annotation for this task. Our findings reveal that even the proprietary GPT-4 model fails to meet at least one constraint on over 21% of instructions, highlighting the limitations of state-of-the-art models. To address the performance gap between open-source and proprietary models, we propose the Decompose, Critique and Refine (DeCRIM) self-correction pipeline, which enhances LLMs’ ability to follow constraints. DeCRIM works by decomposing the original instruction into a list of constraints and using a Critic model to decide when and where the LLM’s response needs refinement. Our results show that DeCRIM improves Mistral’s performance by 7.3% on RealInstruct and 8.0% on IFEval even with weak feedback. Moreover, we demonstrate that with strong feedback, open-source LLMs with DeCRIM can outperform GPT-4 on both benchmarks.

## introduction large language models (llms) have demonstrated impressive instruction following ability across various tasks, such as creative writing, coding, and arithmetic reasoning @xcite @xcite @xcite . despite their remarkable success, recent studies and * work done while interning at amazon agi foundations. benchmarking on real-world requests. these benchmarks often rely on synthetic data either to address data scarcity or to facilitate automatic rulebased evaluation @xcite @xcite . in this work, we argue that synthetic constraints may not accurately capture the complexity and nuances of real-world scenarios, being sometimes artificially difficult. as a result, focusing on synthetic benchmarks may push research in the wrong direction, as improvements on these may not translate to real-world performance and could even degrade it. to address this gap, we introduce the realin-struct benchmark, which evaluates llms using real user requests to ai assistants. it assesses llms' response on individual constraints at a time, as illustrated in figure 1 . our analysis reveals that even strong proprietary model gpt-4 fails to meet at least one constraint in over 21% of the instructions, highlighting the limitations of current llms in handling user's constrained instructions. llm-based evaluation. evaluating real-world instructions is challenging due to their open-ended nature. drawing on recent research on llm-based evaluation (llm-as-a-judge) @xcite ; our experiments show that gpt-4-turbo with chain-of-thought prompting is a cost-effective and reliable alternative to human evaluation. however, open-source models, such as mistral, lag significantly behind proprietary models. weakly supervising mistral with gpt-4-turbo reasoning steps led to 26% relative improvement, though, still remaining insufficient for reliable evaluation. llm self-correction. improving llms by generating and building upon intermediate outputs, known as system 2 approaches, has been explored for constrained instruction following. but many works in this direction, such as branch-solve-merge @xcite and self-refine @xcite , assume constraint independence or focus on specific constraint types, limiting their applicability to real-world use cases. to address this, we introduce decompose, critique, and refine (decrim), a self-correction pipeline designed to enhance llm performance in multiconstrained instruction following without making assumptions about the constraints. as shown in figure 2 , decrim breaks down the original instruction into a main task and granular constraints, and includes a critic model that decides whether the response is final or requires refinement, which is handled by the underlying llm. through extensive benchmarking, we demonstrate the effectiveness of the decrim pipeline. even with weak feedback and no external data, decrim improves mistral's instruction-level performance by 4.8% on realinstruct and 1.2% on ifeval relative to baseline. when perfect instruction decomposition into constraints is provided, improvements increase to 7.3% and 8.0%, respectively. providing stronger feedback further boosts mistral's performance by 22.0% on realinstruct and 33.8% on ifeval, surpassing gpt-4 on both benchmarks. in summary, our work contributes the following: the rest of the paper is organized as follows: sections 2 and 3 present the realinstruct dataset and the decrim self-correction pipeline, respectively. section 4 outlines our hypotheses and experimental setup, while section 5 analyzes the results. final remarks are provided in section 6. we provide extra details and discussions in the appendix. ## realinstruct dataset we introduce realinstruct, a novel dataset consisting of original user-generated instructions, each decomposed into a task and a set of constraints (see figure 1 for an example). the task represents the user's main objective -the outcome they expect from llm -and may optionally include context to aid the model's understanding. constraints are conditions or limitations that guide the llm's generation. formal definitions of these terms are provided in appendix c. since users rarely specify constraints in a structured list format, the decomposition breaks instructions into manageable items, providing the necessary granularity for evaluating llms' ability to follow constraints. using this dataset, we develop a benchmark protocol that evaluates llm performance on each constraint at a time, generating constraint-level scores benchmark instruction source constraints source evaluation size (instructions) constraint types avg. constraints per instruction collie synthetic synthetic rule 2,080 13 n/a ifeval synthetic synthetic rule 541 25 1.4 followbench crowdsourced + synthetic synthetic model / rule 795 6 5 infobench crowdsourced crowdsourced model / rule 500 5 4.5 realinstruct (ours) real users real users model 302 (test) + 842 (val) 20+ 3.5 (test) ## self-correction with decompose, critique, and refine (decrim) in this section, we present our proposed de-crim self-correction pipeline, designed to enhance llm responses to follow user constraints. given a multi-constrained user instruction, the pipeline iteratively refines the llm's response through four key steps: initial response, decompose, critique, and refine, iterating until the response meets all constraints or a maximum number of iterations n max is reached. figure 2 provides an overview of the proposed pipeline and we detail each of these steps below: ## experimental setup this section outlines the experimental setting to evaluate our proposed methods. we first vali-date the use of llm-as-a-judge for the realin-struct benchmark (sec. 4.1). next, we benchmark models on instructions with multiple constraints (sec. 4.2) and, finally, test our proposed self-correct decrim pipeline to improve opensource models in this task (sec. 4.3). all experiments with open llms were performed with hug-gingface's transformers library @xcite on an aws instance with 8 v100 32gb gpus. times are reported in this configuration. ## results and discussion we discuss our results, particularly the reliability of llm-as-a-judge for constraint satisfaction (section 5.1), the ability of various open-source and proprietary llms on follow multi-constrained instructions (section 5.2), and the efficacy of our decrim self-correction pipeline (section 5.3). ## conclusion in this work, we benchmarked llms' ability to follow real multi-constrained user requests with realinstruct. we showed that even strong proprietary models like gpt-4 fail to meet at least one constraint in over 21% of instructions, demonstrating realinstruct's challenging nature and the need for improvement across both proprietary and open-source models. to address this, we proposed the decrim self-correction pipeline, which decomposes instructions into granular requirements, critiques responses, and refines outputs. extensive experiments showed that decrim significantly improves open-source llm performance, with stronger feedback allowing them to surpass gpt-4. overall, our work highlights the underexplored problem of following real-world user requests, as well as advances system 2 techniques with de-crim. future work could refine the decrim's components and integrate the pipeline to other system 2 approaches, such as self-consistency and generate-and-rank, to enhance its effectiveness in tasks where spending more time on generationrefinement iterations would improve performance.