Step-by-Step Mastery: Enhancing Soft Constraint Following Ability of Large Language Models (2025)

Abstract: It is crucial for large language models (LLMs) to follow instructions that involve multiple constraints. In real-world scenarios, user instructions often contain soft constraints, which are semantically related and cannot be rule-based verified, posing challenges for LLMs. To enhance the soft constraint following ability of LLMs, we initially design a pipeline to construct datasets with high-quality outputs for instructions containing soft constraints automatically. Additionally, to fully utilize the positive and negative samples generated during the data construction process, we choose Direct Preference Optimization (DPO) as the training method. Furthermore, taking into account the difficulty of soft constraints indicated by the number of constraints, we design a curriculum learning training paradigm based on the constraint quantity. We experimentally evaluate the effectiveness of our methods in improving LLMs’ soft constraint following ability and analyze the factors driving the improvements.

## introduction in the application of llms, the instruction following ability is of paramount importance, especially when the instructions involve multiple constraints @xcite @xcite . the capability of llms plays a critical role in aligning with human preferences, ensuring the reliability and helpfulness of the models' outputs @xcite . following instructions with soft constraints is imperative for llms @xcite . constraints can be categorized into soft and hard constraints. hard constraints can be explicitly expressed as specific rules and directly verified through programming methods @xcite , @xcite in real-world scenarios, user instructions contain many soft constraints, posing challenges for llms. we use bold to represent soft constraints. existing methods for improving soft constraint following ability of llms have following limitations: first, much of the existing work focuses on evaluating llms' soft constraint following ability @xcite rather than how to improve the ability. second, when constructing soft constraint datasets, current methods typically introduce all constraints at once and directly prompt advanced models to generate responses @xcite . however, these responses may not satisfy all constraints. for example, gpt-4 demonstrates a constraint satisfaction rate of only 74.4% on follow-bench @xcite , making the assurance of high-quality outputs difficult. third, in terms of the training paradigm, existing work ignores the difficulty of constraint following indicated by the number of constraints @xcite . many studies show that organizing training data in a curriculum-based order-starting from simpler examples and gradually increasing complexity-leads to better performance than random shuffling @xcite . therefore, a pipeline constructing datasets with highquality outputs and an efficient training paradigm are required. in this work, we systematically investigate strategies to enhance the ability of llms to follow soft constraints, with the framework shown in fig. 2 , including constructing datasets with high-quality outputs and proposing a reinforcement learning training paradigm based on curriculum learning. to construct datasets with high-quality outputs for soft constraint following, we progressively add constraints to the instructions and incorporate judger to reorder the outputs based on constraint following. to fully leverage the positive and negative samples generated during judger reordering, we apply direct preference optimization (dpo) @xcite as the training method. since soft constraints cannot be verified through rule-based methods, judger reordering can ensure that the constructed preference dataset adheres to the correct constraint-following behavior. to account for the difficulty of soft constraints indicated by the number of constraints, we propose a training paradigm that constructs curricula based on the number of constraints in the instruction. in this paradigm, the model starts by learning simpler tasks (fewer constraints) and progresses to more complex ones (more constraints). our methods can improve llms' soft constraint following ability effectively. moreover, our method is also effective for hard constraints. our contributions are summarized as follows: (1) we design a pipeline to automatically construct datasets with high-quality outputs for soft constraint following. ability of llms by constructing benchmarks @xcite . these benchmarks include a variety of fine-grained constraint types @xcite . these constraints can be categorized into several types: (1) content soft constraints involve restrictions on the scope or depth of the responses @xcite . ( 2 ) situation soft constraints refer to the background limitations @xcite . (3) style soft constraints limit the manner of expressions @xcite . some works directly utilize responses generated by gpt-4 to construct datasets @xcite . different from these, we propose a pipline constructing datasets with highquality outputs to improve llms' soft constraint following ability. curriculum learning curriculum learning is a training strategy that mimics the learning process of humans from simpler to more complex tasks @xcite . current research on llms' curriculum learning can be categorized into two primary paradigms: (1) learning based on data difficulty: this approach organizes the training data sequence according to various evaluation metrics. metrics such as sequence length @xcite , perplexity @xcite have been employed to guide this process. llms can also construct curricula by organizing the training data sequence in a strategic way @xcite . (2) learning based on task difficulty: this paradigm focuses on changing the training tasks @xcite or adjusting the training objectives @xcite . however, our work organizes the curriculum based on the number of constraints . ## method in this section, we provide a detailed explanation of how to obtain datasets with high-quality outputs and how to leverage the dataset by establishing a curriculum learning training paradigm. the framework is shown in fig. 2 . ## experiments we conduct extensive experiments to validate the effectiveness of our method on improving llms' soft constraint following ability. ## conclusion in this paper, we systematically study how to improve llms' ability to follow soft constraints. initially, we design a pipeline to automate the construction of datasets with high-quality outputs for soft constraint following. based on the pipeline, we introduce a reinforcement learning training method utilizing positive and negative samples generated during the pipeline. moreover, we propose a new training paradigm that leverages curriculum learning to enhance llms' soft constraint following ability. the experiment results show that our methods enhance models' ability to follow soft constraints effectively.