R ule R : Improving LLM Controllability by Rule-based Data Recycling (2025)

Abstract: Large language models (LLMs) still lack delicate controllability over their responses, which is critical to enhancing their performance and the user experience. However, curating supervised fine-tuning (SFT) datasets to improve LLM controllability usually relies on human experts or proprietary LLMs, which requires additional costs. To bridge this gap, we propose Rule-based Data Recycling (RuleR), a data augmentation method incorporating multiple constraints into the original data samples according to predefined rules, which creates new training tasks to consolidate the controllability of LLMs. Instead of creating new data from scratch, RuleR “recycles” existing data by simply applying rule-based edits to their responses and appending the rule-instructions in their original instructions. Experimental results demonstrate RuleR’s effectiveness in improving LLM controllability while maintaining general instruction-following capabilities.

## introduction despite the remarkable advancement of the current large language models (llms) and the continuous efforts to build high-quality supervised fine-tuning (sft) datasets, one critical challenge is to generate responses better interacting with humans, with the utility and effectiveness maximized for endusers @xcite @xcite . according to the systematic investigation from @xcite , it is essential for llms to constrain their outputs to follow user-specified formats or characteristics. in various practical applications, free-formed responses are not legal or directly applicable without any constraint or format being enforced. it has also been verified on llm agents @xcite @xcite that enforcing predefined formats is necessary for tasks. however, existing sft datasets are mainly composed of general instructions without userspecified constraints @xcite @xcite @xcite @xcite and thus result in models lacking delicate controllability of the lengths and format of responses @xcite . to enhance the utility of existing sft data in improving the controllability of llms, a potential method is to rewrite or modify instructions and responses by experts such as humans/llms @xcite @xcite @xcite in order to make them fulfilling multiple constraints, as shown in figure 1 (top). however, the curation of new data is not only costly and inefficient, requiring careful editing by human experts or proprietary llms, but also represents a waste of previous efforts: it is impractical to discard all existing data and create brand new data every time we need to add more constraints to the instructions. hence, we raise the question: can we "recycle" existing sft data without human/llm editing and enforce various types of constraints in order to improve llm controllability? drawing inspiration from ifeval @xcite , which utilizes verifiable constraints to evaluate llms' controllability, and the human/modelfree data augmentation in mosaic-it @xcite , we propose rule-based data recycling (ruler), which automatically "recycles" existing sft data for improving llm controllability. as illustrated in figure 1 (bottom), the key insight of ruler is to automatically build constraint-augmented sft datasets at no cost of human/llm efforts, by applying predefined rules to the original instructions and responses. specifically, we manually inspect and construct a diverse set of rules as constraints, which specify the linguistic or formatting constraints on different parts of the response. our predefined rules cover a wide range of diverse constraints generalizable to many application scenarios, ranging from high-level constraints, e.g., controlling the word frequency in the response, to lower-level constraints, e.g., setting specific wrapping formats of some keywords. each rule is composed of (1) multiple templates to produce additional instructions enforcing the constraints, and (2) a piece of code that alternately edits the original instruction and response in order to make the edited response fulfill all the constraints appended to the instruction. for each sample from the original dataset, we randomly draw several rules to be applied to the editing. this produces an augmented sample with the constraints enforced so it can be used for controllability tuning. the complete list of rules and descriptions can be found in appendix e. illustrative examples are provided in figure 2 , which showcase (a) a rule that constrains the number of letters; (b) a rule that specifies the case of specific words (if presenting in the response); and (c) a rule that specifies the wrapping format of specific words. to ensure the consistency between the input constraints and the output response, we modify both the instruction and response based on the characteristics of the original data sample. for each sample, we only sample from the rules applicable to the original response, hence avoiding the potential incorrectness of the edited responses. extensive experimental results on the ifeval benchmark on various base models and datasets demonstrate the effectiveness of ruler in enhancing llm controllability without extra help from humans/models. on the other hand, ruler still preserves the general instruction-following ability promoted by the original sft dataset. this is demonstrated by the instruction-following metrics (pair-wise comparison and open llm leaderboard). to the best of our knowledge, ruler is the first human/model-free data augmentation and recycling approach designed to improve llm controllability under multiple constraints. ## conclusion in this work, we proposed rule-based data recycling (ruler), which modified the original instructions and responses from an existing dataset by rule-defined constraints. the "recycled" data aims to enhance the llms' capability to generate outputs fulfilling the constraints specified in the input, thereby improving the controllability of llms. ruler took the first step of exploring rule-based data recycling, which can serve as a plug-and-play and easy-to-use method that converts any existing sft datasets to new datasets for better controllability.