CARE - ST a R : Constraint-aware Self-taught Reasoner (2025)

Abstract: In real-world applications, large language models (LLMs) often need to handle diverse and complex instructions. Specifically, when instructions are subject to multiple constraints, some of which are somewhat ambiguous, LLMs often fail to produce answers that satisfy all constraints, limiting their effectiveness in various tasks. To address this challenge, we examine the different constraints in the instructions and discover that the difficulty of determining whether an answer meets a constraint varies widely, from extremely straightforward to exceptionally perplexing. Correspondingly, we propose to assign constraints to different constraint levels. Furthermore, inspired by chain-of-thought (CoT) and self-taught reasoner (STaR), we propose a two-stage method named CARE-STaR (Constraint-AwaRE STaR). Our method distinguishes constraints within instructions by generating different CoTs and guides LLMs to autonomously learn optimal answers by setting the positive rewards to the CoTs that are beneficial to generating accurate answers and iteratively optimizing these answers. We have conducted extensive experiments on three instruction-following benchmarks, taking three existing LLMs as base LLMs, respectively. Experimental results indicate that our method substantially enhances the capability of these LLMs to handle complex instructions, outperforming supervised fine-tuning (SFT). Our code is available at https://github.com/lzl0124/carestar.

## introduction in recent years, large language models (llms), which have been used in a wide range of applications, such as dialogue systems, text summarization, and machine translation @xcite , have achieved significant success. as llms are applied to an expanding range of domains, the instruc- ## formulation of instruction following in this section, we borrow the concept of membership function from fuzzy theory @xcite to formulate the instruction-following task in llms, taking into consideration that the optimization of llm has similarities with the fuzzy control process. a multi-constraint instruction refers to an instruction issued by a user to llm that incorporates multiple constraints, all of which should be satisfied simultaneously when generating an answer. let @xmath0, c 1 , c 2 , ..., c m } denote the multiconstraint instruction, where q denotes the question, c i denotes the i-th constraint and m is the total number of constraints. let the set x be the set of all possible answers generated by an llm for a given instruction i. we define a fuzzy set f c i over x, in which each answer @xmath1, c i ) ∈ [0, 1] reflecting the degree to which it satisfies the constraint c i . the larger the value of the membership function µ c (a, c i ), the more a belongs to that fuzzy set f c i . similarly, we introduce the membership function µ q (a, q) ∈ [0, 1] for question q in the instruction i. thus, the ideal goal pursued by the llm is that the answer a to the instruction @xmath2, c 1 , ..., c m } satisfies the following equation: however, it is often hard for llm to achieve this objective. as a result, the instruction-following task is proposed to train the llm so that it satisfies the following requirement: notably, the roles of µ q (a, q) and µ c (a, c i ) can be undertaken by any sota llm. in the light of this definition, we set the fuzziness of constraint for each constraint and a constraint level function l(c i ) to quantify its degree of hardness or softness. a lower value indicates that c i is a highly fuzzy constraint, which means that it allows for flexible interpretation, while a higher value signifies that c i is highly precise, requiring strict adherence. for simplicity, the range of l is currently set to l(c i ) ∈ {1, 2, 3, 4, 5}. c i is identified as a hard constraint if l(c i ) = 5, or as a soft constraint if l(c i ) = 1. in our implementation, we employ chain-ofthought (cot) to perform "constraint analysis", which adopts the llm to be trained to conduct the l(c i ) evaluation. furthermore, we design three loss functions loss talk , loss talk_cot , and loss think as described in the next section to autonomously optimize the generation of cots that most effectively enhance instruction following. ## method we now describe our method, which aims to teach llms to effectively follow instructions and obtain satisfactory answers. our method consists of two stages, i.e., the initial cold-start stage, followed by the self-taught stage. in the first stage, we warm up the llm to be trained to improve its generation capability of highquality cots. in the second stage, whose process is shown in figure 2 , we first guide the model to generate multiple candidate cots for each instruction in the training dataset of stage 2. then, we evaluate the quality of each cot based on its impact on the answer prediction. finally, we adjust the probabilities of generating cots according to the evaluation results. we iterate the process above to automate the generation of "constraints analysis" until finishing the designated number (defaulted to three) of epochs. to be noted, we can adopt any typical instructiontuning dataset @xmath3, y (i) } n @xmath4, which is not required to contain cots. here, x (i) denotes an instruction and y (i) denotes an answer. the dataset is randomly split into two disjoint subsets, which are used for the two stages of the method, respectively. ## conclusion in this paper, we propose the care-star method, which trains llms in two stages to push the upper limit of the capability of the original llms to follow instructions. our method adopts a self-taught mechanism to tackle multiple constraints in the instruction, eliminating the need for any specific instruction dataset during training. experimental results show that our method outperforms sft in handling complex instructions.