Failures Pave the Way: Enhancing Large Language Models through Tuning-free Rule Accumulation (2023)

Abstract: Large Language Models (LLMs) have showcased impressive performance. However, due to their inability to capture relationships among samples, these frozen LLMs inevitably keep repeating similar mistakes. In this work, we propose our Tuning-free Rule Accumulation (TRAN) framework, which guides LLMs in improving their performance by learning from previous mistakes. Considering data arrives sequentially, LLMs gradually accumulate rules from incorrect cases, forming a rule collection. These rules are then utilized by the LLMs to avoid making similar mistakes when processing subsequent inputs. Moreover, the rules remain independent of the primary prompts, seamlessly complementing prompt design strategies. Experimentally, we show that TRAN improves over recent baselines by a large margin.

## introduction large language models (llms) have recently demonstrated remarkable performance across a broad spectrum of natural language processing (nlp) tasks. prominent models, such as chat-gpt @xcite and @xcite , have garnered substantial attention for their proficiency in generating human-like text, driving their increasing adoption in real-world applications @xcite . as these applications involve ever-changing scenarios and specific requirements @xcite , there is a growing interest in exploring approaches to tailor these models to meet specific goals. to address the challenge of aligning llms with human preference, @xcite construct human-written instruction data and conduct instruction tuning @xcite in a reinforcement learning manner. recent works @xcite further gain remarkable performance by employing parameter-efficient tuning @xcite , which avoids fine-tuning the entire model. despite their great success, numerous users engage with llms via apis, posing significant challenges for modifying the parameters @xcite . thus, it is essential to develop tuning-free approaches for effectively adapting llms to specific requirements. instead of tuning the parameters, recent approaches @xcite design crafting prompts to guide llms. @xcite effectively alleviate the harmfulness of generated texts with human-written principles by specialists. in contrast, recent approaches @xcite optimize the prompt globally on the training set by instructing llms to generate guidelines @xcite or criticism based on the current prompt @xcite . however, in real-world scenarios, data arrives in a streaming setting @xcite . as depicted in fig. 1 , llms face a continuous influx of streaming data instances, demanding their adaptation to the changing data distribution, in order to avoid repeating similar mistakes. tuning-free rule accumulation (tran) framework, which enables the self-adaptation of llms to specific scenarios without additional training sets or complementary models in an online learning fashion @xcite . specifically, the framework guides llms to generate rules for subsequent deployment when the generated content is unsatisfactory. by iteratively accumulating rules based on observed mistakes in the streaming data, we construct a comprehensive set of rules. for each input sample, we retrieve relevant rules to provide guidance to the model alongside the initial prompts. additionally, we devise strategies for llms to autonomously manage and maintain the rule collection, ensuring minimal redundancy and contradictions, which further alleviates the potential of excessive growth in the size of the rule collection. to validate our framework, we conduct experiments over various tasks, spanning multi-choice question answering and text classification tasks from different domains. through rule accumulation, tran consistently promotes performance by a significant margin in both zero-shot and fewshot settings. moreover, as rules are independent of the prompt design, tran seamlessly complements prompt design strategies like chain-of-thought @xcite . additionally, by manually adjusting the classification boundary, we construct challenging scenarios that deviate from the distribution of training data, further validating the effectiveness of our approach. we summarize our contributions as follows: foot_0 . ## analysis whether tran complements cot? recent approaches @xcite have achieved remarkable performance gains by improving the prompts. in our tran framework, as mentioned in section 2.2, the rules are concatenated before the base prompt. consequently, we conduct experiments to apply tran to these prompt-design strategies. the results presented in how tran performs when the full training set is given? to further investigate our framework, we conduct experiments in a train-test setting. following wang and li (2023), we randomly select 250 samples from each task within bbq-lite, and we divided the data into the training set and the test set using a 0.8/0.2 split ratio. note that only the samples from the training set are utilized by tran and salam for testing. the comparison results are presented in moreover, by incorporating a training set, our tran provides the model with an initial nonempty rule collection. this mirrors real-world scenarios where humans can predefine basic rules tailored to specific environments. to delve deeper into the influence of the training set, we conducted an ablation study. according to the experimental results in whether tran generalizes to out-of-domain tasks? to assess the generalization capability of our tran, we employ the rule collection constructed during training on the source task to guide the model when evaluating the test set of the target task. the results are illustrated in fig. 4 . notably, tran demonstrates a significant enhancement in performance for out-of-domain tasks across most task pairs. in particular, we observe that utilizing the rule collection from the physical task results in a noteworthy 10% performance improvement on the ses task, whereas the rule collection of the ses task itself fails to provide a performance boost. we find similar outcomes when employing the age task as the source and the sexual task as the target. the top two rules applied to the ses task are demonstrated in top 1: if a person suggests that corporal punishment is always acceptable, then there is a higher likelihood that they may be willing to use physical violence as a form of discipline, which is not considered a good parenting practice. top 2: if a candidate misrepresents their experience and doesn't actually know how to do the job, then they are not capable of working at that job. we notice that although rules are generalized from the physical task, both rules prioritize social knowledge over physical appearance. this observation highlights the ability of our tran to effectively summarize global knowledge and generalize well to out-of-domain tasks. how does tran perform in counterfactual scenarios? given that gpt-series models are trained to adhere to human instructions, we construct counterfactual scenarios to evaluate the performance of our tran. these scenarios consist of data distributions that are different from human preferences. to carry out this evaluation, we manually modify the classification surface of two datasets, offensive and irony, sourced from the tweeteval @xcite . we label all instances containing hashtags (#) as "offensive" or "irony". in total, 476 and 255 instances have been modified, respectively. the comparison results are provided in moreover, we define the rule "if the content contains a hashtag, then it is offensive (irony)", delineating the adjusted classification boundary. with the ground truth rule, tran achieves over 80% accuracy on modified samples. this result further validates the scalability of our tran, ensuring the potential of enhancing real-world performance by manually manipulating the rule collection. ## conclusion and future work in this work, we introduce tran, an innovative tuning-free framework that enhances the selfalignment capabilities of llms in a streaming setting, without additional training data. tran utilizes an iterative process of generating and accumulating rules based on observed mistakes, enabling llms to avoid repeating similar mistakes. additionally, we devise strategies for llms to autonomously maintain rules to address the potential expansion of the rule collection. extensive experiments demonstrate that our tran framework outperforms recent comparative approaches on diverse datasets. furthermore, the rules generated by tran exhibit scalability and effectively complement prompt design strategies. manually crafted counterfactual scenarios further validate the efficacy of our approach. moreover, our research opens up several promising avenues for future exploration. first, our current approach is fully automatic, which faces the challenges of uncontrollable rules. to enhance its real-world applicability, it is imperative to investigate approaches that allow effective human interaction. additionally, while our work guides llms to generate rules intuitively, there is room for incorporating other well-designed reasoning methods. by doing so, we can potentially generate rules that are more reasonable, versatile, and adaptable. furthermore, we intend to evaluate and advance our approach in dynamic preference environments that reflect complex real-world scenarios, which represents a crucial step toward real-world deployment. in conclusion, the impressive performance of tran showcases its potential to augment llms in real-world applications and it remains largely unexplored how to effectively adapt llms to dynamic environments better.