Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models (2024)

Abstract: Aligning Large Language Models (LLMs) traditionally relies on complex and costly training processes like supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF). To address the challenge of achieving alignment without these extensive tuning costs and expensive annotations, we present a novel, tuning-free approach for self-alignment called Dynamic Rewarding with Prompt Optimization (DRPO). Our approach enables self-alignment through a search-based prompt optimization framework, allowing the model to self-improve and generate optimized prompts without additional training or human supervision. The core of DRPO leverages a dynamic rewarding mechanism to identify and rectify model-specific alignment weaknesses, enabling LLMs to adapt quickly to various alignment challenges. Empirical evaluations on eight recent LLMs, including both open- and closed-source, reveal that DRPO significantly enhances alignment performance, enabling base models to outperform their SFT/RLHF-tuned counterparts. Moreover, DRPO’s automatically optimized prompts surpass those curated by human experts, demonstrating its superior alignment capabilities. Our findings envision a highly cost-effective and adaptable solution for future alignment research to be further explored.

## introduction aligning large language models @xcite @xcite with human ethical standards and practical expectations is extremely crucial to prevent unintended consequences and ensure ai's positive contribution to society. traditional alignment methods, such as supervised fine-tuning (sft) and reinforcement learning from human feedback (rlhf) @xcite , are resource-intensive and require extensive human oversight, limiting their scalability and practicality. as llms grow more complex and widespread, the demand for cost-effective, annotation-efficient, and quickly adaptable alignment strategies becomes increasingly urgent. self-alignment seeks to better align llms using the models themselves; for example, by replacing human feedback with model-generated feedback @xcite , synthesizing preference data @xcite , or selfimproving with self-critique @xcite . despite these advancements, such methods still require significant resources, including the costly and unstable rlhf tuning, and some level of human supervision, such as carefully curated alignment rules or in-context learning (icl) prompts @xcite . on the other hand, as shown in figure 1 , a recent line of research focuses on tuning-free alignment, which aims for extremely efficient alignment without incurring any tuning cost. these approaches include techniques like decoding-based alignment @xcite or icl alignment @xcite @xcite . however, these tuning-free methods are often static (e.g., relying on fixed prompts or reward functions) and thus lack the flexibility to self-improve for better alignment. to marry the strengths of both paradigms, in this paper, we propose drpo, dynamic rewarding with prompt optimization, a novel tuning-free approach for llm self-alignment. drpo draws inspiration from two key insights from recent alignment research. first, the superficial alignment hypothesis @xcite posits that llms can be effectively aligned with lightweight tuning or simply prompting @xcite . second, reward models in rlhf often generalize poorly to out-of-distribution samples @xcite , whereas llms, well-known for their superior generalization capabilities, can provide more effective rewards and feedback for alignment. building on these insights, drpo is constructed atop a search-based prompt optimization (po) framework @xcite @xcite , allowing llms to selfcorrect and automatically craft detailed alignment instruction. this steers model behavior more effectively, without relying on any use of human preferences or model training. the core novelty of drpo lies in its dynamic rewarding mechanism, integrated with the optimization framework. this mechanism enables llmbased rewards to be adjusted on the fly based on specific queries, helping to identify and rectify the model's alignment blind spots. for example, if an llm with outdated knowledge pretends to answer a question requiring the latest news, its "knowledge limitation" reward will be low, and the alignment prompt will be updated accordingly. we apply this novel method to automatically craft both the system prompt and responses in icl examples, which have proven highly effective in improving alignment. we conducted comprehensive experiments on 8 recent llms using the standard alignment benchmark, just-eval-instruct, composed of questions from multiple alignment datasets. our results show that drpo can effectively align both base and sft/rlhf tuned models. notably, drpo significantly enhances base models, enabling them to outperform their sft/rlhf-tuned counterparts. figure 2 : comparing drpo with other alignment methods, such as rlhf and urial @xcite . our method consistently outperforms both the baselines for multiple llms. note that we do not have access to gpt-3.5-turbo base model; thus, both drpo and urial are directly applied to its rlhf-tuned version. drpo can further improve sft/rlhf-tuned models, showing its compatibility with other tuningbased alignment techniques. additionally, our automatically optimized prompts substantially outperform those curated by human experts. although tuning-free, the former still requires human curation and often underperforms compared to sft/rlhf-tuned counterparts. the latter, while effective, incurs higher inference costs per query, making it computationally expensive. it is worth mentioning that there is a recent promising direction of cost-efficient alignment, which introduces representation engineering @xcite to steer llm representation vectors for alignment @xcite @xcite . however, these methods typically are not fully tuning-free and require additional data or model training to identify alignment directions in the embedding space. nevertheless, drpo requires no additional annotations or model training and also only needs a one-time optimization for each model to achieve better performance than sft/rlhf-tuned counterparts. prompt optimization. discovering optimal discrete prompts becomes far more crucial nowadays. modern prompts for llms can be generally divided into two parts: in-context learning examples and detailed instructions. the former is usually treated as a retrieval problem with various schemas to select the influential examples @xcite . optimizing the latter has been heavily studied recently, mostly formulated as a sampling or search problem. generally, an initial prompt (e.g., a base prompt, "you are a helpful assistant") is given to start an iterative process, where diverse prompt candidates are generated per turn, and the best ones are kept for the next iteration. various sampling strategies are proposed to diversify the prompt candidates, e.g., back translation @xcite , evolutionary operations @xcite , self-critique @xcite . different search frameworks also have been studied, such as monte carlo search @xcite , evolutionary algorithms @xcite , beam search @xcite , and monte carlo tree search (mcts) @xcite . drpo is built on top of recent search-based prompt optimization methods, but introduces novel techniques, including dynamic rewarding, to solve the alignment problem. ## methodology in this section, we introduce our formulation formally and present drpo for solving the alignment problem by optimizing the alignment prompt. ## conclusion this paper introduced dynamic rewarding with prompt optimization (drpo), a tuning-free approach for self-aligning llms. drpo integrates a novel dynamic rewarding mechanism into a searchbased prompt optimization framework, enabling llms to self-improve its own model-specific alignment weakness adaptively. experiments on eight llms show that drpo-enhanced base models outperform sft/rlhf-tuned counterparts, and its optimized prompts surpass those by human experts. drpo's adaptability and efficiency offer a promising path toward more personalized ai systems.