Self-Renewal Prompt Optimizing with Implicit Reasoning (2024)

Abstract: The effectiveness of Large Language Models (LLMs) relies on their capacity to understand instructions and generate human-like responses. However, aligning LLMs with complex human preferences remains a significant challenge due to the potential misinterpretation of user prompts. Current methods for aligning LLM behaviors fall into two categories: output optimization (such as RLHF, RLAIF, and DPO) and input optimization (like OPRO and BPO). While both approaches aim to guide LLMs towards generating responses that align with desired objectives, the labor-intensive and intentions-inconsistent data annotation, as well as the strict and tedious training supervision, make them struggle to yield optimal results across all models. To address these shortcomings, we introduce a novel self-renewal approach called Prompt Optimization with Implicit Reasoning (POIR). It consists of two key components: 1) a model-specific and self-recirculating data collection method that leverages self-evaluation to enhance prompts in accordance with the model’s intrinsic logits, and 2) a prompt rewrite schema that injects implicit reasoning for direct preference learning. Through self-renewal optimization, POIR refines LLM outputs to better align with human preferences across various LLMs and tasks, without relying on supervised fine-tuning. Extensive experiments on a range of LLMs and tasks demonstrate POIR’s superior performance. We believe this advancement offers a novel paradigm for developing LLMs that are more attuned to user intentions.

## introduction recent advancements in natural language processing (nlp) have been primarily driven by the development of large language models (llms) @xcite @xcite * equal contribution. the second approach, typically like opro, promptagent, and bpo @xcite @xcite , focuses on input alignment through prompt optimization. by refining input prompts, incorporating additional context, and reformulating illogical or ambiguous components, these methods try to bridge the gap between user intentions and llms' interpretations. however, most of them require multiple rounds of prompt rewriting, which is time-consuming during the inference @xcite @xcite . moreover, opro (yang et al., 2023a) needs the task-specific searches, fixed rewriting prompts, and model-specific optimization, which hinder its generalizability and adaptability across diverse tasks and language models. bpo @xcite uses gpt-4 to rewrite the original prompts, and train one specific model to generate optimized prompts. though simple and easy to use, its generalization to unseen text is limited due to excessive rigid supervision. furthermore, a single general optimized prompt may not produce optimal results across all models. to address these, here we introduce a novel self-renewal approach called prompt optimization with implicit reasoning (poir). it contains two components: 1) a model-specific and selfrecirculating data collection method that leverages self-evaluation to enhance prompts, aligning them by the model's intrinsic logits. and 2) a prompt rewrite schema that injects implicit reasoning for direct preference learning. specifically, we employ direct preference optimization (dpo) @xcite , instructing the model to output both optimized prompts and corresponding responses. this design enables the model to implicitly reason about the prompt-response relationship, facilitating more effective prompt optimization. poir addresses the limitations of bpo by employing a model-specific data collection method that generates optimized prompts tailored to different model families. moreover, poir employs dpo instead of sft, reducing the need for high-quality annotated data. by injecting implicit reasoning, poir facilitates more effective prompt optimization and encourages a deeper understanding. extensive experiments across five models and four tasks demonstrate poir's superior performance, with win rates increasing by 9.2% to 36.2% compared to models using bpo and dpo. notably, the effect increases with model size, with poir's win rate increasing by 9.2% on the qwen-7b model and 25.6% on the 14b model compared to bpo. ## related work aligning llms with human preferences has been a focus of extensive research. two main approaches have emerged: output alignment, which employs sft followed by rlhf to optimize model outputs, and input alignment, which focuses on rewriting input prompts. this section provides a detailed discussion of these methods. ## method the objective of prompt optimization is to construct a mapping that transitions the original prompt to its optimized version. bpo @xcite trains a seq2seq prompt optimizer through sft. however, as mentioned earlier, sft has some drawbacks, including its reliance on high-quality data and the use of a strict cross-entropy (ce) loss that supervises the generation of llms on a token-by-token basis. this approach introduces a susceptibility to overfitting. for instance, an optimized prompt like "what is the pythagorean theorem?" holds the same intent as "please provide a detailed explanation of the pythagorean theorem." despite their semantic equivalence, the rigid supervision might lead to a substantial loss for such variations due to its emphasis on matching specific token sequences. this overfitting problem may restrict the flexibility of the model in handling diversely expressed prompts, detracting from the adaptability and utility of the optimized prompts in more generalized applications. as a result, this may potentially constrain the model's overall effectiveness. to address these limitations, poir employs dpo @xcite to train the prompt optimizer, thereby mitigating the issues caused by sft. the comprehensive workflow of poir is illustrated in figure 2 . we first leverage the model's inherent capabilities for prompt generation, response evaluation, and prompt optimization to produce paired data for the upcoming prompt optimizer training. this pipeline is delineated in section 3.1. by utilizing the model's inherent capabilities, poir reduces its reliance on expensive and potentially inconsistent annotated data, addressing another limitation of existing approaches. after acquiring these data, we employ dpo with an implicit reasoning approach to train the prompt optimizer, as detailed in section 3.2. this approach enables the model to implicitly reason about the relationship between the prompt and the response, facilitating more effective prompt optimization without the need for strict token-level supervision. by incorporating the model's inherent capabilities, poir aims to provide a more robust and adaptable approach to prompt optimization that can effectively handle diverse prompts and align with human preferences across a wide range of model families and tasks. ## conclusion in this study, our innovative prompt optimization via implicit reasoning (poir) presents an obvious leap forward in aligning llms with human preferences without the need for expert supervision. by introducing a unique data collection pipeline complemented with a preference learning scheme that hinges on implicit reasoning, poir effectively enhances the quality of model responses. our methodology for data collection showcases the ability to refine prompts autonomously while maintaining the critical information from the ori. prompt. the effectiveness of poir is empirically validated through comprehensive experiments across multiple models and datasets, demonstrating that our approach outperforms conventional ones.