PURE : Aligning LLM via Pluggable Query Reformulation for Enhanced Helpfulness (2024)

Abstract: Aligning large language models (LLMs) with human values and preferences is a significant challenge. Training-based methods, such as reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO), require substantial resources and are impractical for API-based LLMs. Post-processing methods decouple alignment from training but may incur high multiple-time inference costs or rely on less knowledgeable lightweight models for response refinement. In this paper, we propose a new LLM alignment paradigm from the perspective of pre-processing. By reformulating risky queries into highly relevant yet harmless ones before feeding them into LLMs, our method eliminates the high costs of training base LLMs, efficiently applies to both open-source and proprietary LLMs, and achieves a promising balance of harmlessness and helpfulness. For example, with Vicuna-7B as the LLM to align, it enhances helpfulness by 28.52% over DPO while maintaining comparable harmlessness levels. When applied to Gemini-1.5-pro, it increased harmlessness and helpfulness by 7.04% and 29.37%, respectively.

## introduction large language models (llms) depend on vast amounts of text data sourced from the internet for their training @xcite . such uncurated data may contain inappropriate and offensive material, posing the risk of generating harmful content that diverges from human values during llm application @xcite . therefore, aligning llms with human values and preferences to provide helpful and harmless services has become a significant challenge @xcite . numerous approaches have been proposed to address alignment issues, broadly categorized into two types: training-based methods. rlhf (reinforcement learning from human feedback) is a prominent example of a training-based model alignment technique @xcite @xcite . more recently, other trainingbased methods such as dpo @xcite , orpo @xcite , and kto @xcite have emerged. these methods finetune large models based on human preference data, demonstrating significant alignment efficacy. however, they require substantial training resources and high-quality human preference data @xcite . moreover, the alignment process heavily depends on the base llm, necessitating training for each llm to achieve alignment. additionally, training-based methods are impractical for application in api-based llms @xcite . post-processing methods. recent efforts have explored post-processing methods to decouple the alignment process from the base llm, with aligner @xcite and llama guard @xcite as notable examples. beyond achieving independence from model training, these methods also enhance helpfulness in scenarios where training-based techniques may generate excessive refusals @xcite . a specific type of post-processing involves using the base llm itself to refine the initial response multiple times, which incurs substantial inference costs @xcite . instead, aligner primarily involves efficiently rewriting the original response to be more helpful using an additional small rewriter model. however, since the rewriting process does not leverage the rich knowledge of the powerful base llm, the lightweight rewriter itself can become a bottleneck in improving helpfulness. in this paper, we propose a new llm alignment paradigm from the perspective of pre-processing. generalized, lightweight, and flexible. compared to classical training-based alignment methods, pure acts as a frontend plugin that eliminates the high costs associated with training base llms. it can flexibly and efficiently perform alignment across a wide range of llms, including both opensource and proprietary ones. promising harmlessness-helpfulness balance. training-based methods are widely known for their "alignment tax", while post-processing alignment alleviates this tax by enriching simple refusals to improve helpfulness. compared to post-processing, our pre-processing mechanism performs the "enriching" process by coordinating the reformulated query with the robust base llm, rather than relying on an auxiliary lightweight model like aligner @xcite , resulting in better helpfulness. to verify the effectiveness of our methods, we conduct two types of evaluations: (1) assessing the quality of the reformulated risky queries as well as checking reformulation patterns and (2) evaluating how the reformulation improves the helpfulness and harmlessness of llms. for the first evaluation, we annotated a test dataset that aligns with human preferences and designed four reformulation quality metrics, including precision, correlation, safety, and text quality. we found that pure demonstrated near-human performance, significantly surpassing off-the-shelf models of the same size. for the second evaluation, we compared pure with five represenative alignment methods, finding that pure enhanced helpfulness by 28.52% over dpo and 10.32% over orpo, while maintaining comparable harmlessness levels. we also integrated pure as a preprocessing plugin into api-based and open-source models, including aligned and unaligned ones. experimental results showed that pure enhanced the helpfulness of 11 models and maintained or improved the harmlessness of all models. for example, pure increased the harmlessness and helpfulness of gemini-1.5-pro by 7.04% and 29.37%, respectively. notably, pure outperformed postprocessing methods like aligner and demonstrated significant efficiency advantages. our core contributions are as follows: we introduce a pre-processing paradigm to align llms for the first time. compared to training-based and post-processing methods, this perspective offers promising generalization, cost-effectiveness, and efficiency while benefiting the helpfulness of llm. extensive empirical results show that pure provides a robust baseline for future work in pre-processing alignment. ## related work large language model (llm) has been validated to exhibit outstanding capabilities in various domains @xcite @xcite , and as a crucial embodiment of advanced artificial intelligence, they are reshaping the way we interact with the contemporary world @xcite . our research focuses on the safety alignment of llms, and it is closely related to the following alignment methodologies: 1) reinforcement learning from human feedback (rlhf): this approach employs reinforcement-learning techniques to train llms @xcite , ensuring that the models, adjusted through specific instructions, adhere to principles of social good @xcite . numerous methods and optimization strategies have already been proposed for this purpose @xcite @xcite . 2) moderation apis: this method involves labeling and categorizing safety-related content, providing a level of safety filtering, and representing an initial exploration into preemptive filtering @xcite @xcite . 3) llms for alignment: based on the concept of weak to strong generalization @xcite , this approach involves using smaller models to guide or substitute larger models for alignment tasks @xcite @xcite . 4) self-correction: this method guides models to achieve further alignment of initial outputs through iterative self-refinement and self-critique @xcite @xcite . ## methodology in this section, we present the data details and training process of pure. ## pure performance for query reformulation in this section, we focus on pure's performance in the core tasks of risk identification and reformulation, and evaluate its capacity through experiments. ## pure performance as aligning plugin in this section, we investigate the overall performance changes after integrating pure as a plugin for base models. ## conclusion research and experiments demonstrate that pure effectively enhances the aligning generalization and balances the harmlessness and helpfulness of the base model's responses. we are committed to further investigating the intrinsic characteristics and data support for aligning large models with smaller models.