LIRE : listwise reward enhancement for preference alignment (2024)

Abstract: Recently, tremendous strides have been made to align the generation of Large Language Models (LLMs) with human values to mitigate toxic or unhelpful content. Leveraging Reinforcement Learning from Human Feedback (RLHF) proves effective and is widely adopted by researchers. However, implementing RLHF is complex, and its sensitivity to hyperparameters renders achieving stable performance and scalability challenging. Furthermore, prevailing approaches to preference alignment primarily concentrate on pairwise comparisons, with limited exploration into multi-response scenarios, thereby overlooking the potential richness within the candidate pool. For the above reasons, we propose a new approach: Listwise Reward Enhancement for Preference Alignment (LIRE), a gradient-based reward optimization approach that incorporates the offline rewards of multiple responses into a streamlined listwise framework, thus eliminating the need for online sampling during training. LIRE is straightforward to implement, requiring minimal parameter tuning, and seamlessly aligns with the pairwise paradigm while naturally extending to multi-response scenarios. Moreover, we introduce a self-enhancement algorithm aimed at iteratively refining the reward during training. Our experiments demonstrate that LIRE consistently outperforms existing methods across several benchmarks on dialogue and summarization tasks, with good transferability to out-of-distribution data, assessed using proxy reward models and human annotators.

## introduction while a growing plethora of large language models (llms) have exhibited incredible performance in a broadening scope of tasks and applications such as summarization, machine translation, and dialog generation @xcite @xcite , they subsequently, the scored responses with their query are optimized in a listwise manner. the "dashed" line means it is optional to re-initialize the updated model π θ as the sampling policy and generates fresh responses that substitute the prior ones within the candidate pool. existing methods focus on employing reinforcement learning from human feedback (rlhf) to fine-tune the pre-trained llms @xcite @xcite , which introduces a paradigm that involves leveraging supervised fine-tuning (sft) on the initial models, fitting the reward model to human preferences, and then using reinforcement learning (rl) algorithms such as proximal policy optimization (ppo) @xcite to optimize a policy that doesn't drift overly far from the original model. however, ppo is optimized in a pointwise manner based on the sparse rewards, penalizing fragments within a sentence equally and disregarding the truly informative parts. additionally, ppo requires online sampling during training, which im-pairs computational efficiency and stability. alternatively, methods such as dpo @xcite and rrhf @xcite primarily leverage a pairwise comparison between a positive and a negative sample and transform the rl problem into an offline setting. however, the performance of pairwise comparison is heavily dependent on the quality of the sample pairs, and trivial negatives may yield suboptimal results. moreover, given a large candidate pool, performing pairwise comparisons among multiple samples entails a significant computational complexity. to fully exploit the diversity among multiple samples and better identify high-reward segments within sentences, we propose an rl-free listwise approach: listwise reward enhancement for preference alignment (lire). unlike existing algorithms @xcite ) that employ the bradeley-terry model @xcite or plackett-luce (pl) models @xcite to model the preferences, we reformulate the response probability distribution and implicitly model the preferences with the offline rewards. the probabilities of responses are updated under the guidance of the rewards, entailing a more accurate indication of how the responses are preferred than their binary format counterpart of "chosen" and "reject" labels. 1 notably, the concept of listwise in lire is different from the traditional listwise approach proposed in the learning-to-rank literature @xcite , which is largely based on the pl model and requires a ranking permutation determined by the position given any labels. the key idea for the pl model is that the choice in the i-th position in a rank π only depends on the candidates not chosen at previous positions, leading to a time complexity of at least o(n) depending on different surrogate losses. in contrast, in lire we compute the exponential probability distribution only once, making it efficient to compute even if n increases. the overarching concept is illustrated in figure 1 : we first gather responses a for queries q from different sources, after which the rewards are collected before the optimization begins. an optional strategy to further boost performance is 1 in this sense, lire can also be viewed as an optimization objective under the paradigm of minimum bayes risk (mbr) decoding @xcite , where the method for choosing the output is based not on the output with the highest probability, but the output with the lowest risk (highest reward) among multiple candidates. lire is easy to implement, free from heavy parameter tuning, and perfectly fitted within the pairwise paradigm with natural extension towards the multi-response scenario. experiments of the stateof-the-art methods are fairly conducted on multiple benchmarks of dialogue and summarization tasks. the results show that the proposed lire achieves superior and consistent performance in all the experiments, using strong 7b-llms as base models and various proxy reward models, gpt-4 as well as human evaluators for assessment. ## preliminaries in this section, we illustrate the motivation for the lire framework and the related preliminaries. we start with the objective function of the policy gradient (pg) methods: to better approximate p (τ, θ) and thus a better indicator of the expected rewards in equation 1, we propose to leverage multiple pre-allocated responses to reformulate the trajectory probability distribution. we also transform the rl learning problem into an offline setting, improving computational efficiency and stability. ## methodology in this section, we first introduce the construction of the lire objective and its relation with another popular objective from a theoretical aspect. subsequently, we introduce a self-enhancement algorithm that further boosts the model preference. please note that directly training with pairwise preference data in a single stage (without algorithm 1) will suffice to bring out impressive performance and only experiments in section 4.7 leverages iterative sampling and training. ## related work reinforcement learning from human feedback. leveraging human feedback to improve model generation ability toward human desire renders it imperative given the quickly growing family of llms. directly leveraging human feedback to optimize models generally requires an "optimizable" formulation of the feedback @xcite . however, it is expensive and impractical to generate sufficient human feedback for llm training in general cases. alternatively, one line of work relies on models to produce feedback that approximates human perception @xcite @xcite . given enough feedback (preference data), rlhf has been extensively employed to optimize an llm with various training objectives using a unified approach. ## conclusion in this paper, we propose lire, a listwise optimization scheme under the general pg framework for preference alignment tasks, and a self-enhancement algorithm to progressively optimize rewards. lire learns the preferred patterns through iterative maximization of the overall rewards of the diverse candidate pool. our approach is free from heavy parameter tuning and simple to implement, exhibiting commendable performance on dialogue and summarization tasks.