Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Models (2024)

Abstract: The correct specification of reward models is a well-known challenge in reinforcement learning.Hand-crafted reward functions often lead to inefficient or suboptimal policies and may not be aligned with user values.Reinforcement learning from human feedback is a successful technique that can mitigate such issues, however, the collection of human feedback can be laborious.Recent works have solicited feedback from pre-trained large language models rather than humans to reduce or eliminate human effort, however, these approaches yield poor performance in the presence of hallucination and other errors.This paper studies the advantages and limitations of reinforcement learning from large language model feedback and proposes a simple yet effective method for soliciting and applying feedback as a potential-based shaping function.We theoretically show that inconsistent rankings – which approximate ranking errors – lead to uninformative rewards with our approach. Our method empirically improves convergence speed and policy returns over commonly used baselines even with significant ranking errors, and eliminates the need for complex post-processing of reward functions.

## introduction the correct specification of task rewards is a well-known challenge in reinforcement learning (rl) @xcite . complex tasks often necessitate complex, nuanced reward models, particularly as shaping terms may be required to guide exploration. however, hand-crafting these reward functions is difficult and often leads to a phenomenon known as reward hacking, wherein an agent learns to exploit a reward function for increased returns while yielding unexpected or undesired behavior @xcite . reward hacking is symptomatic of the broader challenge of value alignment, in which it is difficult for a human domain expert to fully and accurately specify the desired behavior of the learned policy in terms of a reward function. in this study, we aim to eliminate the dependence on handcrafted reward functions by training agents with reward functions derived from data. a notable method in this domain is reinforcement learning from human feedback (rlhf), where policy trajectories are ranked by humans. these rankings are then used to learn a reward function which guides model training and facilitates value alignment. this process is extremely costly in terms of human effort, however, requiring a significant number of rankings to train accurate reward models @xcite . we can avoid the need for humans-in-theloop by instead generating rankings with pretrained large language models (llms) in a process known as reinforcement learning with ai feedback (rlaif) @xcite @xcite . however, llms are well known to hallucinate and present false information as fact @xcite , which reduces the accuracy and reliability of the resulting rankings. this is often overcome through complex reward postprocessing techniques, which may be task-specific and difficult to tune @xcite . in this work, we propose a simple and effective strategy for reinforcement learning in the face of unreliable llm feedback. the core idea underlying our approach is to issue uninformative rewards for states in which the llm is uncertain. thus, we avoid issuing potentially misleading rewards which allows us to train performant policies even in the face of significant ranking errors. building off the insight that certainty in language models is expressed through output consistency @xcite , we show that rewards issued from a potential-based scoring function learned over repeated rankings naturally reflect an llm's uncertainty. our contributions are as follow, we 1) introduce a methodology for incorporating noisy llm feedback into rl which out-performs prior sota; and 2) provide theoretical and empirical analysis showing that uncertain llm outputs -as given by inconsistent responses -lead to uninformative rewards which improve convergence speed and policy returns in experiments. the codes of this work can be accessed here. ## related works constructing rewards based on human feedback has a long history @xcite . to efficiently use human domain knowledge and provide more generalizable rewards, human preference on episode segments @xcite @xcite and human demonstrations @xcite are distilled into models which serve as reward functions for rl. the method has witnessed great success in complex domains where rewards are difficult to specify such as training large language models (llm) to align with human logic and common sense @xcite . one major drawback of rlhf is its requirement of exhaustive human participation to provide demonstrations and feedback. llms have shown deductive logic abilities comparable to humans in recent years @xcite , and are able to substitute humans in reward issuing @xcite @xcite , or data collection and labeling for reward model training @xcite . while the former suffers from time and resource costs for training rl agents, the latter is becoming promising for training complex rl tasks @xcite . an outstanding challenge with leveraging llmbased feedback is that the performance of rlhf is dependent on the quality of feedback received @xcite . different llms have distinct probabilities of giving wrong feedback, thus leading to rewards of varying quality. @xcite also suggests that comparison-based feedback may not be efficient and adequate to train reward models with noisy llm outputs. in this work, we analyze the training performance of reinforcement learning agents across various llms, each of which produce different error distributions in feedback. another challenge is that of the reward formulation itself. many works train a model distilling llm or human preference and use it as the reward model @xcite @xcite , but in practice, this needs post-processing on outputs of the reward model such as filtering @xcite , and normalization @xcite . our work posits that a reward function trained without complex post-processing and environment rewards would be more general and adaptable to various practical scenarios. at each discrete timestep t in this process, the agent observes environment state s t and takes action a t , leading to the next environment state s t+1 and a reward r t . an mdp is represented as a tuple (s, a, r, t , γ), where s is a set of states, a is a set of actions, r : @xmath0, t (s, a, s ′ ) = p (s ′ |s, a) is a transition function, and γ is a discount factor. a stochastic policy π(a|s) : a × @xmath1, 1] indicates the probability that the agent selects action a given the state s. the agent's goal is to learn π maximizing the expected discounted return through training, given an initial state distribution. preference-based reinforcement learning: our work is based on the framework of preferencebased reinforcement learning, where the reward function is learned from preference labels over agent behaviors @xcite @xcite . for a pair of states (s a , s b ), an annotator gives a preference label y that indicates which state is ranked higher, considering which state is closer to the given task goal. the preference label @xmath2, 1}, where 0 indicates s a is ranked higher than s b , and 1 indicates s b is ranked higher than s a . given a parameterized state-score function σ ψ , which is commonly called the potential function and usually equated with the reward model r ψ , we compute the preference probability of a state pair with the standard bradley-terry agent target a b llm better llm state ranking scoring model (sharing parameters) (running twice) rlhf loss function scalar state score ranking individual states state pair randomly sampled consecutive state pairs worse (a) training the scoring model from llm-ranked state pairs. reward at t-1 scoring model score at t-1 time: t-1 agent behavior time: t time: t+1 time: t+2 time: t+3 potential reward training rl agents using our scoring model score at t reward at t reward at t+1 reward at t+2 reward at t score at t+3 score at t+1 score at t+2 model @xcite , ## methodology despite the success of llms in few-shot task generalization, these models are imperfect and yield suboptimal performance in many areas. one notable issue is the well-documented tendency of llms to hallucinate, which results in llm-generated preference rankings frequently containing errors (see ## performance analysis of potential-difference rewards we evaluate our approach in commonly-used discrete (grid world) and continuous (mu-joco) @xcite benchmark environments. throughout these experiments, we investigate the effectiveness of our potential-based reward function a) as compared to using the score as a reward directly in both single and multi-query settings; and b) its sensitivity to inconsistency in state rankings. ## conclusions in this work, we propose a simple method for incorporating noisy llm feedback into reinforcement learning. our approach is based on learning a potential-based score function over repeated llmgenerated preference rankings which issues uninformative rewards for states in which the llm is uncertain. we show both theoretically and empirically that this results in a natural trade-off between reward sparsity and llm confidence in a variety of discrete and continuous environments.