Reward Gaming in Conditional Text Generation (2023)

Abstract: To align conditional text generation model outputs with desired behaviors, there has been an increasing focus on training the model using reinforcement learning (RL) with reward functions learned from human annotations. Under this framework, we identify three common cases where high rewards are incorrectly assigned to undesirable patterns: noise-induced spurious correlation, naturally occurring spurious correlation, and covariate shift. We show that even though learned metrics achieve high performance on the distribution of the data used to train the reward function, the undesirable patterns may be amplified during RL training of the text generation model. While there has been discussion about reward gaming in the RL or safety community, in this discussion piece, we would like to highlight reward gaming in the natural language generation (NLG) community using concrete conditional text generation examples and discuss potential fixes and areas for future work.

## introduction natural language generation aims to automatically produce text that is fluent, relevant, and factual. to train text generators such that the outputs are aligned with desired behaviors, recent work has used rewards learned from human annotations, such as improving the quality of generated summaries by using learned saliency and faithfulness metrics @xcite ) and by using rewards based on learned question answering systems @xcite ; the recent chat-gpt model also uses an approach in the same class. in general, this class of methods (1) collects a human annotation dataset d reward consisting of, e.g., direct ratings of generations @xcite @xcite , labels of error spans in the generations @xcite , or pairwise comparison of generations given the same source sequence @xcite @xcite ; (2) learns a proxy reward function that scores generations (as opposed to a true reward function which is often given by human judgment) on d reward ; and then (3) learns the text generator on a dataset d task , using rl with the learned reward function. what could go wrong when we obtain the reward signal from humans? the rewards would rarely be robust. when training the text generator, the distribution induced by the policy (i.e., the generator) changes because we frequently update it, which opens up opportunities for exploiting errors in the reward. thus, even if the reward function performs well on the dev/test split of d reward as an evaluator, the reward can still be gamed during rl training of the generator. reward gaming commonly refers to the issue that when the proxy reward increases, the true reward decreases or stays stable @xcite . in this discussion and in the context of nlp, we use "reward gaming" to broadly refer to the phenomenon that as training progresses, models produce low-quality generations that exhibit undesirable patterns while converging to high rewards. reward gaming can happen when an undesirable pattern is associated with a high reward in the learned metric. we identify three ways this phenomenon can happen. (1) a group of examples is misannotated systematically. for instance, suppose we train a model to do effective negotiation and annotators carelessly label all long paragraphs as effective, then the reward model would assign high scores on long generations even if they are nonsensical, and the generator would subsequently exploit this pattern. (2) d reward contains some bias due to the data we select to annotate, or due to the people we select to be annotators. an example in the former case is that suppose every translation that contains "united nations" happens to have high quality/reward, possibly due to the way we collect d reward ; then the neural machine translation model may end up almost always generating the phrase surrounded by some gibberish. an example in the latter case is that due to the selection bias of annotators, certain language varieties may be rated higher (or lower) by annotators, even if the language variety itself is not an indicator of quality @xcite ; subsequently, the generator could learn to favor generating sentences of certain language varieties over others. (3) d reward does not cover certain groups of sentences. a quick example is that a dialogue agent trained to negotiate generates incomprehensible sentences, because those sentences are underspecified by the reward function @xcite . in short, among these three cases, the first two cases induce spurious correlations between the undesirable pattern and the reward, and the third case induces underspecified behavior on uncovered examples. we use synthetic and real-world examples to illustrate the above three cases: even if the learned reward achieves a good performance on d reward , high rewards can still be assigned to undesirable patterns. notably, we show that these patterns get amplified during rl training of the generators. for instance, a synthetic experiment discussed later ( §4.1) shows it is possible that even a reward function that gives the correct reward on 99.3% of the test split of d reward can lead to generation failure after rl. we also review potential fixes ( §5), including restricting the policy -e.g., maximum likelihood regularization which is commonly used in recent work including @xcite and ramamurthy et al. (2023) -and fixing the reward itself like iteratively collecting human annotations. in light of these observations, we would like to bring more attention to reward gaming in the context of conditional text generation. leveraging learned metrics during rl is a promising approach to training aligned text generation systems. but given that the rewards can only reliably improve generators if the sampled texts are within the distribution of d reward , extra caution is needed when interpreting the results when training text generators using learned rewards -quality control or manual inspection is required to ensure good generation quality. ## related work reward gaming or similar ideas have been discussed since @xcite . more recently, it is extensively discussed in @xcite . in this discussion, we avoid the term "reward hacking" because reward tampering @xcite actively changing the reward (e.g., by execution of reward-modifying code under certain circumstances in a video game) -is also reward hacking, but it is not the topic of our discussion. many examples have demonstrated the reward gaming behavior, usually in gameplay or autonomous driving. for example, in a boat racing game in amodei et al., the boat would hit objects in circles mid-way in the race instead of completing the race (the latter being the intended goal), because the reward increases faster by hitting a certain set of objects than completing the race; @xcite find that the reward is gamed in a hideand-seek game -one behavior is that hiders can trap themselves using walls and boxes so the seeker never reaches them; the reward can be gamed in a tic-tac-toe game by making specific moves to cause opponents' out-of-memory crash and lead them to forfeit @xcite . similar reward gaming behaviors have been observed in atari games @xcite , in code/program generation @xcite , in a football simulator @xcite , in a neuromusculoskeletal environment where an agent learns to run @xcite , and so on. reward gaming is rarely concretely discussed in conditional text generation. a quick example by @xcite and @xcite is that a dialogue agent trained to do successful negotiation ends up generating nonsensical sentences, because those generations are underspecified by the reward function that is used to train the dialogue model. recently, there have been two findings that indicate the seriousness of reward gaming, albeit not in the context of nlp. first, more capable models may exacerbate reward gaming: @xcite study the reward gaming problem using traffic control, covid response, blood glucose monitoring, and the river raid game, by designing misaligned proxy reward functions; they find that if an agent is more capable (depending on, e.g., model size, the number of training steps), then it is better at exploiting loopholes in the reward function, and therefore ends up with a lower true reward compared to a less capable model. more recently, @xcite has suggested a strict definition of the hackability of a pair of reward functions, where "a pair" can be understood as an original reward and a proxy reward. 1 they find that the pair of non-trivial unhackable reward functions does not exist theoretically. the question then becomes whether it is safe to use a proxy reward function empirically. in this discussion, we aim to demonstrate the effect of reward gaming in text generation using concrete examples. here are the two main differences of our discussion from the aforementioned examples. first, we focus on conditional text generation; in particular, the experiments in this discussion do not rely on state-of-the-art large language models -we aim to use smaller specialized conditional generation models instead. second, we aim to investigate the reward gaming categories when the reward signal is learned from human annotations. ## background conditional text generation systems usually model p(y | x) where @xmath0, . . . , x ts ) is a source sequence and @xmath1, . . . , y t ) is a target sequence. most models use an autoregressive factorization: log p(y | x) = t @xmath2, x), where @xmath3, . . . , y t-1 ), and p θ is parameterized with a neural network. maximum likelihood estimation (mle) leads to mismatched train/test history and objectives during sequence generation @xcite @xcite @xcite . in addition, recent work aims to better align training objectives with human-annotated quality of generated texts (e.g., translation quality judgments, summarization faithfulness, human preference of generations). the generation process can be considered a sequential decision making process suitable for rl. given state s @xmath4, @xmath5, the policy π θ (i.e., p θ ) takes action a t (a token in the vocabulary), transits to the next state s t+1 , and receives a reward r @xmath6. assume discount factor γ = 1. to maximize the objective j(θ) = e τ ∼π θ r(x, y), where r(x, y) = t @xmath7, one way is to use policy gradient (re-1 in short, reward functions r1, r2 are hackable w.r.t. a policy set and an environment, if there exist policies π, π ′ such that j1(π) < j1(π ′ ) but j2(π) > j2(π ′ ) where ji denotes the expected return corresponding to reward function ri. see definition 1 in @xcite for details. @xcite @xcite : to stabilize rl training, in each rl training run, we first initialize the model using an mle-trained model to ensure a good starting point for rl optimization. in addition, we also use kl regularization which helps rl optimization @xcite @xcite ## examples of reward gaming in conditional text generation as a reminder, we consider the class of conditional text generation learning algorithms where we: reward gaming happens when some undesirable pattern is associated with a high reward. we identify three such scenarios: we use both synthetic and real-world tasks to demonstrate the reward gaming behavior. the full experimental details can be found in the appendix. the experiments in this discussion do not rely on large language models; instead, we aim to build smaller specialized conditional generation models. for synthetic tasks, we simulate all three settings using the following framework. we adapt sudoku as a conditional text generation task. 2 a valid sudoku is a 9x9 grid with each cell containing a number from 1 to 9, such that no rows/columns and none of each of the nine non-overlapping 3x3 regions contains duplicates. for this task, let the input be the first k (k randomly chosen from 36 to 80) cells in a valid sudoku after flattening it row by row. let the reference output be the rest of the cells (i.e., the last 81k cells). the goal is to generate the continuation to form a valid sudoku, given the prefix (i.e., first k cells). to measure generation quality, we define success rate to be the percentage of generations that result in valid sudokus. while the sequence generator can be rule-based without using neural nets in this synthetic setting, to illustrate reward gaming, we consider learning the generator from a learned reward function. ## possible remedies as discussed in §2, @xcite has suggested that a pair of unhackable nontrivial originalproxy reward functions do not exist in theory. then, when is it safe to use the proxy reward function? while this is still an open question, it is possible to reduce the extent of generating undesirable sentences through the following approaches. the fundamental problem is that errors in the reward functions, specifically the over-confident errors where low-quality outputs have high rewards, can be exploited during rl training of text generators. thus, one solution is to avoid ood states that incur such errors by restricting the policy. ## conclusion we use synthetic and real-world tasks to demonstrate that even if a learned reward achieves high performance on d reward , a high reward may still get assigned to undesirable patterns which get amplified during rl training of the conditional text generation model. a critical future direction is to detect obscure or hard-to-interpret gaming behaviors especially in long generations. then, we can investigate when or how easily a spurious feature could be exploited, by exploring the relationship among the minimum description length of a spurious feature @xcite or similar statistics, the proportion of datapoints that contains the spurious feature, the choice of rl algorithm, and the degree of the reward gaming behavior. additionally, research on new approaches of reward or preference learning is needed.