RL 4 F : Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs (2023)

Abstract: Despite their unprecedented success, even the largest language models make mistakes. Similar to how humans learn and improve using feedback, previous work proposed providing language models with natural language feedback to guide them in repairing their outputs. Because human-generated critiques are expensive to obtain, researchers have devised learned critique generators in lieu of human critics while assuming one can train downstream models to utilize generated feedback. However, this approach does not apply to black-box or limited access models such as ChatGPT, as they cannot be fine-tuned. Moreover, in the era of large general-purpose language agents, fine-tuning is neither computationally nor spatially efficient as it results in multiple copies of the network. In this work, we introduce RL4F (Reinforcement Learning for Feedback), a multi-agent collaborative framework where the critique generator is trained to maximize end-task performance of GPT-3, a fixed model more than 200 times its size. RL4F produces critiques that help GPT-3 revise its outputs. We study three datasets for action planning, summarization and alphabetization and show relative improvements up to 10% in multiple text similarity metrics over other learned, retrieval-augmented or prompting-based critique generators.

## introduction correcting model outputs is a pressing challenge in natural language generation @xcite , emerging across many use-cases such as style transfer @xcite ., 1 a significant portion of this work was done while feyza was an intern at allen ai. correspondences to afra feyza akyürek (akyurek@bu.edu). put soap in dishwasher action planning topic-based summarization input ( ): x passage: when did it all begin? where it always does i suppose, with the vows […] question: summarize the plot of the story. model output ( ): ŷ 1. open dishwasher door 2. grab some dish soap 3. put soap in dispenser michael sits in the waiting room, thinking about his family […] he was betrayed by his father bill […] critique ( ): c one cannot put soap in the dispenser without first opening it. bill wasn't michael's father. corrected output ( ): ŷ new 1. open dishwasher door 2. grab some dish soap 3. open dispenser 4. put soap in dispenser michael sits in the waiting room, thinking about his marriage […] he was betrayed by his friend bill […] recently, researchers show that large language models can correct their answer given more sophisticated feedback formulated in natural language @xcite . for example, in fig. 1 , we present sample feedback for two tasks. both of these examples exemplify the case where the initial model outputs ŷ have flaws. in topic-based summarization, an automatically generated summary of a story involves factually incorrect statements such as "... he was betrayed by his father bill ..." where an appropriate critique is "bill is not michael's father". in action planning, given a goal x, the objective is to generate a set of steps y to achieve the goal. the initial sequence of actions in fig. 1 , denoted by ŷ, has a missing a step. the human-written natural language critiques c describe the ways in which ŷ's are incorrect and ŷnew denotes the corrected prediction conditioned on the critique. note that in many situations helpful critiques do not necessarily reproduce an entire answer-they may simply point out one way in which the answer could be improved. researchers use crowd-sourcing to collect critiques for model outputs @xcite . however, collecting feedback from humans is infeasible in an online setting where a model is required to produce a rapid stream of outputs. the goal of this paper is to shed light on whether the task of critiquing language model predictions can be effectively passed on to an external agent while keeping the language model itself intact. our multi-agent collaborative framework involves two language models where one model's job is to criticize the other as the latter performs a task of interest, such as summarization. this setting comprises a task model, denoted by lm task , which learns the mapping from an input x (e.g. passage) to a ground truth output y (e.g. summary); and a critiquing model lm critique which provides natural language critiques for lm task 's outputs ŷ ∼ lm task (x). the framework can additionally involve a separate model (say lm refine ) for repairing model outputs conditioned on critiques. we follow past work @xcite , and merge lm task and lm refine into a single model. hence, in addition to predicting y given x, lm task is also tasked to improve its initial output conditioned on a critique ĉ sampled from lm critique (x, ŷ). we introduce rl4f (reinforcement learning for feedback generation), a cascade @xcite of two language models for automatic cri-tique generation and refinement. rl4f is trained to maximize target task performance of lm task by learning to provide critiques for its outputs via lm critique . rl4f advances retrieval-based methods with learned critique generation @xcite . unlike previous work which teaches lm task to read a crowd-sourced set of critiques @xcite , rl4f learns the particular set of critiques that will steer lm task into improving its predictions without requiring any updates to lm task parameters. treating lm task as fixed is especially important in era of limitedaccess large language models which are costly, if not impossible, to fine-tune. rl4f is illustrated in fig. 2(a , c ). previous work demonstrate that language models smaller than roughly 50 billion parameters lack the ability to understand and act upon a natural language critique @xcite . therefore, we chose gpt-3 as the lm task model which is a clear example of an inaccessible lm that shows this ability. while rl4f is general enough to accommodate an ensemble of feedback generators, in this work we focus one single model as lm critique for simplicity. in summary, this work foot_0 : ## related works past works differ to a large extent with respect to what they call human feedback and how they make use of it. in this section, after elucidating the use of human feedback in previous works, we briefly describe connections of rl4f to the parameterefficient fine-tuning and discrete prompt learning literature. ## background the problem of learning from critiques entails two major challenges: (1) generating critiques and (2) the task of refining initial answers based on a critique. in our experiments (section 6), gpt-3 responds very well to ground-truth critiques. this observation suggests that given quality critique, gpt-3 is indeed able to improve a potentially erroneous prediction for the better. hence, in this study we focus our efforts on (1). our ultimate goal is to reach, and eventually exceed, human-level critique performance using machines. following @xcite , we identify four primary functions towards studying the problem of learning with natural language critiques. a) predict @xmath0: input e.g. goal or passage and question y: ground truth for x : large model for and : small model for : prediction by : natural language critique by : parameters are frozen r: reward function; figure 2: a) a downstream task model takes in an input (e.g. a passage and a question) and predicts the output (e.g. summary). b) past work proposed using a supervised learning scheme @xcite or retrieval @xcite for critique generation (critique) and refinement tasks (refine). in our setting, we only train lmcritique and parameters of the task model are left unchanged. c) rl4f uses lmcritique that was produced as a result of the training in part b. using task data pairs (e.g. passages and summaries) we continue fine-tuning lmcritique with policy gradient such that critiques steer lmtask to produce better outputs. first is predict: the base task of predicting without using critiques to model @xmath1. as an example, if x is a passage, y is the summary (see fig. 1 ). moreover, we refer the task of learning to generate critiques x, ŷ → c where ŷ ∼ lm task (x) as critique. lastly, we call the conditional refinement objective x, ŷ, @xmath2, ŷ → y as direc-trefine 3 . we use ŷ and ĉ notation to indicate an estimate of ground truth y, and similarly for c, from a respective model. ## rl4f: reinforcement learning for feedback generation supervised is straightforward to implement but it does not make use of any final task data (@xmath3. moreover, it fails to provide ground for adaptation when the critiques in the train set are generally applicable but not entirely tailored to improving a target model. we describe rl4f where we follow supervised training with policy gradient learning using end-task data in order to generate critiques. we assume that the task model lm task is already deployed and treat it as a fixed module. in all of our implementations we train the natural language critique generator lm critique alone. in both supervised and rl4f, lm critique takes in the input x and an initial prediction ŷ and produces a (natural language) critique ĉ: fig. 2c provides an illustration of rl4f. we implement lm task as gpt-3 given its adaptability into new tasks using few-shot prompting. our implementation which is primarily based on the rl4lms library 4 @xcite will be publicly available. ## analysis scaling properties while we use t5-large as our main model for all of our experiments to initialize lm critique , we inquire about different model sizes. in fig. 4 , we consider three different model sizes to tackle interscript ranging from 60m to 770m parameters. on the y-axis we provide average of three rouge scores for the generated plans. rl4f greatly benefits from an increase in the model size where a similar trend in super-vised is non-existent. 6 we have identified a handful of examples where a pair of train and test examples differs by only a single concept e.g. all occurrences of "noodle" in the train sample was replaced with "food" to produce the test sample. the goal and steps are the same otherwise. memprompt does exceedingly well on these 7 cases, hence performing occasionally higher, yet fails in the rest of the test/val examples. ## conclusion we have described a collaborative framework involving two language models where one model, the critique generator, is trained to improve the performance of the task model. we train the former via policy gradient learning while treating the task model as a black-box. we show that rl4f leads to superior final performance across three domains compared to other strong baselines without resulting as the critiques remain fluent and natural. future work might focus on generalizing the critique generator into a mixture of experts allowing humans and other models to contribute to critiqueing procedure.