REFINER : Reasoning Feedback on Intermediate Representations (2024)

Abstract: Language models (LMs) have recently shown remarkable performance on reasoning tasks by explicitly generating intermediate inferences,e.g., chain-of-thought prompting. However, these intermediate inference steps may be inappropriate deductions from the initial contextand lead to incorrect final predictions. Here we introduce REFINER, a framework for finetuning LMs to explicitly generate intermediate reasoning steps while interacting with a critic model that provides automated feedback on the reasoning. Specifically, the critic provides structured feedback that the reasoning LM uses to iteratively improve its intermediate arguments. Empirical evaluations of REFINER on three diverse reasoning tasks show significant improvements over baseline LMs of comparable scale. Furthermore, when using GPT-3.5 or ChatGPT as the reasoner, the trained critic significantly improves reasoning without finetuning the reasoner. Finally, our critic model is trained without expensive human-in-the-loop data but can be substituted with humans at inference time.

## introduction large language models (llms) have made significant strides in natural language processing (nlp) tasks @xcite . recent work has shown that explicitly generating intermediate steps during reasoning tasks significantly improves a model's performance and interpretability @xcite @xcite @xcite . producing such intermediate representations provides insight into the model's predictions and allows humans to inspect the model's reasoning process. however, these intermediate representations 1 can be unreliable @xcite and result in poor 1 in a reasoning task, the intermediate representations can be viewed as inference rules, explanations or reasoning steps. * work done at epfl performance on downstream reasoning tasks. most importantly, it is unclear how to meaningfully refine the intermediate representations to further improve the final performance. the standard practice for correcting reasoning errors is to annotate new data and either retrain or finetune the model @xcite . however, fixing such errors by finetuning with more data is not only data-and resource-intensive but can also be insufficient to generalize well in complex reasoning tasks @xcite . other works have explored improving models using feedback by providing a scalar reward @xcite or directly revealing the correct missing answer @xcite @xcite . however, in natural language reasoning tasks, defining a reward that captures different fine-grained reasoning error types (e.g., semantic consistency, logical, etc.) remains an open challenge @xcite . additionally, such a reward provides a relatively sparse training signal. in this work, we instead provide fine-grained and structured feedback on reasoning errors. we present refiner, a novel interaction-based framework that allows a generator lm to iteratively use fine-grained feedback and refine its reasoning. the interaction happens between two models: a generator, which learns to solve the task by first generating the intermediate reasoning steps, and a critic, which provides structured feedback to the generator about errors in the intermediate steps. to provide fine-grained feedback about reasoning errors, we develop a scheme to independently train the critic model on automatically constructed feedback data. more specifically, we create pairs of incorrect intermediate representations and structured foot_0 feedback on their fine-grained reasoning errors. then, we use this data to train the critic to provide fine-grained feedback on erroneous intermediate reasoning steps. finally, the critic interacts with the generator lm, offering feedback both during the training of the generator and during inference. figure 1 illustrates an example of our refiner framework where, given a math word problem, the generator generates an equation as an intermediate representation. the critic identifies the errors in the equation and provides semi-structured textual feedback (e.g., "the operator in #0 is incorrect") to the generator. by interacting with the critic, re-finer enables the generator to reason over the semi-structured feedback and refine its generation. ## related work intermediate representations. while state-ofthe-art lms achieve incredible performances in a wide range of tasks, they have difficulty with many reasoning tasks @xcite , especially ones with multiple constraints or sub-problems or requiring specialized knowledge @xcite such as mathematical problem solving @xcite @xcite @xcite @xcite . for these tasks, both intermediate representations and rationales have been shown to be beneficial in learning mathematical skills @xcite , intermediate program execution computations @xcite , or general reasoning outputs @xcite . our work builds upon the observation that generating intermediate steps are valuable but distinguishes itself in several key aspects. firstly, instead of prompting a large model, we finetune smaller models to learn to generate intermediate steps. secondly, our framework can accommodate tasks that do not necessarily have unique closed-form correct answer, such as the moral norm task (see §3). finally, our framework is trained with a critic providing feedback, improving the model's reasoning process and teaching it how to leverage feedback. ## experimental setup datasets. we evaluate refiner on three diverse tasks (examples in fig. 6 ). we briefly describe the datasets used for each task below.math word generator model eq. (z) ans. (y) uqa-base 34.1 -uqa-base + ppo 31.5 -refiner base 47.2 -uqa-large 46.7 -uqa-large + ppo 48.2 -refiner large 53.8 -gpt-3.5 + cot 64.1 67.1 gpt-3.5 + cot + refiner critic 67.3 70.6 training details. for each task, we train a unifiedqa-t5-base model (uqa-base) @xcite as a critic ( §3.1). for exploration ( §3.2), we use nucleus sampling with @xmath1.5. we select the hyper-parameters by the validation loss: for both the generator and critic model, we use the adam optimizer with a learning rate of 1e -4 . each model is trained for 20 epochs with early stopping based on validation loss. we trained all models on one a100 gpu. we run our models with 3 random seeds and report the average results. for the human study, we selected outputs from the best models (baselines and our model) according to automatic metrics. we train models with @xmath2. at inference time, we use greedy decoding for the generator and critic model with @xmath3. on the mwp and snlr tasks, we use the exact match (em) metric for intermediate steps (equation generation and inference rules) and accuracy (acc) for the final answers. for ms, we conduct a manual evaluation study to assess the relevance of norms and moral actions foot_4 . further evaluation details are provided in appendix g. to train the critic model, we used the feedback data generated using the rulebased perturbation strategy (see §3.1). baselines. we compare our method with three different lms as generator models: uqabase, uqa-large (supervised setting), gpt-3.5text-davinci-003 and chatgpt (few-shot setting). we also compare refiner to proximal policy optimization (ppo) rl-based method @xcite . we use the implementation of ppo from @xcite . for gpt-3.5, we provide 2 for demonstrations per class. we also experimented with chain of thought (cot) prompting @xcite where the model is prompted first to generate the intermediate steps (z) and then the final answer (y). note that the snlr task is a synthetic task where the model needs to perform either one-hop or two-hop reasoning. @xcite showed that fine-tuning large language models (354m parameter size) could achieve (99% accuracy) high performance. hence, we only compare our refiner model with the uqa-base model (220m) (see ## results we evaluate our model on two aspects (i) performance on intermediate steps and (ii) performance on the final answer prediction. tables 2 , 3 , and 4 show the performance comparisons. performance on intermediate steps. for snlr, similar to @xcite , we observe that gpt-3.5 performs poorly (see ## conclusion in this paper, we propose refiner, a framework to improve the reasoning abilities of lms through an iterative feedback loop between two models, a generator and a critic. our evaluation of this framework on three reasoning tasks showed structured and fine-grained feedback on intermediate reasoning errors results in significant performance gains, surpassing scalar value feedback. our trained critic model alone, even when noisy, can improve intermediate representations of lms, showing that re-finer can significantly boost lms' performance on reasoning tasks. our refiner framework is very general and, in principle, might be applied to steer language models in performing different reasoning tasks. more specifically, the critic model can be seen as a tool for llms to refine their generation quality.