LLMR efine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedback (2024)

Abstract: Recent large language models (LLM) areleveraging human feedback to improve theirgeneration quality. However, human feedbackis costly to obtain, especially during inference.In this work, we propose LLMRefine, aninference time optimization method to refineLLM’s output. The core idea is to usea learned fine-grained feedback model topinpoint defects and guide LLM to refinethem iteratively. Using original LLM as aproposal of edits, LLMRefine searches fordefect-less text via simulated annealing, tradingoff the exploration and exploitation. Weconduct experiments on three text generationtasks, including machine translation, long-form question answering (QA), and topicalsummarization. LLMRefine consistentlyoutperforms all baseline approaches, achievingimprovements up to 1.7 MetricX points ontranslation tasks, 8.1 ROUGE-L on ASQA, 2.2ROUGE-L on topical summarization.

## introduction in recent years, large language models (llms) have shown impressive performance on various text generation tasks @xcite . critical to their success has been the ability to incorporate human feedback into the learning process @xcite . nevertheless, human feedback is costly to collect, especially at inference time when the model provides new, unseen input. in the meanwhile, automatic text generation evaluation metrics for a variety of tasks are rapidly improving @xcite @xcite . can we use one of these metrics to rectify llm's generation? in this work, we propose llmrefine, an inference-time optimization method to improve the quality of generated text. our llmrefine starts with llm's initial output, then uses a learned error pinpoint model to provide fine-grained feedback about the location and type of defects in the text. we then use a refinement model (same or another llm) to follow the feedback instruction and generate candidate text. the fine-grained feedback provides more much precise information about what exactly is wrong in the generated text, resulting in higher quality revision. however, due to the large search space, the refinement model is imperfect; it often fails to correct all of the errors identified by the feedback model in one iteration @xcite . we formulate the iterative refinement procedure into a local search problem. it alternates between the feedback generation and refinement in multiple iterations, with the goal of searching for the highest scoring output according to the feedback model. to this end, we develop a simulated annealing technique in llmrefine to trade off between exploring many possible edits and quickly reaching optimal text. figure 1 shows overview of our approach. we evaluate llmrefine on three text generation tasks, including machine translation (wmt @xcite ), long-form question answering (asqa @xcite ) and topic summarization @xcite , because they have a large number of annotated outputs with fine-grained error spans @xcite @xcite . we use those ratings to train an automatic error pinpoint model that generates a list of error spans along with error categories and severities without the aid of a reference text (which is unavailable during inference) @xcite . this model serves as our substitute for human feedback. our experiments show that llmrefine results in higher-quality text compared to baseline methods using other feedback (scalar or binary score) or other search techniques. our contributions are: inference-time optimization approach we divide techniques for incorporating feedback at inference time into two main techniques @xcite : generate-then-ranking and feedback-guided generation. the reranking framework involves generating a large set of candidate text outputs from the base model and utilizing a critic model to select the best output. the integration of the critic model can be achieved through chain-of-thoughts @xcite , binary verifier @xcite , or a utility function @xcite . our approach is complementary to re-ranking or minimum bayes risk decoding (mbr) strategies, offering additional performance beyond these techniques. incorporating fine-grained feedback recent studies have highlighted the benefits of fine-grained error annotation by demonstrating that it can reduce noise in human ratings and increase inter-rater agreement @xcite as well as increase automatic metric correlation to human judgments @xcite . one approach to leveraging these benefits is through the use of large language models to self-correct their own output @xcite . building on this, chen et al. to address this limitation, wu et al. (2023) propose a fine-grained reward model that distinguishes rewards at the spanlevel associating with different error categories. orthogonal to this work, we propose an inference time optimization approach to iteratively refine model's output with fine-grained feedback. ## refinement with fine-grained feedback there are three main components to our framework: a generation model, a feedback model, and a refinement model, each described next. the generation model produces an initial candidate output y i given the input x. x and y i are the source text and a candidate output that is generated by the model. the feedback model f takes x and y i and generates some form of feedback f i that represents the quality of y i , which can be in any form-a scalar value, boolean, free form natural language, or more. we assume f i can always be converted into a scalar quality score via function s(•) (details of our scoring scheme can be found in appendix a). finally, the refinement model uses x, y i , and f i and generates a new, improved output y i+1 . as we will discuss in section 4, the loop between the feedback and refinement model can repeat for multiple iterations to further evaluate and update the generated output. for most of this work, we assume that both the generation and refinement models are an llm that is 0-shot prompted to perform the respective task (see example prompt in ## iterative refinement as search although the refinement model receives the output y i and feedback f i , it is not always guaranteed to generate the best new output in a single step. therefore, we experiment with different methods for iterative refinement in which the feedback and refinement loop is repeated until some stopping condition is met. iterative refinement can be viewed as a search procedure that is trying to find the optimal y i for a given x, where "optimal" is measured by the feedback model. specifically, we model iterative refinement as a local search algorithm in which every possible output is a state in the search space, and each step of the search algorithm starts at some state represented by y i and moves to y i+1 . the goal is to find the highest scoring state. we explore three different local search algorithms, described next. ## experimental setup here, we describe the setup for experiments related to implement/evaluate error pinpoint feedback model and implement/evaluate the proposed refinement and iterative refinement procedures. ## results we explore several research questions in our experiments: 1) how well does our error pinpoint model align with human annotations of translation quality? 2) does fine-grained feedback result in better downstream translations than more coarse feedback? 3) can the feedback and refinement models be used to improve translations generated by unrelated text generation models? 4) does the iterative refinement improve the generated translation quality? ## conclusion in this work, we proposed llmrefine for incorporating a fine-grained error pinpoint into a text generation pipeline via refinement-feedback model loop. we empirically demonstrated that our fine-grained error pinpoint model with simulated annealing achieves superior performance compared to baseline feedback models and other search algorithms across three text generation tasks. lastly, our experimental findings are further solidified by the human evaluation study. humans demonstrate a significant preference for the output of llmrefine over the baseline outputs.