Learning to Refine with Fine-Grained Natural Language Feedback (2024)

Abstract: Recent work has explored the capability of large language models (LLMs) to identify and correct errors in LLM-generated responses. These refinement approaches frequently evaluate what sizes of models are able to do refinement for what problems, but less attention is paid to what effective feedback for refinement looks like. In this work, we propose looking at refinement with feedback as a composition of three distinct LLM competencies: (1) detection of bad generations; (2) fine-grained natural language critique generation; (3) refining with fine-grained feedback. The first step can be implemented with a high-performing discriminative model and steps 2 and 3 can be implemented either via prompted or fine-tuned LLMs. A key property of the proposed Detect, Critique, Refine (“DCR”) method is that the step 2 critique model can give fine-grained feedback about errors, made possible by offloading the discrimination to a separate model in step 1. We show that models of different capabilities benefit from refining with DCR on the task of improving factual consistency of document grounded summaries. Overall, DCR consistently outperforms existing end-to-end refinement approaches and current trained models not fine-tuned for factuality critiquing.

## introduction large language models (llms) have been observed to display inconsistent behavior such hallucinations, not following instructions, and unfaithful reasoning @xcite @xcite @xcite one recent strategy to fix these mistakes is to perform post-hoc refinement of the response with natural language feedback @xcite . these methods either use human feedback @xcite or, … has caused controversy since its implementation in 2005. in response to a lawsuit by 10 students who failed the exam, a judge has allowed thousands of students who did not pass to potentially graduate. ## refining with fine-grained feedback we propose decomposing the task of refining textual responses using natural language feedback into three steps: detect, critique, refine. algorithm 1 concretely shows the cascade of these steps. step 1: detect with m detect for a response r grounded in document d, we split the response into sentences @xmath0, and for each sentence s @xmath1, we determine if there is an error by computing m detect (s i , r, d) ∈ {0, 1}. if all sentences are correct, we do not modify the response. if any sentence is marked with an error, we generate feedback using the critique step. step 2: critique with m critique once we have determined r to have an error, for each sentence s i with m @xmath2, we generate a natural language feedback f @xmath3, r, s i ) that does span localization, reasons about why the span has an error and then suggests a natural language fix for the span. we combine the sentence wise feedback to create f and use it for the refine step. step 3: refine with m refine given the document d, the response with errors r and the natural language feedback f , we use @xmath4, r, f ) to generate a response which targets editing error spans mentioned in f . we refine with combined feedback as opposed to per-sentence to enable m refine to place edits in context and better produce a final coherent response. ## results does the three-step refinement help over standard refinement strategies? ## understanding generated feedback is the proposed detailed feedback form helpful? the feedback used in our proposed method has two important parts: error localization and a reasoning for why it is an error with a suggested fix. to evaluate the importance of this detailed feedback, we fine-tune m critique to generate only the error localization as feedback and refine with a non fine-tuned m refine . focusing on llama3-8b-instruct, we in what kind of edits does the feedback model suggest and the refinement make? we manually examine 50 feedbacks generated from gpt-4, critique-l3-ft, and critique-l2-ft when prompted to refine with dcr. are the edits made by the refinement model faithful to the feedback? we manually examined 50 mediasum/tofueval refinements generated using dcr by gpt-4, refine-l3-ft. and refine-l2-ft. for each instance, we first look at the error span and the reasoning and then look for evidence in the refinement for whether or not the error span was fixed according to the feedback. if the error span was fixed as per the feedback, we mark the edit being faithful to the feedback. for each response, we calculate the percentage of feedback points that were incorporated in the refinement and then average that over the 50 examples for every model. we find that on average gpt-4 is able to incorporate 92% of the feedback, where as refine-l3-ft is able to incorporate 96% and refine-l2-ft is able to incorporate 69%. ## related work several recent evaluation datasets in nlp have followed the trend of collecting explanations alongside evaluation scores. this ranges from model based evaluation datasets @xcite @xcite @xcite @xcite to human evaluation @xcite @xcite . in our work, we test feedback for effectiveness at refinement, which feedback for evaluation is not always optimal for. for refinement specifically, @xcite implement a two step (feed+refine) refinement pipeline for the task of improving factual consistency in summarization.however, their use of the xsum dataset results in several key differences: the high prevalence of errors, simple summarization task, and short summaries mean that no "detect" step is necessary. furthermore, our use of stronger llms and factuality evaluators substantially changes the performance regime of our base model and evaluator methods, leading to different conclusions. fatahi bayat et al. ( 2023 ) implement a two-step (detect+dr) pipeline for claim-level fact-checking. they use an external knowledge source to retrieve the relevant evidence for the claim. they verify using the evidence and also use it to guide the revision. in our experiments, we show that the proposed three-step pipeline outperforms two-step refinement. @xcite also compare feedback modalities like in our work. however, they focus on refining with a general instruction vs. using scalar feedback vs. using a binary feedback. furthermore, they improve overall quality rather than a specific aspect like factuality. to refine outputs for factuality, @xcite train a hallucination detector and editor by fine-tuning the model to localize fine-grained hallucination error types by tagging spans and then removing the tagged spans in post-editing. balachandran et al. (2022); fabbri et al. (2022) ; thorne and vlachos (2021) train post-editing models with techniques like infilling and sentence-compression to train better post-editing models. these approaches do not use natural language feedback, and are most optimized for deletion or replacement of simple errors rather than complex rewriting, where feedback can more clearly articulate a subtle error. ## conclusion in this work, we propose a new post-hoc refinement method: detect, critique and refine (dcr). we showed that our method performs better than existing refinement baselines on the task of improving factual consistency in document-grounded topic-focused summaries. we also showed that smaller models can be fine-tuned to perform finegrained feedback generation for identifying and reasoning about any factual inconsistency. when refining with our proposed method, smaller models perform on par with gpt-4 on our task. we also show that our critic model produces more effective feedback for the task of post-hoc refining as compared to existing models.