MAF : Multi-Aspect Feedback for Improving Reasoning in Large Language Models (2023)

Abstract: Language Models (LMs) have shown impressive performance in various natural language tasks. However, when it comes to natural language reasoning, LMs still face challenges such as hallucination, generating incorrect intermediate reasoning steps, and making mathematical errors. Recent research has focused on enhancing LMs through *self-improvement* using feedback. Nevertheless, existing approaches relying on a single generic feedback source fail to address the diverse error types found in LM-generated reasoning chains. In this work, we propose **Multi-Aspect Feedback**, an iterative refinement framework that integrates multiple feedback modules, including frozen LMs and external tools, each focusing on a specific error category. Our experimental results demonstrate the efficacy of our approach to addressing several errors in the LM-generated reasoning chain and thus improving the overall performance of an LM in several reasoning tasks. We see an improvement of up to 20% in Mathematical Reasoning and up to 18% in Logical Entailment.

## introduction recent research in language models has focused on augmenting them with external tools @xcite , learning from feedback @xcite and iterative-refinement @xcite @xcite . iterative-refinement has been a necessary tool in human evolution and problem-solving. moreover, humans seek feedback from domain-specific knowledge sources. for instance, an architect tasked with creating an environmentally friendly and structurally sound building design will require targeted feedback from civil engineers for structural integrity, and sustainability experts for eco-friendly 1 our source code can be found here design principles. however, previous works on iterative refinement overlook this requirement and collect generic feedback from multiple sources or collect a single one from the language model itself. in this work, we first investigate whether the use of generic feedback is a bottleneck in addressing the diverse range of errors present in the reasoning chains generated by lms. we posit that utilizing generic feedback for a broad spectrum of errors may result in vague or non-actionable feedback for specific errors. this is due to two main factors: (i) the inclusion of multiple error categories within a single prompt, which not only increases the size of the prompt but also poses a challenge to current models that have a limited context length and struggle with long texts, and (ii) the model is burdened with identifying a multitude of error types in a single instance, which degrades the quality of the generated feedback. to surmount these challenges, we introduce multi-aspect feedback (maf), a novel generalpurpose iterative refinement framework that employs a collection of specialized feedback modules, including pre-trained lms and external tools, each tailored to address a specific error category. our framework includes a base model to generate an initial solution, multiple feedback modules, and a refiner model to revise the solution using feedback. contrary to previous works, our feedback modules can use one of two refinement strategies: eager refinement and lazy refinement §3.4. when using the eager-refine mode, the solution is revised immediately before moving on to the next feedback, however, for lazy refinement, we first collect feedback from multiple feedback sources and then refine the solution using this collective feedback. in devising maf, we first classified errors in lmgenerated reasoning chains based on @xcite and also identified some new error categories. subsequently, we decoupled our feedback modules such that each module focuses on a single error category. this strategy not only elevates performance but also allows one to leverage specialized tools tailored for distinct error types, such as utilizing a code interpreter for syntax errors in generated python programs instead of relying on generic lm feedback. through a modular design, we ensure that each module of our framework is replaceable depending on the task. we conduct extensive experimentation on a variety of complex reasoning tasks such as mathematical reasoning @xcite , logical reasoning @xcite , and multi-hop question answering @xcite , and establish that maf effectively addresses the challenges previously discussed, and significantly enhances the performance of language models in complex reasoning tasks. we compare our method against the base lms and self-refine @xcite , which also implements iterative refinement. during our experiments, we find that over-refining §4 the solution can lead to worse performance, and thus we treat the number of iterations as a hyperparameter. we also propose an oracle verifier setting §4, where we assume we have access to an "answer verifier" and stop the refining process once we have reached the correct answer. to avoid unfair comparisons, we test self-refine under this setting and find that maf provides actionable and specific feedback. to summarize, the main contributions of this work are as follows: ## conclusion in this work, we present multi-aspect feedback (maf), a novel iterative refinement framework that decouples the feedback modules and takes advantage of error-specific tools to generate feedback. we demonstrate the performance of our framework on a set of diverse reasoning tasks and compare it with other iterative refinement baselines. contrary to previous works, we found that over-refinement can be a problem in iterative refinement frameworks since models are not certain if their own answer is correct. our work also draws focus on the necessity to devise better feedback methods and call for augmenting language models with them. we hope this work will inspire further research in this area and to this end, we make all our code, data, and prompts available.