The ART of LLM Refinement: Ask, Refine, and Trust (2024)

Abstract: Large Language Models (LLMs) have demonstrated remarkable generative abilities, but can they judge the quality of their own generations and self-improve?A popular concept, referred to as *self-refinement*, postulates that LLMs can detect and correct the errors in their generations when asked to do so. However, recent empirical evidence points in the opposite direction, suggesting that LLMs often struggle to accurately identify errors when reasoning is involved. To address this, we propose a reasoning with a refinement strategy called *ART: Ask, Refine, and Trust*, which *asks* necessary questions to decide when an LLM should *refine* its output, and uses it to affirm or deny *trust* in its refinement by ranking the refinement and the initial prediction. On two multistep reasoning tasks of mathematical word problems (GSM8K) and question answering (StrategyQA), *ART* achieves a performance gain of +5 points over self-refinement baselines, while using a much smaller model as the decision maker. We believe that *ART* with smaller models, making refinement decisions can be a cost-effective alternative to fine-tuning LLMs.

## introduction the ability of large language models (llms) to generate coherent and meaningful text has significantly improved over the years @xcite . however, llms still exhibit inaccuracies in their generations, and it has been posited that iterative refinement of generations, also using the same llms, can help rectify these errors @xcite @xcite . @xcite demonstrated the potential of self-refinement for tasks such as dialogue response generation and sentiment reversal; however, this approach proves less effective on other tasks such as mathematical reasoning. @xcite and @xcite have also demonstrated the challenges llms face in identifying errors in reasoning tasks. while developing models that can self-evaluate and self-correct their errors is certainly a vital step towards reliable llms, building such models is quite challenging. through empirical observation on two multistep reasoning datasets, we find that self-refinement does not reliably improve initial generations, validating the previous findings of @xcite . in fact, in the majority of cases, self-refinement has a detrimental effect on performance. on the other hand, fine-tuning language models usually improves their performance on a given task by facilitating better adaptation to the task objectives @xcite . smaller models can be trained on llms' data to improve their performance, which can further serve as cost-effective alternatives to llms for the given task @xcite @xcite . thus, in this paper, we explore the possibility of training a smaller model as the decision maker for refinement, which can be used to determine when to refine the output of a llm, or use the original output. in our work, we propose a refinement approach called art: ask, refine, and trust, which, given an initial llm response, works in the following three stages: (a) evaluating whether the initial generation requires refinement by asking a series of questions (ask); (b) executing the refinement step based on the evaluation (refine); and finally (c) selecting either the refined result or the initial prediction (trust). on two multistep reasoning tasks, mathematical reasoning and question answering, we illustrate the effectiveness of art by training refiners of different sizes. we observe that a much smaller model (llama 7b; @xcite trained to decide when to refine, can outperform a 10x larger model (llama 70b) in a self-refinement setup (by up to 5 points). we evaluate the cost and accuracy tradeoffs of training a smaller model question: john buys 2 pairs of shoes for each of his 3 children. they cost $60 each. how much did he pay? how many pairs of shoes did john buy? how much did john pay? subquestion: 1. 2. 2 pairs of ... the answer is 720. all subq answered refinement: q: how many pairs of shoes did john buy? a: john buys 2*3=6 pairs of shoes q: how much did john pay? a: john paid 6*60=360. the answer is 360. llm 2 pairs of ... total amount is ... the answer is 720. with art to make the refinement decision for a pretrained llm vs fine-tuning the llm. in many cases, we illustrate the cost-effectiveness of art as a viable alternative to fine-tuning llms. we show that our trained models (asker and truster) can work seamlessly across a wide range of llms (llama 70b @xcite , chatgpt @xcite and gpt-4 (openai, 2023)) without requiring additional modifications. ## art: ask, refine, and trust in this section, we describe our proposed methodology art: ask, refine, and trust in detail. given a query and an initial prediction generated by the llm, art uses a trainable pipeline for refinement as follows: (a) evaluate whether the initial generation requires refinement by asking a series of questions (ask); (b) perform the refinement step based on the evaluation (refine); and finally (c) choose either the refined result or the initial prediction (trust). ## ablation studies importance of asking questions for refinement for this ablation, we trained asker to make only a binary decision of "yes" or "no" to refine, without asking the relevant questions, and found that all versions of the llama models always trusted the predictions and never decided to refine them. llms are often bad at judging their own predictions and often prefer their own predictions @xcite , and our experiments observed a similar phenomenon. however, asking questions leads to a better refinement decision. a qualitative example is presented in figure 3 . ## key findings from the experiments, we observe the following: ## related work strategies that use intermediate computation to solve reasoning tasks such as chain of thought @xcite @xcite @xcite and subquestion decomposition @xcite @xcite have proven to be very effective. however, a lot of times, these llms don't get the correct outputs in the first place, often requiring a refinement. most llm refinement techniques use one of these two strategies @xcite @xcite @xcite or occasionally a combination of the two @xcite . @xcite unified past reasoning with refinement methods under a common umbrella of sampling (given a query, llm generates the initial response), re-sampling (llm refines the initial response), and selection (choose either the refine-ment or rollback to initial response). however, a single llm was used to perform the initial generation, refinement and later selection by using different prompts. this makes the entire process dependent on the capability of a single model. we, on the other hand, propose to train a separate, much smaller capacity model to make refinement decisions and later decide whether or not to trust the refinement over the initial generation and are not limited to prompting-based solutions. asking questions to verify factuality of the llm output has been studied by @xcite . however, this work only deals with hallucinations in the form of directly stated factual inaccuracies. it is important to note that hallucinations can take other forms, including incorrect reasoning steps for reasoning problems considered in our paper. to address this, we train an expert model to verify each reasoning step by asking relevant questions. training a model to rank llm outputs has been studied in the past in various contexts @xcite , including but not limited to open-ended text generation @xcite , mathematical reasoning @xcite , machine translation @xcite , and so on. in our work, we do not study the standard setting of training to rank the quality of generations, but rather to decide if the refinement is inappropriate and needs to be rolled back. choosing one decision over the other has some additional similarities to rejection sampling fine-tuning @xcite , where a model is trained to generate and collect the correct reasoning chains as augmented fine-tuning datasets. on the other hand, we collect both correct and incorrect reasoning chains for ranking the outputs. finally, our work is similar to distilling reasoning skills into smaller models @xcite @xcite . however, instead of teaching smaller models to reason, we train smaller models to ask questions to verify the reasoning and decide whether the reasoning is correct, which differs from asking questions as planning to reason @xcite . ## conclusion in this work, we propose a refinement strategy called art: ask, refine, and trust, which allows smaller models to make refinement decisions for llms and determine whether these refinements are reliable. we empirically demonstrate the effectiveness of our approach on two reasoning tasks, mathematical word problems and question answering. our results show that smaller models, even up to 10x smaller, can outperform larger models in making refinement decisions.