Small Language Models Need Strong Verifiers to Self-Correct Reasoning (2024)

Abstract: Self-correction has emerged as a promising solution to boost the reasoning performance of large language models (LLMs), where LLMs refine their solutions using self-generated critiques that pinpoint the errors. This work explores whether small ( \leq 13 B) language models (LMs) have the ability of self-correction on reasoning tasks with minimal inputs from stronger LMs. We propose a novel pipeline that prompts smaller LMs to collect self-correction data that supports the training of self-refinement abilities. First, we leverage correct solutions to guide the model in critiquing their incorrect responses. Second, the generated critiques, after filtering, are used for supervised fine-tuning of the self-correcting reasoner through solution refinement. Our experimental results show improved self-correction abilities of two models on five datasets spanning math and commonsense reasoning, with notable performance gains when paired with a strong GPT-4-based verifier, though limitations are identified when using a weak self-verifier for determining when to correct.

## introduction recent research shows that large language models (llms) @xcite can self-correct their responses to meet diverse user requirements, ranging from diminishing harmful content to including specific keywords and to debugging code @xcite . self-correction is typically accomplished by first generating a critique that identifies the shortcomings of the initial response, followed by revising it according to the self-critique-a process that can be iterated. self-correction has emerged as an intriguing paradigm for rectifying the flaws in llm's outputs @xcite . however, models that are effective at self-correction are of very large sizes, and many of them are proprietary and accessible only via apis. in this work, we focus on the self-correction abilities of small, open-source language models (lms). 2 previous studies have shown that these smaller models can learn selfcorrection in reasoning through distillation from stronger lms @xcite @xcite . yet this poses security risks for high-stakes domains and hinders the scientific understanding of enhancing lms' ability to correct errors. we thus ask the question: to which degree do small lms require guidance from strong lms to learn self-correction for reasoning? we study this question by leveraging the small model itself to generate supervised training data to enhance its self-correction ability, instead of resorting to stronger lms. to this end, we draw inspiration from the rejection sampling fine-tuning (rft) @xcite ) method where llm's reasoning skills are bootstrapped via diverse chain-of-thought sampling and supervised fine-tuning on the correct reasoning chains. we propose score-an approach to bootstrap small lms' self-correction ability in reasoning tasks. concretely, we devise a pipeline for accumulating high-quality critique-correction data from small lms, which are used for supervised fine-tuning of self-correcting reasoners. first, we leverage correct solutions as hints for the base lms to critique incorrect answers. by reverseengineering from the correct answer, the models generate more effective critiques. second, we filter these critiques for correctness, well-formedness, and clarity using simple rule-based and prompting methods. finally, we fine-tune the same lms to be-come self-refining models using this curated data. by avoiding the use of supervision from stronger lms, we ensure that our method enables a small lm to bootstrap its self-correction capabilities. we evaluate our score fine-tuned refiner under both extrinsic and intrinsic self-correction settings @xcite . the primary difference between these two settings is whether the refiner is allowed to use external signals to determine when to self-correct (i.e., refine the initial solution only when it is believed to be incorrect). identifying when to self-correct involves verifying the solutions' correctness, which is still challenging for current state-of-the-art llms without proper external feedback @xcite . we adopt a simple baseline for the self-verification problem following @xcite . specifically, we fine-tune the same lms to become verifiers with labels based solely on the correctness of the final answer, conditioning on the question and a candidate solution. as for the extrinsic setting, we simulate strong verifiers with gpt-4 and oracle labels, to show the effectiveness of small lms as self-correcting reasoners. we test the score method with the llama-2-13b-chat @xcite and gemma-7b-it @xcite models on five datasets spanning math and commonsense reasoning. we find that our model with score finetuning outperforms the original model by an average of 14.6% when using a gpt-4-based verifier. nevertheless, the model struggles with selfcorrection when subjected to a weak self-verifier fine-tuned on self-generated solutions. our main contributions are summarized below: decoupling (self-)verify and self-refine brings two major advantages over a one-modeldoes-all design. first, we can freely parameterize each module-for example, by using a fine-tuned and a few-shot prompted model. this allows us to carefully examine the impact of strong vs. weak verifiers on the refiners' performance. on the contrary, previous work on self-correction with small lms @xcite @xcite conflates self-verify and self-refine, creating a barrier to fully understanding the distinct capacities of these models in each skill. second, it reduces the difficulty of training each module, since the model only needs to specialize in one kind of ability, which is either verification or refinement. ## the score method our approach is inspired by rejection sampling finetuning (rft): sampling diverse solutions for each question and fine-tune llms on the self-generated solutions that lead to the correct final answer @xcite @xcite . we want to bootstrap the small lm's inherent ability to generate critiques for reasoning steps. we design an end-to-end pipeline to collect selfcorrection data generated by small lms at scale, without any distillation from stronger lms. the self-generated critiques, after filtering, are used to fine-tune the smaller lm itself to bootstrap its ability to self-correct. concretely, the score pipeline consists of two stages shown in figure 1 and described below. stage 1: generating and filtering critiques. we sample n solutions for each question in the training set by few-shot chain-of-thought prompting a base lm (step a). to enable the base lm to reflect on its incorrect solutions, we include a correct solution for the same question (if exists) in the prompt as a hint (step b). we then filter the self-generated critiques based on their correctness and clarity (step c). this process is detailed in section 3.1. stage 2: supervised fine-tuning of the refiner. the filtered critiques obtained from stage 1 are used in the next stage for fine-tuning the small lm itself. we train a refiner that generates critiques and corrections conditioned on questions and initial solutions (step d). we exclude the hints during finetuning. more details are given in section 3.2. ## experimental setup self-correction data collection. as stated in section 3.1, we sample @xmath0, label their correctness, and formulate incorrect-correct solution pairs for critique generation. we separately collect data for each base lm and task. this results in the number of raw critiques shown in verifiers. we experiment with verifiers of different levels of capabilities to gauge their impacts on self-correction performance. first, we adopt a simple baseline for training a self-verifier following @xcite . the self-verifier is a model with the same architecture as the base lm, conditioned on the question and a candidate solution to judge the probability that the solution is correct/incorrect. specifically, we label the solutions sampled from the base lm as incorrect (0) or correct (1) solely based on their final answers and finetune the verifier with a binary classification head on the last-layer representation of the last token in the input sequence "question: {q} \n solution: {s} \n is this solution correct?". since the fine-tuning data is imbalanced between correct and incorrect solution, we re-weight the loss for each class with regarding to its proportion. during inference, the verifier model outputs a probability of the initial solution being incorrect, and the refinement is introduced only when the confidence of the verifier's predictions exceeds a certain threshold, which is automatically chosen in a way that maximizes the accuracy on the dev set and then fixed during test-time predictions. since fine-tuned small lms are still weak verifiers that bottleneck the performance of self-correction, we also experiment with a second option by using gpt-4 as an off-the-shelf strong verifier to demonstrate the potential of our fine-tuned refiner. we do so by few-shot prompting gpt-4 to predict the correctness of the initial solution by smaller lms, with the verifying prompt shown in appendix a. finally, we directly use the gold labels of the initial solutions as signals to determine when to self-refine. this oracle verifier setting provides an upper bound for the refiners' performance. benchmarks and base models. to demonstrate the effectiveness of the score method, we conduct experiments on two popular datasets: gsm8k @xcite for mathematical reasoning and commonsenseqa @xcite for commonsense reasoning. we also conduct transferability studies and evaluate the generalization performance of our fine-tuned refiner on math @xcite for mathematical reasoning, qasc @xcite and rid-dlesense (lin et al., 2021) for commonsense reasoning. specifically, for mathematical reasoning, we train self-verifiers and score refiners using only gsm8k training data and evaluate them on the whole gsm8k test set and a subset of math test set, foot_3 following the practice of @xcite . similarly, for commonsense reasoning, we fine-tune our models using only commonsenseqa training data and evaluate them on the whole dev foot_4 set of commonsenseqa, qasc, and riddlesense. since questions in commonsenseqa, qasc, and riddlesense have a multiple-choice format, we also include a random refiner baseline that randomly picks a choice different from the initial answer, following the practice of @xcite . we explore two open-source smaller language models, namely llama-2-13b-chat @xcite and gemma-7b-it @xcite figure 2 : performance of score models using llama-2-13b-chat and gemma-7b-it as base lms. we show the relationship between f1 scores of the verifiers and final answer accuracies. we also report the difference between the final accuracy after refinement and the corresponding initial answer accuracy (init. accu.) by few-shot prompting for each base lm. for the concrete numbers of these metrics, please see as the base lms to generate self-correction data and evaluate their self-correction abilities. in appendix c, we also investigate whether our selfcorrection fine-tuning can be built on top of other fine-tuning methods (e.g., rejection-sampling finetuning) to further boost the reasoning performance. fine-tuning and evaluation. we fine-tune the base lm using the llama-factory library @xcite with lora @xcite . we set the low-rank dimension as 32, the learning rate as 2e-5, training epochs as 3, batch size as 32. during inference, we set the temperature as 0 (i.e., greedy decoding) and the max sample length as 2,048. all our experiments can be conducted on 4×a40 gpu with 48gb of memory. ## results in this section, we first present the experimental findings of score method on various models and datasets (section 5.1). to better understand the performance changes after self-correction, we then analyze the behaviors of verifiers and refiners (section 5.2) and further highlight several key design decisions of our pipeline with ablation stud-ies (section 5.3). lastly, we show the impact of score fine-tuning data size on self-correction performance (section 5.4). ## conclusion in this study, we investigate how to leverage minimal signals from strong lms to teach small lms to self-correct their reasoning. we propose the score method to collect self-correction finetuning data solely from small lms. we find that score-fine-tuned small lms become better refiner models without relying on knowledge distillation from stronger lms, yet they still need strong verifiers to be successful at self-correcting their reasoning errors. our results highlight that the self-verification limitation of lms currently poses an obstacle to the advancement of intrinsic selfcorrection in reasoning and thus warrants future research.