Large Language Models Can Self-Improve (2023)

Abstract: Large Language Models (LLMs) have achieved excellent performances in various tasks. However, fine-tuning an LLM requires extensive supervision. Human, on the other hand, may improve their reasoning abilities by self-thinking without external inputs. In this work, we demonstrate that an LLM is also capable of self-improving with only unlabeled datasets. We use a pre-trained LLM to generate “high-confidence” rationale-augmented answers for unlabeled questions using Chain-of-Though (CoT) prompting and self-consistency, and fine-tune the LLM using those self-generated solutions as target outputs. We show that without any ground truth label, our approach improves the general reasoning ability of a 540B-parameter LLM (74.4% \rightarrow 82.1% on GSM8K, 90.0% \rightarrow 94.4% on OpenBookQA, and 63.4% \rightarrow 67.9% on ANLI-A3) and can also be adapted to extreme low-resource cases where even training questions and CoT prompts are limited. We conduct ablation studies and show that fine-tuning on diverse reasoning paths is critical for self-improvement.

## introduction scaling has enabled large language models (llms) to achieve state-of-the-art performance on a range of natural language processing (nlp) tasks @xcite @xcite . more importantly, new capabilities have emerged from llms as they are scaled to hundreds of billions of parameters @xcite : in-context few-shot learning @xcite makes it possible for an llm to perform well on a task it never trained on with only a handful of examples; chain-of-thought (cot) prompting @xcite demonstrates strong reasoning ability of llms across diverse tasks with or without few-shot examples; self-consistency @xcite further improves the performance via self-evaluating multiple reasoning paths. despite these incredible capabilities of models trained on large text corpus @xcite , fundamentally improving the model performances beyond few-shot baselines still requires finetuning on an extensive amount of high-quality supervised datasets. flan @xcite and t0 @xcite curated tens of benchmark nlp datasets to boost zero-shot task performances on unseen tasks; instructgpt @xcite crowd-sourced many human answers for diverse sets of text instructions to better align their model to human instructions; minerva @xcite parsed the full arxiv database carefully for relevant articles to excel on challenging competitive math and science datasets. the need for large annotated data for supervised llm training still remains a burden for low-resource applications or specific domains where only limited annotations are available. in this paper, we study how an llm capable of in-context few-shot learning and chain-ofthought reasoning, is able to self-improve its reasoning ability without supervised data. we show that using only input sequences (without ground truth output sequences) from multiple nlp task datasets, a pre-trained llm is able to improve performances for both in-domain and out-of-domain tasks. our method is shown in figure 1 : we first sample multiple predictions using few-shot chain-of-thought (cot) @xcite as prompts, filter "high-confidence" predictions using majority voting @xcite , and finally finetune the llm on these high-confidence predictions. the resulting model shows improved reasoning in both greedy and multi-path evaluations. we call the model fine-tuned in this way as language model self-improved (lmsi). note that lmsi depends on in-context few-shot learning and chain-of-thought reasoning abilities which small language models do not necessarily have. we empirically verify lmsi using a pre-trained 540b palm model @xcite , where our method not only significantly improves training task performances (74.4%→82.1% on gsm8k, 90.0%→94.4% on openbookqa, and 63.4%→67.9% on anli-a3), but also enhances out-of-domain (ood) tasks, without relying on supervised ground truth answers. lastly, we explore more extreme cases where training questions and human-curated cots are also limited, and propose self-generating additional input questions and few-shot cot prompts for model self-improving. we hope our simple approaches and strong empirical results could inspire more future work by the community to investigate optimal performances of pretrained llms without additional human supervision. our contributions are summarized as follows: the rest of this paper is organized as follows. section 2 discusses related work. section 3 lays out our method in detail. section 4 shows our setup for experiments. section 5 demonstrates our experiment results with ablation studies. section 6 concludes our work. the chain-of-thought prompts used in our work are included in appendix a. ## related work learning from explanations. augmenting a machine learning model with explanations has been studied in existing literature extensively. for example, in the supervised learning setting, a model can be fine-tuned using human-annotated rationales @xcite @xcite @xcite . a few works have also looked at how explanations can help the models in various settings, e.g., in-context learning @xcite and in distillation @xcite . @xcite treat explanations as process supervision to train a reward model. in this paper, we focus more on the unsupervised learning setting, where we do not assume we have a rationale-augmented training dataset available, since human-annotated rationales can be expensive. ## method the overview of our method is illustrated in fig. 1 : we are given a pre-trained large language model (llm) m and a question-only train- question: stefan goes to a restaurant with his family. they order an appetizer that costs @xmath020 each. if they tip 20% of the total, what is the total amount of money that they spend? multiple path decoding: output 1: the appetizer costs @xmath120 each so in total 4 * @xmath280. this means the total cost is @@xmath14.2 = @xmath510 and the entrees are @xmath620 * 4 = @xmath780 is @xmath816. the answer is @@xmath15. the entrees cost 4 * @xmath1180. the tip is 20% of the total, so it is 20% of the @xmath1218. the total they spent is @@xmath16. the answer is 108. @xcite . we apply multiple path decoding with a sampling temperature @xmath17, r i 2 , . . . , r im } for each question x i in d train , and use majority voting (selfconsistency) to select the most consistent, highest confidence answer @xcite . we then keep all reasoning paths that lead to the most consistent answer, apply mixed formats of prompts and answers for augmentation, and fine-tune the model on these self-generated reasoning-answer data. we consider our approach as making the model self-improve. in the following sections, we detail important designs within our method, along with additional approaches for the model to selfimprove without supervised data. ## experimental setup tasks and datasets. we demonstrate the effectiveness of our method on three types of tasks foot_0 : ## experiments and results we conduct a series of experiments to demonstrate the effectiveness of our proposed self-improving method. first, we apply our method on each individual dataset (task) and report the results. we then merge the generated data from all datasets and train one model to study the generalization ability of the model on unseen datasets as in @xcite . in addition to the results of using generated cot reasoning paths, we show studies on generating input questions and few-shot prompts. we end with ablation studies on model sizes and hyperparameters. ## conclusions we demonstrated that a large language model (llm) is capable of improving its performance on reasoning datasets by training on its own generated labels, given input questions only. experiments using the palm model with 540 billion parameters show that lmsi improves the accuracy scores by 1.1% to 7.7% on six datasets, without training on ground truth labels. furthermore, we show that 1 5 10 15 20 25 it is possible for the llm to self-improve even on its own generated questions and few-shot cot prompts. as part of our future work, we plan to combine large-scale generated data from lmsi and existing supervised data, to further improve the performance of llms.