The Self-Improvement Paradox: Can Language Models Bootstrap Reasoning Capabilities without External Scaffolding? (2025)

Abstract: Self-improving large language models (LLMs) – i.e., to improve the performance of an LLM by fine-tuning it with synthetic data generated by itself – is a promising way to advance the capabilities of LLMs while avoiding extensive supervision. Existing approaches to self-improvement often rely on external supervision signals in the form of seed data and/or assistance from third-party models. This paper presents Crescent – a simple yet effective framework for generating high-quality synthetic question-answer data in a fully autonomous manner. Crescent first elicits the LLM to generate raw questions via a bait prompt, then diversifies these questions leveraging a rejection sampling-based self-deduplication, and finally feeds the questions to the LLM and collects the corresponding answers by means of majority voting. We show that Crescent sheds light on the potential of true self-improvement with zero external supervision signals for math reasoning; in particular, Crescent-generated question-answer pairs suffice to (i) improve the reasoning capabilities of an LLM while preserving its general performance (especially in the 0-shot setting); and (ii) distill LLM knowledge to weaker models more effectively than existing methods based on seed-dataset augmentation.

## introduction in recent years, large language models (llms) such as gpt-4o @xcite , gemini @xcite , llama @xcite have demonstrated remarkable capabilities, revolutionizing natural language processing and various other tasks. the success of these models can be attributed to the scaling laws @xcite , which dictate the relationship between model parameters, computational resources, and training data size. for instance, the prominent performance of llama-3.1 with 405b parameters @xcite roots in, amongst others, the massive, high-quality datasets for preand post-training. however, as models continue to scale, the available real-world (public) data quickly becomes exhausted; meanwhile, manually crafting high-quality data is time-and labor-intensive. thus, data volume has become a key limiting factor for the effective scaling of new-generation models. in response to this challenge, synthetic data generation and data augmentation have emerged as key methods to further improve the performance of llms while avoiding extensive supervision. these methods leverage the ability of llms to mirror real-world distributions and generate high-quality, pseudo-realistic data @xcite . following this line of research, the problem of selfimprovement naturally arises: can we improve the performance of an llm by fine-tuning it with synthetic data generated by itself? this problem has triggered a recent surge of research results @xcite . these methods, however, rely heavily on external seed datasets for augmentation (e.g., @xcite ) and/or stronger third-party models as classifiers or reward agents (e.g., @xcite ); see fig. 1 . such dependency on external supervision signals limits their ability to achieve true self-improvement. orthogonally, the recently proposed method magpie @xcite suffices to generate high-quality dialogue datasets (i.e., both responses and instructions) entirely through the model itself. nonetheless, the generated data is highly randomized and primarily dedicated to the alignment of base llms. such data may improve instruction-following abilities but will degrade fundamental capabilities like math and reasoning; see @xcite . recent discussions @xcite have explicitly questioned whether genuine self-improvement is feasible, suggesting that when trained solely on self-generated data, llms may fail. can llms achieve true self-improvement? remains an open question in the literature. this paper aims to provide the infrastructure to explore the self-improvement problem of llms: we present crescent -a fully autonomous framework for generating high-quality synthetic question-answer (qa) data that suffice to improve the reasoning capabilities of an llm while preserving its general performance. crescent adopts a simple yet effective workflow: (i) it uses a bait prompt to guide the model to generate raw questions in a specific domain, such as math word problems; (ii) it applies a self-deduplication mechanism based on rejection sampling @xcite to refine and diversify the question pool; and (iii) for each question, it performs majority voting @xcite to identify the most confident answer from the model (thus enhancing the consensus). the so-obtained qa pairs are then used to fine-tune the original llm via, e.g., supervised fine-tuning (sft), to improve its math-reasoning capability. experiments with crescent demonstrate evident self-improvement of llms consistently for three benchmarks on mathematical word problems in both 0-shot and 5-shot settings, without trading off their general capabilities. the improvement is especially prominent for the 0-shot case, thus improving the generalization ability of the model to real-world tasks. ablation studies further demonstrate the superiority of crescent over magpie @xcite in the generation of themed data: the latter tends to generate math-related dialogues, e.g., "could you tell me what type of mathematics you like?" -rather than proper mathematical problems. moreover, our experiments show that crescent can serve as a highly effective and effi-cient distillation method, surpassing the baselines using external data and stronger models. contributions. our main contributions include: ## the crescent approach this section presents crescent -a framework for controlled qa self-generation via diversification and consensus enhancement. crescent suffices to generate high-quality domain-specific qa pairs leveraging only the model itself, with zero external data, nor assistance from third-party models. fig. 2 sketches the general workflow of cres-cent, which consists of three main steps: (i) bait prompting: we use a bait prompt to instruct the original, aligned llm to produce a set of raw questions within a specific domain; (ii) diversification: the raw questions may be semantically analogous to each other (as per some similarity metric), and thus we employ a rejection sampling mechanism to attain a diverse pool of representative questions through self-deduplication; (iii) consensus enhancement: we treat the generated questions as query prompts and feed them back to the llm. then, by majority vote, we obtain the final set of synthetic qa pairs. we show that such qa pairs are of high quality in the sense that they suffice to improve the domain-specific capabilities (mathematical reasoning, in our case) by fine-tuning the original llm with these qa pairs while preserving its general capabilities. can you create some math word problems for me? bait prompting question pool diversification qa pairs consensus enhancement finetuning bait prompt {question} is very similar to {question}, please modify the latter to make it different. question: tom wants to buy some apples. if each apple costs 3 bucks and he has 20 dollars, how many apples can he buy at most? … tom can buy 6 apples. … 6 apples most. … 5 apples in total. below, we first present the technical details of steps (i) to (iii) and then provide the rationale behind the self-improvement achieved by these steps. ## conclusion we presented crescent as a simple yet effective framework -leveraging techniques of bait prompting, diversification, and consensus enhancement -for exploring the self-improvement problem of llms. we show that crescent suffices to improve the mathematical reasoning capabilities of an llm with zero supervision signals while preserving its general performance. moreover, it facilitates more effective and efficient llm knowledge distillation than existing approaches based on seeddataset augmentation.