R eflect E vo: Improving Meta Introspection of Small LLM s by Learning Self-Reflection (2025)

Abstract: We present a novel pipeline, ReflectEvo, to demonstrate that small language models (SLMs) can enhance meta introspection through reflection learning. This process iteratively generates self-reflection for self-training, fostering a continuous and self-evolving process. Leveraging this pipeline, we construct ReflectEvo-460k, a large-scale, comprehensive, self-generated reflection dataset with broadened instructions and diverse multi-domain tasks. Building upon this dataset, we demonstrate the effectiveness of reflection learning to improve SLMs’ reasoning abilities using SFT and DPO with remarkable performance, substantially boosting Llama-3 from 52.4% to 71.2% and Mistral from 44.4% to 71.1%. It validates that ReflectEvo can rival or even surpass the reasoning capability of the three prominent open-sourced models on BIG-bench without distillation from superior models or fine-grained human annotation. We further conduct a deeper analysis of the high quality of self-generated reflections and their impact on error localization and correction. Our work highlights the potential of continuously enhancing the reasoning performance of SLMs through iterative reflection learning in the long run.

## introduction self-reflection involves meditating on, examining, and evaluating one's behaviors, thoughts, motivations, and desires @xcite @xcite . typically, it inspects the reasoning process leading to the current solution, identifies errors in each step, generates critiques on the causes of the failure, and offers advice for refining the solution to improve the problemsolving performance of large language models (llms) @xcite @xcite . unlike the paradigm of learning directly from the reasoning process and final answer, we refer to it as the process of humanlike meta introspection, which explicitly generates self-reflection, providing textual differentiation and gradients as clear critiques and guidance on what to learn and how to improve based on the current state. recent research has demonstrated that llms can self-improve through their intrinsic capability of self-reflection @xcite @xcite . however, conventional approaches rely closely on llms with large model sizes or supervision distilled from a superior model. in this study, we challenge whether the self-reflection capability of slms can be learned effectively from reflection data. however, it usually requires highcost on fine-grained human annotation to acquire high-quality data for fine-tuning and is impractical to scale. therefore, we are also curious whether it is possible to effectively utilize both high-and low-quality self-generated data from weaker models for reflection learning. with this in mind, we aim to investigate the effectiveness of reflection learning via self-training @xcite @xcite and further validate that the improvement of selfreflection can further strengthen llm's inherent reasoning capabilities across various methods and tasks with more interpretability and generalization. we believe that this paradigm can act as a plug-andplay enhancement for various reasoning methods, which emulates human learning through a slower and deeper thought process that iteratively and ultimately derives self-evolution @xcite @xcite @xcite . therefore, in this paper, we propose a novel pipeline reflectevo (sec. 2), to automatically generate self-reflection data and leverage self-training to enhance llm's reflection capability. to the best of our knowledge, we are the first to demonstrate the potential of meta introspection of llms that are asked to explicitly generate reflection as an intermediate step-by-step process supervision rather than directly mapping an initial solution to a revised solution. building on this pipeline, we curate a largescale, diverse, and unsupervised reflection learning dataset reflectevo-460k containing 460k reflection samples derived from 17 source datasets spanning 10 tasks and domains. we explore the diversity of reflection instructions and bootstrap multiple comparative reflections conditioned on the same question and initial solution. based on the data, we develop reflection learning (sec. 3) to further improve the self-reflection and self-correction capabilities of llms. the evaluation results validate the effectiveness of reflection learning in boosting the reasoning of weak models. it shows significant improvements on llama-3, exceeding the original base model by 10% on average tasks and outperforming its strongest counterpart with model size ×8. we conduct a deeper analysis of the self-generated reflection data including various error types identified from the reflection and observe their gains on corrected answers. in summary, our main contributions are: • novel pipeline for self-reflection generation: we propose reflectevo for automatic selfreflection generation and curation, which is the first to explore meta introspection of slms. ## the reflectevo generation scheme in this section, we introduce the end-to-end pipeline reflectevo for collecting self-generated reflections as the training data for sec. 3, leveraging the inherent ability of slms (see fig. 1 ). ## reflection learning on self-generated data in this section, we further investigate the effectiveness of reflection learning on the reflector r by adopting self-training on reflectevo-460k using supervised fine-tuning (sft) @xcite and direct preference optimization (dpo) @xcite . ## conclusion we propose reflectevo to enhance slms through reflection learning by iteratively generating selftraining data, which achieves substantial performance improvements, even surpassing much larger models highlighting its generalization across various models and tasks for future research.