Mitigating Tail Narrowing in LLM Self-Improvement via Socratic-Guided Sampling (2025)

Abstract: Self-improvement methods enable large language models (LLMs) to generate solutions themselves and iteratively train on filtered, high-quality rationales. This process proves effective and reduces the reliance on human supervision in LLMs’ reasoning, but the performance soon plateaus. We delve into the process and find that models tend to over-sample on easy queries and under-sample on queries they have yet to master. As iterations proceed, this imbalance in sampling is exacerbated, leading to a long-tail distribution where solutions to difficult queries almost diminish. This phenomenon limits the performance gain of self-improving models. A straightforward solution is brute-force sampling to balance the distribution, which significantly raises computational costs. In this paper, we introduce Guided Self-Improvement (GSI), a strategy aimed at improving the efficiency of sampling challenging heavy-tailed data. It leverages Socratic-style guidance signals to help LLM reasoning with complex queries, reducing the exploration effort and minimizing computational overhead. Experiments on four models across diverse mathematical tasks show that GSI strikes a balance between performance and efficiency, while also being effective on held-out tasks.

## introduction large language models (llms) have demonstrated impressive ability in performing complex reasoning tasks @xcite @xcite . while fine-tuning models on curated data can further boost performance, it relies heavily on human supervision, limiting scalability and generalization @xcite . to address this, the "self-improvement" paradigm emerges, where rebalanced distribution iter 1 iter 2 iter 3 progressive shift over iterations rebalance after guided sampling tail narrowing tail recover long-tail distribution models generate multiple reasoning paths, filter out incorrect responses and fine-tune themselves on their own outputs without human intervention @xcite @xcite @xcite . despite the benefits of self-improvement, its performance typically reaches a ceiling after a few iterations @xcite . we perform preliminary experiments ( § 4) and find that in reasoning tasks, the most significant gains from self-improvement occur in the first iteration, while subsequent iterations encounter performance bottlenecks or even degradation (figure 2 ). similar performance bottlenecks in synthetic data have also been observed in text generation @xcite and image synthesis @xcite . further, we delve into the self-improvement process and conduct an in-depth analysis (figure 3 ) to investigate the underlying causes behind the per-formance bottlenecks. on the one hand, complex problems with lengthy reasoning chains tend to amplify hallucinations, making it difficult for models to explore the vast search space and sample correct rationales @xcite @xcite . consequently, the models tend to over-sample easy queries and under-sample queries they have yet to master @xcite . on the other hand, as iterations proceed, this imbalance in sampling is exacerbated, leading to a long-tail distribution where solutions to difficult queries almost disappear (figure 1 ). this situation is also referred to as tail narrowing or tail cutting in previous studies @xcite . as a result, the model's self-improvement is limited, since difficult examples are also crucial for further training @xcite . to address this imbalance, a common approach is to allocate more sampling trials to undersampled, challenging queries @xcite , but this can be considerably more costly. in this paper, we propose guided self-improvement (gsi), an efficient method that leverages interactive guidance signals inspired by the socratic method @xcite . specifically, our approach introduces an additional resampling phase, termed distribution re-balancing for difficult queries, which is applied after the generation step in the self-improvement process. during this phase, we incorporate targeted guidance signals derived from oracle answers, stepwise rationales, and strong teacher supervision. these signals help to narrow the sampling space, reduce sampling difficulty, and minimize hallucinations during reasoning @xcite . as a result, gsi enables more effective exploration, increases solution coverage for challenging queries @xcite , and mitigates the issue of tail narrowing during the sampling process. we perform experiments across four models and six mathematical reasoning tasks, including arithmetic reasoning, abstract algebra, and formal logic. the results demonstrate that gsi mitigates the performance bottlenecks of self-improvement while maintaining computational efficiency. further analysis shows that this method leads to a more balanced solution distribution and improved model generalization across multiple reasoning tasks. in addition to natural language reasoning, our method has also been proven effective in program-based reasoning @xcite . our contributions are summarized as follows: ## performance bottleneck and tail narrowing in self-improvement despite extensive research into self-improvement methods @xcite , the performance dynamics across successive training iterations remain underexplored. to this end, we conduct experiments across four backbone models to investigate the effects of sampling and tail data during iterative training. tail data refers to samples for which the model rarely generates correct solutions during sampling. these samples lie in the "long tail" of data distributions. performance trends. to uncover the relationship between performance trends with sampling, we vary the number of sampling k. figure 2 reveals the following: (1) lower sampling times (e.g., @xmath0, leading to negative gains. llama2-7b, mistral-7b, and llama3-8b underperform compared to sft on the original dataset (iteration 1). this decline stems from the models' weaker reasoning abilities, which limit their coverage of challenging queries. consequently, the models tackle only basic problems, resulting in a degradation or collapse of reasoning ability @xcite . (2) with larger sampling numbers k, performance improves across multiple iterations. the most notable gains occur during the first iteration, as shown by the green and orange solid lines in figure 2. however, after the third iteration, progress halts, eventually reaching a performance plateau. impact of tail data. as previously discussed, merely scaling the number of sampling eventually encounters performance bottlenecks. however, challenging low-probability samples, often referred to as tail data, are often considered more crucial for improving model performance @xcite @xcite . to investigate the impact of tail data, we conduct additional experiments targeting these difficult examples. for queries that do not yield a correct response after k sampling attempts, we supplement them with a golden rationale, ensuring that every query has at least one practically correct response and effectively rebalancing the long-tail distribution. we use hollow markers to represent the performance with tail data. as shown in figure 2 , across different numbers of sampling and model variants, the performance represented by the dashed line exceeds that of the solid line representing vanilla self-improve. the results demonstrate that rebalanced data helps mitigate performance degradation under a small number of sampling attempts (e.g., @xmath1. moreover, as k increases, a larger number of sampling alleviates performance plateaus and boosts model efficacy to some extent. this study highlights the value of solutions to challenging queries in overcoming the limits of finite and progressively biased sampling. ## guided self-improvement to mitigate the observed tail-narrowing phenomenon, a straightforward solution is to increase the sampling trials for tail data. however, directly tackling these difficult queries from scratch often results in low success rates and higher costs due to repeated failed attempts. drawing from socratic-style education @xcite , we introduce guidance-based exploration techniques to improve sampling efficiency, such as learning from demonstrations @xcite . we provide the model with tailored assistance in structured contexts, enabling it to address difficult queries more effectively. the following paragraphs outline four guiding strategies we propose. answer-driven. this strategy incorporates the ground-truth answer y i along with the input query x i as context to guide the generation process. it helps the model better align with the expected solution, particularly for challenging queries @xcite . formally, at each iteration t, instead of inputting only x i into the model m t-1 , we extend the prompt by appending y i as a hint: this approach helps the model focus on the reasoning process behind the answer, reducing the overall difficulty of the task. rationale-driven. in this strategy, we further extend the input by introducing a rationale r i , which helps the model derive the correct reasoning process. unlike the answer-driven method, providing a rationale offers a more detailed reference for the model to follow, narrowing the exploration space of reasoning paths @xcite . formally, at iteration t, the input to the model m t-1 is augmented as follows: this approach enables the model to handle queries it has yet to master and ensures a higher coverage of solved problems. importantly, it alleviates the hallucinations when the model tries to provide reasoning paths for problems it doesn't fully understand @xcite , improving the reliability of generated data. interactive sampling. inspired by previous work in the area of interactive rl @xcite , we introduce feedback from a stronger model m s after the model m t-1 fails. instead of providing hints along with the query, this dynamic process ensures that the model can explore its own solution before receiving external guidance. formally, after the model m t-1 generates an incorrect answer, we re-sample by giving it both its prior incorrect output and the feedback f i from m s as additional context: this feedback includes an analysis and correction of the model's errors, reducing its reliance on the correct answer. through this interactive process, we balance exploration and correction, enabling the model to learn from its mistakes without overly restricting its reasoning path. ## conclusion in this work, we delve into the performance bottlenecks in the self-improvement process of llms, identifying the issue of tail narrowing caused by progressively imbalanced data sampling. to mitigate this, we propose guided self-improvement (gsi), a new method that incorporates a distribution re-balancing phase and socratic-style guidance to enhance solution coverage for challenging queries. experimental results across multiple models and mathematical reasoning tasks demonstrate the effectiveness of this method in improving reasoning performance while maintaining computational efficiency. we believe gsi offers a promising direction for enhancing the scalability and generalization of self-improving models in the future.