Can Large Language Models Invent Algorithms to Improve Themselves?: Algorithm Discovery for Recursive Self-Improvement through Reinforcement Learning (2025)

Abstract: Large Language Models (LLMs) have shown remarkable performance improvements and are rapidly gaining adoption in industry. However, the methods for improving LLMs are still designed by humans, which restricts the invention of new model-improving algorithms to human expertise and imagination. To address this, we propose the Self-Developing framework, which enables LLMs to autonomously generate and learn model-improvement algorithms. In this framework, the seed model generates, applies, and learns model-improving algorithms, continuously improving both the seed model and the algorithms themselves. Among model-improving strategies, we focus on model merging algorithms. In mathematical reasoning tasks, Self-Developing discovers novel merging strategies and outperforms human-designed methods. On GSM8k, the discovered algorithms improve the seed model by 6% and surpass human-designed methods by 4.3%. Moreover, they exhibit strong transferability, achieving a 7.4% performance gain on out-of-domain models. These results suggest that LLMs can autonomously develop effective model-improvement techniques beyond human intuition.

## introduction the advancement of large language models (llms) is having a significant impact on society @xcite @xcite . llms have been continuously improved by human experts' knowledge and experience, realizing advanced capabilities such as mathematical reasoning or code generation @xcite . building on these advanced capabilities, researchers are increasingly focusing on developing self-improving methods for llms to autonomously improve their performance without human intervention, with the goal of automating the llm development process it-self. research on self-improvement of llms includes approaches such as fine-tuning using selfgenerated data @xcite @xcite @xcite , self-play @xcite , and planning using feedback from environment @xcite . however, a fundamental limitation is that the exploration of the strategies to improve llms (model-improving algorithms) remains constrained by human knowledge and imagination. regarding this, as an extreme form of selfimprovement, one can ask a question: could we empower llms to autonomously discover and develop algorithms to improve themselves? this approach could potentially uncover novel, high-performance algorithms beyond human knowledge and imagination, as exemplified by alphago's 'move 37' @xcite , thus expanding the frontiers of ai capabilities beyond the limitations of humandesigned algorithms. in this paper, we propose self-developing, an llm-based framework that invents modelimproving algorithms without the use of human expertise or feedback from external stronger models. while our framework can be applied to various types of model improvement algorithms, in this study we specifically focus on model merging algorithms @xcite that create a single improved model from multiple input models, as a concrete instance of model-improving algorithms. our approach iteratively refines two components: the seed model, which is improved using llmgenerated algorithms, and the algorithm factory, which generates these algorithms. in experiments on mathematical reasoning, self-developing invents new model-improving algorithms, which can be considered novel model merging strategies. on the gsm8k task, llmdiscovered algorithms surpass human-designed methods such as task arithmetic @xcite , enhancing the seed model by 6% and outperforming human-designed algorithms by 4.3%. furthermore, the discovered algorithms demonstrate strong transferability to out-of-domain models not used in algorithm generation, surpassing the performance of task arithmetic optimized for these models by 7.4%. notably, our experiments reveal that the iterative refinement of both the seed model and the algorithm factory plays a crucial role in generating increasingly effective algorithms. ## main results in this section, we demonstrate that self-developing can generate algorithms that improve the model itself, and furthermore, these automatically discovered algorithms overperforms the conventional human-designed ones. ## transferability of algorithms we analyze the effectiveness of the algorithms discovered by the algorithm factory on out-of-domain models that were not used in the algorithm generation process. experimental setup to investigate the transferability of llm-discovered algorithms, we maintained the original seed model m 0 while introducing a new set of candidate models (c new 1 , c new 2 , c new 3 ) and applied the discovered algorithms to these new models. from models with capabilities similar to the original merge candidates, we selected wizardmath-7b-v1.1 foot_5 @xcite , biomistral-7b foot_6 (labrak et al., 2024), and starling-lm-7b-alpha foot_7 @xcite as new candidate models for merging. although these models differ from the candidate models used in algorithm generation, they are finetuned based on mistral-7b @xcite and can therefore be merged with the seed model. we apply the top 15 algorithms discovered by the algorithm factory (based on their performance on the development set with the original candidate models) to these new models. for comparison, we chose task arithmetic, which showed the secondbest performance after our proposed method in §3, and apply its top 15 mixing ratios (based on development set with the original candidate models) to these new models. ## related work self-improving the concept of self-improving artificial intelligence was proposed by @xcite and @xcite , and later formalized by @xcite . with the rapid development of llms, the community has shifted towards practical implementations of self-improvement @xcite . many recent self-improvement approaches primarily focus on the generation of finetuning with self-generated training data @xcite @xcite @xcite . their methods do not generate or learn the improvement strategies themselves. additionally, agents that modify outputs based on feedback from the environment have been proposed @xcite ., models gsm8k (%) math (%) base model (seed model) openchat-3.5-1210 (m 0 ) 70.1 0.5 new models for merging wizardmath-7b-v1.1 (c new 1 ) 57.4 0.03 starling-lm-7b-alpha (c new 2 ) 75.5 0.1 biomistral-7b (c new 3 ) 0.0 0.5 task arithmetic (top 3 performances) 1st 71.4 1.2 2nd 71.3 0.6 3rd 70.6 0.4 llm-designed algorithms (top 3 performances) 1st 78.8 2.5 2nd 78.8 2.4 3rd 78.8 2.0 algorithm generation using llms code generation by llms @xcite has been proposed for various applications, such as solving reasoning problems (chen et al., 2023) and generating agent actions @xcite . focusing on llms' code generation capabilities, several approaches have been suggested where llms generate and execute code to enhance their own abilities. for example, @xcite propose a method where llms themselves are used to discover loss functions for preference optimization. @xcite propose a method to improve a code that makes structured calls to a llm. these methods generate improvement algorithms using llms. unlike their approaches, we not only generate model-improvement algorithms but also enhance the llm that generates these algorithms. another major difference is that we do not rely on external llms, like @xcite , other than the seed model, for algorithm generation. our work builds upon these foundations but uniquely focuses on the autonomous improvement of both the algorithms and the model generating them, without relying on external models, thus pushing the boundaries of self-improving ai systems.