Weak-to-Strong Reasoning (2024)

Abstract: When large language models (LLMs) surpass human capabilities, supervising them effectively becomes difficult. Weak-to-strong learning , where a less capable model enhances a stronger one, proves valuable in this context. Yet, the efficacy of this paradigm for complex reasoning tasks is still unexplored. In this paper, we introduce a progressive weak-to-strong reasoning framework that enables the strong model to autonomously refine its training data, maximizing the use of weak signals and unlocking its latent abilities. This framework begins with supervised fine-tuning on a selective small but high-quality dataset, followed by preference optimization on contrastive samples identified by the strong model itself. Experiments on the GSM8K and MATH datasets verify that our method can effectively improve the reasoning capabilities of Llama2-70b using three separate weak models. This work paves the way for a more scalable and sophisticated strategy to enhance AI reasoning powers. All relevant code and resources are available in https://github.com/GAIR-NLP/weak-to-strong-reasoning .

## introduction as the pursuit of artificial general intelligence (agi) advances, the creation of superintelligent systems-models that exceed human cognitive capabilities-remains a key ambition within the field @xcite @xcite . this quest introduces a host of challenges, especially concerning the supervision and learning paradigms for these advanced ai models. conventional supervision methods, which typically depend on human oversight @xcite @xcite or guidance (i.e., distilled knowledge) from more advanced models @xcite @xcite i think you're wrong. i won't let myself make such a mistake. ## methodology in this section, we propose a weak-to-strong training method designed to maximize the use of weak data and to elicit the strong model's innate talent. first, we identify potentially positive samples in the absence of ground truth and external signals. during stage i, we exclusively utilize this subset of data for supervised fine-tuning. then once the strong model has achieved a certain level of reasoning proficiency, we employ the full weak data, particularly the potentially negative samples in stage ii via preference learning-based approaches like dpo @xcite , encouraging the strong model to learn from mistakes made by the weaker model. the whole framework is depicted in fig. 2 . ## conclusion in this paper, we explore the efficacy of weak-tostrong framework in complex reasoning tasks. we introduce a new method that elicits strong capabilities using weak supervision, without relying on annotations from humans or more advanced models. this method focuses on the strong model's ability to autonomously refine its training data, even if it has not learned the task before. by iteratively expanding its learning scope, the strong model continuously broadens its reasoning skills. this selfdirected data curation is crucial for scaling up the enhancement of ai reasoning capabilities, making the model more independent and effective in its developmental trajectory. through this work, we seek to illuminate new pathways for ai development, emphasizing the critical role of innovative model supervision in advancing agi and beyond.