Democratizing Reasoning Ability: Tailored Learning from Large Language Model (2023)

Abstract: Large language models (LLMs) exhibit impressive emergent abilities in natural language processing, but their democratization is hindered due to huge computation requirements and closed-source nature. Recent research on advancing open-source smaller LMs by distilling knowledge from black-box LLMs has obtained promising results in the instruction-following ability. However, the reasoning ability which is more challenging to foster, is relatively rarely explored. In this paper, we propose a tailored learning approach to distill such reasoning ability to smaller LMs to facilitate the democratization of the exclusive reasoning ability. In contrast to merely employing LLM as a data annotator, we exploit the potential of LLM as a reasoning teacher by building an interactive multi-round learning paradigm. This paradigm enables the student to expose its deficiencies to the black-box teacher who then can provide customized training data in return. Further, to exploit the reasoning potential of the smaller LM, we propose self-reflection learning to motivate the student to learn from self-made mistakes. The learning from self-reflection and LLM are all tailored to the student’s learning status, thanks to the seamless integration with the multi-round learning paradigm. Comprehensive experiments and analysis on mathematical and commonsense reasoning tasks demonstrate the effectiveness of our method. The code will be available at https://github.com/Raibows/Learn-to-Reason.

## introduction large language models (llms) with emergent abilities have achieved remarkable success across a wide range of tasks, deeply changed the landscape of both research and applications in natural language processing @xcite 2021; @xcite . and @xcite argue that emergent abilities particularly in reasoning only exist in llms whose parameters are commonly larger than 100b. nevertheless, a line of research @xcite @xcite has indicated that smaller lms with about 7b parameters after supervised fine-tuning such as vicuna @xcite can be comparable to llms in following human instructions, while still falling short of reasoning. in this paper, we aim to harness the untapped reasoning potential of smaller lms to democratize this important emergent ability. chain-of-thought (cot) prompts lms to generate intermediate reasoning steps (i.e., rationale) to reach the final answer, significantly improving the complex reasoning ability @xcite @xcite . however, it is challenging to prompt smaller lms to generate reasoning steps, since such ability appears to be exclusive to llms @xcite , which indicates the necessity of utilizing data annotated with rationales to cultivate smaller lms' reasoning ability. unfortunately, most existing reasoning datasets lack high-quality rationale annotations, and manual labeling them can be costly. inspired by the success of collecting instruction data from llms (e.g., chatgpt) for instruction tuning smaller lms @xcite @xcite , we propose to leverage the rationales generated by llms to train smaller lms to learn to use cot towards reasoning. recently, teaching smaller lms towards reasoning with the help of llms has gained increasing attention. most of these works @xcite @xcite can be summarized in two main steps: (1) employing llms to generate rationales for annotating the training data. (2) fine-tuning smaller lms on these data to enable reasoning with cot. this approach can be viewed as a distant variant of black-box knowledge distillation @xcite . however, these methods only employ llms to annotate the data for training smaller lms, without leveraging the smaller lms to assist llms in return. as a consequence, the llms are not aware of the weaknesses of the smaller lms, thereby hindering their powerful ability to analyze and provide targeted feedback, which undermines the effectiveness of the reasoning distillation. to this end, we propose a multi-round interactive learning paradigm to exploit the potential of black-box llm as a reasoning teacher. in each round of learning, the student (i.e., smaller lm) first provides its learning status to the teacher llm who then can provide customized rationales as the feedback to the student. the data annotated with these rationales serves as our customized training data. such a paradigm is natural as it is in inline with how we human beings learn from teachers. beyond learning from the teacher, another crucial paradigm for human learning lies in selfreflection on self-made mistakes. in parallel, recent studies @xcite @xcite have also shown that llms can self-improve by reflecting on their own mistakes. therefore, we exploit the reasoning potential of smaller lm by eliciting it to take self-reflection on the mistakes. these mistakes can complement correct rationales collected from the teacher llm to teach the student lm to distinguish bad and good reasoning steps, thereby enhancing its reasoning ability. putting them together, as briefly presented in fig. 1 , we propose a tailored multi-round learning paradigm based on the student's learning status and deficiencies, including learning from llm's customized training data and self-reflection. in summary, our contributions are three-fold: ## related work emergence in llm llms show emergent abilities in a wide range of nlp tasks @xcite @xcite , among which the reasoning ability is the most noteworthy as it requires the model to perform multi-hop reasoning like human beings. smaller lms (< 100b) are often considered to be falling significantly short in reasoning, highlighting the superiority of llms in this aspect @xcite . in this paper, we aim to democratize such emergent reasoning ability to smaller lms. ## method as shown in fig. 2 , we propose a multi-round learning paradigm that motivates the student lm and the teacher llm to learn feedback from each other in an interactive manner. specifically, each round of learning consists of three key steps: (1) the student lm undergoes an "exam" on the training set for collecting mistakes which are the wrong generated rationales. existing works @xcite @xcite merely provide the sample question for the llm to collect annotated rationales, neglecting the importance of the student's feedback. however, the student's feedback is crucial in knowledge distillation @xcite @xcite . (2) therefore, we propose to curate a prompt integrated with the student's wrong rationale to ask the teacher llm to generate customized feedback for the student. (3) in the last step, the student learns to reason via training on the tailored training data collected from the llm, and selfreflection on its self-made mistakes. these steps are iterated to improve the reasoning ability of the student lm until convergence. ## conclusion in this paper, we propose a tailored learning approach to cultivate the reasoning ability of the smaller lm, aiming to democratize the emergent reasoning ability of the llm. first, we propose a multi-round interactive learning paradigm that enables the teacher llm to provide customized training data according to the student's feedback. next, we propose the self-reflection learning to motivate the student to distinguish correct rationales from wrong ones. further, the integration of learning from llm's customized feedback and self-reflection can complement each other within the proposed multi-round learning paradigm. the empirical results from mathematical and commonsense reasoning tasks demonstrate the success of unleashing the reasoning potential of smaller lms. we believe that these findings can benefit the opensource and nlp communities in the era of llm.