Making Large Language Models Better Reasoners with Orchestrated Streaming Experiences (2024)

Abstract: Large language models (LLMs) can perform complex reasoning by generating intermediate reasoning steps using chain-of-thought prompting under zero-shot or few-shot settings. However, zero-shot prompting always encounters low performance, and the superior performance of few-shot prompting hinges on the manual-crafting of task-specific demonstrations one by one. In this paper, we present **RoSE** (**R**easoning with **O**rchestrated **S**treaming **E**xperiences), a general framework for solving reasoning tasks that can self-improve as it answers various reasoning questions. To enable RoSE, we describe an architecture that extends an LLM to store all answered reasoning questions and their reasoning steps in a streaming experience pool and orchestrate helpful questions from the pool to assist itself in answering new questions. To set up a question-aware orchestration mechanism, RoSE first calculates the similarity of each question in the pool with the question to be answered. Since the solution to each question in the experience pool is not always correct, RoSE will sort the questions according to their similarity with the question to be answered, and then uniformly divide them into multiple buckets. It finally extracts one question from each bucket to make the extracted questions more diverse. To make the extracted questions help RoSE answer new questions as much as possible, we introduce two other attributes of uncertainty and complexity for each question. RoSE will preferentially select the questions with low uncertainty and high complexity from each bucket. We evaluate the versatility of RoSE in various complex reasoning tasks and LLMs, such as arithmetic and commonsense reasoning, and find that it can achieve excellent performance without any labeled data and pre-set unlabeled data.

## introduction large language models (llms) @xcite @xcite @xcite @xcite @xcite have an emerged ability on performing various complex reasoning tasks. recently, the chain-of-thought (cot) prompting technique @xcite was proposed to have llms generate intermediate reasoning paths before generating the final answers. the prompting makes llms think deeply before giving an answer and further enhances the reasoning power of llms. besides, the zero-shot cot prompt @xcite ) "let's think step by step" also enhances the reasoning power of llms without any manualcrafting demonstrations. after the cot prompting was proposed, more studies tried to manually design better prompts @xcite @xcite to further improve the performance of llms in reasoning. however, no matter how the prompts change, the goal is to have llms generate intermediate reasoning steps. recent works such as react @xcite , reflexion @xcite , remem-berer @xcite , and expel @xcite were presented and have demonstrated the feasibility of autonomous agents that are built on top of an llm core. these methods use llms to generate reasoning paths and "actions". these "actions" can be used in api calls and executed in an environment. besides, some golden feedback will be presented to llms during the reasoning process @xcite or labeled samples are needed to collect correct or false experiences @xcite . overall, these methods still require humans to carefully design some demonstrations and need golden feedback, labeled samples, or external tools to improve the reasoning performance of llms. we investigate how to improve the reasoning performance of llms in a more challenging streaming setting without any labeled data, pre-set unlabeled data, feedback signals, and other external help. inspired by the observation that humans constantly do various exercises to construct a large experience pool in their minds and use the pool to help them quickly and better answer questions in ex-ams, we present rose, a general framework for solving reasoning tasks with only streaming experiences. the greatest characteristic of rose is that it can self-improve by constantly collecting and orchestrating streaming experiences like humans. we build an experience pool for rose to store the answered questions and corresponding reasoning paths. we expect these questions can assist llms in answering new questions, and construct a novel experience orchestration mechanism to extract helpful questions from the pool for each new reasoning question. to achieve this, we consider three attributes for each question in the pool when orchestrating. first, the solution to each question may be incorrect. if we randomly select some answered questions as demonstrations, llms may directly copy the incorrect labels of these questions when they are similar to the questions to be answered. this phenomenon is also known as the copy effect @xcite . to avoid this, we introduce diversity so that the extracted questions are distributed from the highest to lowest similarity to the question to be answered. second, before a question is appended to the pool, we calculate uncertainty for it according to the outputs of llms. the lower the uncertainty, the more confident rose is about its prediction. we first filter questions with higher uncertainty in the pool. however, since the pool is a dynamic system, we also set the dynamic uncertainty threshold to only filter the questions with relatively higher uncertainty in a pool snapshot. third, one intuition is that the more complex the question, the more it can help rose learn how to answer other questions @xcite . therefore, we introduce the complexity as the final attribute. after filtering the questions with high uncertainty, we select the most complex questions as the final demonstrations. we evaluate the versatility of rose on 9 reasoning tasks, 2 llms, and different cot methods. experimental results show that rose significantly improves the reasoning performance of llms. the analysis experiments verify the importance of each experience orchestration process and the stability of rose across various experimental settings. we summarize our contribution as follows: experimental results show that rose can significantly improve the reasoning performance of llms. ## methodology in this paper, we present rose, a framework for collecting and orchestrating streaming experiences to make llms self-improve in various reasoning tasks. our setting is zero-shot (i.e., without any manual-crafting demonstrations) and stream-ing (i.e., test questions arrive one by one and there are no pre-set unlabeled questions). figure 1 shows the overview of the proposed framework. rose incorporates a streaming experience pool to store the answered questions and their reasoning paths. rose will orchestrate the experiences using multiple attributes to extract helpful questions to assist itself in better answering new questions. we construct a novel experience orchestration mechanism for rose that considers the diversity, uncertainty, and complexity of questions. in this section, we introduce how rose collects streaming experiences and how it orchestrates the collected experiences. ## conclusion we present rose, a general framework for improving the performance of llms on reasoning tasks. rose can self-improve by constantly collecting questions into an experience pool and does not need other complex external help. to extract more helpful experience from the experience pool, we propose a systematic and novel experience orchestration mechanism that sequentially regards diversity, uncertainty, and complexity of questions in the pool as important measures to filter experiences. the comprehensive experimental results on 9 reasoning tasks and 2 llms show that rose significantly improves the reasoning performance of llms. moreover, we conduct extensive analysis experiments and verify the importance of each process and the stability of rose across various experimental settings.