M o T : Memory-of-Thought Enables C hat GPT to Self-Improve (2023)

Abstract: Large Language Models (LLMs) have shown impressive abilities on various tasks. However, fundamentally improving them depends on high-quality datasets or computationally expensive fine-tuning. On the contrary, humans can easily improve themselves by self-thinking and memory, without external resources. In this paper, we propose a framework, **MoT**, to let the LLM self-improve through **M**emory **o**f **T**houghts, without annotated datasets and parameter updates. Specifically, MoT is divided into two stages: 1. before the test stage, the LLM pre-thinks on the unlabeled dataset and saves the high-confidence thoughts as external memory; 2. During the test stage, given a test question, the LLM recalls relevant memory to help itself reason and answer it. Experimental results show that MoT can help ChatGPT significantly improve its abilities in arithmetic reasoning, commonsense reasoning, factual reasoning, and natural language inference. Further analyses show that each component contributes critically to the improvements and MoT can lead to consistent improvements across various CoT methods and LLMs.

## introduction large language models (llms) have demonstrated surprising abilities on a wide range of natural language processing (nlp) tasks @xcite @xcite @xcite @xcite @xcite . notably, new abilities emerge in llms as they are scaled to hundreds of billions of parameters, like in-context few-shot learning @xcite , simple digit operation and factual knowledge query @xcite . especially, the general reasoning ability of the llm has impressed the nlp community and relevant techniques have achieved a series of new state-of-the-art @xcite @xcite ; while fine-tuning llm with labeled datasets is costly and needs powerful computational resources, mot can make the llm self-improve via pre-thinking and recalling, without parameter updates and annotated datasets. @xcite . specifically, @xcite and @xcite propose few-shot cot and zero-shot cot, which elicit llm's reasoning by few-shot demonstrations and simple yet effective "let's think step by step" prompting, respectively. based on them, @xcite ; @xcite ; @xcite ; @xcite ; @xcite further propose self-consistency, self-ask, least-tomost, plan-and-solve, etc., to achieve more complicated reasoning in various specialized scenarios. despite the impressive abilities of the llm pretrained on the large corpus, fundamentally improving the llm's performance beyond few-shot / zero-shot baselines highly depends on either highquality annotated datasets or costly fine-tuning of llms. in general, these methods can be divided into three categories: 1. annotated datasets + fine-tuning: @xcite and @xcite propose flan and t0 respectively to enhance the llm's zero-shot ability by tens of curated nlp benchmark datasets. based on @xcite scale up its training in terms of model size and the number of tasks, and demonstrate that the added cot examples with rationales improve the llm's reasoning abilities. instruct-gpt @xcite improves the gpt-3's instruction-following ability by fine-tuning on many diverse crowd-sourced instruction-answer pairs. 2. retrieving annotated data: @xcite , @xcite and @xcite use sentencebert @xcite or bm25 @xcite to retrieve relevant examples from the annotated dataset, to improve llm's in-context learning. @xcite and @xcite leverage annotated datasets to train retrievers by the lm-feedback to retrieve helpful demonstrations for the test example. 3. fine-tuning with llm-generated data: zelikman et al. (2022) let the lm generate rationales for annotated dataset and train itself to enhance the reasoning ability. @xcite , @xcite and @xcite use the reasoning paths generated by large lm to improve the small lm's reasoning capability. more recently, @xcite demonstrate the effectiveness of self-training on palm @xcite . as annotating high-quality data, especially rationales in cot data, is expensive, fine-tuning llm requires extremely powerful computational resources and results in high computational costs. methods above that rely on fine-tuning also face two challenges: 1. since the most powerful llms, e.g., @xcite and palm @xcite , are only publicly available through the inference api, it is not feasible for most of the research community to improve them by these methods. 2. fine-tuning llm for specific capability enhancement is costly and not environmentally friendly. as the llm has massive parameters, fine-tuning them will lead to substantial costs of model storage and deployment. further studies show that fine-tuning the llm with specialized data may significantly decrease its general abilities @xcite . while considerable efforts were dedicated to collecting high-quality annotated datasets and finetuning the llm, which is costly and may decrease its general ability, on the contrary, humans can improve their own reasoning abilities through the metacognition process @xcite and the memory mechanisms @xcite , and preserve their general abilities. for example, memory helps humans improve themselves in terms of decision-making, reasoning, judgment, etc @xcite . inspired by this, we propose mot, shown in figure 1 , a pre-think-then-recall framework to let the llm self-improve through memory-of-thoughts, without supervised data and parameter updates. in the pre-thinking stage, the llm thinks on the unlabeled dataset and saves the thoughts as external memory. in the test stage, the llm recalls relevant memory to help reason and answer the given test question. since we focus on the overall framework and aim to demonstrate its generality and extensibility, we use simple components to instantiate these two stages. specifically, we use the simple few-shot-cot @xcite with multiple-path decoding strategy @xcite in the pre-thinking stage and propose answer-entropy to filter out uncertain thoughts. for memory recall, we propose llm-retrieval, which lets the llm itself retrieve relevant memory to help answer the test question. compared with typical semantic retrievers like sbert @xcite , llm-retrieval can better capture the deep connection of complicated logic and reasoning than semantic embeddings. we summarize our contribution as follows: ## background: chain of thought the large language model has shown impressive reasoning abilities on various tasks. chain-of-thoughts (cot) prompting @xcite is the most prevailing way to q: roger has 5 tennis balls. he buys 2 more cans of tennis balls. each can has 3 tennis balls. how many tennis balls does he have now? let the llm reason, i.e., generate a series of intermediate reasoning steps that lead to the final answer. as shown in figure 2 , few-shot-cot @xcite provides a few demonstrations with rationales, i.e., question/rationale/answer pairs, and prompts the llm to generate the rationale that leads to the final answer. zero-shot-cot @xcite adds the prompt, "let's think step by step", after the test question and elicits the llm's reasoning. specifically, the few-shot-cot gets the answer as: ## method we show the overview of our framework in figure 3 . in this paper, we mainly focus on making llm self-improve in the typical few-shot cot scenario, where we are given a frozen large language model and an unlabeled dataset with a few cot demonstrations @xcite . we further demonstrate mot's effectiveness in zero-shot scenarios in section 4.3. our framework is divided into two stages: 1. pre-think before the test stage, the llm thinks over the unlabeled dataset and keeps the high-confidence reasoning paths as memory. 2. recall in the test stage, given a test question, we propose llm-retrieval to let the llm retrieve relevant memory to help itself reason and answer it. our method does not depend on high-quality labeled datasets and costly finetuning of llm, and it is feasible when the llm is frozen or only available through the inference api. since we let the llm think over the unlabeled dataset, save the self-generated thoughts as external memory and retrieve relevant memory for itself to help reasoning, we consider our method as making the llm self-improve with memory-of-thought. we introduce these two stages below. ## conclusion in this paper, we propose mot, a framework that let the llm self-improve via memory-of-thought, without annotated datasets and parameter updates. experimental results show that mot can help chat-gpt significantly improve its abilities in arithmetic reasoning, commonsense reasoning, factual reasoning and natural language inference. further analyses show that each component contributes critically to the improvements and mot can lead to consistent improvements across various cot methods and llms.