LDM ^2 : A Large Decision Model Imitating Human Cognition with Dynamic Memory Enhancement (2023)

Abstract: With the rapid development of large language models (LLMs), it is highly demanded that LLMs can be adopted to make decisions to enable the artificial general intelligence. Most approaches leverage manually crafted examples to prompt the LLMs to imitate the decision process of human. However, designing optimal prompts is difficult and the patterned prompts can hardly be generalized to more complex environments. In this paper, we propose a novel model named Large Decision Model with Memory (LDM ^2 ), which leverages a dynamic memory mechanism to construct dynamic prompts, guiding the LLMs in making proper decisions according to the faced state. LDM ^2 consists of two stages: memory formation and memory refinement. In the former stage, human behaviors are decomposed into state-action tuples utilizing the powerful summarizing ability of LLMs. Then, these tuples are stored in the memory, whose indices are generated by the LLMs, to facilitate the retrieval of the most relevant subset of memorized tuples based on the current state. In the latter stage, our LDM ^2 employs tree exploration to discover more suitable decision processes and enrich the memory by adding valuable state-action tuples. The dynamic circle of exploration and memory enhancement provides LDM ^2 a better understanding of the global environment. Extensive experiments conducted in two interactive environments have shown that our LDM ^2 outperforms the baselines in terms of both score and success rate, which demonstrates its effectiveness.

## introduction the rapid development of large language models (llms) has led to remarkable revolution in the field of natural language processing (nlp). llms, such as llama @xcite , palm @xcite , and @xcite , have achieved impressive results in a variety of tasks, including text/code generation @xcite , question answering @xcite , and reasoning @xcite , to name a few. besides nlp tasks, llms are also employed as policy agents to accomplish decision-making tasks @xcite . nowadays, most approaches adopt the standard prompting paradigm which uses manually crafted in-context examples to prompt the llms @xcite ) making decisions. however, the standard prompting is not suitable for decisionmaking tasks, as it restricts the llms to merely imitate the provided examples, making the generated decisions highly context-sensitive. worse still, standard prompting cannot generate admissible decisions in complex environments @xcite . in certain cases, the llms may fall into confusion when new situations are greatly different from the examples. in more challenging situations, even humans are unable to provide complete solution examples. meanwhile, once the prompt examples are fixed, the llms can no longer learn from the feedback of the environment, thus cannot further improve the performance. in this paper, we proposed large decision model with memory (ldm 2 ), a framework that enhances the standard llms with dynamic updating memory. the memory mechanism maintains the most valuable state-action tuples when imitating human decisions. as shown in fig. 1 , the prompts generated by the proposed ldm 2 focus on providing llms with sufficient information to guide decision making in the current situation, rather than relying solely on the simple examples in the entire process of decisions. in order to obtain sufficient state-action tuples in the memory, ldm 2 incorporates a memory formation stage that is analogous to the traditional imitation learning @xcite . in this stage, we take human trajectories as training data and instruct the llms to produce numerous standard state-action tuples including the task goals, observations, historical information, and actions. these standard state-action tuples are preserved to form the initial memory. in the inference phrase, we retrieve the most similar state-action tuples from the memory with current observation and then construct the prompt to inspire the llms dynamically. these retrieved state-action tuples 1) inform the llms which actions would be taken by human in the current state and 2) help the llms understand the global environment. differentiate from traditional imitation learning @xcite , ldm 2 is equipped with a dynamic memory refinement stage to enhance the memory with valuable state-action tuples. first, we conduct tree exploration to generate all potential decision processes and evaluate them according to the environment rewards. then, we add the stateaction tuple corresponding to the best decision process into the memory. this exploration-evaluationadding circle mimics the traditional reinforcement learning framework @xcite ). the refinement stage not only expands the action space of the llms, but also enable the llms to deal with new situations not covered by the initial memory. we evaluate the proposed ldm 2 in two interactive environments: webshop and alfworld. ldm 2 outperforms the standard few shots prompting methods and other methods prompted with verbal reasoning. we further analyze the successful examples in both tasks and find that ldm 2 has a more diverse action space compared with meth-ods using fixed examples prompt, this advantage empowers the llms to handle unseen or complex situations. additionally, we conduct ablation experiments to evaluate the memory refinement mechanism. the results demonstrate the effectiveness of adding highly rewarded state-action tuples into the memory. our main contributions can be summarized as: ## related work our paper is closely related to the following three research directions: llms for decision-making, feedback for llms, and memory and retrieval for llms. in this section, we briefly review the literature on these research. llms for decision-making powerful llms are able to act as policy models to make decisions in interactive environments. @xcite constructed a general framework for decision-making that uses llms to encode observations, goals, and history and then generate actions. demonstration of example prompts and utilization of high-level function libraries are employed to explore innovative strategies @xcite @xcite @xcite . prompting structure with pre-defined functions, behaviors, and examples are leveraged to ground llms to generate robotic actions @xcite @xcite . however, these methods use manually crafted examples to prompt the llms which results in decisionmaking in a fixed direction. our ldm 2 leverages dynamic state-action tuples as prompt to improve the effectiveness of decisions. feedback for llms recent techniques have emerged that focus on establishing closed-loop systems which are capable of utilizing the scalar or textual feedback from environment or human to update the llms (christiano et al., 2017; ouyang et al., 2022; chen et al., 2022; bai et al., 2022). madaan et al. (2023) and pryzant et al. (2023) employ an iterative framework for self-refinement to optimize the prompt of llms based on the feedback of self-evaluation. refiner (paul et al., 2023) fine-tunes another critic model to provide intermediate feedback within trajectories to improve the reasoning response. react (yao et al., 2022b) prompts llms with both verbal reasoning traces and actions which guides the models to perform dynamic reasoning according to environmental feedback. reflexion (shinn et al., 2023) converts binary or scalar feedback from the environment into verbal feedback which is then added in the prompt of the next episode. introspective tips (chen et al., 2023) learns tips from the action trajectories and environmental feedback to empower the llm agents with self-optimizing capabilities. these approaches mainly aim to leverage reward feedback to augment the prompt of llms, however, ldm 2 adds highly rewarded state-action tuples into the memory which achieves dynamic learning ability. memory for llms memory could store information perceived from the environment and leverages the recorded memories to facilitate future actions. generative agents (park et al., 2023) maintain a memory stream to record the experience including observations and behaviors. reflexion (shinn et al., 2023) stores experiential feedback in natural language within a sliding window. voyager (wang et al., 2023) employs natural language descriptions to represent skills within the minecraft game, which are directly stored in memory. mem-orybank (zhong et al., 2023) encodes the memory segment into embedding vector which could enhance memory retrieval and reading efficiency. knowledge base is also used as the memory to retrieve relevant information and construct the taskrelated augmented prompts @xcite @xcite . our ldm 2 also constructs a memory to record vast state-action tuples, which forms the retrieval indices list through instructing the llms to cluster different goals and observations. ## conclusion this paper proposed ldm 2 , which enhances the standard llms with dynamic updating memory to maintain the most valuable state-action tuples to imitate human decision. ldm 2 consists of two stage: memory formation and memory refinement. in the formation stage, we take human trajectories as training data and instruct the llms to produce numerous standard state-action tuples. these standard state-action tuples are preserved to form the initial memory. ldm 2 is equipped with dynamic memory refinement stage to enhance the memory through adding valuable state-action tuples. we conduct tree exploration to generate all potential decision processes and add the state-action tuple corresponding to the higher reward decision process into the memory. experiments on two interactive environments illustrated that ldm 2 outperforms the standard few shots prompting methods and the ablation study verified the effectiveness of the memory formation and refinement mechanism.