MSI -Agent: Incorporating Multi-Scale Insight into Embodied Agents for Superior Planning and Decision-Making (2024)

Abstract: Insight gradually becomes a crucial form of long-term memory for an agent. However, the emergence of irrelevant insight and the lack of general insight can greatly undermine the effectiveness of insight. To solve this problem, in this paper, we introduce **M**ulti-**S**cale **I**nsight Agent (MSI-Agent), an embodied agent designed to improve LLMs’ planning and decision-making ability by summarizing and utilizing insight effectively across different scales. MSI achieves this through the experience selector, insight generator, and insight selector. Leveraging a three-part pipeline, MSI can generate task-specific and high-level insight, store it in a database, and then use relevant insight from it to aid in decision-making. Our experiments show that MSI outperforms another insight strategy when planning by GPT3.5. Moreover, We delve into the strategies for selecting seed experience and insight, aiming to provide LLM with more useful and relevant insight for better decision-making. Our observations also indicate that MSI exhibits better robustness when facing domain-shifting scenarios.

## introduction creating agents that can make autonomous decisions in the environment has always been a promising and interesting research direction. @xcite with the emergence of chatgpt and gpt-4 @xcite , large language models (llms) have transformed from specialized models to a general model that can complete multiple types of tasks, hence it can make decisions for agents. @xcite @xcite . this type of agent will transform multi-modal information into natural language as short-term memory. it then prompts large language models with shortterm memory and long-term memory to plan and * corresponding authors. make decisions. with these capabilities, the agent can generate a series of actions that are executable within a given environment. @xcite @xcite insight foot_0 , as a form of long-term memory, has gradually become a crucial part of guiding llm planning and decision-making. @xcite @xcite @xcite . relative to other long-term memory such as examples, insight is more concise and higher-level. although previous work has proposed a method of using llm to summarize and utilize insights @xcite , it either provides llm with too many irrelevant insights or can not summarize the highlevel insights, as shown in figure 1 . the former can interfere with decision-making @xcite figure 2 : the overall pipeline for the msi-agent to complete a task. msi memory refers to the part that deals with insight. in msi memory, experience selection and insight generation will summarize historical experience into insights, while insight selection will select insights to assist the executor in completing future tasks. to address these challenges, we proposed multi-scale insight agent (msi-agent), an embodied agent designed to summarize and utilize insights effectively. inspired by expel @xcite , msi collects the task background, user queries, agent's plans, environmental feedback, and execution results as "experience" from a series of training tasks. these experiences are then organized into the successful experience set or successfailure experience pairs set via an experience selector. subsequently, an insight generator summarizes multi-scale insights based on the organized experience(s). through this method, both high-level and fine-grained insight can be generated. during task execution, the insight will pass an insight selector to filter out the irrelevant insight and the remaining insight prompts the executor to formulate plans and execute tasks within a given environment. the overall pipeline for the msi agent to complete a task is illustrated in figure 2 , while the architecture of the insight part in msi is detailed in figure 3 . this solution effectively mitigates the issues highlighted earlier. by allowing classifying and selecting insights, msi ensures that the llm is not overwhelmed with irrelevant insights. simultaneously, the multi-scale insights generation provides a nuanced understanding at various levels, address-ing the challenge of high-level insights summarization. as a result, msi stands as a robust solution, offering contextual and comprehensive insights tailored to enhance decision-making capabilities. in summary, our contributions are as follows: ## experiment we evaluate msi on the 2 benchmarks foot_4 : teach tfd benchmark @xcite and agentbench alfworld benchmark @xcite . our experiments are designed to address the following research questions (rqs): rq1: does msi outperform other insights methods? rq2: what kind of seed experience selection strategy should be chosen when facing different insight generation strategies and tasks? rq3: what kind of insight selection strategy should be adopted for different future tasks? rq4: how does the robustness of the msi system evolve with the domain shifts?