M eta R eflection: Learning Instructions for Language Agents using Past Reflections (2024)

Abstract: The popularity of Large Language Models (LLMs) have unleashed a new age of Language Agents for solving a diverse range of tasks. While contemporary frontier LLMs are capable enough to power reasonably good Language agents, the closed-API model makes it hard to improve in cases they perform sub-optimally. To address this, recent works have explored techniques to improve their performance using self reflection and prompt optimization techniques. While techniques like self reflection work well in an online setup, contemporary prompt optimization techniques are designed to work on simpler tasks. To address this, we introduce METAREFLECTION, a novel offline reinforcement learning technique that enhances the performance of Language Agents by augmenting a semantic memory based on experiential learnings from past trials. We demonstrate the efficacy of METAREFLECTION by evaluating across multiple domains, including complex logical reasoning, biomedical semantic similarity, open world question answering, and vulnerability threat detection, in Infrastructure-as-Code, with different agent design. METAREFLECTION boosts Language agents’ performance by 4 % to 16.82 % over the raw GPT-4 baseline and performs on par with existing state-of-the-art prompt optimization techniques while requiring fewer LLM calls.

## introduction large language models (llms), such as @xcite , have gained significant popularity in recent years due to their ability to generate human-like text and solve complex tasks across various domains. to leverage these models, users typically craft prompts with instructions that are * equal contribution tailored to a specific task. furthermore, many practical llm applications setup complex multi-step systems with multiple llm calls chained together @xcite or llm calls with different prompts called in succession @xcite . given their close resemblance to reinforcement learning agents, it has become useful to model these systems as language agents @xcite building up on these insights, recent works @xcite have showed that the performance of such language agents can be improved using verbal reinforcement learning across multiple conversational turns, by providing feedback at the end of a failing trajectory and asking the agent to reflect on its mistakes. the reflective text is then stored as episodic memory to improve future trajectories on the same task. parallel to this, there has been recent advancements in developing prompt optimization techniques @xcite . these techniques typically start with a seed human prompt and a train dataset of input-output pairs. leveraging error feedback over failing cases, they deploy various search strategies to find prompt variants that can give the optimal results on train set. while prompt optimization techniques exist for single llm call setups, they are not designed for complex multi-step language agents. here, techniques like self-reflection can help improve a language agent's performance significantly. however, these are online reinforcement processes that depend on the availability of performing multiple turns with a feedback mechanism on the same task instance and aren't applicable to similar new tasks. motivated by the semantic memory observed in humans @xcite @xcite we introduce metareflec-tion, an offline reinforcement learning technique that works by augmenting a semantic memory to represent experiential learnings from trials in an of- ## metareflection we consider a setup of an llm agent a interacting with an environment env to solve a task t, characterized by the start state s 0 . at time step t, the agent a receives an observation o t at state s t and algorithm 1 metareflection require: train data dt , val. data dv , agent a require: environment env ensure: meta-reflection instructions instr 1: instr ← ⊥ 2: for all @xmath0: v ← randomsample(dv ) 4: for all @xmath1, 1 . . . maxtrials} do ▷ iterative refinement 5: initialize a with instr 6: trajectories ← {a.run(i) | @xmath2: ft ← env.getfailingtrajectories(trajectories) 8: if len(ft) == 0 then 9: break ▷ early stopping 10: refl ← { a.selfreflect(t) | @xmath3: instr * ← a.metareflect(instr, b, refl) 12: if a.showsimprovement(instr * , instr, b, v, env) then ▷ validation 13: instr ← instr * ▷ else backtrack 14: return instr samples an action a t using a policy π(a t |t t , p t ) metareflection aims to improve the performance of such language agents by augmenting them with a semantic memory @xcite in the form of rule based instructions instr to store the experiential learnings, replacing π(a t |t t , p) with π(a t |t t , p instr). algorithm 1 shows the outline of the metareflec-tion process. at its core, the algorithm works by starting with an empty set of instructions and iteratively improving the instructions using small training batches. for instance, if the agent a react is a reactstyle agent working on a question answering task over wikipedia, at state s 0 , the agent will be provided by a question q. now, the agent can choose to take a search action, upon which it will receive an observation o. the agent can then either chose to pick further search or lookup actions, or pick an answer action to return an answer. this action will mark a transition to the state s n and a binary reward can be generated depending on the correctness of the answer. an even simpler example can be an agent a s which given a multiple-choice question, picks an answer and receives a positive reward if the answer is correct. agent, environment and self-reflection method metareflection procedure uses the following components at its core: the metareflect step. the metareflect method is designed to take as input a prior set of instructions instr, a set of self-reflections refl, the training data d t and a validation data d v and will produce an updated set of instructions instr * as denoted in line 11 of algorithm 1. for the metareflect method, we use a standard language model with a prompt that instructs the llm to observe the reflections, the training data, and produce a new non-case specific instruction style memory. further, the prior memory is also passed as input so that the output is a generalization of the prior learnings. in our implementation, this meta-reflection and generalization are done in the same prompt for efficiency. alternatively, new instructions can be generated first and then combined with existing ones. we specify that the instructions need to take the form of a list. hence, the meta-reflection step in line 11 typically either (a) updates the list by adding a new item, or (b) combines one or more previous items with learnings from the self-reflections to produce a shorter list. for example, one meta-reflection instruction learned during our hot-potqa experiments suggested including the profession when searching for a person to narrow down results. in a subsequent batch, the selfreflection step produces a reflection that mentions adding search terms like release date when searching for movies. the metareflect step may combine the previous instructions with the current selfreflections either by appending a new item to the list clarifying the strategy to search for movies, or may generalize the previous item to something like "when searching for specific entities, use additional contextual information to augment the primary search terms with secondary keywords corresponding to the characteristics of the entity". validation and backtracking in each iteration, after metareflect, we validate the quality of the new instructions. due to the sparse reward signals leading to poor self-reflections or over-generalization of the meta-reflection instructions, we may end up with instructions that are of a poorer quality than the prior instructions. the poorer instructions may also be due to general capricious, unpredictable nature of large language models. therefore, we validate the new instructions by testing them on training data and a random sample of the validation set to ensure that they perform better than the prior instructions as depicted in line 12 of algorithm 1. ideally, we would do this validation over the full validation dataset. however, in our case, we only validate on the current batch to balance quality of instructions and efficiency. in cases where the updated instructions perform poorly compared to the prior ones, we backtrack to prior instructions as depicted in line 13. as an example, in the previous paragraph the meta-reflection step replaced the specific instruction on how to search for persons with a more general instruction on how to search for entities. however, it is possible that these general instructions are too vague (especially for smaller, less capable models) and the client agent is not able to apply them correctly to the case of searching for persons. iterative refinement motivated by the self refining behaviour of language models as demonstrated in @xcite , we use multiple attempts at meta-reflection for each batch (line 4 through line 13) until we see no failures in the current batch or exhaust a maximum number of trials (set to 3 for all our experiments). similar to how self-reflections help optimize language agents' tra-jectories towards achieving a task, the feedback on failed trajectories from the instr * act as implicit verbal reinforcements in the meta-reflection process. these verbal reinforcements can then be used by the meta-reflection step to guide the instruction search. this trajectory driven iterative refinement strategy reduces the chances of repeating mistakes, such as proposing incorrect refinements that were already tried in the past, during the instruction refinement process. in the single step agent setups where they are directly comparable, we observe that this iterative refinement strategy leads us to similar performance as state of the art prompt optimization techniques, with lesser number of llm calls. ## related work with the increasing ubiquity of black-box large language models @xcite @xcite , there has been a growing interest in the community to develop strategies that can maximize the model's performance on a downstream task. these techniques may involve guiding an llm to arrive at the correct answer @xcite , output selection @xcite , picking up the right in-context examples @xcite or prompt optimization, etc. being closely related to our work we dive deeper into contemporary prompt optimization techniques in the literature. ## conclusion in this work, we presented metareflection, a novel offline reinforcement learning technique that takes inspiration from how human brain store memories to enhance language agents by augmenting them with an experiential semantic memory. we further empirically demonstrated that the instructions learned using metareflection are more effective at capturing task-specific nuances. this behavior helps metareflection perform competitively with state of the art prompt optimization baseline for single-step agent scenarios. the objective driven iterative refinements also provide a significant advantage to capture the best learning at each iteration and save it to its semantic memory which helps the agent to perform better inference in next iteration. in the future, we would like to see how we can leverage our semantic memory in a multi-agent workflow. sharing memories between agents can enhance synergy and improve task performance.