Positive Experience Reflection for Agents in Interactive Text Environments (2025)

Abstract: Intelligent agents designed for interactive environments face significant challenges in text-based games, a domain that demands complex reasoning and adaptability. While agents based on large language models (LLMs) using self-reflection have shown promise, they struggle when initially successful and exhibit reduced effectiveness when using smaller LLMs. We introduce Sweet&Sour, a novel approach that addresses these limitations in existing reflection methods by incorporating positive experiences and managed memory to enrich the context available to the agent at decision time. Our comprehensive analysis spans both closed- and open-source LLMs and demonstrates the effectiveness of Sweet&Sour in improving agent performance, particularly in scenarios where previous approaches fall short.

## introduction intelligent agents, designed to interact with and make decisions in dynamic environments, have become a central focus in ai research, with textbased games (tbgs) emerging as a particularly challenging domain for evaluating these agents' reasoning, adaptability, and learning abilities @xcite . originally popular in the 1970s as text adventure games, 1 tbgs present players with textual descriptions of environments, requiring them to input natural language commands to achieve objectives @xcite . for instance, determining if a metal fork is conductive involves locating the fork, assembling a circuit, and analyzing the result. navigating tbgs demands that agents exhibit a combination of abilities, including planning, memory retention, spatial reasoning, and common sense knowledge @xcite . previously, deep reinforcement learning and behavior cloning were the primary approaches to develop agents to play tbgs @xcite . however, recent research shows that agents based on pretrained large language models (llms) are more effective at navigating tbgs @xcite . a key factor in their success is the integration of internal reflection to improve planning @xcite @xcite . self-reflection, closely related to self-refinement, is a form of reasoning that occurs after receiving binary or scalar feedback from the environment @xcite . in this process, the llm reviews its actions and their outcomes, considering what went wrong and potential ways to improve @xcite . by iteratively adjusting its strategy based on verbal reinforcement, conveyed through textual feedback, the agent refines its planning for subsequent attempts @xcite . however, reflection also has several limitations, including 1) underwhelming performance when agents are correct initially @xcite , 2) significantly worse efficacy when using smaller llms @xcite , and 3) dependence on external feedback @xcite . our contributions. in this work we conduct a comprehensive analysis of llm-based agents employing reflection approaches in tbgs and evaluate their performance across various llms. to address the limitations of reflection when agents are initially successful and the diminished efficacy of smaller llms, we propose sweet&sour (s&s) to leverage positive experiences to create a richer context for self-reflection. we supplement this by proposing a managed memory approach to build context across multiple rollouts. our findings demonstrate that our method improves the performance of agents using reflection, particularly in scenarios where they previously struggled, enabling more robust and generalizable learning. action @xmath0: look @xmath1. you see: [...] action t-1: connect battery anode to black @xmath2: now i need to connect the cathode agent reflection verbalize what lead to failure to improve next attempt next attempt action t: out of actions? / possible action demo + previous reflections task: determine conductivity of fork. action @xmath3: look @xmath4. you see: [...] action t-1: connect battery anode to black @xmath5: now i need to connect the cathode action t: out of actions? / possible action agent reflection verbalize what lead to failure to improve next attempt next attempt demo + previous reflections task: determine conductivity of fork. action @xmath6: look @xmath7. you see: [...] action t-1: connect battery anode to black @xmath8: now i need to connect the cathode action t: out of actions? / possible action subgoal reached? agent reflection next timestep stop act act stop act stop react reflexion sweet&sour stm ltm ltm ## results and discussion the results for scienceworld are shown in sweet&sour improves robustness to tilt. in highly challenging tasks, such as 1-1 and 8-2, all methods tend to struggle, while in simpler tasks, methods succeed based on the llm's inherent capabilities. however, medium-difficulty tasks, such as 6-3 and 10-2, reveal a critical performance gap between our method and previous approaches. for instance, on task 10-2, we could observe reflexion getting off to a strong start, achieving multiple subgoals, though it appeared to get stuck upon encountering its first error. we hypothesize this occurs because methods reflecting only on failures lack context from early successes, leading to performance decline when facing first encountering failure (also known as "tilt"). by contrast, s&s's balanced reflection approach provides richer context, enabling it to build upon initial successes. ## conclusion we introduced sweet&sour, a novel approach that enhances llm-based agents' performance in text-based games through balanced reflection on both successes and failures. our evaluation demonstrates significant improvements over existing methods across different model sizes, particularly for smaller llms. these findings establish the value of positive experience reflection and managed memory in developing more robust and adaptable agents for complex interactive environments.