S tate A ct: Enhancing LLM Base Agents via Self-prompting and State-tracking (2025)

Abstract: Large language models (LLMs) are increasingly used as autonomous agents, tackling tasks from robotics to web navigation. Their performance depends on the underlying ‘base agent‘. Existing methods, however, struggle with long-context reasoning and goal adherence. We introduce ‘StateAct‘, a novel and efficient ‘base agent‘ that enhances decision-making through (1) ‘self-prompting‘, which reinforces task goals at every step, and (2) ‘chain-of-states‘, an extension of chain-of-thought that tracks state information over time. StateAct outperforms ReAct, the previous best ‘base agent‘, by over 10% on Alfworld, 30% on Textcraft, and 7% on Webshop across multiple frontier LLMs. We also demonstrate that StateAct can be used as a drop-in replacement for ReAct with with advanced LLM agent methods such as test-time scaling, yielding an additional 12% gain on Textcraft. By improving efficiency and long-range reasoning without requiring additional training or retrieval, StateAct provides a scalable foundation for LLM agents. We open source our code to support further research at https://github.com/ai-nikolai/stateact.

## introduction leveraging the in-built world and commonsense knowledge foot_0 of large language models (llms), such as @xcite @xcite to perform interactive reasoning tasks has become a frontier in ai research. "ai agents" are now able to solve a range of multi-modal complex tasks @xcite . these include simulated robotics tasks @xcite and digital tasks, such as online shopping @xcite , navigating operating systems @xcite , and playing a variety of games @xcite @xcite . at the core of an llm agent is the base agent foot_1 , such as act @xcite , react @xcite , and adaplanner @xcite . existing efforts to improve llm agents build on top of base agents and are usually quite resourceintensive: @xcite require human expert annotations of rules; @xcite require a code execution environment with carefully crafted code-based prompts; @xcite use additional training data together with retrieval augmented generation (rag) to help the ai agent; @xcite in our work, we propose a new base agent, called stateact. starting with the observation that: i) llm agents fail to follow the original instruction and goal in longer interactions; and ii) llms struggle with long context despite longer available contexts @xcite , we propose two main contributions for improving the base agent. to address the first issue, we propose a mechanism for the agent to 'self-prompt' at every turn of the interaction to improve staying on track with the main goal. concretely, our agent 'reminds' itself of the final goal at every turn. to address the second issue, we propose 'chain-ofstates', an extension of chain-of-thought based on state tracking to help the agent stay on track with the current interaction and context. using this, the agent keeps track of its state in the environment (such as location and inventory). our experiments show that stateact achieves near state-of-the-art performance without using additional training data or additional tools. statact also significantly outperforms react across multiple tasks and eight frontier llms of varying sizes. specifically, stateact improves performance over react by more than 10% on alfworld @xcite , 30% on textcraft @xcite , and 7% on webshop @xcite . additionally, we validate that stateact can serve as a drop-in replacement for existing extension methods. using test-time computation @xcite , we achieve a further 12% performance gain with stateact on textcraft using adapt. ## background due to recent advances, llms are now being used as autonomous agents in interactive environments as an alternative to traditional reinforcement learning (rl) @xcite @xcite . llm agents now tackle tasks in simulated environments such as alfworld, webshop, and textcraft. llm agents consist of the base agent and a possible extensions, such as fine-tuning (ft), retrieval augmented generation (rag), test-time scaling (tts), repeated attempts (reps) or tools; see stateact is a base agent that can be used in combination with extensions. ## method stateact is an llm-based ai agent that works on top of pre-trained llms. it takes the textual 'observation' from the environment and, after a single call to the pre-trained llm, returns the 'action' back to the environment, without the use of additional tools or resources. stateact utilises in-context learning @xcite to make the agent interact with the environment. at the core of the approach is a prompt that consists of few-shot examples of successful interaction traces as well as the current interaction trace up to the current step in the environment. an interaction trace consists of alternating observations from the environment and desired (or actual) outputs from the llm. in the case of stateact, the llm is tasked to generate the 'goal', 'state', 'thought' and 'action'. the action is then extracted and passed to the environment to produce the next observation, see figure 2 . this renders stateact similar to react and therefore an easy replacement for extension methods. ## experimental setup our aim is to study reasoning capabilities of llmbased agents. in line with previous work, we use alfworld @xcite , a household robotics environment; webshop @xcite , an online shopping environment; and texcraft @xcite , a text-based tool crafting game, as our evaluation environments. we mainly evaluate stateact as the base agent. however, we also show that stateact can be used as a drop-in replacement for react and validate it on testtime scaling. rag (autoguide) and fine-tuning (actre)-based extensions are left for future work, as these are expensive and complex to run due to training on additional data and use of additional tools. ## results we present results of state-of-the-art methods such as actre and adapt, as well as the underlying base agent react. in ## analysis and ablations in the results section, we discovered that our methods outperform the previous state-of-the-art base agent, react. stateact shows strong performance with in-context learning without resorting to additional tools or data. in this section, we analyse our results further and also show that self prompting and state-tracking help with long-range reasoning. for this purpose, we compare the original react (thought + action) with the self-prompting included, i.e. stateact (goal + thought + action). in figure 5 we can see that, while the performance of both react and stateact goes down as there are more steps, the goal tracking has better relative performance as the number of steps increases and is able to solve longer tasks of 40 to 50 steps. this finding is in line with our original motivation, that llm agents deteriorate in performance as the prompts and interactions get longer. to verify that this actually means that goal tracking helps with performance, as opposed to just increasing the number of steps it takes to solve a task, we calculate the average number of steps for react foot_6 and stateact. ## conclusion our work is driven by the fundamental challenge that llm agents struggle with long context and keeping on track with instructions. the current state-of-the-art to overcome such challenges propose extensions on top of the base agent, react. in contrast, we introduced a novel base agent, state-act, an in-context learning method that leverages chain-of-states and self-prompting to significantly enhance the capabilities of llm agents. by rethinking how agents track and utilize state information, stateact establishes a new state-of-the-art for base agents, surpassing react by 9% to 20% across different models and tasks. furthermore, we demonstrate that stateact scales effectively when combined with advanced techniques such as adapt, reinforcing its usability. beyond raw performance, our analysis uncovers a crucial insight: stateact not only improves reasoning but also enhances efficiency, allowing agents to achieve better results with fewer steps. this suggests that integrating structured statetracking and self-prompting cues helps mitigate the well-documented long-context issue in llms. our findings suggest that methods like stateact can serve as a practical and efficient way to improve llm-based agents without the need for additional data, tools or other extensions. by enabling agents to manage their own state explicitly, stateact provides a scalable approach to improving reasoning and decision-making across a range of tasks. this makes stateact a viable drop-in solution for current systems, offering both better performance and greater efficiency in llm agent tasks.