Conditional Multi-Stage Failure Recovery for Embodied Agents (2025)

Abstract: Embodied agents performing complex tasks are susceptible to execution failures, motivating the need for effective failure recovery mechanisms. In this work, we introduce a conditional multi-stage failure recovery framework that employs zero-shot chain prompting. The framework is structured into four error-handling stages, with three operating during task execution and one functioning as a post-execution reflection phase.Our approach utilises the reasoning capabilities of LLMs to analyse execution challenges within their environmental context and devise strategic solutions.We evaluate our method on the TfD benchmark of the TEACH dataset and achieve state-of-the-art performance, outperforming a baseline without error recovery by 11.5% and surpassing the strongest existing model by 19%.

## introduction in embodied ai settings, autonomous agents are required to perform complex tasks within environments such as homes or offices. in these settings, large language models (llms) have been employed to decompose natural language instructions (e.g., make breakfast) into a plan of executable actions (e.g., pick_up(cup)), with the objective of ensuring successful task completion @xcite @xcite . prior research has explored generating plans that are robust to failures through few-shot prompting, utilizing an underlying memory of demonstrations @xcite @xcite @xcite . however, the initial llmgenerated plan does not inherently ensure successful task completion, as (a) the plan may contain errors, such as missing or incorrect steps, and (b) the agent may encounter unforeseen challenges within the environment that are difficult to anticipate in the planning phase. thus, grounding the plan within the environmental context and integrating error recovery mechanisms are essential for enabling the agent to adapt and re-plan to address execution challenges. this motivates incorporating feedback from the environment for more robust planning and error recovery. for example, existing approaches incorporated visual information represented through image embeddings @xcite @xcite @xcite or structured scene descriptions 1 @xcite @xcite @xcite . other work used human feedback to correct the agent's behavior @xcite @xcite . another form of feedback involves verifying action preconditions, which can either be explicitly encoded within the execution module, requiring domainspecific expertise @xcite @xcite , or learned via reinforcement learning methods @xcite . however, prior research has not systematically examined the structured process of error recovery or devised strategic frameworks for how agents should handle execution challenges. we propose a conditional multi-stage failure recovery (cmfr) approach for embodied agents that uses zero-shot chain prompting. 2 chain prompting decomposes a complex task into a sequence of interdependent prompts, where the output of one prompt serves as input for the next @xcite . our method leverages llms to assess execution challenges in the given environmental context and devise strategic solutions. cmfr is structured across four distinct stages utilised both during and after task execution. our approach stands out by leveraging the reasoning abilities of llms in a zero-shot manner without relying on external modules, such as example-based memory or domain-specific precondition checks. we evaluate the cmfr approach on the teach benchmark @xcite , which encompasses a diverse set of long-horizon household tasks. our experimental results demonstrate that our cmfr approach enhances task success by 11.5%, achieving state-of-the-art results and surpassing existing models by a significant margin. in addition, we integrate and evaluate an llm-based object search mechanism that leverages object locations mentioned in task dialogues. finally, we conduct an ablation study to highlight the contribution of each stage in cmfr. our approach provides a structured framework for reasoning about and overcoming execution challenges, contributing to research on embodied agents designed to assist humans in household tasks. ## related work embodied ai a substantial body of research has focused on developing embodied agents capable of translating natural language instructions into executable actions, leveraging various simulators and benchmarks designed to support such tasks @xcite @xcite @xcite @xcite . some approaches have focused on fine-tuning multimodal models to encode linguistic and visual inputs for predicting low-level actions @xcite @xcite @xcite @xcite @xcite @xcite @xcite . other work has leveraged prompting techniques to use llms as planners for embodied tasks @xcite @xcite @xcite @xcite @xcite . while llms excel at reasoning and decomposing complex tasks into actionable steps, they require environmental grounding to address execution challenges. to achieve this, various approaches have been proposed to integrate environmental feedback into llm-based planning. some studies utilize perception models or ground-truth simulator data to generate scene descriptions, which are then incorporated into llm prompts @xcite @xcite @xcite . others employ vision-language models to classify scene images based on predefined failure categories @xcite @xcite , while additional research explores learning affordance functions through reinforcement learning @xcite . ## problem definition task-driven embodied agents that chat (teach) @xcite is a dataset focused on long-horizon tasks in household environments. it comprises over 3, 000 gameplay episodes built on top of ai2-thor simulator @xcite . an episode consists of a human-human interactive dialogue between a commander that has oracle information about the task and a follower (agent) that tries to complete the task by navigating and interacting with objects in the simulated environment. teach comprises 12 household tasks with varying granularity (appendix d). 3 furthermore, teach includes three benchmarks: (1) trajectory from dialog (tfd): where given the full dialogue history of the task, the agent predicts the sequence of actions that completes the task successfully. (2) execution from dialogue history (edh): where the task dialogue in tfd is segmented into sessions and the agent is asked to predict the actions that lead to the next session, and (3) two-agent task completion (tatc) where both the commander and follower are modeled to perform the task. in this work, we focus on the tfd benchmark as it poses several challenges. for instance, unlike other datasets that give a single instruction or a high-level task to the agent @xcite @xcite @xcite , the input in tfd is a noisy dialogue that contains information about the task, making it more challenging for the agent to extract relevant task information and convert that to a sequence of executable steps. moreover, in tfd, the average number of actions the human agent takes to solve a task is 117, demonstrating the complexity and long-horizon nature of the tasks. for evaluation, we use the following metrics @xcite . success rate (sr) which is the fraction of episodes in which all task goal-conditions are fulfilled. 4 goal-condition success (gc) which is the ratio of the completed goal-conditions to those necessary to succeed in the task. path length weighted (plw) where both sr and gc metrics have a path length weighted counterpart which penalises the agent for taking more steps than the human-annotated reference path. more details about the dataset are presented in appendix a. the task fails if the agent exceeds 1000 actions or 30 failed actions. ## experiments we assess our approach on both the seen and unseen splits of the tfd benchmark in teach and present the evaluation results using the sr, gc, and pwl metrics (section 3). all experiments use the same initial plans generated by gpt-4o @xcite , allowing a focused assessment of error recovery. we evaluate four llms for cmfr and object search: gpt-4o, o3-mini, qwen2.5-7b @xcite and llama-3.1-8b @xcite . our cmfr method is compared against a baseline where no failure recovery is included. we also conduct an experiment where we apply chain-of-thought (cot) reasoning @xcite in error recovery (cot-gpt-4o). 7 additionally, we compare our results to the following previous models. helper @xcite uses gpt-4 for planning, error recovery and object search, supported by a memory that is expanded with successful examples for few-shot prompting. helper explicitly encodes subgoal preconditions within the executor module and relies on previously established perception models (dong @xcite . to ensure a fair comparison, we reproduce their results using groundtruth perception on the seen split (helper*). danli @xcite fine-tunes a bart-large model @xcite to predict highlevel subgoals that are translated to low-level actions using a pddl planner @xcite . in danli, all preconditions and error recovery mechanisms are hardcoded and perception models @xcite are used. we replicate their experiments using ground-truth perception on the seen split (danli*). msi @xcite builds on top of helper and enhances the performance by collecting the agent's experiences and leveraging them later for future task executions. 8 furthermore, we conduct experiments using fewshot learning in cmfr, where we incorporate a fixed set of examples across the stages. 9 finally, we conduct an ablation study to demonstrate the impact of each cmfr stage and the object search module. the main results and the ablations are shown in tables 1 and 2 respectively. ## results we further conduct an ablation study on the seen split to demonstrate the importance of each stage of error recovery ( we depict in figure 2 the fraction of subgoals passed to each cmfr stage during execution. the figure shows that 19.5% of all subgoals generated in the initial plan fail and hence are passed to cmfr. in the remainder of this section, we further examine the impact of each stage in cmfr in addition to the importance of object search. ## conclusion we presented a conditional multi-stage failure recovery framework for embodied agents, achieving state-of-the-art performance on the teach tfd benchmark with 36% success rate, which further improves to 38% with few-shot prompting. through an ablation study, we demonstrated the importance of each stage within the framework, identifying the preconditions stage and the postexecution stage as the most critical for effective error recovery. our findings also showed the importance of object search, highlighting object localization as a key performance bottleneck that requires further investigation. for future work, we aim to explore the integration of spatial reasoning to enhance error recovery and improve task success rates.